Abstract
Learning generalizable and robust behavior cloning policies requires large volumes of high-quality robotics data. While human demonstrations (e.g., through teleoperation) serve as the standard source for expert behaviors, acquiring such data at scale in the real world is prohibitively expensive. This paper introduces EXPERTGEN, a framework that automates expert policy learning in simulation without any human demonstrations, while preserving strong sim-to-real transfer performance. EXPERTGEN first initializes a behavior prior in using a diffusion policy trained on synthetic data (e.g., Code as Policy). Reinforcement learning is then used to steer this prior in massively parallel simulation toward high task success by optimizing the diffusion model’s initial noise while keeping the denosing process frozen. In this way, EXPERTGEN regularizes exploration to remain within safe behavior manifolds, enabling stable learning from sparse rewards without additional reward engineering.
Extensive experiments on challenging manipulation benchmarks demonstrate that EXPERTGEN reliably acquires high-quality policies with strong robustness and generalization. On long-horizon manipulation tasks, EXPERTGEN achieves success rates of up to 85%, while on industrial assembly benchmarks it achieves a 91.1% overall success rate, outperforming all baseline methods. To validate sim-to-real transfer, the learned state-based expert policies are further distilled into visuomotor policies via DAgger and successfully deployed on real hardware.
