Short human demonstrations can be used to create fully automated training datasets, sidestepping the prompting limitations that hold back vision-language models. Rather than describing objects with language, the system tracks what a person touches and manipulates during a demo, follows those objects through time, and clusters detections to handle objects merging or splitting apart in the scene. This bypasses a core weakness of VLMs, which struggle to reliably detect unusual or novel objects even with repeated, carefully engineered prompts.

The result? Task-specific detectors that are trained in minutes with zero manual labeling and outperform state-of-the-art vision-language models on both detection accuracy and real-world task success. Down the line, this work can help folks who are domain experts in other areas – mechanics, assembly line workers, etc. – more easily program robots without needing to become robotics experts themselves.