Abstract
Handheld devices such as the Universal Manipulation Interface (UMI) enable scalable, embodiment-agnostic data collection for behavior cloning. The resulting policies predict Cartesian end-effector trajectories and delegate execution to a robot-specific controller, which trades off tracking accuracy with compliance. However, current policy evaluation protocols treat the controller as a fixed implementation detail, conflating policy and execution errors. We disentangle the two by replaying real handheld demonstrations on a simulated Franka FR3 while varying only the controller across speeds, payloads, and contact-directed reference errors. We find that controller choice alone substantially changes task success, even for kinematically feasible references; soft impedance closes the drawer in only 6% of trials at nominal speed, whereas tuned impedance always succeeds. Under a 20 mm reference error, stiff position control drops to 0% success on bottle placement and uprighting, while compliant controllers risk applying insufficient force. Force- aware residual control mitigates this trade-off, yet no single controller performs best under learned flow-matching policies. We argue that the community should carefully consider the controller as an explicit “tunable knob” during inference.
