Abstract
Modeling deformable objects – especially continuum materials – in a way that is physically plausible, generalizable,
and data-efficient remains challenging across 3D vision, graphics, and robotic manipulation. Many existing methods
oversimplify the rich dynamics of deformable objects or require large training sets, which often limits generalization.
We introduce embodied MPM (EMPM), a deformable object modeling and simulation framework built on a
differentiable Material Point Method (MPM) simulator that captures the dynamics of challenging materials. From
multi-view RGB-D videos, our approach reconstructs geometry and appearance, then uses an MPM physics engine
to simulate object behavior by minimizing the mismatch between predicted and observed visual data. We further
optimize MPM parameters online using sensory feedback, enabling adaptive, robust, and physics-aware object
representations that open new possibilities for robotic manipulation of complex deformables. Experiments show
that EMPM outperforms spring-mass baseline models.
