EMPM: Embodied MPM for Modeling and Simulation of Deformable Objects

Yunuo Chen1,2*, Yafei Hu1*, Lingfeng Sun1, Tushar Kusnur1, Laura Herlant1, Chenfanfu Jiang2,
1Robotics and AI Institute, 2UCLA.
*equal contribution
arXiv

EMPM Overview. Our differentiable MPM simulator estimates material parameters from real observations such as 3D object shape. It handles diverse, complex deformable objects and can update parameters online from streaming video.

Abstract

Modeling deformable objects -- especially continuum materials -- in a way that is physically plausible, generalizable, and data-efficient remains challenging across 3D vision, graphics, and robotic manipulation. Many existing methods oversimplify the rich dynamics of deformable objects or require large training sets, which often limits generalization. We introduce embodied MPM (EMPM), a deformable object modeling and simulation framework built on a differentiable Material Point Method (MPM) simulator that captures the dynamics of challenging materials. From multi-view RGB-D videos, our approach reconstructs geometry and appearance, then uses an MPM physics engine to simulate object behavior by minimizing the mismatch between predicted and observed visual data. We further optimize MPM parameters online using sensory feedback, enabling adaptive, robust, and physics-aware object representations that open new possibilities for robotic manipulation of complex deformables. Experiments show that EMPM outperforms spring-mass baseline models.

Method Overview

Method Overview. Our MPM simulation engine takes a reconstructed pointcloud and tracked 3D points as inputs. The model parameters are optimized using the discrepancy between the model's prediction and 3D shape reconstruction and tracking. The control inputs applied to the dynamics model of MPM is computed as the velocity extracted from the reconstructions. In the inference stage, we test the model's performance in terms of predicting 3D point positions and RGB image rendering of the 3D Gaussian splats.

Offline Simulation & Optimization

We perform offline simulation and optimzation of objects using collected RGB-D videos.


Online Simulation & Optimization

We perform online simulation and optimization using real-time sensory data.
In the simulation clip, the red part is the segmented object, and the green part is the simulation overlay.


Application

We showcase model rollouts in which optimal controller actions are explored in simulation while the robot tracks the simulated rollout to execute the task.
In the simulation clip, the red part is the segmented object, and the green part is the simulation overlay.


Comparison with PhysTwin

We compare our method with PhysTwin.


BibTeX

@article{chen2026empm,
      title={EMPM: Embodied MPM for Modeling and Simulation of Deformable Objects},
      author={Yunuo Chen and Yafei Hu and Lingfeng Sun and Tushar Kusnur and Laura Herlant and Chenfanfu Jiang},
      journal={arXiv preprint arXiv:2601.17251},
      year={2026},
}