Experiments
Video for motion, audio for force.
Pick a task and a run. Each row plays the generated reference video + audio, our force-aware execution on a Franka arm that tracks the audio-derived force profile, and a kinematic-only baseline for comparison. Turn the sound on — the contact audio is the force signal.
Abstract
Audio is the force signal video can't see.
Recent advances in video generation enable learning robot manipulation trajectories from generated videos. However, these approaches produce purely kinematic trajectories that lack force information, leading to failure in contact-rich tasks where appropriate contact forces are essential for success. Generated audio carries a complementary and underexplored signal: contact sounds encode force dynamics that video alone cannot capture.
We present a pipeline that jointly leverages generated video and audio to recover both motion trajectories and contact force profiles from a single task description. We execute these force-aware trajectories on a Franka Panda robot using a closed-loop force regulator that tracks the audio-derived force profile during contact. Real-robot experiments across four contact-rich tasks — whiteboard wiping, carrot peeling, box stacking, and button pressing — demonstrate that our force-aware pipeline enables successful manipulation from video generation, where a kinematic-only baseline fails.
We further show that the pipeline can serve as an automated force-aware data-generation engine: a closed-loop policy trained on the generated demonstrations generalizes across object placements and appearances.
Pipeline
Audio as the force signal.
Our vision pipeline localizes the object and gripper (MolmoPoint), segments them (SAM 2), and recovers depth and 3D point tracks (TAPIP3D) to detect contacts and infer force directions. The audio pipeline isolates the contact sound (SAM-Audio) and extracts loudness as a force-magnitude proxy. A 1 kHz impedance controller closes the loop on the audio-derived force profile.
Policy Learning
The pipeline is also a data engine.
Beyond single-trajectory execution, we use the pipeline to generate training data: for whiteboard wiping and carrot peeling, we place the object at five positions, run the full force-aware pipeline at each, and collect 50 demonstrations per task. A CNN-based Diffusion Policy trained on these demonstrations — from RGB observations and end-effector pose only, with no force input — succeeds in 8/10 trials on both tasks, generalizing to unseen object placements and appearances.
Because the demonstrations are collected under force-aware execution, the recorded poses already encode the contact behavior. Comparing all demonstrations with all successful rollouts as mean ± std over in-contact task progress (normalized end-effector path length accumulated during contact, which factors out differences in execution speed and object placement) shows the policy entering contact at the same task stage and reaching matching force magnitudes — without any force regulator at test time. A kinematic-only version of the same scheme yields mostly failed interactions, unsuitable for learning.
