Closing the Dynamics Gap through Self-Directed Robotic Interaction

Agents inferred that the table was not flat and added cloth curvature to the simulator. That improved the fit, but small ball imperfections and local surface variations remain difficult to reconstruct from camera views. Parameter tuning cannot capture effects outside the model, and explicitly modeling every imperfection means continually extending the simulator.

A concrete direction is a combination of physics-based simulation and data-driven dynamics models: keep geometry and contact mechanics in the simulator, and learn corrections to its predicted motion from real trajectories. The agent could then select experiments that expose those errors and update the learned component.

While many claim computer vision is solved, agents still struggle to build reliable vision pipelines for real robotic experiments. Even offline tracking of balls on a clean green background, with clear boundaries, proved brittle: tracks jumped from stationary balls onto the moving cue. Agents kept patching the pipeline and needed prompting to reconsider the approach. Agentic robotics creates a concrete opportunity for the vision community: robust, reusable tools that agents can reliably assemble, verify, and use in the physical world.

Our coding agent needed guidance on how to spend time and compute across long physical experiments. In the initial run, its shot choices left the remaining balls against cushions and in corners, making them inaccessible and forcing it to stop exploration after only 17 shots. We need systems that preserve future exploration options and help agents decide when to think, act, refit, or backtrack in the physical world.