Community Trust ScoreVerified
Robot programs developed by Anthropic’s Claude Opus 5.5 coding agent completed 221 of 240 physical evaluation trials, achieving a 92.1% success rate across eight distinct tasks. The study, posted on November 12, details a method named ACE-CaP-0 where the AI agent writes, runs, inspects, and revises code using physical feedback, but is excluded from the execution loop during final evaluation.
More coverage is filed under Technology.
Frozen Code in Physical Trials
The experimental setup utilized two Agilex PIPERX arms and three cameras to perform physical tasks such as inserting a charger, uncapping a bottle, and packing shoes. For each of the eight tasks, the coding agent produced a program that was frozen before 30 separate evaluation trials. No further updates to the code occurred during these trials, and neither a human nor a coding agent remained in the execution loop. The programs relied on live perception and control systems to respond to observations, while their decision rules remained fixed.
During the development phase, the agent had access to human guidance and demonstrations. It could request textual advice or physical demonstrations from a person. These demonstrations provided video, arm trajectories, and key joint poses. The development process allowed for a maximum of ten human-assisted scene resets per task, although resets used specifically to collect demonstrations were excluded from this budget. The budget counted scene resets rather than total robot executions or demonstrations.
Performance and Simulation Impact
Charger insertion emerged as the strongest task, succeeding in all 30 trials. Bread packing recorded the weakest result, with 25 successful trials. Mean physical execution times across the tasks ranged from a minimum of 0.97 minutes to a maximum of 2.27 minutes. The data indicates that the frozen programs were capable of consistent physical execution without real-time AI intervention.
The study also compared development outcomes with and without on-demand simulation. For the charger insertion task, the program developed with on-demand simulation achieved 100% evaluation success, whereas the program developed without simulation achieved 86.7% success. This comparison used one development run per condition. Development took 2.6 hours with on-demand simulation, compared to 3.7 hours without it. However, the use of simulation increased output-token usage from 730,000 to 1,090,000. The researchers note that total development costs should account for the effort involved in demonstrations, a factor that varies between conditions.
