Agentic Robotics Flips the Build Order

In October, University of Illinois researchers gave a coding agent a UR5e robotic arm with force sensing at its wrist and the deceptively challenging job of opening a see-through drawer (deceptively because a camera looks straight through it). FORCA, their agent, running on Claude Fable 5, wrote the Python that controlled the robot and managed to open the drawer four out of ten times using only the camera feed. Once it was allowed to read the force sensor, it was able to iteratively improve its program using the force feedback (a bit like a coding agent using a test suite to identify and squash bugs) until the program opened the drawer in all ten trials. Then Claude left, and the program kept opening the drawer with no model in the loop.

VLAs (vision-language-action models) are single neural networks that map camera images and language instructions to motor commands, and they have been the focus of most robot foundation model efforts at big labs over the last three years. In other words, the labs built the hands first and put a planner on top to give them orders. Figure's Helix splits the job in two. Helix System 2, analogous to what Daniel Kahneman calls slow thinking, works out what to do and runs at 7 to 9 Hz, while Helix System 1, analogous to fast thinking, translates those instructions into motor commands for Figure's arms at 200 Hz. The instructions between them are embeddings, basically a list of numbers that only have meaning to the models trained on them.

Illinois turned it around: start with the System 2 model, have it practise on a real robot's arm, and let it write the code that performs the motions. Ken Goldberg at Berkeley, who says he had "always been cautious and skeptical of near-term adoption of robots", calls this order Agentic Robotics.

My thesis: agentic robotics puts System 2 first and has it write System 1. The learned policy keeps the behaviours an agent can't check with the sensors it has, and agents call it as a tool when they reach one.

Astra writes better than it drives#

Eighteen days after OpenAI released GPT-6 Astra (on 3 September, with a focus on computer use and coding), the RoboDojo team gave it access to only one action (move gripper to target) and a recipe describing each of 42 simulated manipulation tasks. Without any robot-specific fine-tuning¹OpenAI hasn't said whether Astra was trained on any robot data. The RoboDojo paper says there was "no robot-specific or task-specific fine-tuning performed for this evaluation", but whether some of RoboDojo's material made it into Astra's training data is, the authors write, "unquantified rather than ruled out". it achieved a success rate of 22.48%, outperforming the best of 40 trained policies on the 10 September leaderboard (19.34%). (For comparison, GPT-5.5 could only achieve 0.88% using the same action.) This is not a level playing field. The trained policies aren't shown the recipes, Astra does poorly on precision (4%, while the best trained policy in aggregate manages 16.75%), and the simulator waits while it thinks: in RoboDojo, Astra is moving for 8 to 76 seconds while thinking for an average of 55 minutes.

Just one day after appearing on the board, two world-action models from LiberAI overtook Astra, and as of 9 October Astra, directly controlling the gripper, ranked tenth. However, when the PyRUA-Lean team used the same model to generate Python code (with a higher reasoning effort) over some basic geometric functions and a segmentation model, without recipes or learned policies, its average task success rate increased to 51.6%, 2.3 times higher than when controlling the gripper directly and 19.4 percentage points higher than the leaderboard leader at the time.

As the PyRUA-Lean authors stress, these are comparisons of systems, not just models. Changing the planner from GPT-5.5 to Astra changes system performance from 0.88% to 22.48%. (No retraining, since planner and controller communicate via language.) If they communicated via embeddings, changing to a better planner would usually require retraining the controller. Changing the harness changes system performance from 22.48% to 51.6%. In contrast, compiled programs are immutable except for intentional recompilation, which is likely what you want at a factory. Phillip Isola at MIT wrote in September that if LLMs can competently control robots, "Every robot with an internet connection becomes a potential tool for Fable, Astra, and other AIs."

On 26 September, Stanford's HomeBody team showed Astra commanding a Unitree G1 humanoid as it explored a kitchen it had never seen.

On 12 October, Unitree's founder, Wang Xingxing, said his biggest mistake in the last ten years was "not believing in AI enough" (in Zilan Qian's translation).

Altimeter's Freda Duan called Astra robotics' GPT-3 moment, and said some in robotics seem "reluctant to acknowledge it". I'm happy to admit it. The most important robot brain this year came from a chatbot company, and lots of robot foundation-model companies are, whether they like it or not, building hands for rented brains. Jim Fan has gone further:

(Self-reported. Like every head-to-head number in this section, it comes from simulation.)

Back in July I wrote about a reflex that monitors the policy driving a robot, sees when it's about to screw up, and passes control to a bigger model if things start to look dicey, because "A chatbot gets a free ten seconds to think; a robot arm drops what it's holding." I thought of it as System 1 calling in System 2 for backup. I had the order wrong. In October the frontier model ran first and wrote the System 1 code itself, and the reflex survived as an exception that hands control back. I got the clock right, at least: a Unitree G1 controlled directly by a frontier model took a median of 24.84 seconds per request. On the G1, Astra spends about 90% of each run deliberating. A mind that slow can't be System 1.

The chef writes the recipe and leaves#

Robot cells have long been built by a specialist who writes the recipe, programs the robot (often using a teach pendant), commissions the cell, adjusts it to its environment and then walks away. The robot does its job, often in a production line, for the next 12 to 15 years. In 2014, a spot-welding cell cost $133,000, of which the robot's hardware and software accounted for only a quarter, according to BCG estimates; some $46,000 went on installation, programming and integration. An agent can't do the installing; the programming is the part it might automate. Language models have been used to generate robot programs since at least 2022 (see the Code as Policies paper). What's new is that these agents practise on real robots. Where VLA labs are trying to replace the robot's program, these agents are trying to replace the programmer. It is already in production: Goldberg writes that his Ambi Robotics co-founder, Jeff Mahler, applied an agentic robotics harness to a real industrial problem and deployed the result to robots sorting packages across the US.

RAX, released in October, lets a coding agent (either Codex or Claude Code) control a robot through tool calls, with its cameras, its controls and a fixed budget of practice episodes. Given an air fryer, the agent figured out how to open it, load it and close it again, which took 72.3 minutes of model time. It then wrote a Python program of about 1,500 lines, with no learned perception and no learned policy, that does the same thing. The thinking time was greatly decreased (from 72 minutes to 18 seconds), although the arm's movement time was about the same (from 2.9 to 2.5 minutes). This whole session (from exploration to the ninth run of the program) took place in the course of a morning (09:18 to 12:05).

Our first robot was held together with zip ties and delusion. Three engineers iterated it up to version 547, which as of last year was still dishing out thousands of meals a month. RAX's agent debugged its own program in a morning.

If you've built coding agents, this will look familiar. The agent writes programs that use the same tools it has access to when acting, so any program it develops interactively can be run as-is in autonomous mode. If a program wants to do something that breaks a limit, the tool call fails and the agent gets a message telling it which limit it broke, so that it can try a different approach. If a program gets stuck, it can throw an exception to hand control back to the agent. Every robot has a skills repository, each agent session works on its own branch, and changes are merged only with evidence that they worked unattended. The agent scored 4 of the 9 program runs as clean and unattended (although a human still reset the environment in between).

At MicroFactory, Astra and Claude Opus are still in the loop, but "computer vision code written by the model" does the fast work.

On its first run, the agent's program ended in a safety stop: the robot's elbow bumped into one of the cameras and knocked another out of calibration. The agent can add safety constraints on top of those already in place, but it cannot loosen them. RAX already performs self-collision, joint-limit and workspace-bound checks, and won't move unless armed by a human operator. These checks model the robot, not its environment, which is why its elbow could still collide with a camera in its way. After that collision, when a banana was placed in front of the same camera, the agent wouldn't pick it, because its forearm would come within about 1 cm of the camera; it asked for the banana to be moved instead. In Illinois, the researchers hard-coded a 30 N force limit. RoboDojo's authors stopped their own real-robot evaluation after the model "repeatedly issued physically unreasonable or unsafe actions, and some incidents damaged equipment; no person was injured", and concluded that such incidents "motivate independent safety checks and stopping mechanisms rather than reliance on the model's judgment alone." Goldberg himself warns that "exploration can damage hardware or the environment." And LLMs can't always be counted on to say no:

The hands still matter#

In the first route, the chef writes the recipe and leaves. In the other, the chef stays and runs the kitchen, as in Auguste Escoffier's brigade system, introduced in 1890 when he took over the kitchens of the newly opened Savoy Hotel in London: divide the kitchen into stations, each run by a cook who specialises in that station's work, so that the chef de cuisine can, in Britannica's words, "oversee the entire operation".

Harness VLA, likewise, delegates the planning and memory burden to a coding agent (Claude Code or Codex in their experiments) and sees a frozen VLA's performance increase from 50.0% to 82.4% on LIBERO-Pro, a simulation benchmark. It has scripts for the easy parts of the task and invokes the VLA (via a single function) for the contact-rich parts. Which planner does better isn't fixed: Claude Code beat Codex on LIBERO-Pro, 82.4% to 72.1%, and Codex beat Claude Code on RoboCasa365, 55.4% to 48.6%.

How hard to press on a potato is something I've done since I was six in my parents' restaurant and still can't articulate. I assumed that was exactly what you'd hand to a learned policy. The Illinois robot was given a cucumber to peel, held by a person. Its program logged the force on the wrist 100 times a second and attempted to keep it within the range of 4.5 to 7.5 N. When the agent realised there were gaps in the peel, it referred to the force data and added a new rule: stop the blade until it makes contact again. Contact was lost only 2.2% of the time, against 40.6% for the vision-only agent. I guess I could have written it down. Letting go of an object and grasping it again is harder without touch. In PhysLadder, Astra acting directly on arms with no force sensors (the thing that rescued the drawer) went from 80% on placing and aligning, on a real robot, to 0% once it had to break and re-make contact. Its authors say the roles of "model decisions, sensing, and robot execution remain to be isolated".

Rodney Brooks has never been polite about the hands. "Watch any humanoid video. It's really terrible manipulation," he said on a podcast in October. Cloth is the strongest case against this essay, and Physical Intelligence's laundry-folding demos make it. Dyna's robot, it says, can complete a full shift at a commercial laundry, using its learned model DYNA 2 and a higher-level vision-language model to direct it. On cloth, trained policies still have the edge over agents.