Robots Climbing a "Well-Rounded" Mountain,
LLMs Climbing from Four Entry Points

The RoboDojo paper was posted on arXiv on July 5, and the official site launched on July 6. The co-principal investigators are Chen Tianxing from HKU MMLab and Yue Chen from Peking University. The leaderboard is maintained by the non-profit AI MMLab Club, and the website explicitly states it does not accept commercial funding or sponsorship. It aims to fix two old problems: tasks that are too short and too narrow; and evaluations that are either simulation-only or real-robot-only.

Its approach has three components, corresponding to three goals the robotics field is currently pursuing.

Among the paper's July findings, three shaped the subsequent story:

• Unevenness. Even the best models had success rates below 15% on generalization and long-horizon; the best on open-instruction, π0.5, was only 1.67%.
• Randomization as a cleaver. Scene randomization caused nearly all policies to drop heavily: Hy-Embodied-0.5-VLA averaged 21.98 on standard layouts and 1.57 on randomized layouts, a 92.9% drop.
• Real robot ≠ simulation. The two boards only partially agree; real robots also exhibited action jitter and repeated ineffective actions; the paper specifically noted DM0 had unstable control signals during deployment, requiring human supervision.

Four months later, the best non-Agent score rose to 34.97%, roughly 4×. The real-robot board has 14 models; the top three are world action models (WAM, predicting future frames alongside actions): InternW0-Δ, OpenWAM-α, MWA-Phoenix-Preview; the first-place SLA-Preview is a VLA. The official site's "future extensions" lists the next stops: mobile manipulation, dexterous hands, tactile sensing, humanoids, reusing the same evaluation pipeline.

The LLM entries on the board look like a list of names, but categorized by "where the LLM sits in the control loop," there are four approaches:

Reading down the table reveals a pattern: LLMs are moving outside the control loop. In entry point 1, it is in the loop every step; in entry point 3, it only appears during development, spending zero tokens at evaluation time. Scores and precision also increase in this direction. This is my reading of these numbers, not a definitive conclusion: the task panels differ across the four rows, entry point 3's tasks were cherry-picked, and entry point 4's approach is not disclosed.

Setup. The Astra paper (RoboProbe & RoboDojo team, posted on arXiv Sep 21) draws two lines for "LLM as policy": no learned policy in the control chain; LLM weights are not fine-tuned for the robot or these tasks. Each round the model must call move_eef (giving position, orientation, and gripper target for either arm) or give_up; primitives like "grasp" or "place" are not provided; target points are handed to learning-free inverse kinematics and path planning, executed at 25 Hz. Maximum 100 model calls per episode, medium inference strength, input is three camera views plus robot state. The task description also includes a task recipe compiled from RoboDojo documentation, describing the task and scoring phases but not the action sequence. Public policies did not receive this text; the paper explicitly notes that inputs differ between the two sides.

Results. GPT-6 Astra 28.97 / 22.48% (average score / success rate, 42 tasks × 50 trials, 2,100 total), ranking above 40 public policies in the paper's comparison (leaderboard snapshot from Sep 10). Under the same interface and post-processing, GPT-5.5 was 1.13 / 0.88%, DeepSeek-Flash was 2.99 / 1.92% (only 10 runs per task). A ~25× difference—"LLM" is not a single category that can be scored together.

Extremely uneven profile. Success rates across five dimensions: generalization 30.50%, memory 38.67%, open 31.00%, long-horizon 8.25%, precision 4.00%. Looking at individual tasks is more revealing (success rate, right column is best public policy at the time):

The tasks on the left are hard because of understanding rules: sort by number size, solve an equation, follow language instructions, replicate a given sequence. The tasks on the right demand manual dexterity: alignment, contact, avoiding existing objects. make_kong deserves special mention: in the paper's video, Astra chose the correct move, yet success rate was still 0%—understanding correctly does not equal executing correctly. The paper's own failure videos also fall on this side: failing to grasp moving targets on a conveyor, two-arm coordination failures, knocking over a built tower when reaching for the next block.

What helped, what didn't. Providing a demonstration not only didn't help, it hurt: across 340 matched task-layout pairs, zero-shot 22.9%, image + end-effector pose demo 17.9%, text description demo 12.9%; the paper says the cause was not isolated. What helped was online correction. The project page's perturbation test used 8 layouts of general_pickup (1 trial each): vertical flip 8/8, horizontal mirror 6/8, coordinate axis sign inversion 4/8, end-effector pose + 10cm random jitter 3/8. The model can correct sign errors by comparing "I told it to go here, it actually went there"; when jitter exceeds grasping precision, correction fails.

Real robot. Did not complete the official protocol: the model repeatedly gave unreasonable or unsafe actions, equipment was damaged (no injuries), testing was halted. In the remaining 12 tasks and 33 trials, only 1 was fully successful (stack_bowls), 3.03%.

This approach comes from an unofficial community report (report page, listed as a community report on RoboDojo's report page). Authors are Jiayi Su, Yixin Zheng et al.; EP.103 verified the first author is Su Jiayi, advisors are Wang He and Zhang Zhizheng, team is Galbot. The approach: π0.5 first outputs a 50-step joint-space action block; Astra views the same images, execution history, plus the dual-arm forward kinematics trajectory for this action block, then chooses: execute the first 1–15 steps as-is, or correct 1–5 steps using end-effector poses. After each segment, both models re-decide.

Ten tasks, 5 trials each, scenes and random seeds aligned: hybrid approach success rate 48%, average score 62.60; Astra direct control 26% and 37.81; π0.5's own re-weighted reference on the same tasks is 15.67% and 24.43. Astra modified only 14.4% of control steps, following π0.5 for the remaining 85.6%; tokens were 44.8% fewer than direct control. The report summarizes three behaviors: correcting wrongly selected objects or target positions; pushing objects before grasping; inferring from the simulation not ending that some success condition remains unmet, and continuing to correct.

This 48% cannot be compared to the leaderboard. Each approach only has 50 episodes; tasks were stratified by π0.5's success rate, biased toward its weak spots; Astra used maximum inference; the authors themselves wrote: treating the robot as a pausable video game gives Astra the most lenient conditions, but the physical world has no pause button, and latency remains unresolved. In the same report, switching to single-arm Franka on RoboLab (10 tasks × 5 trials), direct control 98%, hybrid 92%, π0.5 36%—the "hybrid beats direct" ordering flips.

Setup. Peking University DAGroup's Agent × Robot blog (Sep 20) had Astra and Fable 5.1 write robot policy code, iterating with evaluator feedback (nominal budget of 10 development rounds), then freezing one policy per task, evaluated on 20 scene layouts. No coding Agent participates during evaluation; perception, planning, checks, and procedural recovery all run locally.

Results. 18 setups were selected from 54, selection criterion being Astra showed progress during development, so this is a cherry-picked panel: Astra 46.94%, Fable 5.1 42.50%, 20 trials per task. Symbolic tasks were strongest: tic-tac-toe both 80%, press-number-buttons 100% and 90%. Randomization hurt: stack blocks Astra dropped from 85% to 40%, fold clothes from 50% to 10%. Coin insertion and xylophone, both 0.

Accounting. Frozen policies average ~60.3s per episode (excluding simulator startup), 0 tokens, 0 API calls; the same Astra in online step-by-step control (70 episodes × 14 setups, including failures) averaged 510.6s API wait, 46.8 calls, ~877k tokens. The online reference succeeded at least once in 32 of 54 setups; frozen policies succeeded in 14 of the selected 18; the two panels differ and cannot be directly compared.

Real robot. The same group's another post used Franka dual arms for stack blocks, Hanoi tower sorting, and cable insertion: Astra 15/15, Fable 5.1 10/15 (cable insertion 0/5, no success within 10-day observation period). Time to first success: stack blocks ~80 min vs. ~40 min, Hanoi tower ~1 day vs. ~7 days, cable ~1.5 days vs. none; these are operator self-reports, development conditions are unequal, "cable success" means mechanically plugged in. Humans did calibration, scene interpretation, and direction-giving; the Agent would spend a long time on one method, rarely switching proactively.

Compared with entry point 1's real-robot 1/33: here the tasks are three specially set-up tasks accompanied by humans for days, with different robots and tasks, so success rates cannot be compared; what can be compared is position. Where LLMs currently stand firm on real robots is during development, not during control.

There is also the most inconspicuous entry point. XPolicyLab's paper describes a comparison: integrating a representative policy into the standard reduced the process from over five hours to two hours; using a packaged Agent skill reduced it to thirty minutes. Here the LLM is the engineer of the robot evaluation ecosystem—no one ranks it on the leaderboard.

The official leaderboard has a separate Agent board. Entry type labels come from submitters' self-descriptions; the protocol asks submitters to choose one or a combination from Agent, VLA, WAM, LLM, VLM, IL. As of November 3, 10 of 66 models on the board carry the Agent label:

Additionally, DeepSeek-Flash (1.92%) and GPT-5.5 (0.88%) are two general-purpose models tested by the RoboDojo team themselves. PhysicalRSI is described as a "baseline approach" in the leaderboard news. PyRUA-Lean's contributor is the same group that wrote the entry point 3 report; the official site has not disclosed whether it uses the same approach.

In six weeks, the best Agent score rose from Astra's 28.97 to Q0-Preview's 74.20 (average score). By dimension, best Agent vs. best non-Agent (success rate):

The largest gaps are in open and memory. In the open column, all 4 entries above 50% success rate are Agents; the two entries labeled VLM + WAM (HWM-0 Preview, Liber-0 Preview) are only 6.08% and 5.33%, far below the best pure WAM (Awomo-0.5, 22.92%) and pure VLA (WALL-OSS-1.0-lite, 19.58%) in this column. Q0-Preview's memory score of 93.67% is near ceiling.

Randomization comparison. The paper's July finding was that scene randomization caused nearly all policies to drop heavily. On the generalization board, learned policies drop steeply: VPP2 standard layout 46.44%, randomized layout 12.56%; Liber-0 Lite from 33.22% to 5.00%. Agent entries hold up: Q0-Preview 58.67% vs. 58.67%, PyRUA-Lean 45.33% vs. 41.00%, HexaShell-Preview 38.00% vs. 31.00%; GO-3 Nano Preview with WAM is 43.00% vs. 24.33%, holding up less well. This is consistent with a "perception, geometry, planning" pipeline being insensitive to layout changes, but the official site has not disclosed these entries' approaches, so this is just a reading of the numbers.

How the rules govern them. The official main board displays Agent entries in muted style, excluding them from rankings. The protocol requires each submission to be "a fixed system," allowing internal Agents or planners for instruction understanding, subgoal decomposition, and closed-loop replanning; routers that switch policies per task are not allowed, nor is "picking the best known policy per task" to assemble a score. To enter the official verified board, entries must be evaluated on the official system, run three random seeds in simulation, pass hidden layout verification, and release training and deployment code, checkpoints, configurations, and reproduction instructions within three days of listing. The official site has not disclosed which of the 10 Agent entries have completed which steps.

Correction to EP.103. The September issue's topic 04 (EP.103) concluded "the upper layer will be disrupted, the lower layer won't for now." The first half is harder six weeks later; the second half needs revision. By late October, the two entries with the highest precision success rate on the board are Agent entries, 76.00% and 58.50%; in July the best precision VLA (X-VLA) was 12.00%, now the best non-Agent is 35.17%. But the way the lower layer is disrupted is not LLMs learning tactile feel (Astra direct control precision is only 4.00%), but Agents writing the lower layer as code, connecting planners, and some carrying WAMs.

• Except for two real-robot instances, everything is simulation. The two instances are the RoboDojo-RealWorld leaderboard and Astra's 33 diagnostic trials; all simulation tasks are tabletop manipulation.
• LLM entries received task recipes compiled from documentation; public policies did not. The paper explicitly notes differing inputs between the two sides; whether Agent entries also received them is not disclosed.
• Agent entries' approaches are not disclosed on the official site. Q0-Preview et al.'s implementations, and whether they passed hidden layout verification, are not disclosed; the official main board displays them in muted style, excluding them from rankings.
• Both community panels were cherry-picked. PKU's 18 setups were ones where Astra showed progress during development; Galbot's 10 tasks were stratified by π0.5's weak spots, 50 episodes per approach. Neither can be compared to the official 42-task scores; the PKU report did not state whether the development layouts and the 20 evaluation layouts overlap.
• Human experts 76.03% is the paper's July口径. Whether Agent entries were evaluated under the same conditions is not disclosed.
• Simulation and real-robot tasks are not paired. OpenWAM-α's real-robot success rate (24.4%) is actually higher than its simulation success rate (11.92%), so "real robot is worse" can only be said in terms of relative ranking, not by directly comparing absolute values.
• The leaderboard changes daily. The official site self-reports a November 3 update; this article's numbers are from the November 5 leaderboard; the Astra paper's comparison is a September 10 snapshot.
• Unverified: Corporate backgrounds of Qoo Tech, HexaFuture, Einsia.AI; whether PKU DAGroup's PyRUA-Lean is the same approach as its Code-as-Policies report; the correspondence between community report authors and Galbot, only referenced via EP.103's verification.