Literature
Display
← All papers

RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies

Pranav Atreya · Karl Pertsch · Tony Lee · Moo Jin Kim · Arhan Jain · Artur Kuramshin · Cyrus Neary · Edward S. Hu · Kanav Arora · Kirsty Ellis · Luca Macesanu · Matthew Leonard · Meedeum Cho · Ozgur Aslan · Shivin Dass · Tony Wang · Xingfang Yuan · Abhishek Gupta · Dinesh Jayaraman · Glen Berseth · Kostas Daniilidis · Roberto Martín-Martín · Youngwoon Lee · Percy Liang · Chelsea Finn · Sergey Levine

· CoRL 2025 ↗

Robotics12 min text · CondensedOriginal paper ↗
Contents

Overview

Keywords: generalist robot-policy evaluation and crowdsourced benchmarking.

Refer to caption
Figure 1: We present RoboArena, a distributed real-world evaluation framework for generalist robot policies. Instead of standardizing environments and tasks, RoboArena aggregates crowd-sourced pairwise A/B policy evaluations across a broad spectrum of environments and tasks to derive a global policy ranking. Its decentralized design makes RoboArena a scalable, comprehensive, and trustworthy framework for generalist robot policy evaluation. We open-source an instantiation of RoboArena on the DROID robot platform [1] and invite community members to participate, both by contributing policies and running evaluations.

At a glance

RoboArena evaluates generalist robot policies through distributed, blinded A/B comparisons on tasks chosen by local evaluators. Initial conditions must match within each pair, but need not be standardized across institutions. A task-aware preference model aggregates the comparisons, while language-model reports summarize qualitative behavior. The DROID study compares seven policies across seven universities using 612 pairs, with 4,284 total rollouts collected to construct an exhaustive progress-score reference.

1 Introduction

The proposal trades global scene standardization for local pairwise control. Evaluators can vary objects, backgrounds, and instructions, making broad coverage easier than reproducing a fixed benchmark everywhere. Blinding policy identity and aggregating many evaluators are intended to reduce individual influence; they do not prove immunity to bias or malicious feedback.

The important distinction is between the 612 pairwise comparisons used by RoboArena and the larger 4,284-rollout dataset used to measure its ranking accuracy. The latter additionally evaluates the other five policies on each task. The resulting “oracle” is an empirical ranking on these sampled tasks, not a universal measure of robot competence.

Related source: Chatbot Arena.

2 Related Work

The motivation connects the reproducibility limits of physical benchmarks with the growing scope of generalist policies and the use of crowdsourced model preferences.

Simulated and real-world robot evaluation.

Simulation enables repeatable, parallel testing but can misrepresent physical performance. Real-world benchmark manuals, remotely hosted robots, and competitions offer different forms of control, usually over narrower tasks or locations. RoboArena broadens coverage by restricting matched conditions to the two runs in each comparison.

Generalist robot policies.

Policies trained on diverse robot data can sometimes execute new tasks and scenes without retraining. That property makes distributed evaluation practical: each site can test the same policy directly rather than train a site-specific model. It also raises the burden of sampling sufficiently diverse evaluation conditions.

Crowd-sourced benchmarks.

The approach adapts crowdsourced preference benchmarking from language and generative models. Earlier distributed robot benchmarks compared learning procedures after retraining for each institution; RoboArena instead compares already trained generalist policies. This changes both the unit being ranked and the cost of adding evaluation sites.

3 Decentralized Robot Policy Evaluation via Pairwise Comparison

A central service assigns policy pairs, while distributed evaluators choose scenes and tasks. Pairwise feedback yields a global ranking; associated videos and language explanations support qualitative analysis. Open-ended task selection is a design feature, with the resulting ranking conditional on the tasks evaluators actually choose.

3.1 Policy Evaluation Protocol

An evaluator requests two anonymous remote policy endpoints, arranges a scene, and specifies a natural-language task. The policies run sequentially until completion or timeout, with the initial state closely reset between them. The evaluator reports maximum progress on a 0–100 scale, a preference, and a free-form explanation; observations and actions are uploaded centrally. Sessions can otherwise proceed asynchronously.

The protocol assumes that all assigned policies can meaningfully and safely operate across the task pool. Preferences are left to evaluator judgment, so they can reflect speed, confidence, or recovery in addition to completion. Although the main description presents binary preferences, the appendix explicitly extends the ranking model to ties.

Original algorithm 1 preserving all source text, layout, and mathematical notation.
Algorithm 1 Policy Evaluation Protocol

3.2 Computing Global Policy Rankings

Standard Bradley–Terry models preference through a difference in policy abilities. RoboArena adds a mixture of latent task buckets, each with a difficulty, sampling weight, and policy-specific offset. These variables let relative policy strengths vary with task type; bucket identities are inferred from preferences without observed task labels. Approximate expectation-maximization alternates responsibilities with clipped Newton parameter updates and ranks global abilities.

This is a modeling assumption rather than a guarantee of recovering a unique task taxonomy or unbiased ability. The printed probability expressions are not explicitly normalized over win, loss, and tie outcomes; the appendix notes below preserve this reproducibility caveat rather than silently repairing the formulas.

Related source: the Bradley–Terry model.

Equation 1
p⁡(πA>πB)=∑t=1Tνt⋅σ⁡(θA+ψAt−τt)⋅(1−σ⁡(θB+ψBt−τt))p(\pi_{A}>\pi_{B})=\sum_{t=1}^{T}\nu_{t}\cdot\sigma(\theta_{A}+\psi_{A_{t}}-\tau_{t})\cdot(1-\sigma(\theta_{B}+\psi_{B_{t}}-\tau_{t})) (1)

3.3 Extracting Qualitative Policy Characteristics

A vision-language model categorizes the instruction and initial scene images, then a language model summarizes task categories, preferences, and evaluator explanations into a policy report. Reports are instructed to cite rollout IDs, which link to videos for verification. This creates an auditable synthesis aid, not an independently validated explanation of policy internals. The methods name GPT-4.5 for categorization and o3 for report generation; Figure 8 instead labels the categorizer GPT-4o.

Refer to caption
Figure 2: Pipeline for extracting qualitative policy characteristics from RoboArena’s rich evaluation data. We use a VLMs to categorize scenes and tasks, and then use an LLM to aggregate information across a large number of evaluation rollouts into a policy report that summarizes qualitative strengths and weaknesses, and cites concrete evaluation rollout videos as evidence.

4 The DROID-RoboArena Evaluation System

The implementation uses the DROID platform: a seven-joint Panda arm, parallel gripper, wrist and external stereo cameras, and a movable height-adjustable table. Existing DROID data and installations make both policy training and distributed deployment feasible. Other embodiments are proposed extensions; the reported evaluation remains on this shared hardware family.

Related source: DROID.

Refer to caption
Figure 3: The DROID robot setup, which we use for the DROID-RoboArena evaluation system. Reproduced with permission from Khazatsky et al. [1].

4.1 RoboArena System Design

Remote policy servers perform inference; lightweight robot clients conduct sessions and upload results; a database stores instructions, scores, explanations, and videos; and a central server assigns pairs and handles policy registration or timeouts. Contributors can host their own policies without releasing weights. Remote latency was acceptable for the tested mostly static manipulation tasks, but dynamic tasks may need local inference. Decentralized evaluation still relies on central coordination and storage.

Refer to caption
Figure 4: The DROID-RoboArena system consists of a pool of remotely hosted policy servers, a pool of distributed evaluator “clients” with real robot setups, a database for storing evaluation results, and a central evaluation management server that orchestrates communication, aggregates the evaluation results, and computes a policy ranking.

4.2 Open-Sourcing RoboArena: Interfaces, Safety and Incentives

The proposed community system combines input/output checks with supervised trial runs before exposing a submitted policy to general evaluators. Evaluation credits reward each contributed pair with a requested pair involving the contributor’s policy, while sponsoring institutions provide a limited budget for users without robots. Separate all-policy and reproducible open-policy leaderboards are planned. These are the paper’s operating design and stated plans, not a verification of present service availability.

5 Experiments

Experiments measure agreement with exhaustive progress-score rankings, the number of comparisons needed, and whether automated reports capture observed qualitative differences. The comparisons test this evaluation framework on a particular policy pool and task sample.

5.1 Experimental Setup

The seven DROID-tuned policies are π0-flow, π0-FAST, and five PaliGemma variants using flow matching, FAST, FAST+, finite scalar quantization, or simple binning. All use the public openpi implementation. After each A/B trial, the remaining five policies are also run on that task, producing 612 × 7 = 4,284 rollouts for the exhaustive reference.

The conventional comparator is the earlier DROID evaluation protocol with 17 tasks and 44 episodes per policy. Comparing distributed samples with this narrower protocol tests coverage as well as ranking algorithms; it does not isolate every possible advantage of decentralization or establish superiority to all controlled benchmarks.

Related source: FAST action tokenization.

5.2 RoboArena Accurately Ranks Policy Performances

Figure 6 reports Pearson correlation of 0.69 for conventional evaluation, 0.90 for distributed Elo, 0.97 for distributed Bradley–Terry, and 0.98 for both task-aware and progress-based rankings. Mean maximum rank violation falls from 13% to 5.7%, 2.7%, 1.8%, and 1.1%, respectively. Thus task awareness improves preference-based ranking, while simple progress averaging has the lowest reported rank-violation score.

π0-FAST leads this policy pool; expressive action representations generally outrank simple bins. These are empirical tendencies within the tested training setups. Progress averaging agrees strongly with an oracle that is itself defined by average progress, while preferences also capture qualities such as decisiveness that the progress score omits. Both forms of feedback are useful and should be reported.

Refer to caption
Figure 5: Left: Examples of RoboArena evaluations. Evaluations span a diverse set of scenes and tasks. Right: “Oracle” policy ranking, aggregated from progress scores of 4284 evaluation rollouts.
Original figure 6 preserving all source text, layout, and mathematical notation.
Figure 6: Policy rankings from RoboArena pairwise comparisons correlate significantly better with oracle rankings than conventional robot evaluation approaches (“Regular”). Our task-aware ranking approach (“TASK”) leads to the best ranking. Ranking with progress scores (“PROG”) also proves effective, but may lose more nuanced information about policy performance.

5.3 RoboArena Evaluation is Sample Efficient

Random subsets of the collected comparisons show high-quality rankings after roughly 100 pairs, with continued improvement as more data is added. One pair comprises two rollouts, so the pair count should not be confused with the plotted number of episodes or the 4,284-rollout reference cost. The sample-efficiency result concerns resampling this dataset, not a prospective guarantee for a new population of policies and tasks.

Refer to caption
Figure 7: Rank correlation as a function of number of evaluation episodes. RoboArena converges to high-quality rankings within just 100 pairwise comparisons.

5.4 Extracting qualitative policy characteristics

Figure 8 reports 94.6% task-category agreement with an expert over 448 samples, labeling the model GPT-4o despite the GPT-4.5 methods description. For π0-FAST, most generated comparative claims agree with category-level win rates; the open/close judgment is a visible exception. This is a limited check of one report and categorization accuracy, rather than a comprehensive factuality or causal-reasoning evaluation.

Model Task Category Acc.
GPT-4o 94.6%
Figure 8: Task categorization accuracy (448 samples).
Refer to caption
Figure 9: Claims made by our LLM-assisted analysis tools about π0\pi_{0}-FAST-DROID outperforming, matching, or underperforming other policies are supported by the win rates in the evaluation data.

6 Discussion

The study demonstrates that broadly sampled pairwise real-robot evaluations can estimate this dataset’s overall policy ranking more accurately than the tested narrow benchmark, while yielding useful qualitative evidence. Its practical contribution is the distributed protocol and supporting infrastructure; continued community operation and broader policy coverage remain future work in the paper.

7 Limitations

Only one robot platform is evaluated. Unrestricted task choice makes one-factor controlled experiments difficult, so RoboArena complements targeted tests of camera, object, or other generalization changes. Robustness to intentionally misleading evaluators is not tested. Leaderboard over-optimization remains possible, and evolving tasks or policy pools may change what the aggregate ranking means.

Related source: targeted generalization evaluation.

Acknowledgments

The authors acknowledge evaluation volunteers, robot maintenance, institutional and agency funding, hardware support, and contributed computing resources. The distributed evaluation depended on both physical infrastructure and substantial human effort.

Appendix A Contributions

Atreya, Pertsch, and Lee lead system design, implementation, and experiment design; a broader cross-institution team runs policy evaluations. The appendix separately credits simulated-environment work, writing, coordination, and advising, making the human evaluation contribution explicit.

Appendix B Policy Ranking EM Procedure

The appendix specifies the latent-bucket model, derivative expressions, clipped-Newton expectation-maximization procedure, optional partial-success extension, baseline estimators, hyperparameters, and a simulated distribution-shift test. Original mathematics and pseudocode are preserved alongside the noted inconsistencies.

B.1 Model Definition

The model assigns each comparison to an unobserved task bucket and combines global ability, policy–bucket offsets, and bucket difficulty. The appendix supports three observed outcomes: loss, tie, and win.

Observed data.

Each observation identifies the two policies and an outcome coded 0, 1, or 2. True task IDs are not observed by the ranking model. The finite set of latent buckets approximates heterogeneity in the comparison data.

Equations
Π={π1,π2,…,πN}.\Pi=\{\pi_{1},\pi_{2},\dots,\pi_{N}\}.
yn∈{0,1,2},0=loss, 1=tie, 2=win,n=1,…,M.y_{n}\;\in\;\{0,1,2\},\quad 0=\text{loss},\;1=\text{tie},\;2=\text{win},\quad n=1,\dots,M.
𝒟p={(in,jn,yn)}n=1M.\mathcal{D}_{p}=\bigl\{\,\bigl(i_{n},j_{n},y_{n}\bigr)\bigr\}_{n=1}^{M}.

Latent tasks and mixture weights.

Nonnegative mixture weights sum to one and represent the prior chance that a comparison belongs to each task bucket. The number of buckets is a chosen hyperparameter rather than a measured number of task classes.

Equation
ν=(ν1,…,νT),νt≥0,∑t=1Tνt=1,\nu=(\nu_{1},\dots,\nu_{T}),\quad\nu_{t}\geq 0,\;\;\sum_{t=1}^{T}\nu_{t}=1,

Per‐policy and per‐bucket parameters.

A policy’s global log-ability is adjusted by its bucket-specific offset and reduced by bucket difficulty. This permits different relative strengths across subsets of tasks, but also introduces parameter ambiguities that require regularization or centering.

Equation
θ=(θ1,…,θN),ψ={ψp,t}p=1..N,t=1..T,τ=(τ1,…,τT).\theta=(\theta_{1},\dots,\theta_{N}),\quad\psi=\{\psi_{p,t}\}_{p=1..N,\;t=1..T},\quad\tau=(\tau_{1},\dots,\tau_{T}).

Tie‐model parameter.

A shared tie parameter scales a Davidson-style draw term. It is distinct from the mixture weights despite the reused Greek-letter notation.

Link function and conditional likelihood.

Sigmoids turn adjusted abilities into independent solve probabilities. A win is modeled as one policy solving while the other does not; a symmetric term gives a loss, and a geometric-mean expression gives a tie. The bucket mixture then marginalizes these scores.

As printed, the three expressions do not generally sum to one: when both solve probabilities are 0.5, their sum is 0.5 + 0.5 times the tie parameter. No outcome-normalization denominator is shown. This source-level specification issue is retained; the notes do not assume that the released implementation has the same behavior.

Equations
zin,t=θin+ψin,t−τt,zjn,t=θjn+ψjn,t−τt.z_{i_{n},t}=\theta_{i_{n}}+\psi_{i_{n},t}-\tau_{t},\quad z_{j_{n},t}=\theta_{j_{n}}+\psi_{j_{n},t}-\tau_{t}.
qin,t=σ⁡(zin,t),qjn,t=σ⁡(zjn,t).q_{i_{n},t}=\sigma(z_{i_{n},t}),\quad q_{j_{n},t}=\sigma(z_{j_{n},t}).
P⁡(yn=2∣t)=qin,t​(1−qjn,t),P⁡(yn=0∣t)=(1−qin,t)​qjn,t,P(y_{n}=2\mid t)=q_{i_{n},t}\,\bigl(1-q_{j_{n},t}\bigr),\quad P(y_{n}=0\mid t)=\bigl(1-q_{i_{n},t}\bigr)\,q_{j_{n},t},
P⁡(yn=1∣t)=2​νtie​qin,t​(1−qin,t)​qjn,t​(1−qjn,t),P(y_{n}=1\mid t)=2\,\nu_{\rm tie}\,\sqrt{q_{i_{n},t}(1-q_{i_{n},t})\,q_{j_{n},t}(1-q_{j_{n},t})}\,,
P⁡(yn)=∑t=1Tνt​P​(yn∣t).P(y_{n})=\sum_{t=1}^{T}\nu_{t}\;P\!\bigl(y_{n}\mid t\bigr).

B.2 Gradient & Hessian Derivation

The derivation computes mixture responsibilities and blockwise first and second derivatives for approximate maximization. The formulas are retained as published. Several sign, outcome-coding, and normalization details require checking against an implementation before reproduction.

Expected complete‐data objective.

The E-step normalizes bucket likelihoods into posterior responsibilities. The expected complete-data objective combines weighted log mixture and outcome terms with quadratic penalties on global abilities and policy–bucket offsets. This normalization is over buckets; it does not by itself resolve the missing normalization across outcomes noted above.

Equations
γn,t=P⁡(t∣yn;Θold)=νt​P​(yn∣t;Θold)∑u=1Tνu​P​(yn∣u;Θold).\gamma_{n,t}=P\bigl(t\mid y_{n};\Theta^{\rm old}\bigr)=\frac{\nu_{t}\,P(y_{n}\mid t;\Theta^{\rm old})}{\sum_{u=1}^{T}\nu_{u}\,P(y_{n}\mid u;\Theta^{\rm old})}.
Q⁡(Θ)=∑n=1M∑t=1Tγn,t​[log⁡νt+log⁡P⁡(yn∣t)]−λθ2​∑p=1Nθp2−λψ2​∑p=1N∑t=1Tψp,t2.Q(\Theta)=\sum_{n=1}^{M}\sum_{t=1}^{T}\gamma_{n,t}\,\Bigl[\log\nu_{t}+\log P(y_{n}\mid t)\Bigr]\;-\;\frac{\lambda_{\theta}}{2}\sum_{p=1}^{N}\theta_{p}^{2}\;-\;\frac{\lambda_{\psi}}{2}\sum_{p=1}^{N}\sum_{t=1}^{T}\psi_{p,t}^{2}.

Gradient w.r.t. θ p .

Only comparisons involving a policy contribute to its ability derivative. The example win derivatives are positive for the winning policy’s solve logit and negative for the losing policy’s. The displayed aggregate formula additionally subtracts the second-policy derivative, a sign convention that is not reconciled in the text; it should be verified before implementing the update.

Equations
∂Q∂θp=∑n∈ℐp∑t=1Tγn,t​∂∂zin,t​log⁡P⁡(yn∣t)−∑n∈𝒥p∑t=1Tγn,t​∂∂zjn,t​log⁡P⁡(yn∣t)−λθ​θp.\frac{\partial Q}{\partial\theta_{p}}=\sum_{n\in\mathcal{I}_{p}}\sum_{t=1}^{T}\gamma_{n,t}\,\frac{\partial}{\partial z_{i_{n},t}}\log P(y_{n}\mid t)-\sum_{n\in\mathcal{J}_{p}}\sum_{t=1}^{T}\gamma_{n,t}\,\frac{\partial}{\partial z_{j_{n},t}}\log P(y_{n}\mid t)-\lambda_{\theta}\,\theta_{p}.
∂∂zi,t​log⁡P⁡(yn=2∣t)=1−σ⁡(zi,t),∂∂zj,t​log⁡P⁡(yn=2∣t)=−σ⁡(zj,t),\frac{\partial}{\partial z_{i,t}}\log P(y_{n}=2\mid t)=1-\sigma(z_{i,t}),\qquad\frac{\partial}{\partial z_{j,t}}\log P(y_{n}=2\mid t)=-\,\sigma(z_{j,t}),

Hessian w.r.t. θ p .

The printed curvature sums responsibility-weighted negative logistic variances and the ability regularizer. It provides a simple blockwise Newton approximation; the source does not separately derive the curvature of every tie and normalization term.

Equation
∂2Q∂θp2=−∑n∈ℐp∑tγn,tσ(zi,t)(1−σ(zi,t))−∑n∈𝒥p∑tγn,tσ(zj,t)(1−σ(zj,t))−λθ.\frac{\partial^{2}Q}{\partial\theta_{p}^{2}}=-\,\sum_{n\in\mathcal{I}_{p}}\sum_{t}\gamma_{n,t}\,\sigma(z_{i,t})(1-\sigma(z_{i,t}))-\sum_{n\in\mathcal{J}_{p}}\sum_{t}\gamma_{n,t}\,\sigma(z_{j,t})(1-\sigma(z_{j,t}))-\lambda_{\theta}.

Gradients/Hessians for ψ p , t .

Policy–bucket offsets receive analogous gradients and curvature, restricted to a single bucket and with their own quadratic penalty. The same second-policy sign convention appears here.

Equations
∂Q∂ψp,t=∑n:in=pγn,t∂log⁡P∂zin,t−∑n:jn=pγn,t∂log⁡P∂zjn,t−λψψp,t,\frac{\partial Q}{\partial\psi_{p,t}}=\sum_{n:i_{n}=p}\gamma_{n,t}\,\frac{\partial\log P}{\partial z_{i_{n},t}}-\sum_{n:j_{n}=p}\gamma_{n,t}\,\frac{\partial\log P}{\partial z_{j_{n},t}}-\lambda_{\psi}\,\psi_{p,t},
∂2Q∂ψp,t2=−∑n:in=pγn,tσ(zi,t)(1−σ(zi,t))−∑n:jn=pγn,tσ(zj,t)(1−σ(zj,t))−λψ.\frac{\partial^{2}Q}{\partial\psi_{p,t}^{2}}=-\sum_{n:i_{n}=p}\gamma_{n,t}\,\sigma(z_{i,t})(1-\sigma(z_{i,t}))-\sum_{n:j_{n}=p}\gamma_{n,t}\,\sigma(z_{j,t})(1-\sigma(z_{j,t}))-\lambda_{\psi}.

Gradients/Hessians for τ t .

Bucket difficulty enters both policy logits negatively, so its update combines contributions from both participants. Algorithm 2 alternates these blockwise clipped-Newton steps with updates to mixture weights and tie rate, then stops on small changes in global ability and sorts those abilities. The prose mentions centering, but the printed pseudocode does not show that step.

Equations
∂Q∂τt=−∑n=1Mγn,t[∂∂zi,tlogP(yn∣t)+∂∂zj,tlogP(yn∣t)],\frac{\partial Q}{\partial\tau_{t}}=-\sum_{n=1}^{M}\gamma_{n,t}\,\Bigl[\tfrac{\partial}{\partial z_{i,t}}\log P(y_{n}\mid t)+\tfrac{\partial}{\partial z_{j,t}}\log P(y_{n}\mid t)\Bigr],
∂2Q∂τt2=−∑n=1Mγn,t[σ(zi,t)(1−σ(zi,t))+σ(zj,t)(1−σ(zj,t))].\frac{\partial^{2}Q}{\partial\tau_{t}^{2}}=-\sum_{n=1}^{M}\gamma_{n,t}\,\Bigl[\sigma(z_{i,t})(1-\sigma(z_{i,t}))+\sigma(z_{j,t})(1-\sigma(z_{j,t}))\Bigr].
Original algorithm 2 preserving all source text, layout, and mathematical notation.
Algorithm 2 EM for Fitting Model Parameters

B.3 Including Partial‐Success Information

When partial-progress scores are available, an extra Gaussian agreement factor compares them with predicted solve probabilities. Its weight and variance determine how much it influences responsibilities and derivatives. This optional extension uses additional continuous supervision; it is not necessary for the preference-only formulation.

B.4 Baseline Ranking Algorithms

The baselines separate offline preference fitting, online rating updates, and direct progress averaging. Their different information sources matter when interpreting agreement with a progress-defined reference ranking.

Bradley–Terry MLE (offline).

Offline Bradley–Terry fits sigmoid ability differences using all comparisons, optionally regularizing abilities and treating a tie as half a win. The printed gradient uses the outcome symbol also described as 0/1/2, without showing conversion to a binary or half-win target; this coding ambiguity should be resolved before implementation.

Equations
P⁡(yn=2)=σ⁡(θin−θjn),P(y_{n}=2)\;=\;\sigma\bigl(\theta_{i_{n}}-\theta_{j_{n}}\bigr),
θp←θp+η∑n:in=p[yn−σ(θin−θjn)]−η∑n:jn=p[yn−σ(θin−θjn)],\theta_{p}\;\leftarrow\;\theta_{p}\;+\;\eta\sum_{n:\,i_{n}=p}\Bigl[y_{n}-\sigma\bigl(\theta_{i_{n}}-\theta_{j_{n}}\bigr)\Bigr]\;-\;\eta\sum_{n:\,j_{n}=p}\Bigl[y_{n}-\sigma\bigl(\theta_{i_{n}}-\theta_{j_{n}}\bigr)\Bigr],

Elo‐style online update.

Elo updates the two participating ratings immediately using the difference between observed outcome and predicted win probability. Here the text explicitly uses 1 for a win, 0 for a loss, and 0.5 for a tie, with a fixed step factor.

Equation
θA←θA+K⁡(yA​B−σ⁡(θA−θB)),θB←θB−K⁡(yA​B−σ⁡(θA−θB)),\theta_{A}\;\leftarrow\;\theta_{A}\;+\;K\,\Bigl(y_{AB}-\sigma(\theta_{A}-\theta_{B})\Bigr),\quad\theta_{B}\;\leftarrow\;\theta_{B}\;-\;K\,\Bigl(y_{AB}-\sigma(\theta_{A}-\theta_{B})\Bigr),

Partial‐Success Averaging.

Progress averaging ranks each policy by the mean fractional success over its rollouts. It discards pair structure and preference nuances, but directly matches the quantity used to construct the study’s exhaustive oracle.

Equation
s¯p=1Np∑n:in=p​ or ​jn=p[sn(p)],\bar{s}_{p}\;=\;\frac{1}{N_{p}}\sum_{n:\,i_{n}=p\text{ or }j_{n}=p}\bigl[s_{n}^{(p)}\bigr],

B.5 Hyperparameter Table

Reported defaults are 60 iterations, 60 latent buckets, unit step clipping, 0.01 penalties on ability and offset, 0.99 step decay, and a 10⁻⁴ stopping tolerance. Table 1 also lists search ranges; the default iteration count of 60 is not among its displayed 50/100/200 search values. The source table is preserved.

Parameter Default Search range
EM_ITERS 60 {50,100,200}\{50,100,200\}
# buckets TT 60 {10​…​200}\{10...200\}
step_clip 1.0 [0.1, 10][0.1,\,10]
l2_theta 10−210^{-2} {10−3,10−2,10−1}\{10^{-3},10^{-2},10^{-1}\}
l2_psi 10−210^{-2} {10−3,10−2,10−1}\{10^{-3},10^{-2},10^{-1}\}
step_decay 0.99 {0.9,0.99,0.999}\{0.9,0.99,0.999\}
tol 10−410^{-4} {10−5,10−4}\{10^{-5},10^{-4}\}
Table 1: EM procedure hyperparameters.

B.6 Simulating Shifts in the Task Distribution and Policy Set

The authors order sampled tasks from easier to harder and replace weaker policies with stronger ones, using progress scores to define both changes. Under this artificial drift, RoboArena correlation is 0.838 versus 0.692 for conventional evaluation, with rank violation 0.058 versus 0.141. This supports robustness to the tested constructed shift, not to arbitrary future drift or adversarial task selection.

Eval Approach Pearson r(↑)r(\uparrow) MMRV (↓)(\downarrow)
Regular 0.692 0.141
RoboArena w/ dist. shifts (ours) 0.838 0.058
Table 2: RoboArena under simulated distribution shifts.

Appendix C Evaluation Data Breakdown

The appendix plots task verbs and object categories and shows 32 sampled evaluation scenes. Instructions span actions beyond picking, while viewpoints, lighting, table coverings, and objects vary even within one physical site. These visuals document diversity; they do not establish uniform coverage or remove evaluator selection bias.

Refer to caption
Figure 10: Top: bar chart of the most common verbs used in the task commands, along with their frequencies in the evaluation data collected. The task commands exhibit a diverse array of instructions, including ”uncapping”, ”unplugging”, ”finding”, ”dusting”, etc. Bottom: visualization of the diversity of object categories being commanded to be interacted with, bucketed by major objects types.
Refer to caption Refer to caption Refer to caption Refer to caption
Refer to caption Refer to caption Refer to caption Refer to caption
Refer to caption Refer to caption Refer to caption Refer to caption
Refer to caption Refer to caption Refer to caption Refer to caption
Refer to caption Refer to caption Refer to caption Refer to caption
Refer to caption Refer to caption Refer to caption Refer to caption
Refer to caption Refer to caption Refer to caption Refer to caption
Refer to caption Refer to caption Refer to caption Refer to caption
Figure 11: Environments in RoboArena are diverse, due to RoboArena’s distributed nature and the eschewing of a standardization of tasks. Here we depict 32 sample environments used for RoboArena across the network of participating institutions. Evaluators were encouraged to scale diversity, leading to a visible heterogeneity of environments. Even when the physical location of the robot was the same, lighting, camera viewpoints, tablecloths, and objects were often altered significantly, again made easy by the fact that exact scenes and tasks need not be standardized.

Appendix D Nuances in Policy Preference Labels

In 11% of A/B trials, preferences differ from what partial-progress scores alone imply, either through unequal preference at tied progress or opposing trends. Table 3 gives concrete explanations involving decisiveness, retries, natural placement, and distraction. Thus the two feedback channels capture overlapping but different objectives. The text’s reference to “figure 3” here corresponds to Table 3.

Table 3: Examples of preference feedback for A/B pairs for which partial success feedback was equal for A and B, illustrating information beyond binary partial‐success scores.
Session ID Preferred Policy Long‐form Feedback
748 A Although both policies don’t put the cloth on the screwdriver, policy A places the cloth close to the banana, while policy B does not seem close to either object.
674 B Both policies completed the entire task but policy B did it on the first try. After the first grasp and lift, it feels like policy A dropped the mouse prematurely. It then picked it up again and moved it further.
660 B Policy B initially hesitated to move long distance but later transitioned to effective and rapid movements. Meanwhile, policy A also succeeded at the task, but it exhibited more sluggish movements.
649 A Both A and B picked up the cup instead of pushing, and both then placed it on the table. After letting go, A returned to a starting pose while B kept repeatedly grabbing the cup, which is sub‐optimal.
648 B While both did take the object out of the bowl (qualifying for full score), B placed the object on the table area next to the bowl. This is the more natural thing to do.
641 A Both policy A and policy B almost solved the task completely. However, policy A displayed more decisive motions with less corrective behavior while policy B solved the task by chance after multiple attempts.
579 A Both policies were able to close the drawer most of the way. However, policy A went straight for the drawer and didn’t stop until it was closed. Policy B got distracted by the plastic food items halfway through (though it eventually returned to close the drawer).
Refer to caption
Figure 12: We observe that policies tend to succeed on tasks involving direct object manipulation, such as placing objects into containers (examples in green), but often fail in tasks that require tool use or nuanced semantic understanding, such as wiping, cleaning or following detailed multi-step instructions (examples in red).

Appendix E Policy Performance Reports

The report pipeline uses o3-2025-04-16 with a full-report prompt, then a separate condensation prompt. Figures 13–15 preserve the complete example report and both templates. The template requests nine dimensions, repeated behavioral evidence, and exact rollout IDs rather than invented citations. The PDF example removes video references; linked interactive reports are external supplementary material, not reviewed footage embedded in this edition.

Appendix F Strengths and Weaknesses of the Evaluated Generalist Policies

Across these policies, direct object manipulation is more reliable than tool use, deformable-object handling, precise alignment, and multistep or attribute-sensitive instructions. Autoregressive policies show stronger language following in picking, sorting, and stacking; flow-based policies can produce fluid motion yet miss precise instructions; simple bins frequently stall. These are observed family-level tendencies in this pool, not controlled causal conclusions about action representations.

The example π0-FAST report describes confident grasps and recovery alongside freezing, incorrect targets, incomplete release, and failures to obey deliberate-inaction commands. Its apparent cloth-handling advantage over competitors can coexist with poor absolute cloth-task reliability. Generated qualitative judgments remain hypotheses to inspect against the referenced rollouts.

Original figure 13 preserving all source text, layout, and mathematical notation.
Figure 13: The full generated policy performance report for π0\pi_{0}-FAST-DROID (video references removed; check https://robo-arena.github.io/leaderboard for interactive reports with references).
Original figure 14 preserving all source text, layout, and mathematical notation.
Figure 14: The prompt template used to generate the full policy performance reports.
Original figure 15 preserving all source text, layout, and mathematical notation.
Figure 15: The prompt template used to generate a concise summary of the full policy performance report.
Concept demonstration

Open image ↗