Literature
Display
← All papers

What Are We Actually Benchmarking in Robot Manipulation?

Tianchong Jiang · Xiangshan Tan · Samuel Wheeler · Luzhe Sun · Tewodros W. Ayalew · Matthew Walter

Preprint posted · CoRL 2026 (accepted) ↗

Robotics16 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers the complete thirty-one-page arXiv v1 dated 2 June 2026, including experimental protocols, uncertainty calculations, hardware traces, statistical proofs and citation-tracker methodology. Xiangshan Tan, Samuel Wheeler, Luzhe Sun and Tewodros W. Ayalew share equal contribution. All six figures and ten numbered tables are retained.

At a glance

The paper asks whether a manipulation benchmark score identifies the capability readers infer from it. Four diagnostics test shortcut solutions, statistically unresolved gains, overfitting to narrow or fixed tests, and the effect of training data being unusually close to evaluation.

Audits cover LIBERO, CALVIN, SimplerEnv, RoboCasa and RoboTwin 2.0. The results expose limitations of score interpretation; they do not show that high-scoring foundation models lack broader capabilities, or that a benchmark passing these particular checks is universally valid.

1 Introduction

A fixed-distribution success average is often treated as evidence for much broader real-world manipulation. Directly validating that inference would require shared training data and representative real-robot evaluations, which the authors consider impractical.

The proposed substitute is four targeted counterchecks: remove presumed capabilities from a probe; bound significance from published aggregates; resample fixed tests or their narrow configurations; and deliberately move training data nearer the test. Figure 1 previews the outcomes.

A citation audit finds 79 March 2026 arXiv papers reporting LIBERO results, illustrating the benchmark’s influence. The count uses first arXiv dates and a 21 May 2026 snapshot, not current publication totals.

Complete original figure with every panel, label and legend.
Figure 1: The panels preview the four diagnostics used in our audit. 1(a) A 0.09B-parameter probe nearly matches SOTA on LIBERO average. 1(b) Only 19.8% of LIBERO SOTA claims are provably significant. 1(c) On CALVIN, resampling block poses within the training distribution lowers average tasks completed (ATC). 1(d) On SimplerEnv, the score cannot distinguish a policy that generalizes from one trained on data close to the test.
Selectable original panel labels

(a) Shortcut solvability

(b) Statistical significance

(c) Creeping overfitting

(d) Data-source dependence

2 Related Work

The audit adapts established concerns from vision and language benchmarks to embodied policy evaluation. It focuses on five frequently reported manipulation benchmarks rather than attempting an exhaustive simulator survey.

2.1 Benchmark Audits in Computer Vision and NLP

Prior studies identify language-prior shortcuts in VQA, copyright-tag shortcuts in image classification, unresolved leaderboard differences, dataset-specific overfitting and train/test overlap. These motivate separating what a model can exploit from what a benchmark is intended to measure.

Fresh test samples probe adaptation to a particular finite set; changed but training-supported conditions probe adaptation to a narrow test distribution. Neither is a direct measurement of all real-world manipulation capabilities.

2.2 Manipulation Benchmarks and Prior Audits

LIBERO tests language-specified task suites; CALVIN chains tasks across scenes; SimplerEnv reproduces real robot setups; RoboCasa varies kitchen scenes; RoboTwin 2.0 varies bimanual scenes and objects.

LIBERO-Pro and LIBERO-Plus perturb inputs beyond the original training distribution. This audit instead targets changes it argues are supported by training data, while distinguishing evidence of support from an exact distribution-matching proof.

Complete original figure with every panel, label and legend.
Figure 2: Cumulative counts of arXiv papers reporting results on CALVIN, LIBERO, SimplerEnv, RoboCasa, and RoboTwin 2.0. Sources, deduplication keys, and per-paper classification criteria are in Appendix D.

3 Shortcut Solvability

Train a small DINOv2 encoder plus MLP action head, without robotics pretraining, diffusion or an action tokenizer. For fixed public task sets, a learned task-ID embedding replaces language understanding; this retains task identity while removing the need to interpret an instruction.

The best-checkpoint LIBERO probe scores 99.0/100.0/98.8/92.4% on Spatial/Object/Goal/Long. It is within one point of the selected best non-RL result on the first three suites, but 6.4 points behind on Long. Test-set tuning is explicit; final checkpoints average 95.1% versus 97.6% after selection.

On CALVIN it reaches 3.123/3.242/3.872 tasks for D→D/ABC→D/ABCD→D, competitive with some foundation baselines but below current best results. RoboTwin and RoboCasa gaps are much larger. Failure of this one probe does not prove that no other shortcut exists.

The standard-data SimplerEnv probe scores zero, but unlike the task-ID probes it uses a pretrained SigLIP text embedding. DINOv2 supplies visual pretraining in all cases; the claim is absence of robotics/language-model pretraining for the fixed-task probe, not learning vision from scratch.

Table 1: Shortcut-solvability probe results across five benchmarks.
Benchmark Protocol/Suite DINO+MLP Foundation VLA Reported (ΔF\Delta_{F}) Best Reported Reported (ΔB\Delta_{B})
LIBERO Spatial 99.0% OpenVLA (7B) 84.7% (−14.3-14.3) π0\pi_{0}+T-MEE (2.3B) 99.8% (+0.8+0.8)
Object 100.0% OpenVLA (7B) 88.4% (−11.6-11.6) π0.5\pi_{0.5}+MoH (3B) 100.0% (+0.0+0.0)
Goal 98.8% OpenVLA (7B) 79.2% (−19.6-19.6) MoLA (1.64B) 99.5% (+0.7+0.7)
Long 92.4% OpenVLA (7B) 53.7% (−38.7-38.7) CORALSimVLA{}_{\text{SimVLA}} (0.8B) 98.8% (+6.4+6.4)
CALVIN† D→\toD 3.123 DeeR-VLA (3B) 2.83 (−0.29-0.29) MDT-V (n.d.) 4.52 (+1.40+1.40)
ABC→\toD 3.242 RoboFlamingo (3B) 2.48 (−0.76-0.76) MMaDA-VLA (8B) 4.78 (+1.54+1.54)
ABCD→\toD 3.872 RoboFlamingo (3B) 4.09 (+0.22+0.22) Xiaomi-0 (4.7B) 4.80 (+0.93+0.93)
SimplerEnv WidowX 0.0% X-VLA (0.9B) 95.8% (+95.8+95.8) CORALSimVLA{}_{\text{SimVLA}} (0.8B) 97.9% (+97.9+97.9)
RoboTwin 2.0 Clean (50+500) 60.4% X-VLA (0.9B) 72.8% (+12.4+12.4) MotuBrain (n.d.) 95.8% (+35.4+35.4)
Randomized (50+500) 59.4% X-VLA (0.9B) 72.8% (+13.4+13.4) MotuBrain (n.d.) 96.1% (+36.7+36.7)
RoboCasa RSS24 18.8% GR00T-N1 (2B) 49.6% (+30.8+30.8) X-WAM (n.d.) 79.2% (+60.4+60.4)

Foundation VLA: most-cited protocol-comparable foundation VLA reporting on the protocol (chosen per protocol for CALVIN).

Best reported: top non-RL-finetuned result (RL-finetuned excluded, as the probe is imitation-only).

ΔF\Delta_{F} and ΔB\Delta_{B}: the Foundation-VLA score and the Best-reported score, respectively, minus the DINO+MLP probe score (ATC for CALVIN, percentage points otherwise); a negative value means the probe scores higher.

CALVIN† reports ATC (out of 5). “n.d.” = params not disclosed. Policy sources are listed in Appendix A.1.

4 Statistical Significance

Compare each new score with the prior best named in that paper. With only aggregate scores, the unseen task-by-task paired outcomes can imply different variances and therefore different decisions under a one-sided, task-stratified Wald test.

For bounded integer outcomes, derive sharp minimum/maximum variance envelopes consistent with the reported counts. Below the feasibility condition no compatible paired table rejects; above the guarantee condition all do; between them the aggregate alone is inconclusive. For independent samples the same bounds are conservative, not sharp.

At nominal α=.05, guaranteed-rejection shares are 19.8% for LIBERO, 43.9% CALVIN, 19.7% SimplerEnv, 53.3% RoboCasa and 73.7% RoboTwin. Figure 3 uses 789/107/122/30/19 comparisons respectively, including nonimprovements; LIBERO pools Spatial/Object/Goal, not Long.

“Provably significant” means forced rejection under this specified nominal Wald rule and its assumptions. The analysis does not turn the asymptotic test into an exact finite-sample guarantee, and inconclusive does not mean no improvement. Fewer submissions and larger gaps partly explain stronger guarantees on newer benchmarks.

Complete original figure with every panel, label and legend.
Figure 3: Reported gains from public scores alone. Each pie sorts one benchmark’s previous-best-to-new comparisons into the legend’s four groups (bounds derived in Appendix C). Each comparison uses the previous best as reported in the new policy’s own paper, rather than the highest earlier score we could find on arXiv. LIBERO pools three suites. nn is the number of comparisons. All bounds use a 5%5\% significance level.
Equations
δt,i=YB,t,i−YA,t,i,dt=∑i=1Sδt,i,st=∑i=1Sδt,i2.\delta_{t,i}=Y_{B,t,i}-Y_{A,t,i},\qquad d_{t}=\sum_{i=1}^{S}\delta_{t,i},\qquad s_{t}=\sum_{i=1}^{S}\delta_{t,i}^{2}.
Z=T​S​(μ^B−μ^A)SS−1​∑t=1T(st−dt2/S).Z=\frac{TS(\widehat{\mu}_{B}-\widehat{\mu}_{A})}{\sqrt{\frac{S}{S-1}\sum_{t=1}^{T}\left(s_{t}-d_{t}^{2}/S\right)}}.
Δ^:=μ^B−μ^A.\widehat{\Delta}:=\widehat{\mu}_{B}-\widehat{\mu}_{A}.
Δ^≥δ∀​(μ^A,T,S,R,α),\widehat{\Delta}\geq\delta_{\forall}(\widehat{\mu}_{A};T,S,R,\alpha),
YA,t,i,YB,t,i∈{0,…,R}Y_{A,t,i},Y_{B,t,i}\in\{0,\dots,R\}
μ^A<μ^B\widehat{\mu}_{A}<\widehat{\mu}_{B}
H0:Δ≤0​ against ​H1:Δ>0,H_{0}:\Delta\leq 0\text{ against }H_{1}:\Delta>0,
Δ^<δ∃​(μ^A,T,S,R,α),\widehat{\Delta}<\delta_{\exists}(\widehat{\mu}_{A};T,S,R,\alpha),

5 Creeping Overfitting

Distinguish fitting the particular released test sample from fitting its narrow distribution. On CALVIN ABC→D, keep the same task-chain manifest but resample block poses within the training range. X-VLA/GR-1/RoboFlamingo lose 1.027/.749/.498 average tasks, with positive paired 95% intervals.

On SimplerEnv stack, reverse color order, add support blocks, or randomize block/arm pose. Matched local calibration controls hardware and step limits. Effects depend on policy: X-VLA loses 28.47 points with stacked support, while CogACT’s corresponding change is −1.39 points with an interval spanning zero.

The authors support training-distribution relevance with Bridge examples and instruction counts. This does not establish identical joint state/visual distributions; single-view depth ambiguity is offered as a plausible explanation, not isolated experimentally.

Fresh LIBERO initial states move scores by at most .62 points; fresh CALVIN chain samples move ATC by at most .107. All reported fresh-sample intervals span zero. These limited checks find no resolved sample-overfitting effect, rather than proving its absence for every model.

Complete original figure with every panel, label and legend.
Figure 4: SimplerEnv stack task with three changes that stay inside the training distribution. Left, rendered initial states for the original protocol and the three changes. Right, original fixed-grid stack success and success after each change.

6 Data-Source Dependence

SimplerEnv is often interpreted as real-Bridge-to-simulation transfer, but training sources are unrestricted. Train a separate roughly 22M DINOv2-S+MLP policy for each of four tasks on simulator demonstrations near its official evaluation grid.

With 120 demonstrations per task, or 480 total, the four policies jointly score 91/96=94.8%, near X-VLA’s cited 92/96=95.8%. This is not one multitask policy or a matched generalization comparison: generator fixes were selected to produce a high fixed-grid score.

The counterexample shows that the score alone does not identify whether a data-source gap was crossed or removed. Restricting training sources or reporting their proximity to test is the proposed remedy. The diagnostic is not applied to the other four benchmarks here.

7 Discussion

Broaden test variation over objects and scenes, require per-instance outcomes, and size evaluations for the improvement resolution being claimed. Difficulty should reflect intended capabilities rather than an uncontrolled distance between training and test data.

The discussion says the probe matches SOTA on LIBERO and CALVIN; Table 1 supports near-best performance on three LIBERO suites but only older-baseline competitiveness on CALVIN. The notes retain that narrower evidence.

These benchmarks remain useful for debugging and within-project iteration. A failed diagnostic limits the inference from a headline score; it need not invalidate every use of the benchmark. Item response theory and adaptive testing are proposed alternatives to fixed-population averages, not evaluated replacements in this paper.

8 Limitations

The diagnostics substitute for, rather than perform, an ideal shared-data and representative real-world ranking study. Available weights and original evaluation scripts limit the policies that can be rerun; some checks cover only three or four policies.

The authors deliberately tune some existence probes on the test protocol. Their results establish possible high-scoring shortcuts, not unbiased estimates or fair model rankings. Repeated-grid uncertainty, approximate test calibration and metric-rounding conventions further bound the conclusions.

9 Conclusion

Before interpreting a score as progress, test whether it requires the intended capability and whether its improvement is statistically resolvable. The released diagnostics make those questions operational; passing them is evidence within their tested scope, not comprehensive certification.

Appendix A Experiment Details

The appendix specifies each probe and evaluation intervention, supplies per-task/count results and gives confidence intervals. Figure 5 retains representative observations from all five benchmark environments.

Complete original figure with every panel, label and legend.
Figure 5: Example rendered observations from the five audited benchmarks, shown only to orient readers to the visual setup. From left to right, LIBERO [29], CALVIN [35], SimplerEnv [27], RoboCasa [39], and RoboTwin 2.0 [6].

A.1 Shortcut Solvability

Fine-tune DINOv2 end to end with an MLP head. LIBERO/CALVIN/SimplerEnv/RoboTwin use ViT-B/14; RoboCasa uses smaller ViT-S/14. Fixed-task multi-task probes use randomly initialized task embeddings, while RoboCasa trains separate per-task models. Images and proprioception follow each benchmark interface.

Sweep learning rate, augmentation and step budgets per suite/protocol; select the best tested checkpoint. LIBERO uses 50 trials/task, 500/suite. Best versus final scores differ by .8/.6/4.0/4.4 points across its four suites. CALVIN uses a 34-way task ID, 1,000 chains and length 360; best/final ABC→D is 3.242/3.149, a .093 gap rounded to “at most .09” in prose.

The authors attribute weaker D→D than ABC→D performance to segmented demonstrations that can start after the approach motion, leaving a task-ID policy without complete approach examples. This is their explanation, not a separately isolated causal ablation.

RoboCasa RSS24 uses 50 human demonstrations/task, AdamW for 1,500 epochs, object split B and five fixed layout/style pairs, with no camera/texture randomization. Its strongest per-task model gets 226/1,200=18.8%; alternative shared/Transformer-head probes score lower.

RoboTwin uses all 50 tasks with 50 clean plus 500 randomized demonstrations/task: 27,500 total. Evaluate 100 rollouts/task/condition; clean/randomized means are 60.4/59.4%. Table 3 retains every task rate, including large differences between near-solved and very weak tasks.

SimplerEnv’s standard-Bridge probe adds SigLIP text features because instruction lookup is unavailable, and gets zero on four WidowX tasks. Foundation baselines are selected by protocol comparability/citation prominence; best results exclude RL fine-tuning. The X-VLA RoboTwin comparison comes from ABot-M0’s matched-data evaluation, not X-VLA’s original differing protocol.

Table 2: Best-checkpoint and final-checkpoint probe scores. The best-checkpoint column is the best of all checkpoints scored on the reported suite and is what the main table reports. The final-checkpoint column is the score at the last training step, with no checkpoint selection. LIBERO values are success rate over 500500 trials per suite. CALVIN values are average tasks completed out of 55 over 10001000 sequences.
Benchmark Suite or protocol Best checkpoint Final checkpoint
LIBERO Spatial 99.0%99.0\% 98.2%98.2\%
Object 100.0%100.0\% 99.4%99.4\%
Goal 98.8%98.8\% 94.8%94.8\%
Long 92.4%92.4\% 88.0%88.0\%
CALVIN D→\toD 3.1233.123 3.1233.123
ABC→\toD 3.2423.242 3.1493.149
Table 3: DINO+MLP probe policy per-task success rates on the RoboTwin 2.0 benchmark.
Task Clean Rand. Task Clean Rand.
adjust_bottle 95.0% 90.0% place_can_basket 50.0% 57.0%
beat_block_hammer 47.0% 42.0% place_cans_plasticbox 97.0% 97.0%
blocks_ranking_rgb 44.0% 55.0% place_container_plate 62.0% 51.0%
blocks_ranking_size 36.0% 40.0% place_dual_shoes 24.0% 27.0%
click_alarmclock 64.0% 68.0% place_empty_cup 90.0% 87.0%
click_bell 83.0% 81.0% place_fan 42.0% 43.0%
dump_bin_bigbin 48.0% 50.0% place_mouse_pad 15.0% 20.0%
grab_roller 100.0% 100.0% place_object_basket 56.0% 62.0%
handover_block 67.0% 67.0% place_object_scale 50.0% 47.0%
handover_mic 86.0% 80.0% place_object_stand 79.0% 76.0%
hanging_mug 8.0% 16.0% place_phone_stand 58.0% 58.0%
lift_pot 99.0% 100.0% place_shoe 69.0% 67.0%
move_can_pot 69.0% 61.0% press_stapler 99.0% 92.0%
move_pillbottle_pad 63.0% 59.0% put_bottles_dustbin 9.0% 7.0%
move_playingcard_away 83.0% 86.0% put_object_cabinet 50.0% 36.0%
move_stapler_pad 6.0% 5.0% rotate_qrcode 42.0% 44.0%
open_laptop 96.0% 95.0% scan_object 82.0% 77.0%
open_microwave 14.0% 21.0% shake_bottle 100.0% 100.0%
pick_diverse_bottles 83.0% 68.0% shake_bottle_horizontally 100.0% 100.0%
pick_dual_bottles 97.0% 93.0% stack_blocks_three 28.0% 31.0%
place_a2b_left 30.0% 24.0% stack_blocks_two 62.0% 58.0%
place_a2b_right 25.0% 32.0% stack_bowls_three 20.0% 25.0%
place_bread_basket 83.0% 80.0% stack_bowls_two 60.0% 58.0%
place_bread_skillet 80.0% 69.0% stamp_seal 33.0% 29.0%
place_burger_fries 91.0% 94.0% turn_switch 46.0% 45.0%
Average 60.4% 59.4%

A.2 Creeping Overfitting

The SimplerEnv interventions use a common local rollout protocol and report changes from matched calibration. The appendix also gives uncertainty for CALVIN pose resampling and fresh LIBERO/CALVIN test samples.

Setup.

Evaluate CogACT-Base, InternVLA-M1, X-VLA-WidowX and Dexbotic/DB-MemVLA on the stack task with the standard 60-step limit. These local results need not equal published results obtained with longer episode limits.

Matched local calibration.

Repeat the 24 official start states twelve times, giving 288 calibration episodes per policy on the same hardware as each intervention. Matching protocol and hardware reduces those confounds, but repeated start states are not 288 distinct scenes.

The three changes.

Run 288 episodes each for reversed yellow-on-green language, target cubes supported by other blocks, and randomized cube/initial-arm poses. Hold scene, camera, robot, success rule and task fixed. These three belong to a released seven-condition, 2,016-episode run per policy.

Why the changes stay in distribution.

Among 53,192 Bridge demonstrations, instruction text mentions block/cube/tower 869/751/282 times; green-on-yellow appears 41 times and yellow-on-green 30. These counts support the presence of similar conditions, but text frequency alone does not prove joint distribution equivalence or coverage of every altered simulator state.

Results.

Table 4 reports calibration followed by each altered condition. Calibration success ranges from CogACT 20.8% to X-VLA 59.7%; no policy wins every intervention. Preserve the common 60-step evaluation when comparing these local numbers.

Table 4: SimplerEnv stack success rate (%) on the matched local calibration and the three in-distribution changes, 288288 episodes per policy per column. Some calibration values differ from the numbers these policies report. This is expected, not an error. Several of these policies report under longer, easier episode step limits, while we hold every policy to the same shorter standard step limit.
Policy Calibration Reverse language Stacked support Random pose + arm
CogACT-Base 20.8 9.7 22.2 10.8
InternVLA-M1 25.3 16.0 20.8 16.0
X-VLA-WidowX 59.7 53.1 31.3 55.6
Dexbotic (DB-MemVLA) 44.8 47.6 27.4 38.9

Confidence intervals.

Define drop as matched calibration minus altered/fresh performance, so positive means degradation. Compute two-sided 95% intervals from per-episode or per-chain artifacts, with different estimators for binary, paired and independent comparisons.

Equation
drop=(matched local calibration)−(altered or fresh result),\text{drop}=(\text{matched local calibration})-(\text{altered or fresh result}),

Interval methods.

For binary success, combine Wilson intervals into a Newcombe difference interval. This treats rollout rows as independent Bernoulli observations; it does not model SimplerEnv’s twelve repetitions of each fixed grid state.

For CALVIN pose changes, use paired sequence differences on the shared 1,000-chain manifest, including a binary five-task-completion version. For fresh CALVIN samples, compare 1,000 calibration chains with two pooled 1,000-chain manifests using an independent two-sample Wald interval. All original formulas remain below.

Equations
center=p+z2/(2​n)1+z2/n,half=z​p⁡(1−p)/n+z2/(4​n2)1+z2/n,\text{center}=\frac{p+z^{2}/(2n)}{1+z^{2}/n},\qquad\text{half}=\frac{z\,\sqrt{p(1-p)/n+z^{2}/(4n^{2})}}{1+z^{2}/n},
[L,U]=[center−half,center+half].[L,U]=[\,\text{center}-\text{half},\ \text{center}+\text{half}\,].
lower=drop−(pc−Lc)2+(Ua−pa)2,upper=drop+(Uc−pc)2+(pa−La)2.\text{lower}=\text{drop}-\sqrt{(p_{c}-L_{c})^{2}+(U_{a}-p_{a})^{2}},\qquad\text{upper}=\text{drop}+\sqrt{(U_{c}-p_{c})^{2}+(p_{a}-L_{a})^{2}}.
di=(calibration count)i−(resampled count)i,drop=meani⁡di,d_{i}=(\text{calibration count})_{i}-(\text{resampled count})_{i},\qquad\text{drop}=\operatorname{mean}_{i}d_{i},
CI=drop±z​sd⁡(di)1000.\text{CI}=\text{drop}\pm z\,\frac{\operatorname{sd}(d_{i})}{\sqrt{1000}}.
drop=mean⁡(calibration)−mean⁡(fresh),\text{drop}=\operatorname{mean}(\text{calibration})-\operatorname{mean}(\text{fresh}),
SE=var⁡(calibration)1000+var⁡(fresh)2000,CI=drop±z​SE.\text{SE}=\sqrt{\frac{\operatorname{var}(\text{calibration})}{1000}+\frac{\operatorname{var}(\text{fresh})}{2000}},\qquad\text{CI}=\text{drop}\pm z\,\text{SE}.

Distribution-overfitting drops.

On SimplerEnv, reversed-language drops resolve above zero for CogACT and InternVLA; stacked-support drops resolve for X-VLA and Dexbotic. Random pose/arm drops resolve for CogACT and InternVLA. Several other intervals span zero, and some point estimates improve.

CALVIN pose resampling reduces ATC by 1.027 [.890,1.164], .749 [.624,.874] and .498 [.372,.624] for X-VLA/GR-1/RoboFlamingo. Five-task completion drops by 25.0/13.5/6.4 percentage points; all corresponding paired intervals are positive.

Table 5: SimplerEnv distribution-overfitting drops. Calibration and altered columns are success counts out of 288288 episodes per policy. Drop and 95% CI are in percentage points, with a positive drop meaning the altered test scored lower than the matched local calibration.
Policy Change Calibration Altered Drop 95% CI
CogACT-Base Reverse language 60/28860/288 28/28828/288 +11.11+11.11 [+5.26,+16.95][+5.26,+16.95]
CogACT-Base Stacked support 60/28860/288 64/28864/288 −1.39-1.39 [−8.09,+5.33][-8.09,+5.33]
CogACT-Base Random pose + arm 60/28860/288 31/28831/288 +10.07+10.07 [+4.13,+15.99][+4.13,+15.99]
InternVLA-M1 Reverse language 73/28873/288 46/28846/288 +9.38+9.38 [+2.76,+15.91][+2.76,+15.91]
InternVLA-M1 Stacked support 73/28873/288 60/28860/288 +4.51+4.51 [−2.38,+11.35][-2.38,+11.35]
InternVLA-M1 Random pose + arm 73/28873/288 46/28846/288 +9.38+9.38 [+2.76,+15.91][+2.76,+15.91]
X-VLA-WidowX Reverse language 172/288172/288 153/288153/288 +6.60+6.60 [−1.49,+14.57][-1.49,+14.57]
X-VLA-WidowX Stacked support 172/288172/288 90/28890/288 +28.47+28.47 [+20.46,+35.96][+20.46,+35.96]
X-VLA-WidowX Random pose + arm 172/288172/288 160/288160/288 +4.17+4.17 [−3.88,+12.14][-3.88,+12.14]
Dexbotic (DB-MemVLA) Reverse language 129/288129/288 137/288137/288 −2.78-2.78 [−10.84,+5.33][-10.84,+5.33]
Dexbotic (DB-MemVLA) Stacked support 129/288129/288 79/28879/288 +17.36+17.36 [+9.54,+24.89][+9.54,+24.89]
Dexbotic (DB-MemVLA) Random pose + arm 129/288129/288 112/288112/288 +5.90+5.90 [−2.14,+13.84][-2.14,+13.84]
Table 6: CALVIN distribution-overfitting drops on the shared 10001000-sequence manifest. ATC drops are in tasks out of 55; 5/5 chain-success drops are in percentage points. A positive drop means the resampled-pose test scored lower than the calibration.
Policy Metric Calibration Resampled Drop 95% CI
X-VLA ATC 4.1654.165 3.1383.138 +1.027+1.027 [+0.890,+1.164][+0.890,+1.164]
X-VLA 5/5 chain success 709/1000709/1000 459/1000459/1000 +25.00+25.00 [+21.29,+28.71][+21.29,+28.71]
GR-1 ATC 3.2443.244 2.4952.495 +0.749+0.749 [+0.624,+0.874][+0.624,+0.874]
GR-1 5/5 chain success 431/1000431/1000 296/1000296/1000 +13.50+13.50 [+10.23,+16.77][+10.23,+16.77]
RoboFlamingo ATC 2.3672.367 1.8691.869 +0.498+0.498 [+0.372,+0.624][+0.372,+0.624]
RoboFlamingo 5/5 chain success 209/1000209/1000 145/1000145/1000 +6.40+6.40 [+3.57,+9.23][+3.57,+9.23]

Sample-overfitting drops.

Fresh LIBERO evaluation uses 10,000 rollouts/policy versus 2,000 for calibration. Drops are +.62 points for Spatial Forcing, −.30 for SimVLA and +.14 for π0.5/LeRobot, each interval including zero.

Fresh CALVIN ATC drops are −.0145 for X-VLA, +.107 for GR-1 and +.069 for RoboFlamingo, again with intervals including zero. This is much smaller than the pose-resampling effect, but does not establish exact equivalence.

Table 7: LIBERO sample-overfitting drops. Calibration is success counts out of 20002000 episodes per policy; the fresh sample redraws the init-state file from the same generator with 10,00010{,}000 rollouts per policy. Drop and 95% CI are in percentage points.
Policy Metric Calibration Fresh sample Drop 95% CI
Spatial Forcing success rate 1956/20001956/2000 9718/100009718/10000 +0.62+0.62 [−0.18,+1.27][-0.18,+1.27]
SimVLA success rate 1946/20001946/2000 9760/100009760/10000 −0.30-0.30 [−1.15,+0.40][-1.15,+0.40]
π0.5\pi_{0.5} / LeRobot success rate 1951/20001951/2000 9741/100009741/10000 +0.14+0.14 [−0.69,+0.82][-0.69,+0.82]
Table 8: CALVIN sample-overfitting drops. The fresh sample pools two predeclared independent fresh 10001000-sequence manifests. ATC is in tasks out of 55, with a positive drop meaning the fresh sample scored lower than the calibration.
Policy Metric Calibration Fresh sample Drop 95% CI
X-VLA ATC 4.1654.165 4.17954.1795 −0.0145-0.0145 [−0.1287,+0.0997][-0.1287,+0.0997]
GR-1 ATC 3.2443.244 3.13703.1370 +0.1070+0.1070 [−0.0373,+0.2513][-0.0373,+0.2513]
RoboFlamingo ATC 2.3672.367 2.29802.2980 +0.0690+0.0690 [−0.0677,+0.2057][-0.0677,+0.2057]

Confidence-interval source artifacts.

Intervals are recomputed from rollout CSV/JSON outputs rather than rounded summaries. Fresh CALVIN manifests are predeclared; a small number of LIBERO cells use documented retry reruns. These artifacts define which episodes enter each estimate.

A.3 Data-Source Dependence

The data-source probe deliberately trains near SimplerEnv’s public evaluation grid. It tests whether the score identifies real-to-simulation transfer, not whether synthetic demonstrations are equivalent to real Bridge data.

Protocol.

Use stack, carrot, spoon and eggplant with 24 official grid instances each, totaling 96 closed-loop evaluations. Evaluation uses unjittered IDs 0–23 and benchmark episode limits and success checks; report counts because each task has few trials.

Scripted demonstrations.

Generate five successful demonstrations around each grid ID, 120/task. Add .0025-m XY jitter and up to 15° yaw jitter, pad terminal open/place behavior, and canonicalize carrot gripper yaw modulo π. Public test centers deliberately shape the training distribution.

Policy and training.

Train four separate DINOv2-S+MLP behavior-cloning models, fine-tuning pretrained visual features with proprioception and no image augmentation. Normalize environment-format actions and execute five-step chunks with replanning every five steps; include terminal padding.

No checkpoint selection is used, but demonstration-generator fixes were selected to obtain a high fixed-grid score. Results are 24/24 stack, 23/24 carrot, 21/24 spoon and 23/24 eggplant: 91/96 overall.

Table 9: SimplerEnv WidowX data-source-dependence results. Each task row uses a separate DINOv2-S + MLP policy trained only on 120120 scripted demonstrations generated in the same simulator around the official evaluation grid; the final row aggregates all four tasks. Evaluation uses the official 2424-instance grid for each task.
Task Scripted demos Eval successes Success rate
Stack 120120 24/2424/24 100.0%100.0\%
Carrot 120120 23/2423/24 95.8%95.8\%
Spoon 120120 21/2421/24 87.5%87.5\%
Eggplant 120120 23/2423/24 95.8%95.8\%
All four 480480 91/9691/96 94.8%94.8\%

Interpretation.

This is an existence construction on a fixed grid. It is neither an unbiased success-rate estimate nor a controlled claim that a small policy outperforms X-VLA. Its purpose is to show that unrestricted data can remove the transfer gap the benchmark score is assumed to measure.

Appendix B Bitwise Nondeterminism in Policy Rollouts

Closed-loop rollouts can amplify tiny hardware-dependent numerical changes into different observations, actions and outcomes. The appendix distinguishes strict bitwise identity from protocol-faithful statistical reproduction.

B.1 Reproduction Targets

Four targets range from bitwise identity, through matching per-episode outcomes and aggregate scores, to documented protocol/statistical compatibility. This paper adopts the last while using bitwise traces diagnostically.

That standard still requires checkpoints, revisions, commands, seeds, initial states, normalization, action/success processing and hardware/software metadata. Agreement of one aggregate decimal does not imply identical episodes or prove significance.

B.2 Diagnostic Evidence

In one LIBERO-Goal task, changing CPU while retaining RTX A6000 first alters simulator state at step 0, images at 54, actions at 58 and reward/done at 88. Contact instrumentation links growth to bowl/table constraints; the plot uses a log floor for zeros.

The broader table covers eight CPU/GPU contrasts across four stacks. LIBERO CPU traces differ in all ten tasks, reaching pixels/actions in five. CALVIN’s CPU sample has zero divergences in fifty sequences, while its GPU contrast diverges immediately. Same-node controls are identical.

These examples show cross-hardware determinism can fail, not that it always fails or necessarily changes success. Both SimplerEnv GPU traces fail despite different paths. Open-loop replay alone omits the feedback mechanism under investigation.

Complete original figure with every panel, label and legend.
Figure 6: Closed-loop propagation in one LIBERO-Goal task 8 hardware comparison. With policy, seed, initial state, wrapper, and rollout code fixed, changing CPU hardware while keeping the GPU class fixed first changes post-step simulator state at step 0; the difference reaches rendered images at step 54, actions at step 58, and reward/done at step 88. State and action curves use L∞L_{\infty} distance; the image curve uses mean absolute pixel difference. Zeros are drawn at the log-scale floor.
Table 10: Diagnostic closed-loop trace-comparison tests. Each row is one CPU or GPU hardware contrast for one benchmark stack. Same-node controls were bitwise identical in the recorded arrays; the goal is not to survey all benchmark stacks, but to show that bitwise determinism can fail even when the benchmark protocol is fixed.
Stack Hardware contrast Observed propagation Bitwise identical?
LIBERO / OpenVLA-OFT CPU: AMD EPYC 74F3 + RTX A6000 versus Intel Xeon Silver 4210R + RTX A6000 10/1010/10 tasks diverged in simulator state; 5/105/10 reached pixels and actions. No
LIBERO / OpenVLA-OFT GPU: same AMD EPYC 74F3 CPU model, RTX A6000 versus RTX 6000 Ada Raw and processed actions diverged at step 10, followed by simulator state, image, and reward/done divergence. No
CALVIN / GR-1 CPU: AMD-CPU node with RTX A6000 versus Intel-CPU node with RTX A6000 0/500/50 official sequence pairs diverged in this diagnostic sample. Yes
CALVIN / GR-1 GPU: same AMD EPYC 74F3 CPU model, RTX A6000 versus RTX 6000 Ada Raw and processed actions diverged at step 0, followed by simulator state and image-hash divergence. No
SimplerEnv / CogACT CPU: AMD-CPU node with RTX A6000 versus Intel-CPU node with RTX A6000 Actions and post-step simulator state diverged at step 0; images at step 3; reward/done/success at step 38. No
SimplerEnv / CogACT GPU: same AMD EPYC 74F3 CPU model, RTX A6000 versus RTX 6000 Ada Actions and simulator state diverged from step 0, followed by image divergence; both traces still ended unsuccessfully. No
RoboCasa / StarVLA CPU: AMD-CPU node with RTX A6000 versus Intel-CPU node with RTX A6000 Simulator and contact traces diverged immediately; image/action divergence followed at step 4 and reward/success at step 19. No
RoboCasa / StarVLA GPU: same AMD EPYC 74F3 CPU model, RTX A6000 versus RTX 6000 Ada Actions, contact traces, simulator state, and images diverged from the first steps; reward/success diverged at step 18. No

B.3 Implications for This Paper

Evaluate reproduced results using protocol fidelity, per-instance outcomes and appropriate uncertainty. A small compatible score mismatch need not refute reproduction, while an exact score match cannot establish a significant improvement.

Release the complete execution stack and outcomes so numerical divergence can be separated from changed protocol. Bitwise identity remains useful when achievable, but is not proposed as the sole cross-hardware pass/fail criterion.

Appendix C Top-Line Cutoffs for Bounded-Score Wald Tests

Derive aggregate-only bounds for a specified bounded-score Wald decision rule. Binary benchmarks use R=1; CALVIN uses R=5, one stratum and 1,000 samples. The derivation concerns fixed task populations, equal samples per task and integer scores.

C.1 Paired Task-Stratified Wald Rule

Treat T tasks as fixed strata with S>1 independent paired samples each. Average paired score differences over tasks; estimate variance from within-task differences. With fixed T and S growing, a nonzero-variance central limit theorem motivates the one-sided Wald rule.

For finite samples this remains a nominal asymptotic procedure. If estimated variance is zero, define the statistic as positive infinity, zero or negative infinity according to the gap sign; reject only above the normal critical value. The later exact envelopes are exact about this rule, not exact type-I-error calibration.

Equations
YA,t,i,YB,t,i∈{0,1,…,R}.Y_{A,t,i},Y_{B,t,i}\in\{0,1,\ldots,R\}.
δt,i=YB,t,i−YA,t,i∈{−R,…,R}.\delta_{t,i}=Y_{B,t,i}-Y_{A,t,i}\in\{-R,\ldots,R\}.
Δt=𝔼⁡[δt,i]=μB,t−μA,t,σt2=Var⁡(δt,i),\Delta_{t}=\mathbb{E}[\delta_{t,i}]=\mu_{B,t}-\mu_{A,t},\qquad\sigma_{t}^{2}=\operatorname{Var}(\delta_{t,i}),
Δ=1T​∑t=1TΔt.\Delta=\frac{1}{T}\sum_{t=1}^{T}\Delta_{t}.
Δ^t=1S​∑i=1Sδt,i,Δ^=1T​∑t=1TΔ^t.\widehat{\Delta}_{t}=\frac{1}{S}\sum_{i=1}^{S}\delta_{t,i},\qquad\widehat{\Delta}=\frac{1}{T}\sum_{t=1}^{T}\widehat{\Delta}_{t}.
Δ^−Δ∑t=1Tσt2T2​S​⟶𝑑​𝒩​(0,1),\frac{\widehat{\Delta}-\Delta}{\sqrt{\sum_{t=1}^{T}\frac{\sigma_{t}^{2}}{T^{2}S}}}\overset{d}{\longrightarrow}\mathcal{N}(0,1),
dt=∑i=1Sδt,i,st=∑i=1Sδt,i2.d_{t}=\sum_{i=1}^{S}\delta_{t,i},\qquad s_{t}=\sum_{i=1}^{S}\delta_{t,i}^{2}.
σ^t2=1S−1​(st−dt2S).\widehat{\sigma}_{t}^{2}=\frac{1}{S-1}\left(s_{t}-\frac{d_{t}^{2}}{S}\right).
V^=∑t=1Tσ^t2T2​S=1T2​S​(S−1)​∑t=1T(st−dt2S),\widehat{V}=\sum_{t=1}^{T}\frac{\widehat{\sigma}_{t}^{2}}{T^{2}S}=\frac{1}{T^{2}S(S-1)}\sum_{t=1}^{T}\left(s_{t}-\frac{d_{t}^{2}}{S}\right),
Z=Δ^V^=∑t=1TdtSS−1​∑t=1T(st−dt2S).Z=\frac{\widehat{\Delta}}{\sqrt{\widehat{V}}}=\frac{\sum_{t=1}^{T}d_{t}}{\sqrt{\frac{S}{S-1}\sum_{t=1}^{T}\left(s_{t}-\frac{d_{t}^{2}}{S}\right)}}.
Z∗={Δ^/V^,V^>0,+∞,V^=0​ and ​Δ^>0,0,V^=0​ and ​Δ^=0,−∞,V^=0​ and ​Δ^<0.Z_{*}=\begin{cases}\widehat{\Delta}/\sqrt{\widehat{V}},&\widehat{V}>0,\\ +\infty,&\widehat{V}=0\text{ and }\widehat{\Delta}>0,\\ 0,&\widehat{V}=0\text{ and }\widehat{\Delta}=0,\\ -\infty,&\widehat{V}=0\text{ and }\widehat{\Delta}<0.\end{cases}
H0:Δ≤0againstH1:Δ>0,H_{0}:\Delta\leq 0\qquad\text{against}\qquad H_{1}:\Delta>0,
Z∗>z1−α.Z_{*}>z_{1-\alpha}.

C.2 Top-Line Count Notation

Convert reported means into integer totals A and B across N=T·S observations, with positive count gap L=B−A. The numerator is fixed; the unknown paired table affects only Q, the sum of within-task squared-difference residuals.

Absorb the sample correction and critical normal quantile into cα. For positive L, rejection is exactly the condition L>cα√Q, including the declared zero-variance convention. Feasible gaps cannot exceed R·N−A.

Equations
N=T​S,A=N​μ^A,B=N​μ^B,L=B−A,N=TS,\qquad A=N\widehat{\mu}_{A},\qquad B=N\widehat{\mu}_{B},\qquad L=B-A,
A,B∈{0,…,R​N},B>A.A,B\in\{0,\ldots,RN\},\qquad B>A.
g=μ^B−μ^A=LN.g=\widehat{\mu}_{B}-\widehat{\mu}_{A}=\frac{L}{N}.
ℒA={1,…,R​N−A}.\mathcal{L}_{A}=\{1,\ldots,RN-A\}.
∑t=1Tdt=L.\sum_{t=1}^{T}d_{t}=L.
Q=∑t=1T(st−dt2S).Q=\sum_{t=1}^{T}\left(s_{t}-\frac{d_{t}^{2}}{S}\right).
cα=z1−α​SS−1.c_{\alpha}=z_{1-\alpha}\sqrt{\frac{S}{S-1}}.
L>cα​Q.L>c_{\alpha}\sqrt{Q}.

C.3 Exact Lower Envelope for Paired Evaluations

Minimize squared differences within each task by making integer differences as equal as possible. Combining remainders across tasks yields the exact lower envelope Qlo=r−r²/S, where r is L modulo S.

Construct constant nonnegative task differences plus one residual task, then allocate baseline scores within remaining capacity to realize A. This proves attainability. If even this minimum-variance table cannot reject, no compatible paired table can.

The first feasible rejecting count gap has the stated floor-function expression; if it exceeds R·N−A, the feasibility cutoff is infinite. All remainder identities, construction constraints and cutoff formulas are retained.

Equations
x=q​S+ρS​(x),q∈ℤ,ρS​(x)∈{0,…,S−1}.x=qS+\rho_{S}(x),\qquad q\in\mathbb{Z},\qquad\rho_{S}(x)\in\{0,\ldots,S-1\}.
st−dt2S≥ρS​(dt)−ρS​(dt)2S.s_{t}-\frac{d_{t}^{2}}{S}\geq\rho_{S}(d_{t})-\frac{\rho_{S}(d_{t})^{2}}{S}.
ψ⁡(r)=r−r2S,0≤r<S.\psi(r)=r-\frac{r^{2}}{S},\qquad 0\leq r<S.
ψ⁡(x)+ψ⁡(y)≥ψ⁡((x+y)modS).\psi(x)+\psi(y)\geq\psi((x+y)\bmod S).
L=q​S+r,q=⌊LS⌋,0≤r<S.L=qS+r,\qquad q=\left\lfloor\frac{L}{S}\right\rfloor,\qquad 0\leq r<S.
Q≥Qlo​(L,T,S),Q\geq Q_{\mathrm{lo}}(L;T,S),
Qlo​(L,T,S)=r−r2S.Q_{\mathrm{lo}}(L;T,S)=r-\frac{r^{2}}{S}.
∑t=1Tct=q.\sum_{t=1}^{T}c_{t}=q.
δt,i=ctfor all ​t,i,\delta_{t,i}=c_{t}\quad\text{for all }t,i,
δt0,i=ct0+1for ​i=1,…,r.\delta_{t_{0},i}=c_{t_{0}}+1\quad\text{for }i=1,\ldots,r.
r−r2S.r-\frac{r^{2}}{S}.
∑t=1T∑i=1S(R−δt,i)=R​N−L≥A,\sum_{t=1}^{T}\sum_{i=1}^{S}(R-\delta_{t,i})=RN-L\geq A,
YA,t,i=Xt,i,YB,t,i=Xt,i+δt,i.Y_{A,t,i}=X_{t,i},\qquad Y_{B,t,i}=X_{t,i}+\delta_{t,i}.
Q=Qlo​(L,T,S).Q=Q_{\mathrm{lo}}(L;T,S).
L≤cα​Qlo​(L,T,S).L\leq c_{\alpha}\sqrt{Q_{\mathrm{lo}}(L;T,S)}.
L∃​(A,T,S,R,α)=min⁡{L∈ℒA:L>cα​Qlo​(L,T,S)},L_{\exists}(A;T,S,R,\alpha)=\min\left\{L\in\mathcal{L}_{A}:L>c_{\alpha}\sqrt{Q_{\mathrm{lo}}(L;T,S)}\right\},
δ∃​(μ^A,T,S,R,α)=L∃​(A,T,S,R,α)N.\delta_{\exists}(\widehat{\mu}_{A};T,S,R,\alpha)=\frac{L_{\exists}(A;T,S,R,\alpha)}{N}.
g<δ∃​(μ^A,T,S,R,α)g<\delta_{\exists}(\widehat{\mu}_{A};T,S,R,\alpha)
Qlo​(L,T,S)=L−L2Sfor ​1≤L≤S,Q_{\mathrm{lo}}(L;T,S)=L-\frac{L^{2}}{S}\qquad\text{for }1\leq L\leq S,
ℓ∃⋆​(S,α)=1+⌊z1−α2​SS−1+z1−α2⌋.\ell_{\exists}^{\star}(S,\alpha)=1+\left\lfloor\frac{z_{1-\alpha}^{2}S}{S-1+z_{1-\alpha}^{2}}\right\rfloor.
L∃​(A,T,S,R,α)={ℓ∃⋆​(S,α),if ​ℓ∃⋆​(S,α)≤R​N−A,+∞,otherwise.L_{\exists}(A;T,S,R,\alpha)=\begin{cases}\ell_{\exists}^{\star}(S,\alpha),&\text{if }\ell_{\exists}^{\star}(S,\alpha)\leq RN-A,\\ +\infty,&\text{otherwise.}\end{cases}

C.4 Exact Upper Envelope for Paired Evaluations

For each task’s marginal totals, maximize squared differences with an integer contingency table over scores 0…R. An equivalent construction packs positive and negative movements into as few bounded-size samples as possible, choosing the largest feasible opposing movement.

For binary outcomes this reduces to maximum discordance minus the squared task-total difference divided by S. Dynamic programming then allocates global A and B across tasks and computes the exact upper envelope Qhi.

If the largest-variance compatible table rejects, all paired tables do. The scalar guarantee is a suffix cutoff requiring rejection for every larger feasible gap; the pointwise Qhi condition can be sharper for a specific gap. Preserve the distinction rather than assuming monotonicity without the suffix definition.

Equations
Mmax(R)(a,b;S)=max{nu​v}∑u=0R∑v=0Rnu​v(v−u)2M_{\max}^{(R)}(a,b;S)=\max_{\{n_{uv}\}}\sum_{u=0}^{R}\sum_{v=0}^{R}n_{uv}(v-u)^{2}
nu​v∈ℤ≥0,n_{uv}\in\mathbb{Z}_{\geq 0},
∑u=0R∑v=0Rnu​v=S,\sum_{u=0}^{R}\sum_{v=0}^{R}n_{uv}=S,
∑u=0R∑v=0Ru​nu​v=a,∑u=0R∑v=0Rv​nu​v=b.\sum_{u=0}^{R}\sum_{v=0}^{R}u\,n_{uv}=a,\qquad\sum_{u=0}^{R}\sum_{v=0}^{R}v\,n_{uv}=b.
qmax(R)​(a,b,S)=Mmax(R)​(a,b,S)−(b−a)2S.q_{\max}^{(R)}(a,b;S)=M_{\max}^{(R)}(a,b;S)-\frac{(b-a)^{2}}{S}.
D=b−a.D=b-a.
ΠR​(x)=qx​R2+rx2,νR​(x)={0,x=0,⌈x/R⌉,x>0.\Pi_{R}(x)=q_{x}R^{2}+r_{x}^{2},\qquad\nu_{R}(x)=\begin{cases}0,&x=0,\\ \left\lceil x/R\right\rceil,&x>0.\end{cases}
0≤j≤a,0≤j≤R​S−b,0\leq j\leq a,\qquad 0\leq j\leq RS-b,
νR​(D+j)+νR​(j)≤S.\nu_{R}(D+j)+\nu_{R}(j)\leq S.
Mmax(R)​(a,b,S)=maxj∈𝒥⁡(a,b)⁡{ΠR​(D+j)+ΠR​(j)},M_{\max}^{(R)}(a,b;S)=\max_{j\in\mathcal{J}(a,b)}\left\{\Pi_{R}(D+j)+\Pi_{R}(j)\right\},
𝒥(a,b)={j∈ℤ≥0:j≤a,j≤RS−b,νR(D+j)+νR(j)≤S}.\mathcal{J}(a,b)=\left\{j\in\mathbb{Z}_{\geq 0}:j\leq a,\;j\leq RS-b,\;\nu_{R}(D+j)+\nu_{R}(j)\leq S\right\}.
qmax(1)​(a,b,S)=min⁡(a+b, 2​S−a−b)−(b−a)2S.q_{\max}^{(1)}(a,b;S)=\min(a+b,\;2S-a-b)-\frac{(b-a)^{2}}{S}.
F0​(0,0)=0,F0​(u,v)=−∞for ​(u,v)≠(0,0).F_{0}(0,0)=0,\qquad F_{0}(u,v)=-\infty\quad\text{for }(u,v)\neq(0,0).
Fj+1​(u+a′,v+b′)=max⁡{Fj+1​(u+a′,v+b′),Fj​(u,v)+qmax(R)​(a′,b′,S)}F_{j+1}(u+a^{\prime},v+b^{\prime})=\max\left\{F_{j+1}(u+a^{\prime},v+b^{\prime}),\;F_{j}(u,v)+q_{\max}^{(R)}(a^{\prime},b^{\prime};S)\right\}
0≤a′≤R​S,0≤b′≤R​S.0\leq a^{\prime}\leq RS,\qquad 0\leq b^{\prime}\leq RS.
Qhi​(A,L,T,S,R)=FT​(A,A+L).Q_{\mathrm{hi}}(A,L;T,S,R)=F_{T}(A,A+L).
Q≤Qhi​(A,L,T,S,R),Q\leq Q_{\mathrm{hi}}(A,L;T,S,R),
L>cα​Qhi​(A,L,T,S,R).L>c_{\alpha}\sqrt{Q_{\mathrm{hi}}(A,L;T,S,R)}.
L≤cα​Qhi​(A,L,T,S,R),L\leq c_{\alpha}\sqrt{Q_{\mathrm{hi}}(A,L;T,S,R)},
L∀​(A,T,S,R,α)=min⁡{ℓ∈ℒA:for every ​L∈ℒA​ with ​L≥ℓ,L>cα​Qhi​(A,L,T,S,R)},L_{\forall}(A;T,S,R,\alpha)=\min\left\{\ell\in\mathcal{L}_{A}:\begin{array}[]{l}\text{for every }L\in\mathcal{L}_{A}\text{ with }L\geq\ell,\\[2.0pt] L>c_{\alpha}\sqrt{Q_{\mathrm{hi}}(A,L;T,S,R)}\end{array}\right\},
δ∀​(μ^A,T,S,R,α)=L∀​(A,T,S,R,α)N.\delta_{\forall}(\widehat{\mu}_{A};T,S,R,\alpha)=\frac{L_{\forall}(A;T,S,R,\alpha)}{N}.
g≥δ∀​(μ^A,T,S,R,α)g\geq\delta_{\forall}(\widehat{\mu}_{A};T,S,R,\alpha)
L>cα​Qhi​(A,L,T,S,R).L>c_{\alpha}\sqrt{Q_{\mathrm{hi}}(A,L;T,S,R)}.

C.5 Sharp Paired Classification

For fixed exact paired totals, the lower envelope identifies impossible rejection, the upper envelope identifies forced rejection, and the interval between permits both decisions. This intermediate case is the third displayed classification, despite the source’s subsequent “middle line” wording.

Equation
L≤cα​Qlo​(L,T,S)⟺no compatible paired outcome table rejects,L>cα​Qhi​(A,L,T,S,R)⟺every compatible paired outcome table rejects,cα​Qlo​(L,T,S)<L≤cα​Qhi​(A,L,T,S,R)⟺some compatible paired outcome tables reject,and some compatible paired outcome tables do not reject.\begin{array}[]{ll}L\leq c_{\alpha}\sqrt{Q_{\mathrm{lo}}(L;T,S)}&\Longleftrightarrow\text{no compatible paired outcome table rejects},\\[6.0pt] L>c_{\alpha}\sqrt{Q_{\mathrm{hi}}(A,L;T,S,R)}&\Longleftrightarrow\text{every compatible paired outcome table rejects},\\[6.0pt] c_{\alpha}\sqrt{Q_{\mathrm{lo}}(L;T,S)}<L\leq c_{\alpha}\sqrt{Q_{\mathrm{hi}}(A,L;T,S,R)}&\Longleftrightarrow\begin{array}[]{l}\text{some compatible paired outcome tables reject,}\\ \text{and some compatible paired outcome tables do not reject.}\end{array}\end{array}

C.6 Independently Sampled Evaluations

Independent evaluations require separate within-task variances, not paired differences. Their denominator still lies between the paired envelopes: integer-valued minimum variance supplies the lower bound, and an attainable nonpositive-covariance pairing supplies the upper bound.

Thus the paired impossible/guaranteed cutoffs remain conservative under the stated identical task/sample structure. The intermediate region is not sharp for independent samples; a tighter two-sample analysis could resolve some comparisons.

Equations
at=∑i=1SYA,t,i,bt=∑i=1SYB,t,i,a_{t}=\sum_{i=1}^{S}Y_{A,t,i},\qquad b_{t}=\sum_{i=1}^{S}Y_{B,t,i},
ut=∑i=1SYA,t,i2,vt=∑i=1SYB,t,i2.u_{t}=\sum_{i=1}^{S}Y_{A,t,i}^{2},\qquad v_{t}=\sum_{i=1}^{S}Y_{B,t,i}^{2}.
Qind=∑t=1T[ut−at2S+vt−bt2S].Q_{\mathrm{ind}}=\sum_{t=1}^{T}\left[u_{t}-\frac{a_{t}^{2}}{S}+v_{t}-\frac{b_{t}^{2}}{S}\right].
Zind=LSS−1​Qind,Z_{\mathrm{ind}}=\frac{L}{\sqrt{\frac{S}{S-1}Q_{\mathrm{ind}}}},
Qind=∑t=1T[at−at2S+bt−bt2S].Q_{\mathrm{ind}}=\sum_{t=1}^{T}\left[a_{t}-\frac{a_{t}^{2}}{S}+b_{t}-\frac{b_{t}^{2}}{S}\right].
Qlo​(L,T,S)≤Qind≤Qhi​(A,L,T,S,R).Q_{\mathrm{lo}}(L;T,S)\leq Q_{\mathrm{ind}}\leq Q_{\mathrm{hi}}(A,L;T,S,R).
𝔼⁡[Y−X]=bt−atS,\mathbb{E}[Y-X]=\frac{b_{t}-a_{t}}{S},
S​Var⁡(Y−X)=ut−at2S+vt−bt2S.S\operatorname{Var}(Y-X)=u_{t}-\frac{a_{t}^{2}}{S}+v_{t}-\frac{b_{t}^{2}}{S}.
ut−at2S+vt−bt2S≥ρt−ρt2S.u_{t}-\frac{a_{t}^{2}}{S}+v_{t}-\frac{b_{t}^{2}}{S}\geq\rho_{t}-\frac{\rho_{t}^{2}}{S}.
Qind≥Qlo​(L,T,S).Q_{\mathrm{ind}}\geq Q_{\mathrm{lo}}(L;T,S).
Qpair,t=(ut−at2S+vt−bt2S)−2​Ct≥ut−at2S+vt−bt2S.Q_{\mathrm{pair},t}=\left(u_{t}-\frac{a_{t}^{2}}{S}+v_{t}-\frac{b_{t}^{2}}{S}\right)-2C_{t}\geq u_{t}-\frac{a_{t}^{2}}{S}+v_{t}-\frac{b_{t}^{2}}{S}.
Qind≤Qhi​(A,L,T,S,R).Q_{\mathrm{ind}}\leq Q_{\mathrm{hi}}(A,L;T,S,R).
L≤cα​Qlo​(L,T,S)⟹no compatible independently sampled outcome table rejects,L\leq c_{\alpha}\sqrt{Q_{\mathrm{lo}}(L;T,S)}\quad\Longrightarrow\quad\text{no compatible independently sampled outcome table rejects},
L>cα​Qhi​(A,L,T,S,R)⟹every compatible independently sampled outcome table rejects.L>c_{\alpha}\sqrt{Q_{\mathrm{hi}}(A,L;T,S,R)}\quad\Longrightarrow\quad\text{every compatible independently sampled outcome table rejects}.

C.7 Non-Realizable Metric Rounding

If a published mean averaged over runs cannot correspond to a single pass’s integer count, round it to the nearest realizable count before applying the rule. This is a chosen convention, not a treatment of the additional multi-run uncertainty; classifications near boundaries can depend on it.

Appendix D Citation Tracker Methodology

The adoption tracker counts papers with confirmed numerical benchmark results, not all citations. Its snapshot is 21 May 2026 and it supplies row-level evidence through the project site.

D.1 Sources

Candidate sources include citation graphs, metadata and text search from Semantic Scholar, OpenAlex, CORE, arXiv, OpenReview, Europe PMC, OpenCitations, OpenAIRE, Crossref, DataCite, DBLP and proceedings/project repositories.

Google Scholar/Publish or Perish, Lens, Scopus, Web of Science and Dimensions are excluded. Papers only discoverable through excluded sources are missed, so the authors describe counts as lower bounds.

D.2 Deduplication

Deduplicate by DOI, arXiv and service IDs plus normalized title. Preserve every discovery source in a “Found via” field so a single candidate is counted once while its provenance remains auditable.

D.3 Per-Paper Classification

Classify each paper separately for each benchmark: reports numerical results, citation only, unconfirmed term match, data use without evaluation, unavailable/unclear full text, or seed benchmark work. Only confirmed reports-results entries count.

Numeric success/task completion must be tied to an identified protocol or suite. A citation edge or ambiguous name such as LIBERO/CALVIN cannot promote a record without full-text evidence.

D.4 Date Convention

Assign month by first arXiv posting, not venue date, and exclude records lacking an arXiv ID from monthly bins. This tracks when the claim became public; it is not a count of conference publications.

D.5 Reproducing the Headline Count

At the snapshot, LIBERO has 1,053 candidates and 551 confirmed result-reporting papers. March 2026 contributes 79 confirmed papers among 117 candidates; the other 38 are mostly citation-only. The headline follows these filters, not total search hits.

D.6 Figure 2

Figure 2 applies the same result/date filter to five independent trackers from July 2023 through March 2026. A paper using several benchmarks counts once in each relevant series, so summing the stack is not a count of unique papers across the field.

Concept demonstration

Open image ↗