Overview
This edition covers the complete thirty-one-page arXiv v1 dated 2 June 2026, including experimental protocols, uncertainty calculations, hardware traces, statistical proofs and citation-tracker methodology. Xiangshan Tan, Samuel Wheeler, Luzhe Sun and Tewodros W. Ayalew share equal contribution. All six figures and ten numbered tables are retained.
At a glance
The paper asks whether a manipulation benchmark score identifies the capability readers infer from it. Four diagnostics test shortcut solutions, statistically unresolved gains, overfitting to narrow or fixed tests, and the effect of training data being unusually close to evaluation.
Audits cover LIBERO, CALVIN, SimplerEnv, RoboCasa and RoboTwin 2.0. The results expose limitations of score interpretation; they do not show that high-scoring foundation models lack broader capabilities, or that a benchmark passing these particular checks is universally valid.
1 Introduction
A fixed-distribution success average is often treated as evidence for much broader real-world manipulation. Directly validating that inference would require shared training data and representative real-robot evaluations, which the authors consider impractical.
The proposed substitute is four targeted counterchecks: remove presumed capabilities from a probe; bound significance from published aggregates; resample fixed tests or their narrow configurations; and deliberately move training data nearer the test. Figure 1 previews the outcomes.
A citation audit finds 79 March 2026 arXiv papers reporting LIBERO results, illustrating the benchmark’s influence. The count uses first arXiv dates and a 21 May 2026 snapshot, not current publication totals.
Selectable original panel labels
(a) Shortcut solvability
(b) Statistical significance
(c) Creeping overfitting
(d) Data-source dependence
2 Related Work
The audit adapts established concerns from vision and language benchmarks to embodied policy evaluation. It focuses on five frequently reported manipulation benchmarks rather than attempting an exhaustive simulator survey.
2.1 Benchmark Audits in Computer Vision and NLP
Prior studies identify language-prior shortcuts in VQA, copyright-tag shortcuts in image classification, unresolved leaderboard differences, dataset-specific overfitting and train/test overlap. These motivate separating what a model can exploit from what a benchmark is intended to measure.
Fresh test samples probe adaptation to a particular finite set; changed but training-supported conditions probe adaptation to a narrow test distribution. Neither is a direct measurement of all real-world manipulation capabilities.
2.2 Manipulation Benchmarks and Prior Audits
LIBERO tests language-specified task suites; CALVIN chains tasks across scenes; SimplerEnv reproduces real robot setups; RoboCasa varies kitchen scenes; RoboTwin 2.0 varies bimanual scenes and objects.
LIBERO-Pro and LIBERO-Plus perturb inputs beyond the original training distribution. This audit instead targets changes it argues are supported by training data, while distinguishing evidence of support from an exact distribution-matching proof.
3 Shortcut Solvability
Train a small DINOv2 encoder plus MLP action head, without robotics pretraining, diffusion or an action tokenizer. For fixed public task sets, a learned task-ID embedding replaces language understanding; this retains task identity while removing the need to interpret an instruction.
The best-checkpoint LIBERO probe scores 99.0/100.0/98.8/92.4% on Spatial/Object/Goal/Long. It is within one point of the selected best non-RL result on the first three suites, but 6.4 points behind on Long. Test-set tuning is explicit; final checkpoints average 95.1% versus 97.6% after selection.
On CALVIN it reaches 3.123/3.242/3.872 tasks for D→D/ABC→D/ABCD→D, competitive with some foundation baselines but below current best results. RoboTwin and RoboCasa gaps are much larger. Failure of this one probe does not prove that no other shortcut exists.
The standard-data SimplerEnv probe scores zero, but unlike the task-ID probes it uses a pretrained SigLIP text embedding. DINOv2 supplies visual pretraining in all cases; the claim is absence of robotics/language-model pretraining for the fixed-task probe, not learning vision from scratch.
| Benchmark | Protocol/Suite | DINO+MLP | Foundation VLA | Reported () | Best Reported | Reported () |
| LIBERO | Spatial | 99.0% | OpenVLA (7B) | 84.7% () | +T-MEE (2.3B) | 99.8% () |
| Object | 100.0% | OpenVLA (7B) | 88.4% () | +MoH (3B) | 100.0% () | |
| Goal | 98.8% | OpenVLA (7B) | 79.2% () | MoLA (1.64B) | 99.5% () | |
| Long | 92.4% | OpenVLA (7B) | 53.7% () | CORAL (0.8B) | 98.8% () | |
| CALVIN† | DD | 3.123 | DeeR-VLA (3B) | 2.83 () | MDT-V (n.d.) | 4.52 () |
| ABCD | 3.242 | RoboFlamingo (3B) | 2.48 () | MMaDA-VLA (8B) | 4.78 () | |
| ABCDD | 3.872 | RoboFlamingo (3B) | 4.09 () | Xiaomi-0 (4.7B) | 4.80 () | |
| SimplerEnv | WidowX | 0.0% | X-VLA (0.9B) | 95.8% () | CORAL (0.8B) | 97.9% () |
| RoboTwin 2.0 | Clean (50+500) | 60.4% | X-VLA (0.9B) | 72.8% () | MotuBrain (n.d.) | 95.8% () |
| Randomized (50+500) | 59.4% | X-VLA (0.9B) | 72.8% () | MotuBrain (n.d.) | 96.1% () | |
| RoboCasa | RSS24 | 18.8% | GR00T-N1 (2B) | 49.6% () | X-WAM (n.d.) | 79.2% () |
Foundation VLA: most-cited protocol-comparable foundation VLA reporting on the protocol (chosen per protocol for CALVIN).
Best reported: top non-RL-finetuned result (RL-finetuned excluded, as the probe is imitation-only).
and : the Foundation-VLA score and the Best-reported score, respectively, minus the DINO+MLP probe score (ATC for CALVIN, percentage points otherwise); a negative value means the probe scores higher.
CALVIN† reports ATC (out of 5). “n.d.” = params not disclosed. Policy sources are listed in Appendix A.1.
4 Statistical Significance
Compare each new score with the prior best named in that paper. With only aggregate scores, the unseen task-by-task paired outcomes can imply different variances and therefore different decisions under a one-sided, task-stratified Wald test.
For bounded integer outcomes, derive sharp minimum/maximum variance envelopes consistent with the reported counts. Below the feasibility condition no compatible paired table rejects; above the guarantee condition all do; between them the aggregate alone is inconclusive. For independent samples the same bounds are conservative, not sharp.
At nominal α=.05, guaranteed-rejection shares are 19.8% for LIBERO, 43.9% CALVIN, 19.7% SimplerEnv, 53.3% RoboCasa and 73.7% RoboTwin. Figure 3 uses 789/107/122/30/19 comparisons respectively, including nonimprovements; LIBERO pools Spatial/Object/Goal, not Long.
“Provably significant” means forced rejection under this specified nominal Wald rule and its assumptions. The analysis does not turn the asymptotic test into an exact finite-sample guarantee, and inconclusive does not mean no improvement. Fewer submissions and larger gaps partly explain stronger guarantees on newer benchmarks.
Equations
5 Creeping Overfitting
Distinguish fitting the particular released test sample from fitting its narrow distribution. On CALVIN ABC→D, keep the same task-chain manifest but resample block poses within the training range. X-VLA/GR-1/RoboFlamingo lose 1.027/.749/.498 average tasks, with positive paired 95% intervals.
On SimplerEnv stack, reverse color order, add support blocks, or randomize block/arm pose. Matched local calibration controls hardware and step limits. Effects depend on policy: X-VLA loses 28.47 points with stacked support, while CogACT’s corresponding change is −1.39 points with an interval spanning zero.
The authors support training-distribution relevance with Bridge examples and instruction counts. This does not establish identical joint state/visual distributions; single-view depth ambiguity is offered as a plausible explanation, not isolated experimentally.
Fresh LIBERO initial states move scores by at most .62 points; fresh CALVIN chain samples move ATC by at most .107. All reported fresh-sample intervals span zero. These limited checks find no resolved sample-overfitting effect, rather than proving its absence for every model.
6 Data-Source Dependence
SimplerEnv is often interpreted as real-Bridge-to-simulation transfer, but training sources are unrestricted. Train a separate roughly 22M DINOv2-S+MLP policy for each of four tasks on simulator demonstrations near its official evaluation grid.
With 120 demonstrations per task, or 480 total, the four policies jointly score 91/96=94.8%, near X-VLA’s cited 92/96=95.8%. This is not one multitask policy or a matched generalization comparison: generator fixes were selected to produce a high fixed-grid score.
The counterexample shows that the score alone does not identify whether a data-source gap was crossed or removed. Restricting training sources or reporting their proximity to test is the proposed remedy. The diagnostic is not applied to the other four benchmarks here.
7 Discussion
Broaden test variation over objects and scenes, require per-instance outcomes, and size evaluations for the improvement resolution being claimed. Difficulty should reflect intended capabilities rather than an uncontrolled distance between training and test data.
The discussion says the probe matches SOTA on LIBERO and CALVIN; Table 1 supports near-best performance on three LIBERO suites but only older-baseline competitiveness on CALVIN. The notes retain that narrower evidence.
These benchmarks remain useful for debugging and within-project iteration. A failed diagnostic limits the inference from a headline score; it need not invalidate every use of the benchmark. Item response theory and adaptive testing are proposed alternatives to fixed-population averages, not evaluated replacements in this paper.
8 Limitations
The diagnostics substitute for, rather than perform, an ideal shared-data and representative real-world ranking study. Available weights and original evaluation scripts limit the policies that can be rerun; some checks cover only three or four policies.
The authors deliberately tune some existence probes on the test protocol. Their results establish possible high-scoring shortcuts, not unbiased estimates or fair model rankings. Repeated-grid uncertainty, approximate test calibration and metric-rounding conventions further bound the conclusions.
9 Conclusion
Before interpreting a score as progress, test whether it requires the intended capability and whether its improvement is statistically resolvable. The released diagnostics make those questions operational; passing them is evidence within their tested scope, not comprehensive certification.
Appendix A Experiment Details
The appendix specifies each probe and evaluation intervention, supplies per-task/count results and gives confidence intervals. Figure 5 retains representative observations from all five benchmark environments.
A.1 Shortcut Solvability
Fine-tune DINOv2 end to end with an MLP head. LIBERO/CALVIN/SimplerEnv/RoboTwin use ViT-B/14; RoboCasa uses smaller ViT-S/14. Fixed-task multi-task probes use randomly initialized task embeddings, while RoboCasa trains separate per-task models. Images and proprioception follow each benchmark interface.
Sweep learning rate, augmentation and step budgets per suite/protocol; select the best tested checkpoint. LIBERO uses 50 trials/task, 500/suite. Best versus final scores differ by .8/.6/4.0/4.4 points across its four suites. CALVIN uses a 34-way task ID, 1,000 chains and length 360; best/final ABC→D is 3.242/3.149, a .093 gap rounded to “at most .09” in prose.
The authors attribute weaker D→D than ABC→D performance to segmented demonstrations that can start after the approach motion, leaving a task-ID policy without complete approach examples. This is their explanation, not a separately isolated causal ablation.
RoboCasa RSS24 uses 50 human demonstrations/task, AdamW for 1,500 epochs, object split B and five fixed layout/style pairs, with no camera/texture randomization. Its strongest per-task model gets 226/1,200=18.8%; alternative shared/Transformer-head probes score lower.
RoboTwin uses all 50 tasks with 50 clean plus 500 randomized demonstrations/task: 27,500 total. Evaluate 100 rollouts/task/condition; clean/randomized means are 60.4/59.4%. Table 3 retains every task rate, including large differences between near-solved and very weak tasks.
SimplerEnv’s standard-Bridge probe adds SigLIP text features because instruction lookup is unavailable, and gets zero on four WidowX tasks. Foundation baselines are selected by protocol comparability/citation prominence; best results exclude RL fine-tuning. The X-VLA RoboTwin comparison comes from ABot-M0’s matched-data evaluation, not X-VLA’s original differing protocol.
| Benchmark | Suite or protocol | Best checkpoint | Final checkpoint |
|---|---|---|---|
| LIBERO | Spatial | ||
| Object | |||
| Goal | |||
| Long | |||
| CALVIN | DD | ||
| ABCD |
| Task | Clean | Rand. | Task | Clean | Rand. |
|---|---|---|---|---|---|
| adjust_bottle | 95.0% | 90.0% | place_can_basket | 50.0% | 57.0% |
| beat_block_hammer | 47.0% | 42.0% | place_cans_plasticbox | 97.0% | 97.0% |
| blocks_ranking_rgb | 44.0% | 55.0% | place_container_plate | 62.0% | 51.0% |
| blocks_ranking_size | 36.0% | 40.0% | place_dual_shoes | 24.0% | 27.0% |
| click_alarmclock | 64.0% | 68.0% | place_empty_cup | 90.0% | 87.0% |
| click_bell | 83.0% | 81.0% | place_fan | 42.0% | 43.0% |
| dump_bin_bigbin | 48.0% | 50.0% | place_mouse_pad | 15.0% | 20.0% |
| grab_roller | 100.0% | 100.0% | place_object_basket | 56.0% | 62.0% |
| handover_block | 67.0% | 67.0% | place_object_scale | 50.0% | 47.0% |
| handover_mic | 86.0% | 80.0% | place_object_stand | 79.0% | 76.0% |
| hanging_mug | 8.0% | 16.0% | place_phone_stand | 58.0% | 58.0% |
| lift_pot | 99.0% | 100.0% | place_shoe | 69.0% | 67.0% |
| move_can_pot | 69.0% | 61.0% | press_stapler | 99.0% | 92.0% |
| move_pillbottle_pad | 63.0% | 59.0% | put_bottles_dustbin | 9.0% | 7.0% |
| move_playingcard_away | 83.0% | 86.0% | put_object_cabinet | 50.0% | 36.0% |
| move_stapler_pad | 6.0% | 5.0% | rotate_qrcode | 42.0% | 44.0% |
| open_laptop | 96.0% | 95.0% | scan_object | 82.0% | 77.0% |
| open_microwave | 14.0% | 21.0% | shake_bottle | 100.0% | 100.0% |
| pick_diverse_bottles | 83.0% | 68.0% | shake_bottle_horizontally | 100.0% | 100.0% |
| pick_dual_bottles | 97.0% | 93.0% | stack_blocks_three | 28.0% | 31.0% |
| place_a2b_left | 30.0% | 24.0% | stack_blocks_two | 62.0% | 58.0% |
| place_a2b_right | 25.0% | 32.0% | stack_bowls_three | 20.0% | 25.0% |
| place_bread_basket | 83.0% | 80.0% | stack_bowls_two | 60.0% | 58.0% |
| place_bread_skillet | 80.0% | 69.0% | stamp_seal | 33.0% | 29.0% |
| place_burger_fries | 91.0% | 94.0% | turn_switch | 46.0% | 45.0% |
| Average | 60.4% | 59.4% | |||
A.2 Creeping Overfitting
The SimplerEnv interventions use a common local rollout protocol and report changes from matched calibration. The appendix also gives uncertainty for CALVIN pose resampling and fresh LIBERO/CALVIN test samples.
Setup.
Evaluate CogACT-Base, InternVLA-M1, X-VLA-WidowX and Dexbotic/DB-MemVLA on the stack task with the standard 60-step limit. These local results need not equal published results obtained with longer episode limits.
Matched local calibration.
Repeat the 24 official start states twelve times, giving 288 calibration episodes per policy on the same hardware as each intervention. Matching protocol and hardware reduces those confounds, but repeated start states are not 288 distinct scenes.
The three changes.
Run 288 episodes each for reversed yellow-on-green language, target cubes supported by other blocks, and randomized cube/initial-arm poses. Hold scene, camera, robot, success rule and task fixed. These three belong to a released seven-condition, 2,016-episode run per policy.
Why the changes stay in distribution.
Among 53,192 Bridge demonstrations, instruction text mentions block/cube/tower 869/751/282 times; green-on-yellow appears 41 times and yellow-on-green 30. These counts support the presence of similar conditions, but text frequency alone does not prove joint distribution equivalence or coverage of every altered simulator state.
Results.
Table 4 reports calibration followed by each altered condition. Calibration success ranges from CogACT 20.8% to X-VLA 59.7%; no policy wins every intervention. Preserve the common 60-step evaluation when comparing these local numbers.
| Policy | Calibration | Reverse language | Stacked support | Random pose + arm |
|---|---|---|---|---|
| CogACT-Base | 20.8 | 9.7 | 22.2 | 10.8 |
| InternVLA-M1 | 25.3 | 16.0 | 20.8 | 16.0 |
| X-VLA-WidowX | 59.7 | 53.1 | 31.3 | 55.6 |
| Dexbotic (DB-MemVLA) | 44.8 | 47.6 | 27.4 | 38.9 |
Confidence intervals.
Define drop as matched calibration minus altered/fresh performance, so positive means degradation. Compute two-sided 95% intervals from per-episode or per-chain artifacts, with different estimators for binary, paired and independent comparisons.
Equation
Interval methods.
For binary success, combine Wilson intervals into a Newcombe difference interval. This treats rollout rows as independent Bernoulli observations; it does not model SimplerEnv’s twelve repetitions of each fixed grid state.
For CALVIN pose changes, use paired sequence differences on the shared 1,000-chain manifest, including a binary five-task-completion version. For fresh CALVIN samples, compare 1,000 calibration chains with two pooled 1,000-chain manifests using an independent two-sample Wald interval. All original formulas remain below.
Equations
Distribution-overfitting drops.
On SimplerEnv, reversed-language drops resolve above zero for CogACT and InternVLA; stacked-support drops resolve for X-VLA and Dexbotic. Random pose/arm drops resolve for CogACT and InternVLA. Several other intervals span zero, and some point estimates improve.
CALVIN pose resampling reduces ATC by 1.027 [.890,1.164], .749 [.624,.874] and .498 [.372,.624] for X-VLA/GR-1/RoboFlamingo. Five-task completion drops by 25.0/13.5/6.4 percentage points; all corresponding paired intervals are positive.
| Policy | Change | Calibration | Altered | Drop | 95% CI |
|---|---|---|---|---|---|
| CogACT-Base | Reverse language | ||||
| CogACT-Base | Stacked support | ||||
| CogACT-Base | Random pose + arm | ||||
| InternVLA-M1 | Reverse language | ||||
| InternVLA-M1 | Stacked support | ||||
| InternVLA-M1 | Random pose + arm | ||||
| X-VLA-WidowX | Reverse language | ||||
| X-VLA-WidowX | Stacked support | ||||
| X-VLA-WidowX | Random pose + arm | ||||
| Dexbotic (DB-MemVLA) | Reverse language | ||||
| Dexbotic (DB-MemVLA) | Stacked support | ||||
| Dexbotic (DB-MemVLA) | Random pose + arm |
| Policy | Metric | Calibration | Resampled | Drop | 95% CI |
|---|---|---|---|---|---|
| X-VLA | ATC | ||||
| X-VLA | 5/5 chain success | ||||
| GR-1 | ATC | ||||
| GR-1 | 5/5 chain success | ||||
| RoboFlamingo | ATC | ||||
| RoboFlamingo | 5/5 chain success |
Sample-overfitting drops.
Fresh LIBERO evaluation uses 10,000 rollouts/policy versus 2,000 for calibration. Drops are +.62 points for Spatial Forcing, −.30 for SimVLA and +.14 for π0.5/LeRobot, each interval including zero.
Fresh CALVIN ATC drops are −.0145 for X-VLA, +.107 for GR-1 and +.069 for RoboFlamingo, again with intervals including zero. This is much smaller than the pose-resampling effect, but does not establish exact equivalence.
| Policy | Metric | Calibration | Fresh sample | Drop | 95% CI |
|---|---|---|---|---|---|
| Spatial Forcing | success rate | ||||
| SimVLA | success rate | ||||
| / LeRobot | success rate |
| Policy | Metric | Calibration | Fresh sample | Drop | 95% CI |
|---|---|---|---|---|---|
| X-VLA | ATC | ||||
| GR-1 | ATC | ||||
| RoboFlamingo | ATC |
Confidence-interval source artifacts.
Intervals are recomputed from rollout CSV/JSON outputs rather than rounded summaries. Fresh CALVIN manifests are predeclared; a small number of LIBERO cells use documented retry reruns. These artifacts define which episodes enter each estimate.
A.3 Data-Source Dependence
The data-source probe deliberately trains near SimplerEnv’s public evaluation grid. It tests whether the score identifies real-to-simulation transfer, not whether synthetic demonstrations are equivalent to real Bridge data.
Protocol.
Use stack, carrot, spoon and eggplant with 24 official grid instances each, totaling 96 closed-loop evaluations. Evaluation uses unjittered IDs 0–23 and benchmark episode limits and success checks; report counts because each task has few trials.
Scripted demonstrations.
Generate five successful demonstrations around each grid ID, 120/task. Add .0025-m XY jitter and up to 15° yaw jitter, pad terminal open/place behavior, and canonicalize carrot gripper yaw modulo π. Public test centers deliberately shape the training distribution.
Policy and training.
Train four separate DINOv2-S+MLP behavior-cloning models, fine-tuning pretrained visual features with proprioception and no image augmentation. Normalize environment-format actions and execute five-step chunks with replanning every five steps; include terminal padding.
No checkpoint selection is used, but demonstration-generator fixes were selected to obtain a high fixed-grid score. Results are 24/24 stack, 23/24 carrot, 21/24 spoon and 23/24 eggplant: 91/96 overall.
| Task | Scripted demos | Eval successes | Success rate |
|---|---|---|---|
| Stack | |||
| Carrot | |||
| Spoon | |||
| Eggplant | |||
| All four |
Interpretation.
This is an existence construction on a fixed grid. It is neither an unbiased success-rate estimate nor a controlled claim that a small policy outperforms X-VLA. Its purpose is to show that unrestricted data can remove the transfer gap the benchmark score is assumed to measure.
Appendix B Bitwise Nondeterminism in Policy Rollouts
Closed-loop rollouts can amplify tiny hardware-dependent numerical changes into different observations, actions and outcomes. The appendix distinguishes strict bitwise identity from protocol-faithful statistical reproduction.
B.1 Reproduction Targets
Four targets range from bitwise identity, through matching per-episode outcomes and aggregate scores, to documented protocol/statistical compatibility. This paper adopts the last while using bitwise traces diagnostically.
That standard still requires checkpoints, revisions, commands, seeds, initial states, normalization, action/success processing and hardware/software metadata. Agreement of one aggregate decimal does not imply identical episodes or prove significance.
B.2 Diagnostic Evidence
In one LIBERO-Goal task, changing CPU while retaining RTX A6000 first alters simulator state at step 0, images at 54, actions at 58 and reward/done at 88. Contact instrumentation links growth to bowl/table constraints; the plot uses a log floor for zeros.
The broader table covers eight CPU/GPU contrasts across four stacks. LIBERO CPU traces differ in all ten tasks, reaching pixels/actions in five. CALVIN’s CPU sample has zero divergences in fifty sequences, while its GPU contrast diverges immediately. Same-node controls are identical.
These examples show cross-hardware determinism can fail, not that it always fails or necessarily changes success. Both SimplerEnv GPU traces fail despite different paths. Open-loop replay alone omits the feedback mechanism under investigation.
| Stack | Hardware contrast | Observed propagation | Bitwise identical? |
|---|---|---|---|
| LIBERO / OpenVLA-OFT | CPU: AMD EPYC 74F3 + RTX A6000 versus Intel Xeon Silver 4210R + RTX A6000 | tasks diverged in simulator state; reached pixels and actions. | No |
| LIBERO / OpenVLA-OFT | GPU: same AMD EPYC 74F3 CPU model, RTX A6000 versus RTX 6000 Ada | Raw and processed actions diverged at step 10, followed by simulator state, image, and reward/done divergence. | No |
| CALVIN / GR-1 | CPU: AMD-CPU node with RTX A6000 versus Intel-CPU node with RTX A6000 | official sequence pairs diverged in this diagnostic sample. | Yes |
| CALVIN / GR-1 | GPU: same AMD EPYC 74F3 CPU model, RTX A6000 versus RTX 6000 Ada | Raw and processed actions diverged at step 0, followed by simulator state and image-hash divergence. | No |
| SimplerEnv / CogACT | CPU: AMD-CPU node with RTX A6000 versus Intel-CPU node with RTX A6000 | Actions and post-step simulator state diverged at step 0; images at step 3; reward/done/success at step 38. | No |
| SimplerEnv / CogACT | GPU: same AMD EPYC 74F3 CPU model, RTX A6000 versus RTX 6000 Ada | Actions and simulator state diverged from step 0, followed by image divergence; both traces still ended unsuccessfully. | No |
| RoboCasa / StarVLA | CPU: AMD-CPU node with RTX A6000 versus Intel-CPU node with RTX A6000 | Simulator and contact traces diverged immediately; image/action divergence followed at step 4 and reward/success at step 19. | No |
| RoboCasa / StarVLA | GPU: same AMD EPYC 74F3 CPU model, RTX A6000 versus RTX 6000 Ada | Actions, contact traces, simulator state, and images diverged from the first steps; reward/success diverged at step 18. | No |
B.3 Implications for This Paper
Evaluate reproduced results using protocol fidelity, per-instance outcomes and appropriate uncertainty. A small compatible score mismatch need not refute reproduction, while an exact score match cannot establish a significant improvement.
Release the complete execution stack and outcomes so numerical divergence can be separated from changed protocol. Bitwise identity remains useful when achievable, but is not proposed as the sole cross-hardware pass/fail criterion.
Appendix C Top-Line Cutoffs for Bounded-Score Wald Tests
Derive aggregate-only bounds for a specified bounded-score Wald decision rule. Binary benchmarks use R=1; CALVIN uses R=5, one stratum and 1,000 samples. The derivation concerns fixed task populations, equal samples per task and integer scores.
C.1 Paired Task-Stratified Wald Rule
Treat T tasks as fixed strata with S>1 independent paired samples each. Average paired score differences over tasks; estimate variance from within-task differences. With fixed T and S growing, a nonzero-variance central limit theorem motivates the one-sided Wald rule.
For finite samples this remains a nominal asymptotic procedure. If estimated variance is zero, define the statistic as positive infinity, zero or negative infinity according to the gap sign; reject only above the normal critical value. The later exact envelopes are exact about this rule, not exact type-I-error calibration.
Equations
C.2 Top-Line Count Notation
Convert reported means into integer totals A and B across N=T·S observations, with positive count gap L=B−A. The numerator is fixed; the unknown paired table affects only Q, the sum of within-task squared-difference residuals.
Absorb the sample correction and critical normal quantile into cα. For positive L, rejection is exactly the condition L>cα√Q, including the declared zero-variance convention. Feasible gaps cannot exceed R·N−A.
Equations
C.3 Exact Lower Envelope for Paired Evaluations
Minimize squared differences within each task by making integer differences as equal as possible. Combining remainders across tasks yields the exact lower envelope Qlo=r−r²/S, where r is L modulo S.
Construct constant nonnegative task differences plus one residual task, then allocate baseline scores within remaining capacity to realize A. This proves attainability. If even this minimum-variance table cannot reject, no compatible paired table can.
The first feasible rejecting count gap has the stated floor-function expression; if it exceeds R·N−A, the feasibility cutoff is infinite. All remainder identities, construction constraints and cutoff formulas are retained.
Equations
C.4 Exact Upper Envelope for Paired Evaluations
For each task’s marginal totals, maximize squared differences with an integer contingency table over scores 0…R. An equivalent construction packs positive and negative movements into as few bounded-size samples as possible, choosing the largest feasible opposing movement.
For binary outcomes this reduces to maximum discordance minus the squared task-total difference divided by S. Dynamic programming then allocates global A and B across tasks and computes the exact upper envelope Qhi.
If the largest-variance compatible table rejects, all paired tables do. The scalar guarantee is a suffix cutoff requiring rejection for every larger feasible gap; the pointwise Qhi condition can be sharper for a specific gap. Preserve the distinction rather than assuming monotonicity without the suffix definition.
Equations
C.5 Sharp Paired Classification
For fixed exact paired totals, the lower envelope identifies impossible rejection, the upper envelope identifies forced rejection, and the interval between permits both decisions. This intermediate case is the third displayed classification, despite the source’s subsequent “middle line” wording.
Equation
C.6 Independently Sampled Evaluations
Independent evaluations require separate within-task variances, not paired differences. Their denominator still lies between the paired envelopes: integer-valued minimum variance supplies the lower bound, and an attainable nonpositive-covariance pairing supplies the upper bound.
Thus the paired impossible/guaranteed cutoffs remain conservative under the stated identical task/sample structure. The intermediate region is not sharp for independent samples; a tighter two-sample analysis could resolve some comparisons.
Equations
C.7 Non-Realizable Metric Rounding
If a published mean averaged over runs cannot correspond to a single pass’s integer count, round it to the nearest realizable count before applying the rule. This is a chosen convention, not a treatment of the additional multi-run uncertainty; classifications near boundaries can depend on it.
Appendix D Citation Tracker Methodology
The adoption tracker counts papers with confirmed numerical benchmark results, not all citations. Its snapshot is 21 May 2026 and it supplies row-level evidence through the project site.
D.1 Sources
Candidate sources include citation graphs, metadata and text search from Semantic Scholar, OpenAlex, CORE, arXiv, OpenReview, Europe PMC, OpenCitations, OpenAIRE, Crossref, DataCite, DBLP and proceedings/project repositories.
Google Scholar/Publish or Perish, Lens, Scopus, Web of Science and Dimensions are excluded. Papers only discoverable through excluded sources are missed, so the authors describe counts as lower bounds.
D.2 Deduplication
Deduplicate by DOI, arXiv and service IDs plus normalized title. Preserve every discovery source in a “Found via” field so a single candidate is counted once while its provenance remains auditable.
D.3 Per-Paper Classification
Classify each paper separately for each benchmark: reports numerical results, citation only, unconfirmed term match, data use without evaluation, unavailable/unclear full text, or seed benchmark work. Only confirmed reports-results entries count.
Numeric success/task completion must be tied to an identified protocol or suite. A citation edge or ambiguous name such as LIBERO/CALVIN cannot promote a record without full-text evidence.
D.4 Date Convention
Assign month by first arXiv posting, not venue date, and exclude records lacking an arXiv ID from monthly bins. This tracks when the claim became public; it is not a count of conference publications.
D.5 Reproducing the Headline Count
At the snapshot, LIBERO has 1,053 candidates and 551 confirmed result-reporting papers. March 2026 contributes 79 confirmed papers among 117 candidates; the other 38 are mostly citation-only. The headline follows these filters, not total search hits.
D.6 Figure 2
Figure 2 applies the same result/date filter to five independent trackers from July 2023 through March 2026. A paper using several benchmarks counts once in each relevant series, so summing the stack is not a count of unique papers across the field.