Literature
Display
← All papers

Robot World Models Are Not Invariant to How the Actions Are Written

Ahmed Karim · Leon Chlon

Preprint posted · arXiv (preprint) ↗

World models12 min text · CondensedOriginal paper ↗
Contents

At a glance

A small world model trained with absolute robot commands can fail when the same trajectories are written as relative commands. Across three datasets, future-frame retrieval ranks worsen by 2.6–13.4 times. On ALOHA cabinet, offline action selection falls from 53.1% to 15.4%. Training on both encodings largely repairs the average; an added consistency penalty chiefly improves the worst agreement observed in the test set. These are frozen-encoder, offline experiments, and PushT retains a trade-off between agreement and prediction accuracy.

1 Introduction

Absolute joint targets and offsets from the current state can encode the same commanded motion. A model trained on only one convention receives no direct incentive to recognize the other. The paper audits that action-input sensitivity while keeping visual observations fixed.

Why this is not the usual brittleness story.

The proposed distinction is between losing information and changing its representation. On these data, joint state-and-action probes reconstruct one encoding from the other with R² = 0.996. This supports testing whether convention changes unnecessarily degrade prediction, while finite regression probes do not establish a universal invariance guarantee.

The channel nobody audits.

Most familiar invariance tests change images or spatial coordinates. Here a frozen I-JEPA encoder isolates learning in the action-conditioned latent predictor: there is no image decoder and no encoder update during the experiment. The contribution is an action-channel audit, rather than a new general-purpose world-model architecture. [3] [4] [6] [7]

Contributions.

The paper combines an action-encoding stress test, checks intended to distinguish equivalent inputs from lossy views, and a comparison of pooled training, output averaging, and consistency regularization. Its strongest empirical distinction is between average recovery and agreement on the observed difficult cases.

2 Why a predictor has no reason to respect a notation

A minimum-description-length account compares the complexity saved by restricting a predictor with the predictive information that restriction discards. More data need not teach an equivalence absent from the training examples. The proposed “make/break” audit requires reduced prediction dispersion and no task-metric loss beyond a chosen tolerance, assessed with held-out bootstrap intervals.

Equation 1
MDLn​(ℱinv)−MDLn​(ℱ)=L⁡(ℱinv)−L⁡(ℱ)⏟architectural+n⋅IP^ℱ​(Y;G∣Z),\mathrm{MDL}_{n}(\mathcal{F}_{\mathrm{inv}})-\mathrm{MDL}_{n}(\mathcal{F})=\underbrace{L(\mathcal{F}_{\mathrm{inv}})-L(\mathcal{F})}_{\text{architectural}}+\ n\cdot I^{\mathcal{F}}_{\hat{P}}(Y;G\mid Z), (1)

3 Is it a re-parameterization, or is it a different input?

The theory groups equivalent presentations into a quotient variable Q. A predictor forced to agree across that quotient cannot achieve lower optimal log loss than the best predictor using each individual view. In particular, unequal predictive information across views creates a lower bound on the cost of invariance. The lower bound alone does not establish that equal view-wise performance makes invariance free.

Equation
Rinv≥maxv⁡R⁡(φv),soRinv−minv⁡R⁡(φv)≥δ:=maxv⁡R⁡(φv)−minv⁡R⁡(φv).R_{\mathrm{inv}}\ \geq\ \max_{v}R(\varphi_{v}),\qquad\text{so}\qquad R_{\mathrm{inv}}-\min_{v}R(\varphi_{v})\ \geq\ \delta:=\max_{v}R(\varphi_{v})-\min_{v}R(\varphi_{v}).

Predictive parity (necessary).

If two views predict the target differently well, forcing agreement can discard the better view’s information. Predictive parity is therefore a necessary check, not by itself a sufficient guarantee. Different cameras, for example, can expose different evidence.

Reconstructibility (sufficient, not necessary).

The authors propose mutual reconstruction as a sufficient certificate. For unlabeled views sharing one input space, that implication additionally requires the quotient to preserve target-relevant information: invertibility of the individual encodings alone does not ensure it. The experiments provide evidence about these particular conversions, rather than a general proof that any bijective encoding change is free to ignore.

Evaluate on the joint input.

Absolute and relative commands are interchangeable only given the current state. Reconstruction rises from R² = 0.757 using actions alone to 0.996 using the joint input; task-prediction R² differs by 0.003. The authors reject camera choice, photometric jitter, and lossy action-window summaries as candidate equivalences. Shuffling actions across unrelated examples is also a different intervention.

4 Repair: averaging over the two encodings

Train one predictor against the true target under every candidate encoding, averaging the task losses. Optionally add a penalty for disagreement between its predictions. A zero penalty gives ordinary pooled two-view training, the crucial baseline for separating exposure to both conventions from explicit consistency.

Equation 2
ℒ(θ)=1K∑vℓ(gθ(z,ϕv(a)),y)+λ⋅2K⁡(K−1)∑v<wd(gθ(z,ϕv(a)),gθ(z,ϕw(a))),\mathcal{L}(\theta)=\frac{1}{K}\sum_{v}\ell\!\left(g_{\theta}(z,\phi_{v}(a)),\,y\right)\;+\;\lambda\cdot\frac{2}{K(K-1)}\sum_{v<w}d\!\left(g_{\theta}(z,\phi_{v}(a)),\,g_{\theta}(z,\phi_{w}(a))\right), (2)

Where the average belongs.

Averaging probability predictions cannot worsen their average log loss, but the same argument does not apply to normalized latent directions scored by cosine similarity. Their mean can point away from every member’s useful direction or be undefined when vectors cancel. The repair averages losses rather than constructing a normalized mean target. Output averaging’s probability guarantee is relative to the average member, not necessarily the best member.

Equation
J^​(x,y):=1K​∑k=1K(−log⁡f⁡(y∣gk⋅x))−(−log⁡p¯​(y))≥ 0\hat{J}(x;y):=\frac{1}{K}\sum_{k=1}^{K}\big(-\log f(y\mid g_{k}\cdot x)\big)-\big(-\log\bar{p}(y)\big)\ \geq\ 0

5 Experiments

The main experiment predicts a future visual embedding from the current embedding and an action window. It compares two ALOHA tasks and the planar PushT dataset, with episode-disjoint train/test splits and a fixed camera. [5] [9] [10]

5.1 Absolute versus relative action encoding

The prediction horizon is four seconds. Commands are absolute targets or the same targets offset by the current state; both presentations use the same frozen visual representation. A teacher trained only on the absolute convention is evaluated under both. The example retrievals show mismatches from other episodes rather than merely adjacent, visually similar frames.

Equation
A:at:t+Δ(absolute joint targets)B:at:t+Δ−st(relative to current state).A:\ a_{t:t+\Delta}\ \text{(absolute joint targets)}\qquad B:\ a_{t:t+\Delta}-s_{t}\ \text{(relative to current state)}.
Refer to caption
Figure 1: The same failure on frames. Each row is one test instance: the observation at tt, the true frame at t+Δt+\Delta, and the frames retrieved when the predictor is handed the identical action window in absolute and in relative encoding. Green marks a correct retrieval, orange an incorrect one. Rows are restricted to misses from a different episode, since a same-episode miss is a near-duplicate frame and would show the reader nothing. Of 384384 test instances, 125125 are absolute-correct and relative-wrong, 9595 of them from a different episode; two are shown.

A certified special case of covariate shift.

Changing the command encoding is a distribution shift, even when the underlying motion is unchanged. Mean retrieval rank rises from 6.19 to 61.5 on ALOHA cabinet, 3.97 to 53.3 on ALOHA coffee, and 66.2 to 169.8 on PushT, out of 384 candidates. PushT’s mean cross-encoding cosine is 0.067, with an observed minimum of −0.377.

Table 1: Orbit dispersion under the action-encoding axis. Δ=4\Delta=4s, 3 seeds, episode-disjoint splits, pool size 384. Collapse is held-out rank over canonical rank; cos\cos is agreement between the two encodings’ predictions, mean and worst case. ALOHA datasets (Zhao et al., 2023); PushT (Chi et al., 2023); all via Cadene et al. (2024).
Retrieval rank cos\cos
Dataset Morphology Canon. Held out Collapse mean worst
aloha_mobile_cabinet 14-D bimanual, 50 fps 6.19 61.5 9.9×9.9\times 0.370 −0.178-0.178
aloha_static_coffee 14-D bimanual, new task 3.97 53.3 13.4×13.4\times 0.371 −0.096-0.096
pusht 2-D planar pusher, 10 fps 66.2 169.8 2.6×2.6\times 0.067 −0.377\mathbf{-0.377}

The failure is a control failure.

For each query, the model ranks 32 recorded candidate action windows by how well their predicted future matches the goal. ALOHA cabinet’s top-choice accuracy drops from 53.1% to 15.4% after changing notation. This is an offline action-selection proxy, not measured success from executing commands on a robot.

Original figure
Figure 2: Left: retrieval rank on each encoding. Right: cross-encoding agreement, mean and worst case over the evaluation set. The teacher is competent on the encoding it was trained on and collapses on the equivalent one. Pooled training (λ=0\lambda=0) is shown because it, not mean-matching, is the baseline that competes: it recovers accuracy but leaves worst-case agreement at 0.7790.779, which only the consistency term lifts. The teacher’s worst case is −0.178-0.178: on its worst input the two notations point in opposite directions.
Table 2: Repair on aloha_mobile_cabinet, 3 seeds, all rows from the same runs. λ\lambda is the disagreement weight of Eq. (2); λ=0\lambda=0 is the same two-view pooled training with the penalty off, which is the baseline any practitioner would reach for once the axis is known; the orbit-mean row fits a student to the averaged teacher prediction instead. cos\cos is mean and worst-case pairwise agreement across the orbit. Action selection is top-1 over 32 candidates, chance 0.0310.031.
Retrieval rank Agreement Action selection
Canon. Held out mean worst Canon. Held out
Teacher 6.19 61.5 0.370 — 0.5310.531 0.1540.154
Orbit-mean student 11.55 15.1 0.942 0.804 — —
λ=0\lambda=0 5.46 5.98 0.937 0.779 0.443\mathbf{0.443} 0.4010.401
λ=1\lambda=1 5.51 5.58 0.981 0.939 0.4080.408 0.4060.406
λ=4\lambda=4 5.25 5.31 0.993 0.973 0.4260.426 0.4020.402
λ=16\lambda=16 5.17 5.19 0.999 0.995 0.4110.411 0.411\mathbf{0.411}

Repair, and what each part of it buys.

Pooled training alone improves held-out-encoding rank from 61.5 to 5.98 and action selection from 15.4% to 40.1%. With penalty weight 16, these become 5.19 and 41.1%; the one-point action-selection gain is smaller than the reported four-point seed variation. The main benefit is observed minimum agreement, 0.779 to 0.995, at a cost in canonical action selection from 44.3% to 41.1%. This minimum summarizes the evaluation set rather than guaranteeing behavior on future inputs.

The mean is stable over a horizon; the worst case is not.

Latent rollouts feed predictions back while supplying true state and action windows at every hop. Prediction quality against the true future decays, so agreeing rollouts can still be wrong. On ALOHA cabinet, the final observed minimum cosine is 0.242 for the teacher, 0.896 for pooled training, and 0.997 with the penalty. PushT remains harder: Table 3 reports final mean/minimum agreement of 0.677/−0.348 for pooling and 0.872/0.139 with the penalty. These are not closed-loop control results.

Table 3: Latent rollout on held-out episodes, 3 seeds, three datasets and two morphologies: cross-encoding cosine at the final hop (three hops of 44s, two for PushT, whose episodes are short). Mean and worst case over the evaluation set. Per-hop trajectories are in Table 4. Chains need horizon×Δ\text{horizon}\times\Delta of runway and so start earlier in an episode than the pairs of Table 2, so read the comparison across models, not against that table.
Teacher λ=0\lambda=0 λ=16\lambda=16
Dataset mean worst mean worst mean worst
ALOHA cabinet 0.805 0.242 0.988 0.896 1.000 0.997
ALOHA coffee 0.898 0.614 0.991 0.955 1.000 0.998
PushT 0.178 −0.497-0.497 0.677 −0.348-0.348 0.872 0.139

On the value of finding the axis.

Exposure to both encodings works only after identifying which transformations are worth treating as equivalent. Pooling inputs that remove predictive information could purchase consistency by sacrificing accuracy.

Where repair is not free.

PushT’s repair improves the unseen-encoding retrieval rank from 169.8 to 85.7 but worsens the canonical rank from 66.2 to 79.2. The accuracy cost makes the intervention dataset dependent.

5.2 The defect is flat in n over the range we can sweep

Increasing cached training examples from 150 to 1,200 improves canonical retrieval while cross-encoding cosine stays near 0.34–0.37. The collapse ratio therefore grows from 2.5 to 9.9 without showing growing disagreement itself. Larger and longer-trained models also overfit the fixed data, so those sweeps do not independently establish a clean capacity-scaling result.

Locating I ⁡ ( Y ; G ∣ Z ) .

Strong penalties help more with scarce data and can become costly as canonical prediction improves. At weight 256, the rank difference relative to pooling changes from −6.59 with 150 examples to +0.38 with 1,200. The reported practical weights are 4–16.

5.3 Why this axis is the hard case

Some representations remove a transformation before it reaches the predictor: angle features using sine/cosine erase full-turn changes, and excluding absolute time erases time-origin shifts. A molecular example likewise shows that representation choice changes the outcome of attempted invariance repair. A known action convention could also be explicitly normalized in a robot pipeline; this paper studies predictors exposed to both conventions without such a canonicalization step.

6 Discussion

The discussion relates the encoding audit to invariant learning, architectural symmetry, and the limits of the small offline study.

Discovering rather than assuming invariance.

Invariant-risk and causal-invariance methods infer stable prediction across environments. This work instead begins with a proposed transformation and asks whether agreement under it is justified. Automatically proposing useful transformations remains future work. [1] [2]

Relation to prior work.

Architectural equivariance and data augmentation already offer ways to enforce known symmetries. The paper’s focus is diagnosing an overlooked input convention and separating the effects of pooled training from a consistency penalty. [11] [12] [13] [14]

Where repair is unavailable.

The implemented loss enumerates a small set of presentations. Exact enumeration becomes impractical for large permutation groups or continuous transformations. This limits this implementation, rather than proving that sampling or structural alternatives can never help.

Limitations.

Evidence comes from a small MLP, frozen embeddings, three seeds, and offline proxies. Ridge-regression probes may miss nonlinear relationships, and cannot establish all relevant equivalences. Agreement is distinct from predictive correctness; no deployed robot safety or closed-loop task success is established.

Conclusion.

Audit action conventions before trusting latent predictions across interfaces. On the studied datasets, exposure to both representations recovers much of the average performance, while consistency regularization chiefly improves agreement in the observed tail.

Reproducibility statement

The authors describe a Mezzanine repository with cached latents, configurations, seeds, and stored results. Three-seed variation is reported; CPU/GPU retrieval ranks can differ by 2–4% because of floating-point ordering even when dispersion is stable.

Ethics statement

The study reuses public robot-learning data without new collection or human-subject experiments. Its diagnostic is intended for evaluation before control deployment; the measured accuracy–agreement trade-off requires application-specific assessment.

Appendix A Experimental detail

These details specify the datasets, frozen features, predictor, repair loss, and evaluation protocol.

Datasets.

ALOHA cabinet uses 85 episodes, ALOHA coffee 50, and PushT 206, with episode-disjoint splits. The horizon is 200 frames at 50 frames per second for ALOHA and 40 frames at 10 frames per second for PushT. The complete dataset table is preserved.

Dataset Camera Episodes Frames fps Δ\Delta (steps) train/test eps.
aloha_mobile_cabinet cam_high 85 127,500 50 200 68 / 17
aloha_static_coffee cam_high 50 55,000 50 200 40 / 10
pusht image 206 25,650 10 40 165 / 41

Encoder.

A frozen I-JEPA ViT-H/14 encodes 224-pixel frames. Features concatenate patch-token means and standard deviations from the fourth-from-last block. No gradient updates the encoder; cached features are tied to the model and preprocessing configuration. [3]

Predictor.

A two-layer MLP of hidden width 1,024 predicts a normalized future embedding from the current latent and 16 sampled action points. Training uses 800 InfoNCE steps, 1,200 examples, and training-only centering statistics; evaluation uses 384 examples.

Repair.

Repair keeps the same training duration while varying the cosine-disagreement weight over 0, 1, 4, and 16. Weight zero retains both action encodings and removes only the consistency term.

Metrics.

Retrieval rank is lower when prediction better identifies the true future among 384 latents. Agreement compares two encodings’ predictions; it does not compare either prediction with truth. Action selection chooses among 32 windows, with chance probability 1/32.

Seeds and hardware.

Seeds 0, 1, and 2 control sampling, initialization, and batching. Runs use one RTX 5090; reported short sweep times exclude the initial video-decoding and feature-caching work.

What is not controlled.

Three seeds give limited uncertainty estimates. Capacity and optimization sweeps use the same 1,200 examples, so worse canonical performance confounds conclusions about scaling.

Appendix B Action ablations are not orbit-constancy tests

Shuffling actions between examples tests whether a model uses its action input. It changes the commanded trajectory and therefore cannot establish invariance to an equivalent notation. The supplementary action-ablation results should be read separately from the main encoding experiment.

Repair beyond robotics.

In a node-permutation example, target-based pooled training reduces dispersion more than fitting averaged teacher outputs, with slightly lower test error. Periodic integration changes little because its features already remove the relevant nuisance.

Appendix C Rollout divergence, per hop

The per-hop table retains complete rollout trajectories. Chains start earlier in episodes to leave sufficient runway, so their absolute agreement should not be compared directly with the single-step sample. ALOHA uses three hops and PushT two.

Table 4: Per-hop cross-encoding cosine for Table 3: 3 seeds, three datasets and two morphologies; cross-encoding cosine between the absolute and relative rollouts at each hop. PushT episodes are short and admit two hops rather than three. Read the trend across hops, not the absolute values: chains need horizon×Δ\text{horizon}\times\Delta of runway and so start earlier in an episode than the pairs of Table 2, where the encodings are more similar at the input (cos⁡0.46\cos 0.46 against 0.270.27).
Mean cos\cos Worst-case cos\cos
Dataset Model hop 1 hop 2 hop 3 hop 1 hop 2 hop 3
ALOHA Teacher 0.836 0.803 0.805 0.515 0.350 0.242
cabinet λ=0\lambda=0 0.992 0.990 0.988 0.972 0.951 0.896
λ=16\lambda=16 1.000 1.000 1.000 1.000 0.999 0.997
ALOHA Teacher 0.916 0.889 0.898 0.707 0.676 0.614
coffee λ=0\lambda=0 0.995 0.993 0.991 0.987 0.973 0.955
λ=16\lambda=16 1.000 1.000 1.000 1.000 0.999 0.998
Teacher 0.246 0.178 — −0.298-0.298 −0.497-0.497 —
PushT λ=0\lambda=0 0.633 0.677 — −0.054-0.054 −0.348-0.348 —
λ=16\lambda=16 0.946 0.872 — 0.666 0.139 —

Appendix D Additional results

Additional tables and figures compare scaling, penalty strength, and transformations whose information is already removed by the representation.

D.1 Symmetry breaking along data, optimization and capacity

The table contrasts larger datasets, longer optimization, and wider predictors. The latter two also worsen canonical performance on this small dataset, limiting their interpretation as evidence about model scale.

Table 5: The incentive strengthens along all three axes the threshold implicates. Cross-encoding cosine; lower is more broken.
Axis Varied From To
Data n=150→1200n=150\to 1200 collapse 2.5×→9.9×2.5\times\to 9.9\times
Optimization 800→8000800\to 8000 steps cos⁡0.370\cos 0.370 cos⁡0.242\cos 0.242
Capacity hidden 1024/2→2048/31024/2\to 2048/3 cos⁡0.370\cos 0.370 cos⁡0.125\cos 0.125

D.2 When the orbit is too large to enumerate

Two action conventions can be enumerated cheaply. The following examples explain why that convenience does not extend automatically to large or continuous transformation families.

Discrete orbits grow factorially.

There are N! permutations of N interchangeable items. Independent Monte Carlo averaging reduces standard deviation only as the inverse square root of sample count, so demanding near-exact agreement can be expensive.

Continuous orbits are not finite sums at all.

A continuous transformation family requires integration rather than a finite exact sum. Finite sampling can approximate it, but does not enumerate the full orbit.

Where that leaves robot dynamics.

Small convention sets suit the implemented pooled loss. Larger symmetries motivate encoders or architectures that remove or respect the transformation directly.

D.3 True symmetry breaking at finite n

For a synthetic permutation-invariant classification target, invariant models retain high accuracy while unrestricted models show finite-data dispersion. More data reduces that dispersion; increasing dimension makes the unrestricted task harder. This controlled example differs from the one-convention robot training setup.

Table 6: True symmetry breaking at finite nn. Target Y=𝟏{median(𝐱)>0∧std(𝐱)>0.8∧range(𝐱)>2}Y=\mathbf{1}\{\mathrm{median}(\mathbf{x})>0\;\wedge\;\mathrm{std}(\mathbf{x})>0.8\;\wedge\;\mathrm{range}(\mathbf{x})>2\} is exactly SdS_{d}-invariant. The unconstrained model’s orbit dispersion under random coordinate permutations (TV to the orbit mean) is large at finite nn and grows with |Sd||S_{d}|, confirming that compression selects non-invariant predictors even when the target is invariant.
dd |Sd||S_{d}| ntrainn_{\mathrm{train}} Acc (unconstr.) Acc (invariant) Orbit disp.
8 4×1044\times 10^{4} 1,000 0.858 0.979 0.141
8 4×1044\times 10^{4} 10,000 0.942 0.993 0.059
8 4×1044\times 10^{4} 100,000 0.968 0.995 0.024
16 2.1×10132.1\times 10^{13} 1,000 0.756 0.974 0.229
16 2.1×10132.1\times 10^{13} 10,000 0.828 0.991 0.163
16 2.1×10132.1\times 10^{13} 100,000 0.918 0.994 0.062
32 2.6×10352.6\times 10^{35} 1,000 0.708 0.978 0.277
32 2.6×10352.6\times 10^{35} 10,000 0.757 0.991 0.224
32 2.6×10352.6\times 10^{35} 100,000 0.810 0.993 0.151

D.4 Collapse ratio against training-set size

The data-size figure separates canonical and alternative-encoding retrieval. Their ratio worsens because the trained convention improves, while the measured directional disagreement stays approximately flat.

Original figure
Figure 3: More data improves the trained encoding and barely improves the equivalent one, so the collapse ratio grows from 2.5×2.5\times to 9.9×9.9\times. Panel b is a quotient of the two curves in panel a, and should be read as the growing cost of a fixed disagreement, not as a growing disagreement: cross-encoding cosine is flat across the same sweep (0.357,0.337,0.374,0.3700.357,0.337,0.374,0.370).

D.5 Cost of orbit-constancy versus λ

The penalty sweep plots rank changes relative to pooled training. Negative values indicate improvement; large penalties become less favorable with more data or optimization.

Original figure
Figure 4: The cost of orbit-constancy as a function of λ\lambda, relative to λ=0\lambda=0. Negative is better. Left: the benefit shrinks and finally reverses as nn grows. Right: the same, earlier, with more optimization.

D.6 Admissibility certificates on four numerical axes

Angle wrapping and time-origin shifts show almost no dispersion because their representations already omit the nuisance. Circular shifts and node permutations show measurable differences that the tested repair reduces.

Table 7: Admissibility certificates on four further numerical axes, 3 seeds each. All are admissible; two exhibit no measurable departure because the featurization already removes the nuisance.
Axis Family Teacher dispersion Reduction Certificate
Angle wrap (Kepler) gauge 8.6×10−158.6\times 10^{-15} — bijective
Time origin (Lorenz) gauge 4.2×10−124.2\times 10^{-12} — bijective
Circular shift gauge 2.6×10−22.6\times 10^{-2} 100%100\% quotient
Node permutation (Ax=b) exchangeability 3.3×1023.3\times 10^{2} 83.4%83.4\% quotient

Appendix E Description-length account of the incentive

The appendix decomposes a description-length objective into model-description cost and best-achievable empirical prediction loss. It distinguishes empirical predictive benefit from approximation error and population information; these algebraic identities do not themselves demonstrate how a neural optimizer will behave. [15] [16]

Three regimes, not one conclusion.

Three cases matter: the changed coordinate carries target information; the target is truly invariant but finite data or model misspecification create gaps; or training never varies the coordinate at all. The robot convention experiment primarily concerns the third case. The original definitions and formal statements remain available in the source paper.

Equations
MDLn​(ℱ):=L⁡(ℱ)+n⋅inff∈ℱ𝔼P^​[−log⁡f⁡(Y∣X)].\mathrm{MDL}_{n}(\mathcal{F}):=L(\mathcal{F})+n\cdot\inf_{f\in\mathcal{F}}\mathbb{E}_{\hat{P}}[-\log f(Y\mid X)].
MDLn​(ℱinv)−MDLn​(ℱ)=L⁡(ℱinv)−L⁡(ℱ)⏟description-length diff.+n⋅IP^ℱ​(Y;G∣Z).\mathrm{MDL}_{n}(\mathcal{F}_{\mathrm{inv}})-\mathrm{MDL}_{n}(\mathcal{F})=\underbrace{L(\mathcal{F}_{\mathrm{inv}})-L(\mathcal{F})}_{\text{description-length diff.}}+n\cdot I^{\mathcal{F}}_{\hat{P}}(Y;G\mid Z).

E.1 Orbit-constancy

Under a genuinely invariant target distribution and a strictly proper scoring rule, the Bayes-optimal prediction shares that invariance. Independent orbit samples reduce the variance of their mean by the sample count. Both statements depend on their stated distributional and sampling assumptions.

Appendix F Proof of Theorem 2

The proof defines a quotient over presentations identified as equivalent and derives a lower bound from the information that quotient discards.

Step 1: orbit-constant predictors are exactly the Q -measurable ones.

An orbit-constant prediction depends only on the quotient class Q. Its optimal log loss is consequently the conditional entropy of the target given Q.

Equation
Rinv=inff~𝔼⁡[−log⁡f~​(Y∣Q)]=H⁡(Y∣Q).R_{\mathrm{inv}}\;=\;\inf_{\tilde{f}}\,\mathbb{E}\big[-\log\tilde{f}(Y\mid Q)\big]\;=\;H(Y\mid Q).

Step 2: Q is a coarsening of every view.

Each individual view determines Q, so passing to Q cannot improve the optimal prediction risk over that view.

Equation
Rinv=H⁡(Y∣Q)≥H⁡(Y∣φv​(X))=R⁡(φv)for every ​v.R_{\mathrm{inv}}\;=\;H(Y\mid Q)\;\geq\;H(Y\mid\varphi_{v}(X))\;=\;R(\varphi_{v})\qquad\text{for every }v.

Step 3.

Taking the worst individual-view risk and subtracting the best produces the stated lower bound on the cost of forcing agreement.

Equality, and the two certificates.

The authors propose mutual reconstructibility and predictive parity as certificates for zero invariance cost. As qualified in Section 3, these checks alone do not prove target sufficiency for a shared, unlabeled view space; the quotient must also retain the information needed to predict the target.

Beyond log loss.

The coarsening argument extends to strictly proper scoring rules through their generalized Bayes risk. Its lower bound similarly compares the best achievable risks before and after imposing agreement.

Appendix G Direction-valued prediction has no Jensen guarantee

For probability predictions, concavity of the logarithm bounds mixture log loss by the average component loss. For unit-vector predictions, three vectors scoring 1, −0.9, and −0.9 against the target can have a normalized mean scoring −1. Opposite vectors can cancel entirely. These counterexamples explain why the robot experiment averages losses instead of normalizing averaged outputs.

Equations
J^​(x,y):=1K​∑k=1K(−log⁡f⁡(y∣gk⋅x))−(−log⁡p¯​(y))≥0\hat{J}(x;y):=\frac{1}{K}\sum_{k=1}^{K}\big(-\log f(y\mid g_{k}\cdot x)\big)-\big(-\log\bar{p}(y)\big)\geq 0
u1=(1,0),u2,3=(−910,±1−81100),u_{1}=(1,0),\qquad u_{2,3}=\big(-\tfrac{9}{10},\ \pm\sqrt{1-\tfrac{81}{100}}\big),

Audit and repair.

The retained algorithm first evaluates the proposed equivalence, then measures dispersion and task performance, fits the pooled objective, and checks bootstrap intervals against a tolerance. A favorable finite-data audit supports the tested operating point, without proving universal invariance or future task safety.

Algorithm 1 Audit and repair for a certified symmetry
0:  base model fTf_{T}, nuisance sampler μ\mu, orbit samples KK, metric MM, dispersion statistic AA, tolerance ε\varepsilon
1:  Certify the transformation family (§3); abort unless predictive parity holds
2:  Measure A⁡(fT)A(f_{T}) and M⁡(fT)M(f_{T}) on held-out data
3:  Fit gθg_{\theta} by Eq. (2) for probability- or direction-valued outputs
4:  Bootstrap confidence intervals for Δ​M\Delta M and Δ​A\Delta A
5:  return make if CI⁡[Δ​A]<0\mathrm{CI}[\Delta A]<0 and CI⁡[Δ​M]≥−ε\mathrm{CI}[\Delta M]\geq-\varepsilon, else break

Appendix H Supplementary experiments

Supplementary experiments examine periodic integration, node permutation, and molecular representations as contrasting invariance cases.

H.1 Action-conditioned dynamics: robotics and iPhyRE

Supplementary ablations show benefits from action conditioning on ALOHA, LIBERO-10, and iPhyRE, with degradation after cross-example action shuffling. PushT instead has worse mean retrieval rank when actions are added. These experiments test action usefulness, not agreement under equivalent representations.

Table 8: Action conditioning: action-shuffle degrades performance, confirming causal use.
Task rank↓\downarrow (no-act) rank↓\downarrow (act) rank↓\downarrow (shuf) R@10↑\uparrow (no-act) R@10↑\uparrow (act) Verdict
ALOHA 186.2 97.7 635.8 0.068 0.124 make
LIBERO-10 80.9 49.6 55.5 0.136 0.242 make
PushT 179.9 349.9 547.7 0.079 0.093 break
Table 9: iPhyRE: 5.5×5.5\times retrieval improvement with action conditioning.
Metric persist no action action shuf
mean rank↓\downarrow 167.5 172.6 31.3 173.9
Recall@10↑\uparrow 0.203 0.185 0.471 0.124

H.2 Molecular dynamics: representation ablation

For Lennard-Jones dynamics, an invariant radial-distribution representation preserves classification accuracy under the tested distillation, roughly 0.894 to 0.893. Flattened coordinates instead fall from 0.856 to 0.456. The result shows the importance of representation and finite approximation in this setup, rather than an impossibility theorem for all symmetry-aware models.

Table 10: Same symmetry, two representations. The defect is caused by the representation.
Encoder Acc (base) Acc (stud.) TV-mean (base) TV-mean (stud.) Verdict
RDF (σ=0.01\sigma{=}0.01) 0.894 0.893 0.017 0.016 make
RDF (σ=0.03\sigma{=}0.03) 0.894 0.893 0.012 0.011 make
Flatten (σ=0.01\sigma{=}0.01) 0.856 0.456 0.449 0.433 break
Concept demonstration

Open image ↗