Literature
Display
← All papers

ActionCodec: What Makes for Good Action Tokenizers

Zibin Dong · Yicheng Liu · Shiduo Zhang · Baijun Ye · Yifu Yuan · Fei Ni · Jingjing Gong · Xipeng Qiu · Hang Zhao · Yinchuan Li · Jianye Hao

Preprint posted · CoRL 2026 (accepted) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers arXiv v2, dated 8 October 2026, including all six appendices. Zibin Dong and Yicheng Liu contributed equally; Dong’s work was completed during an internship at Knowin AI. “Without VLA pretraining” describes the policy backbone: the tokenizer still learns from robotic datasets.

Complete original Figure 1.
Figure 1: ActionCodec provides a comprehensive analysis of VQ action tokenizers that directly impact VLA training and summarizes the best practices. When utilized for the fine-tuning of SmolVLM2-2.2B without additional architectural designs, ActionCodec achieves performance on LIBERO benchmark that far exceeds other tokenizers, particularly in terms of training efficiency.

At a glance

ActionCodec optimizes action tokens for downstream policy learning, not just trajectory reconstruction. It favors consistent tokens for adjacent chunks, a compact vocabulary/token budget, language/temporal alignment and limited query self-attention. Embodiment prompts support shared tokenizer training; residual quantization post-training improves reconstruction.

A SmolVLM2-2.2B policy reaches 95.5% average LIBERO success; a blockwise variant reaches 97.4%. These models fine-tune a pretrained VLM without prior VLA training. Separate appendix experiments add large-scale VLA pretraining. The information-theoretic identities motivate experiments, but do not prove policy improvements.

1 Introduction

A good reconstruction target can still be difficult for a VLM to predict. The paper evaluates tokenizers through learning speed, closed-loop success and late-stage degradation, comparing learned VQ representations with binning, strings and FAST.

The four design choices jointly affect stability, fidelity and token usage. Their controlled studies concern a small VQ-family setup; broad claims of an optimal tokenizer are not established by those experiments.

Complete original Figure 2.
Figure 2: Neural Network Architecture of ActionCodec. We employ a Perceiver-like transformer architecture due to its inherent flexibility, which facilitates the modeling of diverse token relations and supports the encoding of variable-length action sequences.

2 Preliminaries

An encoder maps a T-step, D-dimensional action chunk to n latent queries. Nearest-codebook lookup produces n discrete indices; a decoder reconstructs the action sequence. Training combines reconstruction, codebook and commitment losses with stop-gradient terms. The original definitions and objective remain below.

Equations 1
ℒV​Q=‖A−A^‖22+‖sg​[Z]−ec‖22+‖Z−sg​[ec]‖22,\mathcal{L}_{VQ}=\|{A}-\hat{{A}}\|_{2}^{2}+\|\text{sg}[{Z}]-{e}_{{c}}\|_{2}^{2}+\|{Z}-\text{sg}[{e}_{{c}}]\|_{2}^{2}, (1)

πθ​(A|V,L)\pi_{\theta}(A|V,L)

A∈ℝT×DA\in\mathbb{R}^{T\times D}

C=[c1,c2,…,cn]C=\left[c_{1},c_{2},\dots,c_{n}\right]

𝒦={1,…,S}\mathcal{K}=\{1,\dots,S\}

Z=[z1,…,zn]∈ℝn×dZ=[{z}_{1},\dots,{z}_{n}]\in\mathbb{R}^{n\times d}

ℬ={ej}j=1S⊂ℝd\mathcal{B}=\{{e}_{j}\}_{j=1}^{S}\subset\mathbb{R}^{d}

ck=arg⁡minj∈{1,…,S}⁡‖zk−ej‖2c_{k}=\arg\min_{j\in\{1,\dots,S\}}\|{z}_{k}-{e}_{j}\|_{2}

A^=𝒢⁡(ec1,…,ecn)\hat{{A}}=\mathcal{G}({e}_{c_{1}},\dots,{e}_{c_{n}})

3 A VLA Perspective on Action Tokenization

Expected token cross-entropy separates model mismatch from conditional target entropy. For a deterministic tokenizer, action-conditioned entropy is zero; sensitivity to perturbations is a different quantity. The token-capacity bound grows with token count and vocabulary size, while conditioning on images/language reduces uncertainty.

Context information and autoregressive-prefix information are complementary chain-rule terms. The paper measures proxies and policy behavior, not their mutual information. Its design studies train tokenizers on all LIBERO data, then fine-tune SmolVLM2-256M on LIBERO-Goal.

Equations 2–4
𝔼[ℒNLL]=DK​L(Pdata∥Pθ)+H(C|V,L),\mathbb{E}\left[\mathcal{L}_{\text{NLL}}\right]=D_{KL}(P_{\text{data}}\parallel P_{\theta})+H(C|V,L), (2)
H⁡(C|V,L)=H⁡(C|A)⏟Deterministic Term+I⁡(C,A)⏟Capacity−I⁡(C,V,L)⏟Perceptual AlignmentH(C|V,L)=\underbrace{H(C|A)}_{\text{Deterministic Term}}+\underbrace{I(C;A)}_{\text{Capacity}}-\underbrace{I(C;V,L)}_{\text{Perceptual Alignment}} (3)
I⁡(ck,V,L,c<k)⏟Total Information=I⁡(ck,V,L)⏟Visual-Language Alignment+I(ck;c<k|V,L)⏟Residual Grammar\underbrace{I(c_{k};V,L,c_{<k})}_{\text{Total Information}}=\underbrace{I(c_{k};V,L)}_{\text{Visual-Language Alignment}}+\underbrace{I(c_{k};c_{<k}|V,L)}_{\text{Residual Grammar}} (4)

H⁡(C|A)=0H(C|A)=0

Hϵ​(𝒯⁡(A+ϵ)|A)H_{\epsilon}(\mathcal{T}(A+\epsilon)|A)

I⁡(C,A)=H⁡(C)≤∑k=1nH⁡(ck)≤n​log2​SI(C;A)=H(C)\leq\sum_{k=1}^{n}H(c_{k})\leq n\log_{2}S

3.1 Impacts of Overlap Rate

Changing a perturbation-contrastive weight produces 26%, 40% and 70% adjacent-chunk overlap. At 500 policy updates, 70% overlap gives 33.4% success, versus 8.2% for the basic VQ tokenizer and 2% for FAST. Four-task t-SNE plots show earlier task separation and less late-stage drift.

These are associations under controlled horizon, vocabulary and token budget; changing the contrastive loss also changes reconstruction and token usage. Maximizing overlap without a fidelity constraint could erase useful motion distinctions.

Complete original Figure 3.
Figure 3: LIBERO-Goal results for different design choices. All VLA models are based on SmolVLM2-256M, following vocabulary expansion and full-parameter fine-tuning without additional architectural modifications. The suffix notations are defined as follows: acc (L1 error), tok (token budget), cb (vocabulary size), OR (overlap rate), SA (self-attention), Causal (SA w/ causal mask), and SP (training w/ soft-prompt).

3.2 Impacts of Vocabulary Size and Token Budget

Token counts 8/16/32/64 and vocabularies 512/1024/2048/4096 are compared while overlap is held near 40% by retuning contrastive strength. Larger token counts are particularly associated with late-stage rollout degradation. Very small representations lose reconstruction fidelity; 16 tokens and 2,048 codes are a practical compromise, not a unique optimum.

Equation

n∈{8,16,32,64}n\in\{8,16,32,64\}

S∈{512,1024,2048,4096}S\in\{512,1024,2048,4096\}

3.3 Impacts of Vision-Language Alignment and Residual Grammar

Time-contrastive learning aligns neighboring action chunks; a language-contrastive objective aligns action embeddings with instructions, with an additional L1 regularizer. Attention maps suggest language alignment emphasizes instruction-relevant objects, while temporal alignment can capture demonstration-specific patterns. Neither visualization measures mutual information.

A Perceiver using cross-attention alone is compared with bidirectional or causal query self-attention. Shuffling one predicted token before continuing generation hurts the cross-attention variant less. “Independent” means no query self-attention; all queries share the action input and may remain statistically dependent.

Equations 5–6
ℒTCL\displaystyle\mathcal{L}_{\text{TCL}} =−∑iloge𝒮⁡(Zi,Zi+)e𝒮⁡(Zi,Zi+)+e𝒮⁡(Zi,Zi−),\displaystyle=-\sum_{i}\log\frac{e^{\mathcal{S}(Z^{i},Z^{i+})}}{e^{\mathcal{S}(Z^{i},Z^{i+})}+e^{\mathcal{S}(Z^{i},Z^{i-})}}, (5)
ℒCLIP\displaystyle\mathcal{L}_{\text{CLIP}} =−∑i∑jlog11+e(li​j​(−t​Zi​yj+b)),\displaystyle=-\sum_{i}\sum_{j}\log\frac{1}{1+e^{\left(l_{ij}(-tZ^{i}y^{j}+b)\right)}}, (6)
Complete original Figure 4.
Figure 4: t-SNE visualization of the VLA’s last hidden states for the action [BOS] token on four LIBERO-Goal tasks.
Complete original Figure 5.
Figure 5: (Left) VLA attention maps for CLIP vs. TCL-trained tokenizers. (Right) t-SNE visualization of the structured latent space across 40 LIBERO tasks.
Complete original Figure 6.
Figure 6: Error propagation under token perturbation. We measure the L1 reconstruction error when noise is injected at specific positions during generating. VQ-Independent maintains a stable, lower error compared to SA and Causal architectures. This supports reducing architectural token coupling for prefix-error robustness.

4 ActionCodec

Embodiment-specific learned queries and timestamp encodings let one Perceiver tokenizer use LIBERO, BridgeData and DROID with different control rates. Cross-embodiment trajectory demonstrations illustrate transfer of motion patterns; they do not establish arbitrary task transfer.

After stable single-codebook training, freeze the encoder and primary codebook and train residual codebooks plus the decoder for reconstruction. The full three-level tokenizer supports blockwise prediction. Ordinary ActionCodec copies the refined decoder back while retaining its original 16-token encoding; this preserves the first-level labels exactly.

5 Experiments

Experiments cover controlled tokenizer comparisons, action-generation architectures, two real robot tasks and component ablations. All original figures and comparison tables are retained.

5.1 Comparison with Mainstream Action Tokenizers

With a shared SmolVLM2-2.2B backbone, ActionCodec reaches 89.5% LIBERO success after 5,000 steps, versus FAST’s 38.6%; final averages are 95.5% and 90.6%. Smaller 256M/500M ActionCodec policies reach 92.3%/92.5%.

Table 2 reports 16 tokens, 72% overlap, 0.9 s policy latency and 22 actions/s for 20-step chunks. Throughput counts decoded actions, not independent reactive decisions at 22 Hz. Baseline horizons and token budgets differ; these are complete-method comparisons rather than matched-budget ablations.

Table 1: Benchmark Results. Panels (a)–(c) report LIBERO success rates (%) for pre-trained VLA backbones, scratch baselines, and SmolVLM2 tokenizer/variant comparisons, respectively. Panel (d) reports SimplerWidowX success rates (%). Pt. denotes VLA pre-training of the policy backbone, excluding tokenizer pre-training. Avg. Rank indicates relative performance across tasks. Best results in each block are bolded, second best are underlined.
(a) LIBERO: Pre-trained VLA Backbones
Model Pt. Goal Spatial Object Long Avg. Avg.
Rank
Octo-Base [27] Y 84.6 78.9 85.7 51.1 75.1 10.5
SpatialVLA [30] Y 78.6 88.2 89.9 55.5 78.1 9.5
GR00T-N1 [3] Y 93.0 94.4 97.6 90.6 93.9 6.6
UniVLA [5] Y 95.6 96.5 96.8 92.0 95.2 5.4
VQ-VLA [36] Y 75.2 - - 60.0 - -
OpenVLA [19] Y 79.2 84.7 88.4 53.7 76.5 10.0
OpenVLA-OFT [18] Y 97.9 97.6 98.4 94.5 97.1 2.5
π0\pi_{0} [4] Y 95.8 96.8 98.8 85.2 94.2 4.5
π0.5\pi_{0.5} [15] Y 98.0 98.8 98.2 92.4 96.8 2.5
π0\pi_{0} FAST [29] Y 88.6 96.4 96.8 60.2 85.5 7.4
ActionCodec (2.2B) N 95.4 96.2 99.6 90.6 95.5 5.1
ActionCodec-BAR N 97.6 97.2 99.8 94.8 97.4 2.0
(b) LIBERO: Scratch Baselines
Model Pt. Goal Spatial Object Long Avg. Avg.
Rank
Diffusion Policy [7] N 68.3 78.3 92.5 50.5 72.4 13.8
MiniVLA [2] N - - - 77.0 - -
OpenVLA-OFT [18] N 91.7 94.3 95.2 86.5 91.9 7.8
SmolVLA 2.2B [31] N 91.0 93.0 94.0 77.0 88.8 10.2
π0\pi_{0} FAST [29] N 89.0 87.0 63.0 48.0 71.8 14.2
π0.5\pi_{0.5} [15] N 94.6 96.6 97.2 85.8 93.3 6.4
VLA-0 [12] N 96.2 97.0 97.8 87.6 94.7 3.8
(c) LIBERO: SmolVLM2 + Tokenizers and Variants
Model Pt. Goal Spatial Object Long Avg. Avg.
Rank
SmolVLM2-2.2B (256M/500M) + Tokenizers
Binning N 27.4 64.2 71.0 50.8 53.4 15.5
String N 30.8 69.6 71.6 26.4 49.6 15.5
VQVLA’s N 60.5 56.0 69.8 51.4 60.5 15.2
MiniVLA’s N 82.1 82.8 90.2 75.2 82.6 12.8
FAST N 89.2 90.2 97.2 85.8 90.6 9.4
ActionCodec (256M) N 91.2 94.2 97.0 86.8 92.3 7.5
ActionCodec (500M) N 96.4 91.6 96.0 85.8 92.5 7.8
ActionCodec (2.2B) N 95.4 96.2 99.6 90.6 95.5 3.8
Variants
ActionCodec-PD N 97.0 95.2 98.4 91.6 95.6 3.0
ActionCodec-KI N 95.0 93.6 97.6 90.8 94.3 5.5
ActionCodec-BAR N 97.6 97.2 99.8 94.8 97.4 1.0
(d) SimplerWidowX Benchmark Results
Model Pt. Spoon Carrot Block Eggplant Avg. Avg.
Rank
Octo-Base [27] Y 12.5 8.3 0.0 43.1 16.0 7.3
SpatialVLA [30] Y 16.7 25.0 29.2 100.0 42.7 4.0
π0\pi_{0} [4] Y 66.7 58.3 58.3 88.3 67.9 2.0
OpenVLA-OFT [18] Y 12.5 4.2 8.3 0.0 6.3 8.1
UniVLA [5] Y 54.2 66.7 50.0 4.2 43.8 3.5
OpenVLA [19] Y 32.0 30.0 18.0 38.0 29.5 4.8
VQ-VLA [36] Y 12.5 8.0 6.0 0.0 6.6 8.1
π0\pi_{0} FAST [29] Y 29.1 21.9 10.8 66.6 32.1 5.3
ActionCodec-BAR N 71.7 64.2 55.0 70.0 65.2 2.0
MiniVLA [2] N 68.0 44.0 70.0 14.0 49.0 1.8
ActionCodec-BAR N 71.7 64.2 55.0 70.0 65.2 1.3
Table 2: Action Tokenization Performance and Efficiency. Comparison of token budget, overlap rate (OR), and inference metrics. All models are evaluated on the LIBERO benchmark.
Tokenizer Token
Budget
OR Horizon Latency
(s)
Throughput
(action/s)
SR (%)
Binning [12] 140.0±0.0 39% 20 4.9 4.1 3.5
Binning [12] 56.0±0.0 38% 8 2.6 3.1 53.4
String [13] 779.2±17.3 28% 20 33.8 0.6 20.4
String [13] 311.8±7.8 34% 8 12.7 0.6 49.6
VQVLA’s [36] 4.0±0.0 45% 5 0.7 7.7 60.5
MiniVLA’s [2] 7.0±0.0 40% 8 0.7 11.1 82.6
FAST [29] 24.3±9.1 19% 20 1.5 13.1 90.6
ActionCodec 16.0±0.0 72% 20 0.9 22.0 95.5

5.2 Integration with Prevailing VLA Paradigms

A 300M action expert enables parallel decoding (PD), knowledge isolation (KI) or blockwise autoregression (BAR). LIBERO averages are 95.6%, 94.3% and 97.4%, respectively, against 95.5% native autoregression. KI is weaker here despite outperforming its FAST-based counterpart.

On Simpler-WidowX, BAR averages 65.2%. It ties π0’s average rank within the mixed-pretraining block but falls below π0’s 67.9% average; it leads the separately ranked non-VLA-pretrained block. The headline ranking is not a win on every task or metric.

Complete original Figure 7.
Figure 7: Success Rates on SO100-ShapeSorter. We report success counts for Pick (solid) and Place (hatched) across 10 tasks. While recovery actions are inherently present in our task demonstrations due to the difficulty of slot insertion, only ActionCodec (with CT) successfully learns to reproduce these corrective strategies. The performance gap is most prominent in the Place phase, where co-training facilitates the execution of complex recovery behaviors that task-only models fail to capture.

5.3 Real-world Evaluation

SO100-ShapeSorter tests selecting ten numbered blocks and inserting them into tight slots. Adding 22.9K community episodes particularly improves insertion/recovery, with qualitative corrective sequences retained in Figure 8. Appendix B clarifies that the “without co-training” variant still uses LIBERO, BridgeData and DROID; it excludes the SO100 community data, so it is not strictly task-only.

On xArm vegetable picking, initialization from the shared tokenizer improves reconstruction and overlap; policy success is 82.5%, versus 74.1% without tokenizer pretraining, 75.0% for π0 and 72.5% for π0-FAST. The source’s Table 4 reference for these numbers should be Table 3. Datasets: SO100 and xArm.

Table 3: Success Rates on xArm-PickVeg. ActionCodec w/ PT leverages heterogeneous priors to achieve the highest success rate.
Model ActionCodec
w/ PT
ActionCodec
w/o PT
π0{\pi_{0}} π0{\pi_{0}} FAST
SR (%) 82.5 74.1 75.0 72.5
Table 4: Ablations. We ablate various design choices on LIBERO. We report the average success rate over the four suites.
ID Co-Training Soft Prompt RVQ Post Training VLM LIBERO Avg.
0

×\bm{\times}

×\bm{\times}

✓\bm{\checkmark}

SmolVLM2 92.0 (-3.5%)
1

✓\bm{\checkmark}

×\bm{\times}

✓\bm{\checkmark}

SmolVLM2 92.7 (-2.8%)
2

✓\bm{\checkmark}

✓\bm{\checkmark}

×\bm{\times}

SmolVLM2 95.2 (-0.3%)
3

✓\bm{\checkmark}

✓\bm{\checkmark}

✓\bm{\checkmark}

SmolVLM2 95.5
4

✓\bm{\checkmark}

✓\bm{\checkmark}

✓\bm{\checkmark}

Qwen2.5VL 95.1
5

✓\bm{\checkmark}

✓\bm{\checkmark}

✓\bm{\checkmark}

InternVL3.5 94.6

5.4 Ablation Study

LIBERO success rises from 92.0% without co-training to 92.7% with co-training but no soft prompts, then 95.2% with prompts. RVQ decoder post-training adds only 0.3 points to reach 95.5%, despite reducing reconstruction error. This supports fidelity/stability tradeoffs in this setting, not a universal error threshold.

SmolVLM2, Qwen2.5VL and InternVL3.5 versions reach 95.5%, 95.1% and 94.6%. The original backbone logos in Table 4 remain; their identities are stated here for clarity.

Complete original Figure 8.
Figure 8: Comparison of recovery behaviors on SO100. After an initial missed shot, ActionCodec with co-training successfully executes a corrective adjustment to complete the insertion, whereas the variant without co-training stalls after a failure.
SmolVLM2Qwen2.5VLInternVL3.5

6 Conclusion

The main contribution is a downstream-policy view of token quality: temporal consistency, compactness, context alignment and limited architectural coupling can matter more than minimum reconstruction error. Broader embodiment coverage and improved architectures/alignment remain open.

7 Limitations

The architecture prioritizes ablation flexibility rather than final performance. Two auxiliary contrastive losses add tuning choices. Even the additional 15K-hour, 14-embodiment study evaluates limited benchmarks and does not establish broad in-the-wild generalization.

Impact Statement

The authors frame the work as advancing machine learning and identify no particular societal consequence requiring special discussion. This is an impact statement, not a deployment safety evaluation.

Appendix A Related Works

Continuous VLA approaches use regression, diffusion or flow experts; discrete approaches preserve a language-model-style prediction objective but can increase sequential latency. Hybrid models also use discrete action supervision.

Binning and raw strings are simple but token-intensive. FAST compresses frequency coefficients with BPE; VQ-VLA, MiniVLA and other learned schemes adapt their latent codes. ActionCodec studies properties of those learned targets rather than treating reconstruction as the only quality metric.

The original bibliography omits two cited works, leaving question marks in the PDF. Their source keys identify Fast-WAM and G0.5; verified entries are appended without renumbering the 39 existing references.

Complete original Figure 9.
Figure 9: Cross-embodiment action transfer. To evaluate the transferability of ActionCodec, a 1-second WidowX action sequence sampled from BridgeData is encoded into action tokens and subsequently decoded into 1-second trajectories for LIBERO-Franka, DROID-Franka, and xArm. After deploying these sequences twice (totaling 2 seconds) and recording the resulting motions, all three robotic platforms exhibit highly consistent action patterns. This demonstrates that ActionCodec effectively captures hardware-agnostic action semantics.

Appendix B Experimental Setup and Benchmark Details

The appendices specify demonstrations, evaluation trials, model inputs, hardware, optimization and the specialized attention masks. All four hyperparameter tables and the complete prompt are retained.

B.1 Benchmarks

LIBERO has four ten-task suites, 50 demonstrations per task and 50 evaluation initial states per task. Simpler-WidowX uses four Bridge-trained out-of-domain tasks with 120 trials each after four fine-tuning epochs.

SO100 uses overhead/wrist RGB, absolute joint angles, 30 Hz demonstrations, 50 examples per numbered block and 20 randomized trials each. Pick and insertion are separate outcomes. The co-training ablation removes only SO100 community episodes.

xArm uses a third-person RGB camera, end-effector proprioception and delta-pose actions at 30 Hz, with 30 demonstrations and 20 randomized trials per vegetable. Pretrained tokenizer initialization uses LIBERO/BridgeData/DROID; the comparison initializes the tokenizer randomly on xArm. Both real-task policies train for 30,000 steps.

Complete original Figure 10.
Figure 10: All evaluation benchmarks. We evaluate VLA models on 4 LIBERO task suites, Bridge-WidowX tasks on SimplerEnv, SO100-ShapeSorter tasks, and xArm tabletop manipulation tasks.

B.2 Details of Validation Experiments

Perceiver query attention can be removed or made bidirectional/causal while retaining shared action input. SmolVLM2-256M is fully fine-tuned after adding action vocabulary entries. Invalid generated sequences become zero-action chunks; FAST instead receives padding/truncation only when its BPE-decoded length is invalid.

Tokenizers see all LIBERO tasks, but design-study policy learning uses LIBERO-Goal. Curves inspect 500–20,000 steps, while Table 6 specifies a 30,000-step schedule. The [BOS] label denotes the prompt context token immediately before action generation.

Equation

<|action_0|>​…​<|action_(S-1)|>\texttt{<|action\_0|>}\dots\texttt{<|action\_(S-1)|>}

B.2.1 Impacts of Overlap Rate

An InfoNCE-style auxiliary loss brings an action’s latent representation close to a Gaussian-perturbed copy, with other batch samples as negatives. Varying its weight controls overlap. The final model uses temporal and language contrastive objectives because they give more stable training in the reported comparisons.

Equation

ℒVQ+λ⋅ℒInfoNCE,\mathcal{L}_{\text{VQ}}+\lambda\cdot\mathcal{L}_{\text{InfoNCE}},

Z=ℱ⁡(A)Z=\mathcal{F}(A)

Z+=ℱ⁡(A+η),η∼𝒩⁡(0,σ2)Z^{+}=\mathcal{F}(A+\eta),~\eta\sim\mathcal{N}(0,\sigma^{2})

Definition and controls.

Overlap is one minus normalized Hamming distance at corresponding token positions for adjacent one-step-shifted chunks, averaged over pairs. It is not set overlap. The overlap ablation fixes 16 tokens, 2,048 codes, horizon, stride and policy; capacity sweeps retune contrastive strength to keep overlap at 40%.

All three overlap variants use every code. Marginal perplexity falls from 2,016 to 1,290 as overlap increases, while reconstruction L1 error changes from 0.020 to 0.028. This indicates concentrated usage rather than codebook collapse; neither perplexity nor overlap directly estimates conditional entropy.

Equation 7
OR=𝔼t​[1−dH​(Ct,Ct+1)n],\mathrm{OR}=\mathbb{E}_{t}\!\left[1-\frac{d_{H}(C_{t},C_{t+1})}{n}\right], (7)
Table 5: Codebook usage in the controlled OR experiment. All variants use n=16n=16 and S=2048S=2048. Utilization is measured on the validation set; PPL is the marginal token-usage perplexity exp⁡(H⁡(c))\exp(H(c)), with entropy in nats. Reconstruction errors are from Figure 3.
OR (%) Codebook utilization (%) PPL L1L_{1} error SR at 500 steps (%)
26 100 2016 0.020 8.2
40 100 1583 0.029 12.4
70 100 1290 0.028 33.4
Equation

At=(at,…,at+T−1)A_{t}=(a_{t},\ldots,a_{t+T-1})

At+1=(at+1,…,at+T)A_{t+1}=(a_{t+1},\ldots,a_{t+T})

B.2.2 Impacts of Residual Grammar

Cross-attention-only, self-attention and causal self-attention tokenizers differ in direct latent-query interaction. During policy generation, one token is shuffled, the remaining sequence is generated and decoded, and L1 action error is measured. The first variant is more robust in these tests; statistical independence and a universal optimal attention pattern are not implied.

Complete original Figure 11.
Figure 11: Comparison of Perceiver architectures for Residual Grammar: (left) Independent (cross-attention only), (middle) SA (with self-attention), and (right) Causal (with causal self-attention)

B.3 Details of Tokenizer Comparison Experiments

The shared tokenizer trains for 100,000 steps with batch 8,192 and learning rate 2×10⁻⁴, encoding one second through timestamp-aware position embeddings. Tokenizer robotic pretraining is separate from whether the policy backbone has VLA pretraining.

Binning and strings use eight-step horizons to contain sequence length; VQ baselines use official implementations/checkpoints with aligned statistics, and FAST uses its universal release. All comparison policies fully fine-tune SmolVLM2-2.2B jointly across LIBERO for 30,000 steps. Different tokenizer pretraining datasets remain a comparison qualification.

Table 6: VLA Training Hyperparameters. Detailed configurations for the validation experiments on LIBERO-Goal.
Hyperparameters Values
GPUs 8×\timesNVIDIA RTX 3090 (24 GB VRAM)
Learning Rate 2×10−42\times 10^{-4} peak (1k steps linear warmup, 30k steps cosine decay to 2×10−52\times 10^{-5})
Global Batch Size 128 (16 per GPU)
Training Steps 30k (LIBERO-Goal only)
Input Modalities 1×\times Third-person camera, 1×\times Wrist camera (concatenated)
Input Image Size 224 ×\times 448
Use Proprioception True
Image Augmentation Random crop (87.5%), Brightness/Contrast/Saturation (0.2), Hue (0.05)
Table 7: VLA Training Hyperparameters. Detailed configurations for the tokenizer comparison experiments on LIBERO.
Hyperparameters Values
GPUs 4×\timesNVIDIA A100 (80 GB VRAM)
Learning Rate 10−410^{-4} peak (1k steps linear warmup, 30k steps cosine decay to 10−510^{-5})
Global Batch Size 128 (32 per GPU)
Training Steps 30k (applied to all four LIBERO task suites)
Input Modalities 1×\times Third-person camera, 1×\times Wrist camera (concatenated)
Input Image Size 224 ×\times 448
Use Proprioception True
Image Augmentation Random crop (87.5%), Brightness/Contrast/Saturation (0.2), Hue (0.05)
Table 8: VLA Training Hyperparameters. Detailed configurations for the Simpler-WidowX experiments.
Hyperparameters Values
GPUs 8×\timesNVIDIA H100 (94 GB VRAM)
Learning Rate 2×10−52\times 10^{-5}
Global Batch Size 128 (16 per GPU)
Training Epochs 4
Input Modalities 1×\times Third-person camera
Input Image Size 224 ×\times 224
Use Proprioception True
Image Augmentation Random crop (90%), Brightness/Contrast/Saturation (0.2), Hue (0.05)
Table 9: VLA Training Hyperparameters. Detailed configurations for the real-world experiments on SO100 and xArm.
Hyperparameters Values
GPUs 4×\timesNVIDIA A100 (80 GB VRAM)
Learning Rate 5×10−55\times 10^{-5} peak (1k steps linear warmup, 30k steps cosine decay to 5×10−65\times 10^{-6})
Global Batch Size 64 (16 per GPU)
Training Steps 30k
Input Modalities SO100: 1×\times Third-person camera, 1×\times Wrist camera (concatenated), xArm: 1×\times Third-person camera
Input Image Size 224×\times448 for SO100, 224×\times224 for xArm
Use Proprioception True
Image Augmentation Brightness/Contrast/Saturation (0.2), Hue (0.05)
Complete original Figure 12.
Figure 12: VLA paradigm architectures.
Complete original Figure 13, including the indented VLM prompt.
Figure 13: Prompt template for VLM fine-tuning.
Copyable prompt
[
  {
    "role": "system",
    "content": "Analyze the image and predict robot actions."
  },
  {
    "role": "user",
    "content": [
      {"type": "image"},
      {"type": "text",
       "text": "**State**: {state}, **Task**: {task}."}
    ]
  },
  {
    "role": "assistant",
    "content": [
      {"type": "text",
       "text": "{’action’: <|action_n|> ... <|action_m|>}"}
    ]
  }
]

B.4 Details of Paradigm Adaptation Experiments

The 300M action expert consumes the VLM KV cache and handles specialized masks. PD predicts all token logits with bidirectional attention in one pass. KI trains a flow-matching action expert alongside autoregressive supervision, stopping expert gradients into the VLM.

BAR predicts each RVQ level in parallel but proceeds autoregressively across levels. One level reduces to PD; later levels refine residuals. Lower prefix-error sensitivity is empirical, not a proof that errors always damp. The expert adds capacity, so architecture gains are not tokenizer-only ablations.

Complete original Figure 14.
Figure 14: Learning curves of ActionCodec with and without pre-training. Compared to random initialization, initializing with pre-trained ActionCodec weights and fine-tuning on novel robotic data results in significantly lower L2 loss (higher reconstruction accuracy) and a higher Overlap Rate (OR). These results demonstrate that the pre-trained ActionCodec possesses a unified and semantically rich latent space, enabling rapid adaptation to new robotic action patterns.

Appendix C Details of Soft-Prompt

Each embodiment index selects learned query embeddings. Control frequency and encoded duration determine timestamps, which receive Fourier embeddings for encoder keys/values and decoder queries. These mechanisms distinguish embodiment-specific control conventions while sharing network weights.

Complete original Figure 15.
Figure 15: Illustration of the Soft-Prompt integration within the Perceiver architecture. Learnable embeddings represent embodiment IDs, while fourier embeddings provide temporal grounding for varying control frequencies.

Appendix D Details of RVQ Post-training

Train a 16-token single-level VQ model first. Initialize a three-level RVQ version from it, freeze the encoder and primary codebook, and optimize residual codebooks plus decoder with reconstruction/RVQ losses. Copying the decoder back preserves the original single-level token identities; BAR alone uses all three levels.

“No additional cost” concerns the ordinary tokenizer’s encoding/prediction interface, not the extra post-training computation or BAR’s extra levels. The paper demonstrates improved fidelity empirically rather than proving zero downstream degradation.

Complete original Figure 16.
Figure 16: Procedure of RVQ post-training.

Appendix E Scope of the Information-Theoretic Analysis

The scope appendix distinguishes exact information identities from measured proxies and empirical design guidance.

E.1 Deterministic and Perturbation-Induced Entropy

A deterministic tokenizer has zero action-conditioned entropy. A separately defined Gaussian perturbation induces randomness near quantization boundaries; its entropy is not substituted into the deterministic identity. Adjacent-chunk overlap probes local consistency only when neighboring motion and context are sufficiently similar.

Equations 8
Hϵ​(Cϵ|A)=𝔼a∼PA​[Hϵ​(𝒯⁡(a+ϵ))].H_{\epsilon}(C_{\epsilon}|A)=\mathbb{E}_{a\sim P_{A}}\!\left[H_{\epsilon}\bigl(\mathcal{T}(a+\epsilon)\bigr)\right]. (8)

C=𝒯⁡(A)C=\mathcal{T}(A)

H⁡(C|A)=0H(C|A)=0

I⁡(C,A)=H⁡(C)I(C;A)=H(C)

Cϵ=𝒯⁡(A+ϵ)C_{\epsilon}=\mathcal{T}(A+\epsilon)

ϵ∼𝒩⁡(0,σ2​I)\epsilon\sim\mathcal{N}(0,\sigma^{2}I)

E.2 Capacity Subject to Fidelity

The practical bottleneck is to reduce token entropy subject to acceptable expected reconstruction distortion. Reducing token count/vocabulary too far violates fidelity. Fixing overlap during capacity sweeps removes one confound, while reconstruction and token-usage distributions still vary.

Equations 9
min𝒯,𝒢⁡H⁡(C)subject to𝔼⁡[d⁡(A,A^)]≤ε,A^=𝒢⁡(C),\min_{\mathcal{T},\mathcal{G}}H(C)\quad\text{subject to}\quad\mathbb{E}[d(A,\hat{A})]\leq\varepsilon,\qquad\hat{A}=\mathcal{G}(C), (9)

I⁡(C,A)≤H⁡(C)≤∑kH⁡(ck)≤n​log2​SI(C;A)\leq H(C)\leq\sum_{k}H(c_{k})\leq n\log_{2}S

E.3 Alignment and Architectural Coupling

Attention maps, temporal/language contrastive comparisons and shuffled-prefix errors are qualitative/behavioral probes, not mutual-information estimators. Temporal alignment avoids direct image–action CLIP computation but can learn demonstration-specific patterns. Cross-attention-only queries remain coupled through their shared input.

Appendix F Additional Experimental Results and Comparison Protocols

Additional results separate large-scale VLA pretraining from the main no-VLA-pretraining setting, then clarify baseline costs and overfitting evidence.

F.1 Large-Scale VLA Pre-training and Transfer

A separate Qwen3.5-2B policy receives 15K hours across 14 embodiments with a π0.5-style KI objective. Reported success averages are 87.7% Simpler-WidowX fine-tuning, 98.2% LIBERO fine-tuning and 82.5% DROID zero-shot. These exceed the listed baseline averages, but dataset composition, trial/seed uncertainty and broad real-world transfer are not established by the aggregate table.

Table 10: Large-scale VLA pre-training and transfer. Benchmark-average success rates (%). FT denotes downstream task fine-tuning; ZS denotes zero-shot evaluation. A dash denotes an unreported result.
Method Simpler-WidowX FT LIBERO FT DROID ZS
π0\pi_{0}-FAST 32.1 85.5 –
π0.5\pi_{0.5} 57.1 96.8 57.5
MolmoAct2-DROID – – 52.0
ActionCodec (Qwen3.5-2B) 87.7 98.2 82.5

F.2 Baseline Fairness and Encoding Cost

FAST and ActionCodec both encode 20-step LIBERO chunks, averaging 24.3 versus 16 tokens; other baselines have different horizons. FAST length repair preserves valid outputs. At batch 1,024, encoding costs are 0.11 versus 0.09 ms per chunk; these exclude policy generation and do not prove equal total training time.

Training-step convergence is useful under the shared VLA recipe, but tokenizer pretraining, token-dependent compute and wall-clock hardware costs remain distinct. The smaller-backbone result supports representation quality, not a fully matched total-compute comparison.

Table 11: Action encoding cost. Per-chunk encoding time at batch size 1024. These measurements concern tokenizer encoding, separately from policy generation latency in Table 2.
Tokenizer Encoding time per chunk (ms)
FAST 0.11
ActionCodec 0.09

F.3 Optimization and Overfitting Evaluation

Raw token NLL is incomparable across different vocabularies and lengths. The paper instead tracks closed-loop success, where predicted actions alter later observations and errors accumulate. Late-stage success drops motivate its overfitting discussion; latent plots provide supporting evidence, not a separate causal diagnosis.

Concept demonstration

Open image ↗