Literature
Display
← All papers

RoboTTT: Context Scaling for Robot Policies

Yunfan Jiang · Yevgen Chebotar · Ruijie Zheng · Fengyuan Hu · Yunhao Ge · Jimmy Wu · Tianyuan Dai · Scott Reed · Li Fei-Fei · Yuke Zhu · Linxi “Jim” Fan

Preprint posted · CoRL 2026 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the complete 22-page arXiv v1 paper, including the architecture, training and evaluation appendices. RoboTTT stores long robot histories in continually updated fast weights. Read its results as task-specific evidence: partial completion scores, full successes and recovery counts measure different outcomes. All 16 original figures, three tables, five equations and 83 references remain.

Original S0.F1 artwork, restored from the PDF with all panels and labels.
Figure 1: RoboTTT, a long-context visuomotor policy that integrates Test-Time Training (TTT) into robot foundation models, with context scaled to 8K timesteps. RoboTTT exhibits capabilities such as one-shot in-context imitation from human videos and on-the-fly policy improvement.

At a glance

RoboTTT extends a pretrained robot policy with Test-Time Training (TTT) layers whose fast weights summarize observation/action history. Independently noising action chunks and truncating training gradients make sequences up to 8K timesteps tractable. On three bimanual assembly tasks, the reported mean completion score is 79%, versus 42% for the single-step baseline. Separate experiments show human-video-conditioned imitation, recovery after perturbations and benefits from training with failures as context. Full Gear Bot assembly succeeds in 2/10 trials; the 79% score is not a full-task success rate.

1 Introduction

Long manipulation tasks may revisit visually similar states, need previously seen object information or recover from failed actions. Simply appending a few frames can miss these dependencies or introduce misleading correlations. RoboTTT makes a small learned network’s weights the recurrent memory: updates encode history and queries retrieve information. The outer training objective learns how this memory should adapt. The paper tests context lengths from 128 to 8K timesteps, rather than demonstrating unlimited retention or universal adaptation.

2 Preliminaries

In TTT, learned projections map each input to query, key and value vectors. An inner squared-error loss updates fast-model weights so keys predict values; the updated network then processes the query. The outer action objective learns the projections and initial fast weights. At deployment, slow parameters stay fixed while the fast weights change. A robot sequence contains images, proprioception and action chunks, and the policy conditions on current observations plus preceding history. Fixed-size fast weights replace storage and repeated attention over an ever-growing token cache.

Equations 1–2
Wt←Wt−1−η​∇WℒFW​(fWt−1​(Kt),Vt),W_{t}\leftarrow W_{t-1}-\eta\nabla_{W}\mathcal{L}_{\mathrm{FW}}\left(f_{W_{t-1}}\left(K_{t}\right),V_{t}\right), (1)
Ot=fWt​(Qt).O_{t}=f_{W_{t}}(Q_{t}). (2)
Original S2.F2 artwork, restored from the PDF with all panels and labels.
Figure 2: RoboTTT model architecture, training, and inference. TTT layers are added after the attention layers in the DiT action head: attention operates within each timestep, while TTT layers operate across timesteps. Training uses a sequence flow-matching loss with sequence action forcing, sampling the noise level independently per action chunk. Inference starts from the learned initialization W0W_{0}, updating fast weights on each observation and propagating them forward.

3 Method: Test-Time-Training Robot Policies

The method adds temporal TTT layers to an existing action model, trains them on trajectories with independent action-chunk noise and truncated gradients, and uses selectively masked action losses to learn from videos and corrective interventions. These are distinct architectural, optimization and data-use choices.

3.1 Model Architecture

The implementation starts from GR00T N1.7, with a vision-language backbone and diffusion-transformer action head. At each timestep, self-attention processes proprioception, noised actions and 16 learned register tokens, while cross-attention reads current visual/language tokens. TTT then processes the state, action and register tokens across time. Visual/language tokens do not directly traverse TTT; the compact registers carry that information. Each action-head layer adds the TTT output through a learned tanh gate initialized at 0.001, allowing a small initial departure from the pretrained computation.

Original S3.F3 artwork, restored from the PDF with all panels and labels.
Figure 3: Computation flow with tanh\tanh gating. The TTT output is weighted by the learned gate tanh⁡(α)\tanh(\alpha) before being added to the attention output.
Equation 3
O=tanh⁡(α)⊙OTTT+Oattn,O=\tanh(\alpha)\odot O_{\mathrm{TTT}}+O_{\mathrm{attn}}, (3)

3.2 RoboTTT Sequence Training

An inner loop updates fast weights on a trajectory; the outer loop averages flow-matching action loss over timesteps and differentiates through those updates, including into the learned initialization. Sequence action forcing independently samples each chunk’s noise level, rather than using one noise level for the entire trajectory. The schedule uses a Beta(1.5, 1) draw and scale 0.999; equations retain the exact objective.

Truncated backpropagation carries fast weights between segments but detaches their gradients at boundaries. Context therefore persists beyond a segment while gradient credit assignment does not; activation memory depends on segment length, not the entire history. The initialization receives gradients through the first segment. Each deployment rollout resets to the learned initialization, then carries updated fast weights forward while generating action chunks by iterative denoising. Fixed-state recurrence makes per-step history processing independent of accumulated context length; the paper does not supply a latency-versus-context measurement series.

Equations 4–5
ℒfm​(ξ,W0)=1T​∑t=1Tℓt​((l,ot,qt,At),Wt−1),\mathcal{L}_{\mathrm{fm}}\left(\xi;W_{0}\right)=\frac{1}{T}\sum_{t=1}^{T}\ell_{t}\left(\left(l,o_{t},q_{t},A_{t}\right);W_{t-1}\right), (4)
ℒfm​(ξ,W0)=1T​∑t=1Tℓt​(ξt,Wt−1)=1T​∑t=1T𝔼τt,ϵ​[‖vθ​(Φt,Atτt,qt,Wt−1)−(At−ϵ)‖2],\mathcal{L}_{\mathrm{fm}}\left(\xi;W_{0}\right)=\frac{1}{T}\sum_{t=1}^{T}\ell_{t}\left(\xi_{t};W_{t-1}\right)=\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{\tau_{t},\epsilon}\left[\left\|v_{\theta}\left(\Phi_{t},A_{t}^{\tau_{t}},q_{t};W_{t-1}\right)-\left(A_{t}-\epsilon\right)\right\|^{2}\right], (5)
Original figure
Figure 4: TBPTT. Gradients are truncated at segment boundaries; fast weights carry over, so TTT continues over the entire sequence.

3.3 Effective Learning from Context

Masking a timestep’s action loss lets it contribute context without becoming an imitation target. For one-shot imitation training, a human video is followed by a robot trajectory of the same configuration: only the robot portion receives action supervision. The resulting policy can condition on a new configuration’s demonstration at test time.

DAgger Distillation applies this distinction to intervention trajectories. Both failed robot actions and human corrections update fast-weight history, but only human corrections receive the action loss. This teaches failure-to-correction mappings using the DAgger data, in the spirit of algorithm distillation. At test time, the learned policy supplies its own corrections without new human intervention. This behavior is trained from corrective data; it is not unrestricted reward-based learning or a guaranteed monotonic improvement process.

Original S3.F5 artwork, restored from the PDF with all panels and labels.
Figure 5: Evaluation tasks. Three long-horizon assembly tasks on a YAM bimanual setup; each row shows one rollout. Top, Pup Go Car: toy vehicle assembly, 2-minute average episodes. Middle, Circuit: target configuration specified by a language prompt or a one-shot human video demonstration; 1-minute episodes. Bottom, Gear Bot: ten-stage assembly spanning five minutes, our longest-horizon task.
Original S3.F6 artwork, restored from the PDF with all panels and labels.
Figure 6: DAgger Distillation. During sequence training, all executed actions update the fast weights, but the flow-matching loss is computed only on human corrections.

3.4 Implementation Details

Each of the 16 action-head layers receives a TTT layer with a two-layer fast MLP. Pretraining mixes tabletop bimanual robot trajectories and egocentric human video, gradually increasing context up to the target length. Runs use 30,000 pretraining steps on 16 GB200 GPUs, followed by 20,000 task-specific steps at 1K context. Appendix A gives which parameters are frozen and the changing batch sizes.

Original figure
Figure 7: Main evaluation: task completion scores on three assembly tasks. Scores are rubric-based, reported in percent; higher is better.

4 Experiments

Setup and main results. A bimanual YAM robot uses top, bottom and two wrist cameras. Pup Go Car, Circuit and Gear Bot provide 8, 6 and 5 hours of task data; typical episodes last 2, 1 and 5 minutes. Circuit trains on 20 configurations and tests on the remaining 60. Baselines are single-step GR00T, GR00T with one history frame and a matched-placement/parameter-count Gated DeltaNet (GDN) recurrent variant. Completion is rubric-based; evaluation uses 20 rollouts per task except ten for Gear Bot. RoboTTT averages 79% completion versus 42% for single-step GR00T and 56% for GDN. Its full successes are 9/20, 13/20 and 2/10, respectively, versus 3/20, 3/20 and 0/10 for GR00T. These small counts accompany, rather than equal, the completion-score improvements.

Context scaling. Figure 8 evaluates models before the additional DAgger training used for Pup Go Car in the main results. RoboTTT reaches 71.5% mean completion at 8K pretraining context versus 43.9% at 1K—about a 63% relative gain. The abstract says 62%; the body’s 63% agrees with these rounded values. The best short-context reference is 45.6%, so the caption’s claim of surpassing it “from 1K onward” is inconsistent with the reported 1K value. GDN does not show a similar scaling trend here. Longer context also changes training work and, above 4K, batch size; this is not a fixed-total-compute scaling experiment. Proposed explanations about memory expressiveness are hypotheses, not proof that only gradient-updated states can use history.

Video conditioning and perturbations. Circuit uses the same generic prompt across configurations, with the target supplied only by a human video. Training adds 5–20 videos per training configuration. On ten held-out-configuration trials, RoboTTT obtains 65% completion and 6/10 full successes, versus 33% and 0/10 for GDN. Perturbation experiments include 30 minutes of relevant training data. After a human removes an assembled roof or tire, RoboTTT reinstalls it in 15/20 and 18/20 trials; GDN obtains 13/20 and 18/20. Thus tire recovery is tied, and these are trained perturbations, not an unqualified zero-shot robustness result.

Corrective data and ablations. All methods share 100 intervention trajectories, half collected using RoboTTT and half using GR00T. DAgger Distillation improves RoboTTT by a reported 36% and GDN by 29%, whereas simply imitating all executed actions does not improve GR00T over corrections-only training. Failure actions are useful as context in this experiment. Removing independent chunk noise severely degrades behavior; a linear fast model performs 27% worse than the nonlinear version. Adding action tokens and then register tokens gives reported relative gains of 23% and 18%. Register tokens alone do not help GR00T, supporting their interaction with temporal modeling in this setup.

Method Pup Go Car Circuit Gear Bot
RoboTTT 9 / 20 13 / 20 2 / 10
GR00T N1.7 3 / 20 3 / 20 0 / 10
GR00T N1.7 Hist. 0 / 20 8 / 20 0 / 10
GDN 3 / 20 8 / 20 0 / 10
Table 1: Main evaluation: fully successful trials on three assembly tasks, out of 20 per task (10 for Gear Bot). RoboTTT is the only method with full successes on Gear Bot.
Original figure
Figure 8: Closed-loop performance scales with pretraining context length. Average task completion score across the three tasks as pretraining context length grows from 128 to 8K timesteps. RoboTTT improves steadily, surpassing the best short-context baseline from 1K onward with no sign of saturation; GDN does not benefit from longer context. Single-Step Context and Short Context denote GR00T N1.7 and GR00T N1.7 Hist.. All evaluations in this figure predate the DAgger training used for Pup Go Car in the main results.
Original S4.F9 artwork, restored from the PDF with all panels and labels.
Figure 9: One-shot imitation from an in-context human video. Continuous footage: a human demonstrates an unseen configuration (frames 1–3), the scene is reset (frame 4), and RoboTTT reproduces the assembly (row 2). The prompt is identical across configurations, so the target is identifiable only from the video.
Task Completion Score Number of Successful Rollouts
RoboTTT 65% 6 / 10
GDN 33% 0 / 10
Table 2: One-shot imitation on Circuit. Task completion score and fully successful trials, conditioning on one in-context human video of an unseen configuration.
Method Roof Perturbation Tire Perturbation
RoboTTT 15 / 20 18 / 20
GR00T N1.7 10 / 20 11 / 20
GR00T N1.7 Hist. 3 / 20 5 / 20
GDN 13 / 20 18 / 20
Table 3: Robustness to external perturbations. Each entry reports successful recoveries, i.e., where the policy reassembled the removed part, out of 20 per condition.
Original figure
Figure 10: DAgger Distillation results on Pup Go Car. Task completion after fine-tuning on a pool of 100 DAgger trajectories (50 collected with RoboTTT, 50 with GR00T N1.7). DAgger Distillation applies to the sequence models RoboTTT and GDN.
Original S4.F11 artwork, restored from the PDF with all panels and labels.
Figure 11: On-the-fly recovery learned through DAgger Distillation. While tightening the roof screw on Pup Go Car, RoboTTT misses the screw (frames 1–3), raises the arm and re-attempts, getting closer but missing again (frames 4–5), then re-adjusts once more and succeeds (frames 6–8).
Original figure
Figure 12: Ablation results on the Pup Go Car task. We ablate sequence action forcing and the fast-model architecture (MLP vs. linear), and trace each component added during development (state tokens, action tokens, register tokens). GR00T N1.7 with register tokens is included for a matched-token comparison.

5 Related Work

Prior policies use short visual windows, explicit semantic memory or autoregressive rollout history. Full attention retains detailed history but grows its decoding work with context; fixed-state recurrent models compress that history. RoboTTT makes the recurrent state a learned fast network rather than a fixed vector, extending TTT mechanisms developed in language, video and vision to visuomotor action generation. The demonstrated backbone is GR00T; broader architectural compatibility is a proposed possibility. The GDN comparison tests one matched recurrent alternative, and does not establish a general impossibility result for other recurrent update rules.

6 Limitations and Conclusion

The evidence supports long-context fast-weight policies on the three studied assembly tasks and the specified video/correction training protocols. Longer training contexts still increase cost, and the fast-weight objective is generic rather than designed specifically for robotics. The method does not recover from every deployment failure; the authors suggest improved training techniques and reinforcement learning. Fixed memory and truncated gradients enable long streams, but do not guarantee retention, causal use of every past observation or reliable adaptation outside the evaluation settings.

Acknowledgments

The acknowledgments credit research discussions, compute support, real-robot operation, data collection/evaluation and release support from collaborators at NVIDIA and Stanford.

Appendix A Model Architecture, Training, and Deployment Details

The appendix specifies the model’s parameter additions, pretraining versus task tuning, optimization schedules and real-robot deployment hardware. These details constrain claims about scaling cost and transfer.

A.1 Model Architecture

Eagle supplies the vision-language backbone. The action head grows from 538M to 690M parameters after adding 16 TTT layers, each roughly 10M parameters. Each fast model is a two-layer MLP with GeLU activation, updated by ordinary gradient descent. The inner learning rate is learned on top of base rate 0.1; rotary embeddings use base 10,000. More elaborate inner optimizers are proposed future work.

A.2 Training

Only newly added sequence layers are trained during pretraining; the GR00T components are frozen. Task-specific post-training tunes all parameters. Pretraining uses 16 GPUs, batch four per device (64 global) for context up to 4K, then one per device (16 global) above 4K. Task tuning uses eight GPUs, batch one per device and 1K context. AdamW weight decay is 10−5; pretraining uses a warmup/stable/decay schedule peaking at 2×10−5, and post-training cosine decay peaking at 5×10−5. The 30K/20K step counts do not imply equal total tokens or computation across context lengths.

Original figure
Figure A.1: Trajectory length distribution of the pretraining mixture.

A.3 Deployment

Deployment uses four RealSense D405 cameras at 480p color resolution, an RTX 5090 workstation and 30 Hz control on the bimanual tabletop platform. This is the reported operating setup, not a cross-hardware throughput comparison.

Appendix B Task Definition and Experiment Details

Task completion scores are normalized partial-progress rubrics. Full-success counts are reported separately, and initial object placements are reproduced across methods.

B.1 Task Definition

Pup Go Car: assemble and tighten the roof, hand over the drill, flip the chassis, install/tighten one wheel and return the drill. Its cumulative rubric progresses through 0.05, 0.10, 0.25, 0.30, 0.35, 0.45, 0.50, 0.55, 0.60, 0.75, 0.80, 0.85, 0.90 and 1.00 at the corresponding pick/place/align/tighten/handoff milestones. At most two wheel-assembly attempts are allowed.

Gear Bot: install one gear and two wheels per side, flip the chassis twice, attach the head and drive it by remote. Each flip, each of the six gear/wheel installations, head attachment and successful remote operation contributes 0.1.

Circuit: install two or three ordered components from roughly 80 configurations, then activate a switch/button when present. Two-part tasks without activation give 0.5 per installation; three required outcomes use the printed 0.33 each; three installations plus activation give 0.25 each. Wrong assembly order receives no partial credit. The printed 0.33 values are rounded rubric weights, while the paper describes scores as normalized to one.

Original A2.F2 artwork, restored from the PDF with all panels and labels.
Figure A.2: The “Pup Go Car” task. The robot assembles the roof and first wheel of a model car across multiple stages, including screwing, drilling, bimanual handoffs, and a car flip.
Original A2.F3 artwork, restored from the PDF with all panels and labels.
Figure A.3: The “Gear Bot” task. The robot installs gears and wheels on both sides of the chassis, flips it twice, attaches the robot head, and drives it with a remote.
Original A2.F4 artwork, restored from the PDF with all panels and labels.
Figure A.4: The “Circuit” task. The robot assembles two or three circuit components on a board and powers the circuit on when a switch or button is present.

B.2 Experiment Details

GR00T uses the official implementation, with a history-input extension for its short-context variant. GDN substitutes Flash Linear Attention’s Gated DeltaNet layers while matching placement, gating and parameter count. Evaluation repeats recorded initial placements for fair comparisons: 20 rollouts each for ordinary Pup Go Car/Circuit, ten for Gear Bot, and ten for video-conditioned Circuit. These controlled comparisons leave broader placement and configuration generalization unmeasured.

Concept demonstration

Open image ↗