Overview
This edition follows the complete 22-page arXiv v1 paper, including the architecture, training and evaluation appendices. RoboTTT stores long robot histories in continually updated fast weights. Read its results as task-specific evidence: partial completion scores, full successes and recovery counts measure different outcomes. All 16 original figures, three tables, five equations and 83 references remain.
At a glance
RoboTTT extends a pretrained robot policy with Test-Time Training (TTT) layers whose fast weights summarize observation/action history. Independently noising action chunks and truncating training gradients make sequences up to 8K timesteps tractable. On three bimanual assembly tasks, the reported mean completion score is 79%, versus 42% for the single-step baseline. Separate experiments show human-video-conditioned imitation, recovery after perturbations and benefits from training with failures as context. Full Gear Bot assembly succeeds in 2/10 trials; the 79% score is not a full-task success rate.
1 Introduction
Long manipulation tasks may revisit visually similar states, need previously seen object information or recover from failed actions. Simply appending a few frames can miss these dependencies or introduce misleading correlations. RoboTTT makes a small learned network’s weights the recurrent memory: updates encode history and queries retrieve information. The outer training objective learns how this memory should adapt. The paper tests context lengths from 128 to 8K timesteps, rather than demonstrating unlimited retention or universal adaptation.
2 Preliminaries
In TTT, learned projections map each input to query, key and value vectors. An inner squared-error loss updates fast-model weights so keys predict values; the updated network then processes the query. The outer action objective learns the projections and initial fast weights. At deployment, slow parameters stay fixed while the fast weights change. A robot sequence contains images, proprioception and action chunks, and the policy conditions on current observations plus preceding history. Fixed-size fast weights replace storage and repeated attention over an ever-growing token cache.
Equations 1–2
| (1) |
| (2) |
3 Method: Test-Time-Training Robot Policies
The method adds temporal TTT layers to an existing action model, trains them on trajectories with independent action-chunk noise and truncated gradients, and uses selectively masked action losses to learn from videos and corrective interventions. These are distinct architectural, optimization and data-use choices.
3.1 Model Architecture
The implementation starts from GR00T N1.7, with a vision-language backbone and diffusion-transformer action head. At each timestep, self-attention processes proprioception, noised actions and 16 learned register tokens, while cross-attention reads current visual/language tokens. TTT then processes the state, action and register tokens across time. Visual/language tokens do not directly traverse TTT; the compact registers carry that information. Each action-head layer adds the TTT output through a learned tanh gate initialized at 0.001, allowing a small initial departure from the pretrained computation.
Equation 3
| (3) |
3.2 RoboTTT Sequence Training
An inner loop updates fast weights on a trajectory; the outer loop averages flow-matching action loss over timesteps and differentiates through those updates, including into the learned initialization. Sequence action forcing independently samples each chunk’s noise level, rather than using one noise level for the entire trajectory. The schedule uses a Beta(1.5, 1) draw and scale 0.999; equations retain the exact objective.
Truncated backpropagation carries fast weights between segments but detaches their gradients at boundaries. Context therefore persists beyond a segment while gradient credit assignment does not; activation memory depends on segment length, not the entire history. The initialization receives gradients through the first segment. Each deployment rollout resets to the learned initialization, then carries updated fast weights forward while generating action chunks by iterative denoising. Fixed-state recurrence makes per-step history processing independent of accumulated context length; the paper does not supply a latency-versus-context measurement series.
Equations 4–5
| (4) |
| (5) |
3.3 Effective Learning from Context
Masking a timestep’s action loss lets it contribute context without becoming an imitation target. For one-shot imitation training, a human video is followed by a robot trajectory of the same configuration: only the robot portion receives action supervision. The resulting policy can condition on a new configuration’s demonstration at test time.
DAgger Distillation applies this distinction to intervention trajectories. Both failed robot actions and human corrections update fast-weight history, but only human corrections receive the action loss. This teaches failure-to-correction mappings using the DAgger data, in the spirit of algorithm distillation. At test time, the learned policy supplies its own corrections without new human intervention. This behavior is trained from corrective data; it is not unrestricted reward-based learning or a guaranteed monotonic improvement process.
3.4 Implementation Details
Each of the 16 action-head layers receives a TTT layer with a two-layer fast MLP. Pretraining mixes tabletop bimanual robot trajectories and egocentric human video, gradually increasing context up to the target length. Runs use 30,000 pretraining steps on 16 GB200 GPUs, followed by 20,000 task-specific steps at 1K context. Appendix A gives which parameters are frozen and the changing batch sizes.
4 Experiments
Setup and main results. A bimanual YAM robot uses top, bottom and two wrist cameras. Pup Go Car, Circuit and Gear Bot provide 8, 6 and 5 hours of task data; typical episodes last 2, 1 and 5 minutes. Circuit trains on 20 configurations and tests on the remaining 60. Baselines are single-step GR00T, GR00T with one history frame and a matched-placement/parameter-count Gated DeltaNet (GDN) recurrent variant. Completion is rubric-based; evaluation uses 20 rollouts per task except ten for Gear Bot. RoboTTT averages 79% completion versus 42% for single-step GR00T and 56% for GDN. Its full successes are 9/20, 13/20 and 2/10, respectively, versus 3/20, 3/20 and 0/10 for GR00T. These small counts accompany, rather than equal, the completion-score improvements.
Context scaling. Figure 8 evaluates models before the additional DAgger training used for Pup Go Car in the main results. RoboTTT reaches 71.5% mean completion at 8K pretraining context versus 43.9% at 1K—about a 63% relative gain. The abstract says 62%; the body’s 63% agrees with these rounded values. The best short-context reference is 45.6%, so the caption’s claim of surpassing it “from 1K onward” is inconsistent with the reported 1K value. GDN does not show a similar scaling trend here. Longer context also changes training work and, above 4K, batch size; this is not a fixed-total-compute scaling experiment. Proposed explanations about memory expressiveness are hypotheses, not proof that only gradient-updated states can use history.
Video conditioning and perturbations. Circuit uses the same generic prompt across configurations, with the target supplied only by a human video. Training adds 5–20 videos per training configuration. On ten held-out-configuration trials, RoboTTT obtains 65% completion and 6/10 full successes, versus 33% and 0/10 for GDN. Perturbation experiments include 30 minutes of relevant training data. After a human removes an assembled roof or tire, RoboTTT reinstalls it in 15/20 and 18/20 trials; GDN obtains 13/20 and 18/20. Thus tire recovery is tied, and these are trained perturbations, not an unqualified zero-shot robustness result.
Corrective data and ablations. All methods share 100 intervention trajectories, half collected using RoboTTT and half using GR00T. DAgger Distillation improves RoboTTT by a reported 36% and GDN by 29%, whereas simply imitating all executed actions does not improve GR00T over corrections-only training. Failure actions are useful as context in this experiment. Removing independent chunk noise severely degrades behavior; a linear fast model performs 27% worse than the nonlinear version. Adding action tokens and then register tokens gives reported relative gains of 23% and 18%. Register tokens alone do not help GR00T, supporting their interaction with temporal modeling in this setup.
| Method | Pup Go Car | Circuit | Gear Bot |
|---|---|---|---|
| RoboTTT | 9 / 20 | 13 / 20 | 2 / 10 |
| GR00T N1.7 | 3 / 20 | 3 / 20 | 0 / 10 |
| GR00T N1.7 Hist. | 0 / 20 | 8 / 20 | 0 / 10 |
| GDN | 3 / 20 | 8 / 20 | 0 / 10 |
| Task Completion Score | Number of Successful Rollouts | |
|---|---|---|
| RoboTTT | 65% | 6 / 10 |
| GDN | 33% | 0 / 10 |
| Method | Roof Perturbation | Tire Perturbation |
|---|---|---|
| RoboTTT | 15 / 20 | 18 / 20 |
| GR00T N1.7 | 10 / 20 | 11 / 20 |
| GR00T N1.7 Hist. | 3 / 20 | 5 / 20 |
| GDN | 13 / 20 | 18 / 20 |
5 Related Work
Prior policies use short visual windows, explicit semantic memory or autoregressive rollout history. Full attention retains detailed history but grows its decoding work with context; fixed-state recurrent models compress that history. RoboTTT makes the recurrent state a learned fast network rather than a fixed vector, extending TTT mechanisms developed in language, video and vision to visuomotor action generation. The demonstrated backbone is GR00T; broader architectural compatibility is a proposed possibility. The GDN comparison tests one matched recurrent alternative, and does not establish a general impossibility result for other recurrent update rules.
6 Limitations and Conclusion
The evidence supports long-context fast-weight policies on the three studied assembly tasks and the specified video/correction training protocols. Longer training contexts still increase cost, and the fast-weight objective is generic rather than designed specifically for robotics. The method does not recover from every deployment failure; the authors suggest improved training techniques and reinforcement learning. Fixed memory and truncated gradients enable long streams, but do not guarantee retention, causal use of every past observation or reliable adaptation outside the evaluation settings.
Acknowledgments
The acknowledgments credit research discussions, compute support, real-robot operation, data collection/evaluation and release support from collaborators at NVIDIA and Stanford.
Appendix A Model Architecture, Training, and Deployment Details
The appendix specifies the model’s parameter additions, pretraining versus task tuning, optimization schedules and real-robot deployment hardware. These details constrain claims about scaling cost and transfer.
A.1 Model Architecture
Eagle supplies the vision-language backbone. The action head grows from 538M to 690M parameters after adding 16 TTT layers, each roughly 10M parameters. Each fast model is a two-layer MLP with GeLU activation, updated by ordinary gradient descent. The inner learning rate is learned on top of base rate 0.1; rotary embeddings use base 10,000. More elaborate inner optimizers are proposed future work.
A.2 Training
Only newly added sequence layers are trained during pretraining; the GR00T components are frozen. Task-specific post-training tunes all parameters. Pretraining uses 16 GPUs, batch four per device (64 global) for context up to 4K, then one per device (16 global) above 4K. Task tuning uses eight GPUs, batch one per device and 1K context. AdamW weight decay is 10−5; pretraining uses a warmup/stable/decay schedule peaking at 2×10−5, and post-training cosine decay peaking at 5×10−5. The 30K/20K step counts do not imply equal total tokens or computation across context lengths.
A.3 Deployment
Deployment uses four RealSense D405 cameras at 480p color resolution, an RTX 5090 workstation and 30 Hz control on the bimanual tabletop platform. This is the reported operating setup, not a cross-hardware throughput comparison.
Appendix B Task Definition and Experiment Details
Task completion scores are normalized partial-progress rubrics. Full-success counts are reported separately, and initial object placements are reproduced across methods.
B.1 Task Definition
Pup Go Car: assemble and tighten the roof, hand over the drill, flip the chassis, install/tighten one wheel and return the drill. Its cumulative rubric progresses through 0.05, 0.10, 0.25, 0.30, 0.35, 0.45, 0.50, 0.55, 0.60, 0.75, 0.80, 0.85, 0.90 and 1.00 at the corresponding pick/place/align/tighten/handoff milestones. At most two wheel-assembly attempts are allowed.
Gear Bot: install one gear and two wheels per side, flip the chassis twice, attach the head and drive it by remote. Each flip, each of the six gear/wheel installations, head attachment and successful remote operation contributes 0.1.
Circuit: install two or three ordered components from roughly 80 configurations, then activate a switch/button when present. Two-part tasks without activation give 0.5 per installation; three required outcomes use the printed 0.33 each; three installations plus activation give 0.25 each. Wrong assembly order receives no partial credit. The printed 0.33 values are rounded rubric weights, while the paper describes scores as normalized to one.
B.2 Experiment Details
GR00T uses the official implementation, with a history-input extension for its short-context variant. GDN substitutes Flash Linear Attention’s Gated DeltaNet layers while matching placement, gating and parameter count. Evaluation repeats recorded initial placements for fair comparisons: 20 rollouts each for ordinary Pup Go Car/Circuit, ten for Gear Bot, and ten for video-conditioned Circuit. These controlled comparisons leave broader placement and configuration generalization unmeasured.