Overview
T²Mem adds an observation-based fast-weight memory to π0.5 and trains the policy to use it for decisions that cannot be made from the current image alone. This edition retains all 12 original figures, four tables, 17 numbered equations and all five appendices. The three demonstration grids preserve all 54 original panels. The strongest evidence combines benchmark performance with interventions that remove writes, substitute conflicting memories and change when observations can be stored.
At a glance
Within a single vision-language-action policy, self-supervised online updates encode observation history into fixed-size fast weights. An interface extracts vision-language features and supplies retrieved history to the action expert, while outer action supervision teaches what information is useful. Alternating memory and policy optimization raises average success on 16 RoboMME tasks to 56.83%, versus the quoted memory-free baseline’s 17.93%. Controlled profiling reports faster inference than two explicit-memory configurations. These results concern the evaluated benchmark and workloads, without external reasoning calls or memory-content labels.
1 Introduction
Useful robot memory must preserve evidence needed for later actions, such as a hidden object’s identity or a demonstrated sequence. Retaining a long record does not ensure that a controller can use it. T²Mem learns an internal memory interface from action demonstrations, while observation-only self-supervision updates episode state online. Slow parameters remain fixed during deployment and fast state resets between episodes. The work targets memory-dependent decisions rather than merely longer context; it does not establish general temporal reasoning or unlimited retention.
2 Related Work
Related work contrasts explicit observation banks, recurrent summaries and test-time learning. The central design question is how historical evidence reaches the action policy in a form shaped by downstream supervision.
Memory in Robotics
Spatial maps emphasize geometry, while stored observations preserve richer evidence at a retention/retrieval cost. Methods such as SAM2Act+, MemoryVLA and recurrent VLA heads provide alternative representations. RoboMME evaluates whether these representations support decisions beyond the visible scene. Reconstructing an old image and retaining exactly the cue needed for an action are different objectives.
Test-Time Training
Test-time-training layers represent history as rapidly updated weights rather than a growing token buffer. Titans adds update/forgetting mechanisms, and RoboTTT applies fast weights to robot context scaling. T²Mem places the interface on vision-language features and shapes it through control supervision. The paper’s comparison with an action-expert-side integration tests this placement under its specified task-specific training protocol.
3 T²Mem
The memory pathway connects the vision-language backbone to the action expert, alongside the original current-observation connection. Demonstrated actions supervise slow learning; deployment writes use observations alone. Memory retrieval enriches action context without requiring an external planner or task-state annotations.
3.1 Preliminaries
The formulation separates tasks that require history from online learning used to encode that history. It also distinguishes cross-episode slow parameters from each episode’s temporary fast weights.
Memory-Dependent Tasks
A task is defined as memory-dependent when the best history-conditioned policy achieves strictly higher expected return than the best current-observation policy under the same task distribution and interaction budget. Both policy classes may depend on time, so the comparison is about access to earlier evidence, not merely the availability of a clock.
Equation 1
| (1) |
Test-Time Training
Learned projections produce queries, keys and values. A neural memory is trained online to reconstruct values from keys, and queried to obtain context for actions. Reads precede the current write; zero step size skips a write. Outer action loss learns the projections, update mechanism and initialization. At test time the slow parameters stay fixed, memory updates require no expert action labels, and each episode begins from the learned initialization rather than arbitrary zero weights.
Equations 2–3
| (2) |
| (3) | ||||
3.2 Architecture of T²Mem
The implementation extends π0.5 with an interface, fast-weight networks and gated fusion into the action expert. Slow parameters encode how to store and use evidence; fast state holds the current episode’s associations. Tasks share slow parameters but not fast state. The original current-scene pathway remains available. Appendix B specifies the freezing precisely: the visual encoder and base vision-language weights stay frozen, while low-rank adapters and the action expert are trained in designated phases.
3.3 Observation-Grounded Memory Interface
Sixteen learned interface tokens summarize valid vision-language features at each layer. Their attention excludes action, proprioceptive and other interface tokens, blocking a direct action-history shortcut through memory formation. Separate projections form observation-conditioned queries and key/value bindings. Reads are fused with current features through channel-wise gates initialized to contribute weakly, followed by residual/feed-forward processing. The enriched interface supplies context to subsequent action-attention layers, alongside normal current-observation and policy inputs. A write changes later memory reads; the current decision still has direct access to its current observation.
Equations 4–8
| (4) |
| (5) |
| (6) |
| (7) | ||||
| (8) |
3.4 Alternating Memory–Policy Learning
Stage 1 adapts the policy without memory. Stage 2A freezes policy parameters while action gradients train the interface, projections, initialization, update strengths and fusion gates through the action expert. Stage 2B freezes those memory slow parameters while adapting policy parameters to use the resulting history. Fast states continue reading and writing in both phases. Repeated alternation gives each trainable component a temporarily fixed counterpart, reducing the moving-target problem observed with joint optimization. Both phases use the same expert-action objective, without memory-content labels.
3.5 Adaptive Self-Supervised Memory
A nonlinear associative memory writes observation-derived bindings. Adaptive scale β lies between 0.1 and 1 and depends on reconstruction residual and alignment of the descent direction with displacement from the learned initialization. It attenuates repeated aligned updates, while permitting stronger writes for less aligned, poorly reconstructed evidence. A separate curvature-calibrated, learned step size determines update magnitude. The β=1 control disables adaptive scaling but still uses that step-size rule. Fixed-size state avoids an ever-growing history buffer, without guaranteeing that all earlier evidence survives.
Equation 9
| (9) |
4 Experiments
Experiments test benchmark success, architectural/training choices, causal dependence on online history and inference cost. Main multi-task results and task-specific diagnostic policies use different protocols and should not be conflated.
4.1 Experimental Setup
Training covers all 16 RoboMME tasks and reportedly uses 32 H200 GPUs for 48 hours. Evaluation spans Counting, Permanence, Reference and Imitation, with 50 test episodes per task and three evaluation seeds. The policy predicts 64 actions and executes 32 before replanning. Table 1 quotes comparison scores from the original benchmark, with a separate Stage-1 row for this work’s own memory-free adaptation. Symbolic, perceptual and recurrent methods have different information access and training annotations; human and simulator-oracle results are reference points, not equivalent deployable policies.
4.2 Performance on Memory-Dependent Tasks
T²Mem averages 56.83% success, a 38.90-point gain over the quoted π0.5 baseline and a 40.45-point gain over its own 16.38% Stage-1 row. It exceeds the reported recurrent baselines, whose best average is 22.35%, and leads non-privileged methods on VideoRepick, InsertPeg and RouteStick. Direct recall and counting are stronger than swap tracking: VideoUnmask is 88%, but VideoUnmaskSwap and ButtonUnmaskSwap are 29.33% and 20%. The authors hypothesize that temporal reasoning and feature correspondence limit these tasks; memory retention alone does not establish that explanation. InsertPeg reaches 36%, with qualitative failures during final insertion even after approaching the correct target, separating memory-dependent choice from motor precision.
4.3 Ablation Studies
Single-task architecture ablations yield 84%/71% on VideoUnmask/MoveCube for the observation-grounded interface, versus 30%/26% for the tested action-expert-side memory. Joint rather than alternating training gives 30%/28%, compared with memory-free 23%/28%. These controlled integrations and task-specific checkpoints do not substitute for a complete independent evaluation of every cited memory system. In a separate 20-episode VideoUnmask Hard retention probe, adaptive writing reduces early-binding reconstruction error by 18.8–46.9% after replaying later observations. Every episode favors adaptation at measured nonzero offsets. This probes interference in key/value reconstruction, not a direct task-success effect of adaptive writing alone.
4.4 Contribution of Online Memory
Suppressing all episode writes while retaining reads and learned initialization lowers success by 82.67 points on VideoUnmask, 64.67 on SwingXtimes and 41.33 on MoveCube. In 50 generated counterfactual episode pairs, both directions and three seeds produce 300 evaluations per condition: correct memory succeeds 300/300, conflicting memory 1/300 and empty memory 101/300. The target instruction, environment and diffusion seed are fixed, supporting content-specific use of history. A separate 66-frame prefix experiment permits 16 total writes or eight writes before/after occlusion: pre-occlusion and full-prefix conditions both reach 76.67%, versus 20.67% afterward. The truncated-prefix protocol differs from unrestricted writing, especially on longer Hard demonstrations; its percentages are not interchangeable with the first intervention.
4.5 Inference Efficiency
On batch-one RTX A5000 profiling, T²Mem takes 146.6 ms per measured inference versus 440.0 ms for FrameSamp+Modul and 10,135.6 ms for MemER with 64 generated tokens: approximately 3.0× and 69.2× differences. Memory-related stages alone take 7.7, 148.9 and 9,890.9 ms. Measurements synchronize devices after warm-up and exclude initialization, compilation, communication, simulation and action execution. The ratios compare specified model/workload configurations and do not establish whole-robot end-to-end speedups or matched policy quality.
5 Conclusion
Fast-weight memory, an observation-grounded interface and alternating action-supervised training provide an internal history mechanism for the evaluated policy. The interventions support reliance on stored evidence rather than a fixed learned shortcut. Remaining issues include evolving object associations, temporal reasoning, information extraction and execution precision. Broader environments, physical deployment and longer-horizon retention are not demonstrated by these benchmark results.
Appendix A Architecture and Adaptive Memory Updates
The appendix specifies the interface, memory network, parameter footprint and adaptive update rule. These details matter because fast-state allocation, effective layer use and learned step scaling are distinct quantities.
Interface and layer correspondence.
The vision-language backbone and action expert each have 18 layers. Sixteen interface tokens propagate through layers without reinitialization. Memory is allocated at all 18 layers, but only 17 modules affect subsequent action computation because an interface update is consumed after its own layer. Fast state persists between observations within one episode, while the initial state is shared through learned slow parameters.
| Component | Reference configuration |
| VLM / action expert | 18 layers each; widths 2,048 / 1,024 |
| Interface | 16 tokens of width 1,024, initialized from ; rank-16 attention adapters |
| Fast-weight memory | 16 heads; per-head MLP , exact GeLU, biases |
| Learned initialization | : matrix standard deviation 0.02, zero biases |
| Read/write projections | Separate biased Q/K/V projections |
| Normalization / position | Q/K/V RMS normalization (); Q/K interleaved RoPE, base |
| Fusion / state precision | Channel-wise initialized to 0.01; FP32 fast state |
Attention and parameterization.
Vision-language tokens cannot attend to the policy suffix; interface queries see only valid vision-language keys. Action/proprioceptive queries retain normal suffix attention, including interface access. The approximately 67.90 million added parameters comprise 56.678 million projections, 9.529 million fast-state initialization values, 1.622 million interface adapters and about 0.069 million remaining parameters. Each episode allocates 9,529,344 FP32 fast-state scalars, or 36.35 MiB, excluding attention caches and activations. Table 2 retains the 16-head, 64→256→64 network, normalization, positional encoding and gate settings.
A.1 Adaptive Write Rule
Each head’s association loss averages squared key-to-value reconstruction error. The update scale combines normalized residual with cosine alignment between negative gradient and displacement from initialization. Clipped alignment defines novelty, and a smooth cubic maps the resulting score to β between 0.1 and 1. Near-zero descent directions set novelty to zero; otherwise near-zero displacement sets it to one. The exact thresholds and equations are retained. This is an adaptive write-strength heuristic, not an explicit stored-history retrieval index.
Equations 10–12
| (10) |
| (11) |
| (12) |
Curvature calibration.
Curvature calibration estimates a candidate step from gradient norm and a Jacobian-vector product, evaluates directional curvature at that step, and caps the calibrated length at 1,000. The final step is half that length times a learned positive softplus multiplier; the cap does not bound the final learned product. β and curvature calibration are stop-gradient quantities, but gradients remain through the fast-state recurrence, loss gradient, projections and learned multiplier. There is no additional inner gradient clipping. Fixing β=1 therefore does not make the step size constant or remove all higher-order training paths.
Equations 13–15
| (13) | ||||
| (14) | ||||
| (15) |
Appendix B Training Algorithm and Hyperparameters
Training uses an inner observation-reconstruction objective to update episode memory and an outer masked flow-matching action objective to learn slow parameters. Figure 7 preserves the complete training/inference order, including writes during demonstrations without action supervision.
Shared action objective.
At valid execution frames, a noisy action chunk interpolates expert actions and Gaussian noise; the policy predicts their flow velocity under a mask of valid future actions and dimensions. Each loss is normalized by its valid elements. Memory state is built chronologically from preceding observations, so outer gradients can train current retrieval and earlier writes through the retained sequence. Demonstration-prefix observations may update memory but do not carry an execution-action loss.
Equation 16
| (16) |
Stage-specific parameter groups.
Stage 1 updates vision-language low-rank adapters and the complete action expert, including action/time projections. Stage 2A trains the memory interface and its adapters, projections, initialization, step multipliers and fusion gates with policy parameters fixed. Stage 2B updates vision-language adapters, action expert, action/time and proprioceptive projections with memory slow parameters fixed. Changing vision-language adapters can change inputs to the fixed memory mechanism. Base vision-language weights and the visual encoder remain frozen throughout; fast-state updates continue whenever scheduled.
Reference recipe.
The released multi-task recipe initializes memory from Stage-1 step 20,000, then alternates 500 memory and 500 policy updates per cycle. Learning rates are 0.00002 for Stage 1, 0.000075 for memory, 0.0005 for gates and 0.00003 for the policy phase. Effective batches are 64 individual frames in Stage 1 and 32 sequences in either memory-enabled phase. The original table preserves each initialization and active group; the source does not specify a total number of alternating cycles here.
| Stage 1 | Stage 2A | Stage 2B | |
| Purpose | Memory-free adaptation | Memory learning | Memory-conditioned policy learning |
| Initialization | Pretrained | Stage-1 step 20,000; new memory | Preceding memory phase |
| Updated group | VLM LoRA, complete AE, action input/output and time projections | Interface, memory projections and initialization, step multipliers, gates | VLM LoRA, AE, action/time and proprioceptive projections |
| Budget | step 20,000 | 500 updates per cycle | 500 updates per cycle |
| Learning rate | ; gate | ||
| Effective batch | 64 frames (8 accumulation 8) | 32 sequences | 32 sequences |
Training: one alternating-phase update
-
Select the active slow-parameter group and a task-homogeneous batch.
-
Initialize independent fast states and loss .
-
Traverse observations chronologically. Read memory and accumulate masked action loss at valid execution frames.
-
At scheduled writes, update memory from the observation using Eq. 12. Demonstration frames may write but have no action loss.
-
Backpropagate through the sequence; update only the active slow group and its optimizer state.
Inference: one episode
-
Reset ; process any demonstration prefix chronologically.
-
At replanning, hold fixed throughout action denoising. Predict 64 actions using repeated memory reads.
-
Commit one observation-conditioned write if scheduled. Do not write once per denoising iteration.
-
Execute 32 actions, process the midpoint observation for its stride-16 write, and replan at the next chunk boundary.
Time units.
Prediction horizon 64, execution horizon 32 and write stride 16 are physical environment steps. At replanning, fast state is held fixed through all denoising iterations, with repeated reads, and one observation-conditioned write is committed if scheduled. A midpoint execution observation can supply the stride-16 write without replanning. Writes do not occur once per denoising iteration or at every intermediate video frame. Diagnostic stride-four schedules use their own explicitly indexed demonstration timeline.
Appendix C Evaluation Protocols and Baseline Configurations
Main testing uses 26 easy, 12 medium and 12 hard episodes for each task, repeated over three evaluation seeds. Scores pool 150 rollouts per task and macro-average the 16 task success rates. Only the terminal success predicate counts, so partial motor progress is not credited. These are evaluation seeds, not reported independent training-seed replicates.
Symbolic memory.
SimpleSG supplies language subgoals; GroundSG additionally supplies front-view target coordinates. These augment π0.5 instructions using prompted Gemini, a subgoal-annotated Qwen model, or privileged simulator state. Learned symbolic predictors use the current image and previous subgoals rather than a retained visual history. Their supervision and oracle access differ from T²Mem’s action-only outer supervision.
Perceptual and recurrent memory.
TokenDrop retains patches selected by image differences; FrameSamp samples historical frames. TTT compresses history into updated weights and RMT into recurrent tokens. Context appends memory to the vision-language input, Modul uses it for adaptive normalization of the action expert, and Expert adds a memory transformer attended to by that expert. Table 1 retains all combinations instead of merging their scores.
Additional reference methods.
The past-action π0.5 variant appends action history to language tokens. SAM2Act+ uses a segmentation-model memory bank and keyframe waypoints executed through a simulator motion planner. MemER retains images and generates grounded subgoals; the benchmark adaptation fine-tunes Qwen with subgoal and keyframe annotations. It has accumulated visual evidence beyond the current-image/subgoal-history inputs of the simpler symbolic baselines.
Appendix D Memory Intervention and Ablation Details
The diagnostic appendices specify fixed task checkpoints, matched sampling seeds, write windows and the distinction between memory reconstruction and task success. The original image grids illustrate what evidence is available in each task.
Fixed diagnostic policies.
All three intervention policies use step-2,000 task-specific checkpoints and 64/32 prediction/execution horizons. VideoUnmask uses stride-four writes; SwingXtimes and MoveCube use stride 16. Write suppression preserves the learned initialization, reads and random-number/inference-call cadence, isolating the removal of episode updates. Suppression and timing tests use all 50 test episodes for each seed.
| Intervention | Protocol |
| Disable writing | Zero effective updates from reset, including demonstration and execution. Retain , reads, and normal inference-call/RNG cadence. |
| Replace content | Build memory from the target demonstration, a conflicting donor demonstration, or no demonstration. Keep the target environment, instruction, and sampling seed fixed across conditions. Empty means learned , not all-zero weights. |
| Restrict write times | On a 66-frame prefix, permit all writes, only , or only . The write-timing protocol restricts writes to a 66-frame prefix, which truncates longer Hard demonstrations; the full-prefix condition therefore differs from the unrestricted setting in panel (a). |
Content and timing controls.
The content test uses 50 generated pairs outside training—25 easy and 25 medium—with conflicting pre-occlusion targets and matched target conditions. Both pairing directions over three seeds give 300 evaluations per memory condition; no Hard cases are included. Empty memory means the learned initialization, not zero weights. For the timing test, eight inspected trajectories place the dominant occlusion transition at frame 32; pre- and post-occlusion windows each allow eight writes. Restricting to 66 frames truncates longer demonstrations, explaining why full-prefix success differs from unrestricted writing.
Adaptive-writing diagnostic.
Eight prefix observations at frames 2, 6, …, 30 create a common adaptive memory state. Both conditions clone it, then replay frames 34, 38, …, 330. The 100/200/300-frame offsets therefore correspond to 25/50/75 subsequent writes. Normalized reconstruction error on saved early key/value bindings is averaged over frames and allocated layers, then episodes. Only β differs; curvature calibration and cadence stay fixed. All 20 episodes show lower adaptive error at every measured nonzero offset.
Equation 17
| (17) |
Architecture and training controls.
The architecture comparison matches initialization, task data, alternating training, adaptive writes and total update count. Step 2,000 contains 1,000 updates for each slow group under 500/500 alternation. Joint training updates all trainable groups each step; it starts from Stage-1 step 20,000 with fresh memory. Its task-specific memory/gate/policy rates are 0.00006/0.0005/0.00003 with 25 warm-up steps, distinct from the main multi-task memory rate. Figures 8–10 retain complete 18-panel examples of occluded-color recall, copying a demonstrated movement method and counting repeated cycles; Figure 11 shows histories that require different answers despite matching post-occlusion observations.
Appendix E Inference Profiling
The profiling comparison measures steady-state model calls under controlled memory and generation budgets. It should be interpreted separately from full task runtime and benchmark success.
Setup.
Batch-one RTX A5000 measurements use three warm-up and 30 timed calls for T²Mem and FrameSamp; MemER uses three/20 for its high-level model and four/30 for its low-level policy. Initialization, compilation, remote calls, simulation and execution are excluded. FrameSamp uses 512 memory tokens from 32 images. MemER uses Qwen3-VL-4B-Instruct, rank-16 adapters, BF16 attention, eight recent plus eight retained images and 64 generated tokens. Figure 12 varies output budget: 16/64/128 tokens yield 3,312.8/10,135.6/19,701.5 ms per measured inference. The large speed ratio is therefore sensitive to external-planner generation workload.