Literature
Display
← All papers

Gated Memory Policy: In-Context Memorization and Adaptation

Yihuai Gao · Jeff Jinyun Liu · Shuang Li · Shuran Song

· arXiv (preprint) ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

At a glance

Gated Memory Policy (GMP) lets a robot decide when historical observations and actions are useful. A calibrated binary gate activates cross-attention to cached history; diffusion noise on past actions reduces overfitting. The authors report a 30.1% average success-rate improvement over long-history baselines on MemMimic and 26.6% over vision-language-action baselines on MIKASA-Robo, while retaining competitive performance on largely memory-free RoboMimic tasks. Memory remains bounded by a finite buffer.

1 Introduction

Memory demands vary: ordinary pick-and-place may need only the current observation, returning an object can require remembering earlier events in the same trial, and adjusting a throw may require learning from previous attempts. Simply feeding longer histories to a policy can increase overfitting, distribution shift, and latency.

GMP separates when to recall from what to retrieve. It learns retrieval directly from visual and action histories and calibrates a gate on held-out demonstrations. MemMimic evaluates both within-trial recall and adaptation across repeated attempts, including simulation and physical robots.

Related references: [30], [26], [5], [9], [20], [14], [6], [43].

Refer to caption
Figure 1: Memory Requirements in robotics range from (a) Markovian tasks requiring no memory; (b) in-trial memory for in-context memorization within a single execution; to (c) cross-trial memory that summarizes information across multiple attempts for in-context adaptation1. Naively increasing policy history often degrades policy performance in Markovian tasks and is computationally expensive for cross-trial memory tasks. Our Gated Memory Policy (GMP) selectively recalls task-relevant history, achieving strong performance across tasks.

2 Related Work

Earlier robot memory systems use hand-designed traces, trajectories, textual plans, or retrieval databases. Longer-context neural policies can learn memory implicitly, but often add prediction tasks or retain irrelevant history. GMP uses a lightweight cross-attention module over visual and action tokens, with a separate gate to suppress memory when it hurts generalization.

Related references: [12], [6], [10], [49], [21], [29], [1], [30], [13], [5], [7], [3], [28], [19], [18], [15], [27], [47], [36], [25], [42], [39], [16], [4], [38], [43], [22], [23].

3 Method

3.1 Background: Transformer-based Diffusion Policy

The backbone is a Transformer diffusion policy: current images and proprioception condition denoising of a future action chunk. Training minimizes action reconstruction error; DDIM performs iterative denoising at deployment. Extending full self-attention to every historical token increases computation quadratically and can amplify distribution shift.

Related references: [7], [28], [37].

Equation 1
ℒaction=𝔼A0t:t+h,ϵ,k[∥At:t+h0−φθ(At:t+hk,It,Pt,k)∥22],\mathcal{L}_{\text{action}}=\mathbb{E}_{A^{0}_{t:t+h},\epsilon,k}\left[\|A^{0}_{t:t+h}-\varphi_{\theta}(A_{t:t+h}^{k},I_{t},P_{t},k)\|_{2}^{2}\right], (1)
Refer to caption
Figure 2: Gated Memory Policy Network (a) Based on Diffusion Transformer (DiT) [34, 28], we add a gated attention module to selectively recall memory. (b) The gated attention module features three key designs: (1) Binary memory gate μt\mu_{t} that determines whether history cross-attention is skipped or applied. (2) Noised history action condition to improve robustness and reduce overfitting. (3) Cached history tokens during inference time to reduce computational cost.

3.2 Gated Memory Policy

An MLP encodes historical action chunks. A pretrained ViT and multi-head attention pooling compress each sampled image into one token. Cached image and action tokens form a sliding history window queried by cross-attention; the history need not repeatedly attend to itself.

Historical actions receive diffusion noise during both training and inference. At each denoising step they are one step cleaner than the future actions, providing informative context without relying on unrealistically clean history.

A gate computed from the current image and proprioception either adds the history-attention output to the residual stream or skips it. Calibration first trains memory-on and memory-off policies, compares their action errors on validation trajectories, and labels states where memory substantially helps. The learned gate is then frozen while the final policy is trained on all demonstrations. End-to-end gate training instead tends to overuse history unless strongly regularized, which can suppress needed recall.

Related references: [11], [44], [46], [32], [4].

Refer to caption
Figure 3: Memory Gate Calibration. We split the dataset into 𝒟train\mathcal{D}_{\mathrm{train}} and 𝒟val\mathcal{D}_{\mathrm{val}}. 1) Train two policies on 𝒟train\mathcal{D}_{\mathrm{train}} with memory gate always off π{\color[rgb]{0.4,0.4,0.4}\pi} or always on πmem{\color[rgb]{0,0.4,1}\pi_{\mathrm{mem}}}. 2) Evaluate the two policies on 𝒟val\mathcal{D}_{\mathrm{val}} for NN rounds and calculate the error of the predicted actions at each timestep tt. 3) At timestep tt, if the error of the no-memory policy δt{\color[rgb]{0.4,0.4,0.4}\delta_{t}} is significantly larger than the error of the memory policy δtmem{\color[rgb]{0,0.4,1}\delta^{\mathrm{mem}}_{t}}, we label memory-required (μt=1\mu_{t}=1) for this timestep; otherwise, μt=0\mu_{t}=0. This stage is referred to as Calibration of the memory gate. 4) We freeze the memory gate weights and retrain the policy on the full dataset to obtain the final Gated Memory Policy πgated\pi_{\mathrm{gated}}.
Refer to caption
Figure 4: Task 1: Match Color (Sim). (a) The robot picks up a cube while observing four randomly colored bins. After lifting the cube, the bin colors are randomized; the robot must place the cube into the bin that matches the cube’s original color. (b) Baselines: Diffusion Policy (DP) [7] and Past-Token-Prediction (PTP) [43] are evaluated across no, medium, and long-history [nh, mh, lh] settings. Our method, GMP, uses the [lh] configuration. (c) Visualization: We visualize the gated cross-attention weights for each 8-step action chunk. At t=80t=80, when the robot is placing the cube back, the attention focuses primarily on t=48t=48, when the robot first observed the bin colors. Blue indicates the memory gate is on (μt=1\mu_{t}=1); gray indicates the gate is off (μt=0\mu_{t}=0).
Refer to caption
Figure 5: Task 2: Discrete Place Back (Sim). (a) The cube is randomly placed in one of the 4 bins. The robot must pick up the cube, hold it in the air for 2 seconds (creating a memory challenge), and return it to the original bin.

4 Experiments

MemMimic tests remembering object locations or colors and adapting pushing, flinging, and casting over successive trials. RoboMimic checks whether memory harms Markovian manipulation; MIKASA-Robo adds memory-intensive tasks and comparisons with vision-language-action policies. Controlled diffusion-policy, past-token-prediction, and recurrent baselines share the visual encoder.

Longer history alone hurts several Markovian tasks and does not reliably solve harder physical or cross-trial tasks. GMP attends to earlier informative views and to overshoot or undershoot outcomes, while often ignoring history: the gate is off for 73% of Match Color and 58% of Iterative Pushing steps. Attention visualizations support selective retrieval, but are not a separate causal test.

Validation-based gate calibration avoids the overuse/underuse tradeoff of joint gate training. Cross-attention scales linearly with history length, and skipping it reduces inference cost further. Noise ablations favor the one-step-cleaner history schedule over clean histories, arbitrary noise levels, or a train/test noise mismatch. Results and original benchmark plots follow.

Related references: [30], [5], [36], [19], [31], [35], [7], [43], [44], [2], [45], [4].

Refer to caption
Figure 6: Task 3: Flip and Place Back (Real). (a) After the robot picks up the cup and places it on the saucer, a human flips it over by 90 degrees. The robot must flip it back then return it to the original position. (b) We use an portable ARX X5 robot arm with an iPhone for visual observation. (c) We evaluate our policy on 6 challenging unseen environments and over 10 cups, showing robustness and generalizability across objects and scenes.
Refer to caption
Figure 7: Task 4: Iterative Pushing (Sim). (a) The robot pushes a cube (with unknown friction) into the red box over 6 trials per episode. The environment resets after each trial. By observing how far the cube moves, the policy learns the physical dynamics and adjusts the pushing velocity accordingly for subsequent trials. (c) Visualization of gated cross-attention weights for each 8-step action trajectory throughout one episode. Blue indicates the memory gate is on (μt=1\mu_{t}=1) and gray indicates the memory gate is off (μt=0\mu_{t}=0). During the pushing stage in Trial 4, attention focuses primarily on the outcomes from Trial 3 (undershoot) and Trial 2 (overshoot). The policy uses these past outcomes to adjust its action and achieve a successful push.
Refer to caption
Figure 8: Task 5: Iterative Flinging (Sim). (a) The robot flings a cloth (with unknown mass) so that the far edge lands on the black target. (b) Multi-attempt flinging. Flinging too slowly leaves the cloth not fully extended; flinging too fast causes the cloth to fold back on itself. The policy learns to adjust the flinging velocity to match the unknown mass of the cloth.
Refer to caption
Figure 9: Task 6: Iterative Casting (Real). (a) The robot casts the object (with unknown friction) so that it stops between the two lines. After the object stops, the robot moves back for the next trial. (b) Two object contact surfaces with different friction coefficients are used across episodes, requiring the policy to infer object dynamics and apply different casting velocities.
Refer to caption
Figure 10: Evaluation results on RoboMimic. We evaluate 3 tasks from the RoboMimic benchmarks [30]: Tool Hang, Square, and Transport. (ph, mh) indicate that the data were collected from proficient-human (ph) or multi-human (mh). While most long-history policies experience performance drops on these Markovian tasks, GMP maintains competitive performance by leveraging the gating mechanism.
Refer to caption
Figure 11: Evaluation results on MIKASA-Robo. We evaluate 5 tasks on the MIKASA-Robo benchmarks [5], out-performing prior work MemoryVLA [36] by 26.6% on average. The baseline performance are reported in [5] and [36].
Refer to caption
Figure 12: Calibration of Binary Memory Gate.
Refer to caption
Figure 13: Inference Time Comparison.
Refer to caption
Figure 14: Noise Injection Ablation.

5 Limitations and Future Work

The fixed attention window can discard information needed much later. Selective token replacement or higher-level memory could extend the horizon; these are future directions. Results concern the tested tasks and demonstration distributions, rather than unrestricted lifelong adaptation.

6 Conclusion

Selective recall, efficient cached retrieval, and noise-augmented action histories together improve memory-dependent manipulation without forcing memory into every control decision.

7 Acknowledgments

The authors acknowledge collaborators and funding from Stanford, the Toyota Research Institute, and other supporting organizations; full credits remain in the original paper.

Related references: [33], [17].

8 Supplementary Materials

8.1 How Long History Can GMP Recall?

In a delayed Match Color variant, the policy recalls a cue after waits of 5–600 seconds. With 6,000 image frames and 6,000 actions, the authors report 99.0% ± 1.0% success and 0.16-second inference for eight denoising steps on an RTX 5090. This experiment freezes the visual encoder and trains on cached features; it does not establish unbounded memory.

Related references: [43].

8.2 Overlapped Trajectory Training

Overlapping trajectory batches share most observations. Encoding each unique image once per batch and reusing its features across subsequences reduces training memory while still updating the visual encoder end to end. Features are recomputed after each optimization step.

Related references: [43], [46].

Equation
τs={(Is,As),…,(Is+H−1,As+H−1)}\tau_{s}=\{(I_{s},A_{s}),\dots,(I_{s+H-1},A_{s+H-1})\}

8.3 MemMimic Details

MemMimic distinguishes memory of a single execution from adaptation across attempts. Training usually uses SigLIP2-B/16 vision features and chunked absolute or relative pose actions; timing and observation conventions vary with each environment.

Match Color and Discrete Place Back require remembering a cube’s origin after a pause. Short-history policies can remain stuck in the pause or choose a random destination; sufficiently long history solves these simpler tasks. Continuous Place Back uses 262 real demonstrations and random action intervals to tolerate timing variation. Its evaluation covers 40 initial states.

Flip and Place Back uses 2,166 trajectories, 12 training cups, and more than 30 scenes. Deployment tests unseen settings and cups. The gate remains always on for this task because the ungated memory policy already works; surface-height errors and difficult geometry still cause failures.

Iterative Pushing uses 500 simulated episodes with hidden friction and six attempts. Demonstrations adjust velocity by binary search, and GMP uses a longer buffer spanning two trials. Iterative Flinging also uses 500 episodes with variable cloth mass; its success criterion is relaxed because the cloth simulator can be unstable. These adaptation behaviors are learned from demonstrated trial-and-error procedures.

Iterative Casting uses 80 real training episodes across two friction surfaces and three attempts per episode. Timing and camera latency are randomized during training. Twenty evaluation episodes test both surfaces. Baselines can collapse to an intermediate velocity or fail to maintain a corrected velocity across trials; GMP uses previous outcomes to adjust.

Related references: [24], [41], [8], [48], [33], [40].

Refer to caption
Figure 15: Task 3’: Continuous Place Back (Real). (a) The robot picks up the cup, places it on the saucer, picks it up again, and returns it to the original position. (b) Initial positions of the cup and saucer tested across the continuous workspace.
Refer to caption
Figure 16: Training objects and scenes for In-the-wild Flip and Place Back. We show the cups used for data collection in the real-world Flip and Place Back task and a subset of the environments used for data collection.
Refer to caption
Figure 17: Position Control for Different Casting Velocities. We present examples of waypoint-based position control to achieve different casting velocities. While starting and ending at the same position and running at the same frequency, the heuristic policy adjusts the waypoint distribution to modify the casting velocity. Before the robot decelerates, sparser waypoints lead to faster casting velocities, while denser waypoints lead to slower casting velocities.

8.4 RoboMimic Evaluation Details

RoboMimic evaluation covers Square, Transport, and Tool Hang. Policies train for 400 epochs; checkpoints every 20 epochs are each tested on 100 episodes, and the best success rate is reported.

Related references: [30].

8.5 MIKASA-Robo Evaluation Details

Five MIKASA-Robo tasks use 250 demonstrations per task from a state-based PPO oracle. Policies train for 500 epochs, and the best checkpoint and action-horizon configuration are reported. Converting noisy joint deltas to absolute joint actions improves supervision. The small training set particularly limits RememberColor9; these checkpoint-selection and data constraints matter when comparing results.

Related references: [5].

8.6 Baseline Details

Baselines vary both history and prediction objectives: no-history diffusion policy, 16-step and 120-step histories, past-token prediction, and a recurrent behavioral-cloning policy. GMP normally retrieves 120 actions and 15 sampled images, with longer buffers for iterative tasks. Real-world long-history baselines sometimes omit proprioception because noisy histories produce severe jitter.

Related references: [43], [30].

8.7 Visualization of the Memory Gate Calibration

Calibration averages repeated validation predictions, smooths action errors over 20 timesteps, and marks memory as useful when the no-memory error exceeds ten times the memory-policy error. Labels can activate before the visibly critical moment because the prediction target is an entire future action chunk.

Refer to caption
Figure 18: Example Statistics of Memory Gate Label Generation. We present the action prediction errors for both the no-memory policy (δt{\color[rgb]{0.4,0.4,0.4}\delta_{t}}) and memory policy (δtmem{\color[rgb]{0,0.4,1}\delta^{\mathrm{mem}}_{t}}) for one example episode per memory requirement category. The x-axis represents the timestep of each episode. We use a blue background to indicate when the memory gate label is set to 1, and no background when it is set to 0. We observe that in (a) Match Color, δt>θ​δtmem{\color[rgb]{0.4,0.4,0.4}\delta_{t}}>\theta{\color[rgb]{0,0.4,1}\delta^{\mathrm{mem}}_{t}} in the middle of the episode when the robot is placing the cube back; in (b) Iterative Pushing, δt>θ​δtmem{\color[rgb]{0.4,0.4,0.4}\delta_{t}}>\theta{\color[rgb]{0,0.4,1}\delta^{\mathrm{mem}}_{t}} in every trial before the robot pushes; while in (c) Square, the action prediction error is similar between the two policies, indicating that the memory gate should remain off throughout.

8.8 Why Not Use Continuous Gating

Continuous gates trained jointly with the policy also tend to saturate or require task-sensitive regularization. A separately calibrated continuous gate can perform similarly, but fractional gates prevent the clean computational skip available with a binary decision.

Refer to caption
Figure 19: Continuous Memory Gate Ablation. When training a continuous memory gate jointly with the policy, using no regularization, which encourages the gate to stay on, degrades performance on Markovian tasks, while using high regularization, which encourages the gate to be off, hurts performance on non-Markovian tasks.

8.9 Training Cost and Calibration Overhead

Calibration can use early checkpoints and only part of the validation data; full convergence of the two preliminary policies is unnecessary. The authors recommend testing an always-on memory policy first and adding calibration when overfitting is apparent. The preserved table reports the additional training cost.

Train π\pi Train πmem\pi_{\mathrm{mem}} Rollout Train Gate Train GMP
1.2h 6.7h 6.4h 1.3h 31.2h
Concept demonstration

Open image ↗