Literature
Display
← All papers

Scaling Short-Term Memory of Visuomotor Policies for Long-Horizon Tasks

Rutav Shah · Rajat Kumar Jenamani · Xiaohan Zhang · Lingfeng Sun · Roberto Martín-Martín · Yuke Zhu · Deva Ramanan · Karl Schmeckpeper

Preprint posted · arXiv (preprint) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

PRISM adds input-conditioned attention gates and hierarchical token compression to a causal visuomotor policy. It uses recent sensory and action history to address partial observability while reducing the cost of long contexts. ReMemBench supplies eight household tasks covering four memory categories. The main matched baseline comparison uses four of those tasks: PRISM averages 41% success there, versus 36% for the strongest baseline, while its all-eight-task average is 32%. A physical wash-and-return task reaches 30% versus 15% for the past-token-prediction baseline. These gains leave substantial failure rates.

I Introduction

A current image cannot reveal where an occluded object was placed, which container an object came from, or how long a stove has been on. Simply extending a transformer’s context adds irrelevant inputs and expensive visual tokens. PRISM addresses these problems through feature-wise gating after attention and compression before global temporal attention. Its memory is a sliding window of recent interaction, not a persistent record spanning different days or episodes.

Refer to caption
Fig. 1: We propose ReMemBench, a benchmark for evaluating visuomotor policies with short-term memory (top), and PRISM, an architecture for visuomotor policies with gated attention and a hierarchical architecture, enabling robots to use short-term memory for partially observable long-horizon tasks (bottom).
Benchmark Domain Episode Length Short-term memory categories
Spatial Prospective Object-Associative Object-Set
ALFRED [20] Manipulation 1000 ✓ ✗ ✗ ✗
Habitat MultiON [21] Navigation 2500 ✓ ✗ ✗ ✗
FindingDory [22] Navigation + manipulation 3500 ✓ ✓ ✗ ✗
Memory Maze [17] Navigation 4000 ✓ ✗ ✗ ✗
Memory Gym [23] Synthetic gridworlds -1 ✓ ✗ ✓ ✗
POPGym [24] Synthetic games 1024 ✓ ✗ ✓ ✓
MemoryBench [18] Manipulation 12 ✓ ✗ ✓ ✗
Mikasa-Robo [25] Manipulation 180 ✓ ✓ ✗ ✗
RoboMME [26] Manipulation 1300 ✓ ✗ ✓ ✓
ReMemBench Navigation + manipulation 2500 ✓ ✓ ✓ ✓
TABLE I: Comparison of memory-focused embodied AI benchmarks. Rows highlighted in blue explicitly require memory by design (i.e., identical observations can correspond to different actions depending on history). Rows in orange involve memory implicitly but do not explicitly test for it. ReMemBench uniquely covers all short-term categories in visuomotor tasks.

II Related Work

Related work spans recurrent memory, attention architectures, robot policies, and benchmarks that require recall.

Short-Term Memory in Machine Learning

Recurrent networks retain history in a fixed hidden state; transformers expose stored tokens directly but increase context costs. Gating, sparse or hierarchical attention, and segment recurrence offer ways to control information flow and resource use. PRISM adapts gating and hierarchical attention to closed-loop imitation learning.

Short-Term Memory in Visuomotor Policies

Robot policies often condition on only recent frames. Alternatives store spatial states, select keyframes, compress trajectories, or add auxiliary reconstruction objectives. Past-Token Prediction [15] is an important comparison. PRISM changes the policy backbone and uses the standard behavior-cloning objective, without a separate memory-reconstruction loss.

Memory Benchmarks

Benchmarks differ in whether memory is explicitly necessary or merely helpful alongside perception and exploration. ReMemBench organizes longer navigation-and-manipulation episodes around spatial, prospective, object-associative, and object-set memory. Its teleoperated dataset targets imitation learning, while other suites emphasize shorter episodes or other learning settings. These differences complicate direct cross-benchmark claims.

III PRISM : A P olicy A r ch i tecture for S hort-Term M emory

PRISM combines a conventional history-conditioned policy formulation with two architectural changes: filtering retrieved features and compressing the tokens used for global memory.

Refer to caption
Fig. 2: PRISM applies gated attention to filter information retrieved from history and hierarchical summarization to scale attention over long interaction histories, improving causal transformer policies trained with behavior cloning by improving robustness to noisy histories and reducing computation.

III-A Formulation, Background, and Challenges

The policy learns from instructed expert trajectories under partial observability and predicts action chunks from recent observations and actions.

Problem Formulation

Each demonstration pairs a language instruction with observations and expert actions. The policy conditions on the available interaction history rather than treating the current observation as a fully observable state.

Equation
at∼πθ​(a∣o≤t,a<t,l).a_{t}\sim\pi_{\theta}\!\left(a\mid o_{\leq t},a_{<t},l\right).

Training Objective

Behavior cloning maximizes the likelihood of demonstrated actions. The action head predicts a chunk of future actions, preserving the original loss and policy equations below.

Equation
ℒ(θ)=−𝔼(τ,l)∼𝒟train∑t=1T⁡(τ)logpθ(at∣o≤t,a<t,l).\mathcal{L}(\theta)=-\mathop{\mathbb{E}}\limits_{(\tau,l)\sim\mathcal{D}_{\text{train}}}\sum_{t=1}^{T(\tau)}\log p_{\theta}\!\left(a_{t}\mid o_{\leq t},a_{<t},l\right).

Evaluation

At inference, the instruction is a static prefix and new observation/action tokens enter the context. Once the configured memory budget is reached, the oldest interaction steps are discarded. Evaluation varies object identities and placements within the specified tasks.

Model Architecture

Image, proprioceptive, past-action, and language encoders create causally ordered tokens with positional information. A transformer processes the truncated history, and an action head reads an observation hidden state to parameterize the next action chunk. The language encoder is pretrained.

Equations 1
Ht=(ok,ak,ok+1,ak+1,…,ot−1,at−1,ot),H_{t}=(o_{k},a_{k},o_{k+1},a_{k+1},\ldots,o_{t-1},a_{t-1},o_{t}),
1:kl,[pk]1:kp,[vk]1:kv,[ak]1:ka,…,[pt]1:kp,[vt]1:kv.\displaystyle{}_{1:k_{l}},[p_{k}]_{1:k_{p}},[v_{k}]_{1:k_{v}},[a_{k}]_{1:k_{a}},\ldots,[p_{t}]_{1:k_{p}},[v_{t}]_{1:k_{v}}. (1)

Challenges

Long contexts create two distinct difficulties: irrelevant history can correlate with actions in the training data, and storing/attending to many visual tokens increases compute and memory. The following components address information flow and resource use respectively.

Refer to caption
Fig. 3: Guided by the cognitive science literature, ReMemBench decomposes short-term memory into several functional categories, such as Spatial, Prospective, Object-Set, and Object-Associative.

III-B Selecting Relevant Features

Following gated attention [12], a sigmoid multilayer perceptron maps current input features to an element-wise gate on the attention output. Gating occurs after value aggregation, before the output projection and residual connection. It can suppress retrieved feature dimensions without removing old frames or directly selecting a subset of stored timesteps.

The ablations support useful filtering, although the architecture alone does not guarantee removal of spurious correlations. The feature-conditioned gate is more expressive than a single learned scalar per layer.

Equations
z=Softmax​(Q​K⊤d)​Vz=\text{Softmax}\!\left(\frac{QK^{\top}}{\sqrt{d}}\right)V
z~=g⁡(x)⊙z,\tilde{z}=g(x)\odot z,

III-C Reducing Memory and Compute Footprint

A local encoder summarizes the current sensory tokens, and global attention integrates these compact summaries over time. Global information is then broadcast back and combined with original local features through a residual skip, preserving detailed current inputs. The hierarchy reduces the length of global attention and the cached history. Its theoretical savings concern the global component; practical end-to-end gains also depend on local encoding, visual processing, and implementation overhead.

Equations
𝒪⁡((n​ktot)2)​, where ​ktot=kp+kv+ka\mathcal{O}\!\big((nk_{\text{tot}})^{2}\big)\text{, where }k_{\text{tot}}=k_{p}+k_{v}+k_{a}
O⁡(n​m​ktot+(n​ktot/m)2)O\big(nmk_{\text{tot}}+(nk_{\text{tot}}/m)^{2}\big)

Summary.

Together, gating controls the retrieved representation and hierarchy makes longer contexts affordable. Both are trained as part of the imitation policy without an auxiliary memory objective.

TABLE II: Success rates on ReMemBench task categories (20 trials × 3 seeds). The four tasks are Retrieve Fruit, Cook Meat, Wash and Return to Container, and Microwave Breadsticks (see section IV).
Group Method Spatial Prospective Obj-Assoc. Obj-Set Avg
ReMemBench LSTM [49] 0.140.14 0.060.06 0.150.15 0.110.11 0.12±0.040.12\pm 0.04
Mamba [29] 0.210.21 0.520.52 0.150.15 0.140.14 0.25±0.030.25\pm 0.03
(Four Tasks) TrXL [34] 0.060.06 0.280.28 0.120.12 0.120.12 0.15±0.030.15\pm 0.03
GTrXL [35] 0.230.23 0.380.38 0.230.23 0.210.21 0.26±0.030.26\pm 0.03
Linear-Attn. [50] 0.190.19 0.300.30 0.220.22 0.160.16 0.22±0.030.22\pm 0.03
PTP [15] 0.280.28 0.580.58 0.160.16 0.120.12 0.28±0.050.28\pm 0.05
SAM2Act++ [18] 0.280.28 0.080.08 0.280.28 0.240.24 0.22±0.030.22\pm 0.03
SMT [41] 0.41\pagecolor{CustomLightGreen}{0.41} 0.400.40 0.300.30 0.330.33 0.36±0.020.36\pm 0.02
PRISM 0.280.28 0.770.77 0.330.33 0.250.25 0.41±0.040.41\pm 0.04
ReMemBench (All) PRISM 0.390.39 0.430.43 0.220.22 0.250.25 0.32±0.080.32\pm 0.08
Original figure
TABLE III: Comparison with alternative mechanisms to the proposed feature-wise gating in PRISM.

IV ReMemBench: A Benchmark for Short-Term Memory in Visuomotor Policies

ReMemBench extends RoboCasa [16] with eight tasks and fifty teleoperated demonstrations per task. Spatial tasks retrieve a fruit or oil bottle whose location must be remembered. Prospective tasks turn a stove off after a delay or time the addition of a second ingredient. Associative tasks wash an object and return it to its original container or countertop position. Set-memory tasks move all breadsticks into a microwave or all bowls, excluding distractor plates, into a cabinet.

Object identities, placements, counts, or timing requirements vary across trials. The suite combines recall with difficult perception and control, so failure does not by itself isolate a defective memory mechanism.

V Experiments

The principal baseline table evaluates four representative tasks with twenty trials per task across three seeds. PRISM averages 0.41 ± 0.04, versus 0.36 ± 0.02 for the scene-memory transformer, 0.28 for past-token prediction, 0.25 for Mamba, and 0.12 for the long short-term memory model. PRISM’s strongest relative gain is prospective timing; the scene-memory transformer is better on the spatial and object-set tasks. The separate all-eight-task PRISM row is 0.32 ± 0.08 and has no corresponding all-eight-task baseline rows.

On a physical wash-and-return adaptation trained from one hundred demonstrations, success is 30% for PRISM, 15% for past-token prediction, and 0% without memory. The source does not specify the number of physical test trials. This is evidence from one task, with most attempts still failing.

On four RoboCasa atomic tasks, memory raises success from 0.32 to 0.43. On the five listed LIBERO suites, the average rises from 0.77 to 0.89. Published baseline numbers are taken from earlier works rather than a fully matched training comparison. The RoboCasa table labels the GR00T baseline N1-2B, whereas the abstract/introduction call it N1-3B; the reported 0.32 score is retained without resolving that naming discrepancy. Although no large robot-policy pretraining is used, PRISM does use pretrained visual and language encoders.

Increasing the memory window from one to 512 steps raises the plotted success from 0.16 to 0.42. Feature-wise gating obtains 0.42, versus 0.29 for blind gating, 0.26 without gating, and 0.13 with attention sinks. These ablation values should not be substituted for the main table’s separately reported 0.41 result.

At 512 steps, the reported peak-memory and compute-time ratios are 0.20 and 0.60 relative to the compared full-attention designs. The appendix measures parameter-normalized repeated unit blocks, so these are architecture microbenchmarks, not whole-policy robot latency or throughput measurements.

Refer to caption
Fig. 4: (Left) Successful real-world rollout of PRISM on Wash and Return to Container. (Right) Without memory, the policy cannot disambiguate ‘just picked up’ vs. ‘about to place.’
Method Success Rate
GR00T-N1-2B 0.32
Diffusion Policy 0.26
PRISM (no memory) 0.32
PRISM (n=256n=256) 0.43
Method LIBERO-90 LIBERO-10 Object Spatial Goal Avg.
OpenVLA 0.62 0.54 0.88 0.85 0.79 0.74
Diffusion Policy – 0.72 0.93 0.78 0.68 –
PRISM (no memory) 0.77 0.75 0.89 0.85 0.61 0.77
PRISM (n=256n=256) 0.85 0.81 0.93 0.92 0.95 0.89
TABLE IV: Success rates on RoboCasa (Left) and LIBERO (Right). PRISM with memory (n=256n=256) outperforms its no-memory variant and strong pretrained baselines finetuned on RoboCasa and LIBERO. Baseline numbers are adopted from prior works [6, 19, 61].
Refer to caption
Fig. 5: (Top) Mean predicted gating value across timesteps during decision-making. Higher values align with intuitive task moments, especially when turning the stove on and off, where memory recall is most important for disambiguation. This trend is consistent across successful “Cook Meat” trials. (Bottom) Qualitative visualization of the gating effect. With gating, the policy pays attention to the correct observations and the spatial region where the orange was previously observed from memory. Without gating, attention shifts to task-irrelevant regions, such as the sink.
Original figure
Original figure
Fig. 6: (Left) PRISM is both computationally and memory efficient at handling long contexts. (Right) PRISM demonstrates strong scalability as memory size increases on ReMemBench tasks.

VI Limitations and Conclusion

Success remains low on the full memory benchmark, and perception, control, and recall can all limit performance. The method stores a recent window within an episode; persistent long-term memory and retrieval across a robot’s lifetime remain future work. Larger video-based memory pretraining is also proposed rather than evaluated. The results support a useful architecture and challenging benchmark, while not establishing dominance on every task or memory setting.

VII Acknowledgements

The authors acknowledge feedback and resources from the Robotics and AI Institute and Texas Advanced Computing Center, along with federal research support. Two authors conducted much of the work during institute internships.

VIII Appendix

The appendices examine attention sharpness and dropping, gating controls, another benchmark, compression, complexity, and implementation.

IX Effect of Attention Scale Factor (Sharpness)

On the Retrieve Fruit task, making attention very flat reduces success to 0.22 at scale 0.125, compared with 0.64 at scales 1 and 2. Sharper settings vary non-monotonically, with 0.54 at scale 4 and 0.62 at 8. This single-task sweep supports sensitivity to weak selectivity; it is not a benchmark-wide average or a guarantee that increasing sharpness improves performance.

V=scale_factor⋅dheadV=\text{scale\_factor}\cdot\sqrt{d_{\text{head}}} 0.125 0.25 0.5 1.0 (PRISM) 2.0 4.0 8.0
Retrieve Fruit 0.22 0.54 0.54 0.64 0.64 0.54 0.62
TABLE V: Effect of attention sharpness on PRISM performance. We scale attention logits by Vdhead\frac{V}{\sqrt{d_{\text{head}}}} before softmax; PRISM uses the default setting 1.01.0.

X Effect of Attention Dropping

Dropping attention tokens with probability 0.1 raises average success from 0.45 to 0.49 with gating and from 0.26 to 0.39 without it. Gating remains helpful in both conditions. These appendix runs use a different reported baseline level from the main 0.41 table; the source does not fully reconcile the settings, so their values remain distinct.

Method Spatial Prospective Associative Object-Set Average
PRISM w/o Gating 0.08 0.35 0.30 0.30 0.26
PRISM w/o Gating (Attn Drop) 0.44 0.80 0.10 0.20 0.39
PRISM 0.40 0.80 0.25 0.35 0.45
PRISM (Attn Drop) 0.60 0.64 0.30 0.40 0.49
TABLE VI: Effect of noisy attention (token dropping with probability p=0.1p=0.1) on success rates across the four ReMemBench categories. While noise improves both gated and ungated models, PRISM with learned gating remains strongest overall.

XI Effect of Gating Module

A scalar gate and a feature-wise gate both include sigmoid nonlinearities and have similar parameter counts: 86 million versus 90 million. Feature-wise gating reaches 0.45 versus 0.28 for the scalar version. This narrows the explanation beyond simply adding any nonlinearity, though the capacity and architecture are not exactly identical.

Method Spatial Prospective Associative Object-Set Average
Blind Gating 0.35 0.30 0.25 0.20 0.28
Feature Gating (PRISM) 0.40 0.80 0.25 0.35 0.45
TABLE VII: Blind vs. feature-wise gating. Blind gating learns a single scalar gate per layer with similar non-linearity and parameter count, yet feature-wise gating in PRISM achieves substantially higher average success.

XII Evaluation on Mikasa-Robo

On Mikasa-Robo, PRISM improves some tasks but is comparable overall to the long short-term memory baselines. The authors attribute the difference from ReMemBench to shorter horizons, fewer distractors, and information concentrated in early frames. These are plausible explanations requiring further controlled study, rather than established causal findings.

Original figure
Fig. 7: Comparison of PRISM vs. LSTM on Mikasa-Robo.

XIII Analysis of the compression factor ( m )

Compression factors near 2, 9, 80, and 200 yield average success of 0.31, 0.40, 0.41, and 0.45. The experiments share a 384 H100 GPU-hour budget; the less-compressed models complete fewer gradient steps, so the comparison mixes representation and optimization-budget effects. At 512 steps, the most compressed configuration uses 13% less compute and 56% less peak memory than the least compressed in the reported unit-block measurements.

Compression (mm) Spatial Prospective Associative Object-Set Average
m≈2m\approx 2 0.30 0.40 0.28 0.24 0.31
m≈9m\approx 9 0.40 0.56 0.24 0.40 0.40
m≈80m\approx 80 0.40 0.60 0.24 0.40 0.41
m≈200m\approx 200 (PRISM) 0.40 0.80 0.25 0.35 0.45
TABLE VIII: Effect of compression factor mm on success rates across ReMemBench tasks. Larger mm corresponds to more aggressive compression.
Original figure
Fig. 8: Effect of compression factor mm on test-time efficiency. The plot reports peak GPU memory and computation time, both normalized with respect to the m≈200m\approx 200 (PRISM) configuration, across different sequence lengths.

XIV Memory and Compute Footprint

The complexity analysis separates local processing from attention over the compressed temporal sequence.

Training.

Training cost contains a local term linear in sequence length and a quadratic global term over the compressed tokens. Reducing the global sequence can save activations, but optimized attention implementations already reduce that memory cost, limiting realized savings relative to simple asymptotic estimates.

Equation
𝒪⁡(n​m​ktot)⏟local+𝒪⁡((n​ktot/m)2)⏟global.\underbrace{\mathcal{O}(n\,m\,k_{\text{tot}})}_{\text{local}}\;+\;\underbrace{\mathcal{O}\!\big((n\,k_{\text{tot}}/m)^{2}\big)}_{\text{global}}\!.

Evaluation.

At inference, compact summaries reduce the global key/value cache and the tokens attended to at each step by approximately the compression factor. This does not imply the same factor of improvement for the full policy, whose other components remain.

Baselines

The compute comparison runs one architecture block twenty times, records wall-clock duration and peak memory, and normalizes measurements by parameter count. Full global attention and a global-plus-sliding-window hybrid are the baselines. This protocol bounds interpretation of the efficiency plots.

XV Implementation Details

Implementation details specify the actual token compression, demonstration counts, and training budget.

PRISM Architecture Details

Two cameras supply 396 visual tokens, proprioception one, and the previous action one. The 397 sensory tokens compress to one summary, leaving two tokens per step and a factor of 199. The backbone follows a LLaMA-style transformer; the action head is a two-layer perceptron. Visual features come from frozen pretrained CrossMAE [63].

Data Collection

A human collects fifty space-mouse demonstrations per ReMemBench task and one hundred for the physical task. These are the imitation data used in the reported setups, separate from encoder pretraining.

TABLE IX: Training hyperparameters.
Config Value
Optimizer AdamW
Base learning rate 5×10−45\times 10^{-4}
Effective batch size 256
Weight decay 0.010.01
Warmup epochs 22
Total epochs 200200
Action prediction horizon 3232
Proprioception noise (std) 0.0050.005
Brightness augmentation Uniform⁡(−0.1, 0.1)\mathrm{Uniform}(-0.1,\,0.1)
Contrast augmentation Uniform⁡(0.8, 1.2)\mathrm{Uniform}(0.8,\,1.2)
Number of cameras 22

Training Details

Training uses about 384 H100 GPU-hours for the eight ReMemBench tasks and 192 for four RoboCasa atomic tasks. The retained table specifies AdamW with learning rate 0.0005, batch size 256, 200 epochs, two warmup epochs, and a 32-step action horizon, along with image and proprioceptive augmentation.

Concept demonstration

Open image ↗