Literature
Display
← All papers

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Marcel Torne · Karl Pertsch · Homer Walke · Kyle Vedder · Suraj Nair · brian ichter · Allen Z. Ren · Haohuan Wang · Jiaming Tang · Kyle Stachowicz · Karan Dhabalia · Michael Robert Equi · Quan Vuong · Jost Tobias Springenberg · Sergey Levine · Chelsea Finn · Danny Driess

Preprint posted · CoRL 2026 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

MEM combines compressed recent video with a running language summary so a robot can remember both detailed manipulation failures and long-horizon task progress.

[Uncaptioned image]
Fig. 1: Multi-Scale Embodied Memory (MEM) equips Vision Language Action Models (VLAs) with memory for solving long-horizon tasks at scale. It has two key components: an efficient video encoder for short-horizon image-based memory, and a language-based memory mechanism for capturing long-horizon memory. After training on a diverse corpus of robot and non-robot data, MEM VLAs can solve tasks that require up to fifteen minutes of memory, handle partial observability, and perform in-context adaptation of manipulation strategies.

At a glance

Multi-Scale Embodied Memory equips π0.6 with two complementary memories: a video encoder for recent images and robot states, and a learned language summary of earlier semantic events. Physical experiments include kitchen tasks lasting up to fifteen minutes, correction-conditioned grasping and door opening, and tests of counting, hidden objects, timing, and dexterous manipulation. Evidence concerns memory within an episode; continual learning across weeks or months is proposed future work.

I Introduction

A recent occluded grasp and an ingredient already collected require different kinds of memory. Dense video can retain the geometry of the former but is expensive over many minutes; language can compactly record the latter but loses fine detail. MEM joins both in a hierarchical robot policy. Its central result is that this combination improves the tested long tasks while preserving useful short-term context and the base policy’s dexterous manipulation performance.

II Related Work

Prior policies retain state histories, image tokens, keyframes, point tracks, latent memories, or language summaries. Each trades detail against inference cost. MEM combines modalities rather than requiring one representation to retain everything. Its video encoder draws on space–time separable attention [3]; its experimental comparisons implement representative pooling and state-history approaches on the same policy backbone.

Memory can also encourage copying past actions instead of responding to the current scene. The authors hypothesize that diverse pretraining reduces this causal-confusion problem, but the reported ablations do not isolate data diversity as its cause.

III Multi-Scale Embodied Memory for VLAs

The architecture divides memory by the kind of information required. Recent camera frames and proprioception support occlusion handling and manipulation adjustment. A recursively updated text summary tracks completed subtasks and other relevant events beyond the video window. This avoids passing an entire long episode of image tokens to the action policy.

III-A Multi-Scale Embodied Memory ( MEM )

A high-level policy reads the task, recent observations, and previous language summary, then predicts both the next subtask and an updated summary. A low-level policy generates a continuous action chunk from the goal, subtask, and recent observation window. The text summary substitutes for dense distant observations; it is a learned compressed state, not a lossless episode record. The retained factorization specifies the interface between these levels.

Refer to caption
Fig. 1: The MEM memory system equips VLAs like π0.6\pi_{0.6} with long-horizon memory via two key components: (1) a high-level policy is trained to keep track of long-horizon semantic events by updating a language memory mtm_{t} (left, Section III-B), (2) the low-level policy uses a short-horizon observation-based memory that is efficiently encoded via a video encoder (right, Section III-C).
Equation
π(at:t+H,\displaystyle\pi(a_{t:t+H}, lt+1,mt+1|ot−T:t,mt,g)\displaystyle\,l_{t+1},m_{t+1}|o_{t-T:t},m_{t},g)
≈\displaystyle\approx~ πLL(at:t+H|ot−K:t,lt+1,g)\displaystyle\pi_{\text{LL}}(a_{t:t+H}|o_{t-K:t},l_{t+1},g)
πHL(lt+1,mt+1|ot,mt,g),\displaystyle\pi_{\text{HL}}(l_{t+1},m_{t+1}|o_{t},m_{t},g),

III-B Language Memory for Long-Term Memory

Training summaries are produced by an off-the-shelf language model from annotated subtask sequences and success/failure indicators. Labels compress facts that remain relevant, such as the number and destination of bowls instead of every bowl’s appearance. The robot’s trained high-level policy generates its own updates at execution. Compression also reduces repeated-failure histories that differ from mostly successful training demonstrations, although discarded information and mistaken summaries can still limit recall.

III-C Video Encoder for Dense Short-Term Visual Memory

Every fourth layer of the image transformer adds causal temporal processing to spatial attention, reusing its query/key/value projections and adding fixed sinusoidal time encodings. Factorized attention costs O(Kn² + nK²) for K frames with n patches each, versus O(K²n²) for joint space–time attention. Only current-frame representations enter the language backbone, keeping its image-token count fixed as history grows.

The authors report no added learnable encoder parameters and initialize the current-frame positional offset to zero to preserve single-image behavior. Total video-encoding work still grows with history. The latency plot uses four camera streams on one H100 and compares against a 300 ms reference threshold; this is a specified hardware/model measurement, not a universal control-rate guarantee. Execution uses real-time action chunking [6] or its training-time variant [7].

Original figure
Fig. 2: Naively passing a sequence of observations into the backbone of a VLA rapidly increases inference latency. Our efficient video encoder architecture allows us to use many observation frames while remaining under critical real-time inference thresholds [6, 7]. Timings measured for the π0.6\pi_{0.6} VLA, with four input camera streams on one NVIDIA H100 GPU.
Refer to caption
Fig. 3: We propose to use an efficient video encoder architecture for compressing short-horizon, image-based memory. Our architecture expands standard ViTs for encoding video inputs by interleaving layers that apply bidirectional spatial attention within each observation (white arrows) with layers that additionally apply causal-temporal attention operations across observations (black arrows). We drop observation tokens for past timesteps in upper layers of the ViT to compress the inputs and reduce the number of tokens passed to the VLA backbone.

III-D Integrating MEM into the π 0.6 VLA

The implementation starts from a Gemma3-4B vision–language model and uses an 860-million-parameter flow action expert alongside discrete FAST action-token training. Action-expert gradients do not enter the vision–language backbone. Up to four 448-by-448 camera streams and linearly projected state-history tokens supply observations. Pretraining mixes demonstrations, rollouts, human corrections, vision–language tasks, and video–language tasks, using six observations at one-second spacing. Post-training expands memory to as many as eighteen frames, described as up to fifty-four seconds. The paper does not provide a complete quantitative accounting of this large training mixture.

IV Experimental Evaluation

Experiments ask whether memory supports long kitchen tasks, whether recent failure context helps manipulation adjustment, and how the video representation compares with other memory designs. The long-task study uses ten rollouts per policy and task or recipe, with mean and standard error. Many graphs show normalized task progress rather than full-episode success, so those outcomes should not be conflated.

IV-A MEM Solves Tasks Requiring Long-Horizon Memory

Recipe setup retrieves specified ingredients and cookware from named locations; full kitchen cleanup combines storage, wiping, and washing. Training covers forty-two recipes; evaluation uses five seen recipes in unseen kitchens and with unseen objects. Both short video memory and long language memory contribute to progress. Removing either lowers performance on these tasks.

A concatenated subtask-history ablation also performs worse than learned compressed summaries. Repeated failures create histories uncommon in successful demonstrations; recording only relevant completed events can reduce that mismatch. This is supported by the ablation and examples, rather than a guarantee that the summary always retains the information required for future recovery.

Refer to caption
Fig. 4: We test MEM policies across multiple challenging, long-horizon dexterous manipulation tasks that require retaining memory for up to fifteen minutes, including setting up a recipe, cleaning up a kitchen (Section IV-A), and making a grilled cheese sandwich (Section IV-C).
Original figure
Fig. 5: Performance of policies on challenging, long-horizon manipulation tasks. Without memory, even state-of-the-art generalist policies like π0.6\pi_{0.6} struggle to perform such tasks. We ablate the memory components and show these tasks are solvable by combining short-horizon, observation-based memory, with long-horizon, language-based memory. Naive memory of past language instructions, without compression, struggles with training-inference distribution shifts.

IV-B In-Context Adaptation of Manipulation Strategies

The policy is finetuned on targeted corrective demonstrations for chopstick grasps at unusual table height and fridge doors with ambiguous opening direction. Failed attempts remain in the memory model’s training context; a memoryless policy receives the same correction data. Remembering a failed strategy improves subsequent adjustment. This is adaptation through observations during execution, learned from intervention data beforehand, not online gradient learning or untrained discovery of new manipulation skills.

Refer to caption
Fig. 6: VLAs with memory can perform in-context adaptation of manipulation strategies, like adjusting grasp height or door opening direction. Without memory, policies get stuck with a suboptimal strategy.

IV-C Analysis Experiments

The comparison uses partial observability, counting, timing, wiping coverage, and dexterous tasks across several robot configurations. Pool Memory compresses separately encoded past images into an averaged memory token; Proprio Memory retains only projected state history. All use the π0.6 backbone, and this comparison omits the long-horizon language memory to focus on the observation representation. The pooling design follows ContextVLA [20]; the state-history design follows TA-VLA [53].

Video memory performs well across the tested memory capabilities. State history cannot directly remember which drawer a person used, and aggressive image pooling struggles with several longer visual histories. Introducing the video encoder only at task post-training is worse than using it during pretraining, even though both variants start from a base policy trained on the same broad robot corpus. Dexterous-task results remain broadly comparable to memoryless π0.6, with task-level variation. These controlled comparisons support the specific representations and training schedules tested, not every possible memory method.

Refer to caption
Fig. 7: Comparison of approaches for equipping VLAs with memory across core memory capabilities (handling partial observability, counting, visual memory). VLAs without memory struggle to perform these tasks. Only MEM’s memory approach performs well across all core capabilities.
Original figure
Fig. 8: Pre-training MEM on a diverse dataset of robot and non-robot video data makes it more effective at using its memory to solve diverse manipulation tasks. Introducing memory during post-training only results in substantially lower performance.
Refer to caption
Fig. 9: In addition to effectively using its memory, MEM also matches the performance of state-of-the-art non-memory VLAs across challenging manipulation tasks that do not require memory.

V Conclusion

Combining detailed recent observations with compressed semantic history supports the demonstrated long tasks and failure-conditioned adjustments. Current evidence stays within individual episodes and the evaluated task/data distribution. Longer-lived memory, continual deployment learning, reliable summary-error correction, and retention of details beyond the video window remain open extensions; the paper does not demonstrate memory across days or years.

Acknowledgements

The acknowledgments credit robot operators, annotators, operations and hardware teams, and assistance with the video encoder, illustrations, filming, and editing.

-A Contributions

The contribution statement assigns video-memory design, language-memory design, evaluation, pretraining/data work, long-context infrastructure, inference optimization, project advice, and writing to the listed authors. These roles cover both the memory architecture and the substantial robot-data and evaluation effort.

-B Task Details

The appendix specifies initial conditions and scoring for the long tasks, correction tests, and analysis suite. These rubrics distinguish partial-progress points from binary success and define what information must be remembered.

-B 1 Long-horizon Tasks ( Section IV-A )

Long tasks test recipe-item retrieval and multi-step kitchen cleaning. The four original example prompts are retained in full, including object locations and required destinations.

Preparing the items for a recipe

Recipe prompts name all required items, their initial locations, and their target surfaces. One point is awarded per correctly retrieved and placed item; most recipes contain six or seven points. Evaluation therefore measures item-level progress as well as task duration, using seen recipes in new scenes. The examples cover mashed potatoes, fried rice, and pizza.

Mashed potatoes Take out supplies for the recipe. Remove the lid from the pot on the stove and place the lid on the countertop to the left of the sink, and pick up the pot and place it in the sink. Get the potatoes, butter, milk from the fridge and place them on the countertop to the left of the sink. Get the masher from the top drawer to the left of the sink and place it on the countertop to the left of the stove. Fried rice Take out the supplies for the recipe. Get the bag of rice and the spam from the cabinet on the bottom right of the dishwasher and place them on the countertop to the left of the sink. Get the soy sauce from the fridge and place it on the same section of the countertop to the left of the sink. Get the pan from the bottom cabinet to the right of the dishwasher and put it on the stovetop. Get the spatula from the top drawer to the right of the dishwasher and place it on the pan on the stovetop. Pizza Take out the supplies for the recipe. Get the baking tray from the bottom cabinet on the left of the dishwasher and place it on the countertop to the left of the sink. Get the pizza dough, package of pepperoni, and bag of cheese from the fridge and place them on the countertop to the left of the sink. Get the dough roller from the top drawer to the right of the stove and place it on the countertop to the left of the sink.

Cleaning the kitchen

Cleanup includes wiping and drying counters, refrigerating food, putting away dishes, and washing sink contents. Each completed subtask earns one point, with about eight subtasks per episode. The example prompt explicitly names relevant objects, receptacles, and dishes.

Clean kitchen Clean up the kitchen. Wipe the countertop with the sponge and dry it with the paper towel and throw the towels in the trash can under the sink. Put the mustard in the fridge. Put the dishes in the bottom cabinet on the left of the fridge. Wash the blue plate and black plate and place them in the dishrack.

-B 2 In-context adaptation ( Section IV-B )

Both adjustment tests use corrective data and assess whether recent attempts help choose a better subsequent manipulation. Their success conditions are task-specific.

Chopstick Pick Up

Training places chopsticks at random locations with table heights in the upper half of the range and includes interventions after mis-grasps. Evaluation uses the lowest table setting. Picking up the chopstick and depositing it in the bin each earn one point; full success requires both.

Open Refrigerator

The fridge has no obvious visual cue for hinge direction. Training includes corrections of wrong-direction attempts. Evaluation counts success only if the door opens within four grasps, limiting the benefit of repeatedly trying random actions.

-B 3 Analysis Experiments ( Section IV-C )

The analysis tasks separately probe object-history recall, hidden scene state, counting, elapsed time, coverage, and dexterous control. Several are scored by partial completion, while others require the entire sequence to succeed.

Three-way swap mugs

Three mugs must visit the coffee machine one at a time without repeats or tipping. Each correct distinct mug earns one point, and previously used mugs return to random table locations, making current position insufficient to identify what remains.

Find object

A person hides an object in one of four drawers while the robot observes. Success requires opening the correct drawer, retrieving the object, and placing it on the table without opening any wrong drawer. Random drawer selection has a one-in-four chance.

Unpack groceries

A bag contains a variable number of items, invisible to the external camera. The policy must remember earlier wrist-camera views, retrieve every item, and signal completion by returning home. Success requires both complete unpacking and appropriate termination.

Scoop coffee

The robot removes the grinder lid, adds exactly two scoops of beans, and replaces the lid. The binary success condition checks the count and final lid placement, so a visually plausible final grinder state alone is insufficient.

Grilled cheese

The robot assembles, cooks, flips, and plates a sandwich. Scoring awards progress for these stages, including cooking between thirty seconds and three minutes per side. This broad accepted interval tests coarse timing, not precise temperature or cooking-quality control.

Window cleaning

The robot sprays cleaner, tears off a paper towel, wipes the entire booth window, and discards the towel. Success requires every stage and full coverage; unwiped regions count as failure.

Table bussing

Twelve tabletop objects are sorted into a bussing bin for tableware/utensils and a trash can for disposable containers and wrapping. Each correctly sorted item contributes one point.

Shirt folding

A flattened shirt must become a rectangular fold without protruding seams or excessive wrinkles. Success is judged on the final folded result.

Clean up counter

The robot opens a drawer, places every counter item inside, and closes it. All stages must finish for success.

Make bed

The robot straightens the blanket and places every pillow at the head of the bed. Both conditions are required.

Kitchen cleanup

This smaller cleanup task moves plates, bowls, and a cutting board from a counter into a sink. The stated success rubric requires all plates in the sink; it is distinct from the long-horizon washing/storage task.

Batch folding

Clothes start in a hamper. The robot retrieves, flattens, folds, and stacks shirts or shorts with existing folded clothes. Accepted folds are rectangular, without protruding seams or excessive wrinkling.

Box building

A flat cardboard cutout must be assembled into an intact box through coordinated two-arm manipulation. Full assembly without damage defines success.

-C Video encoder with Space-Time separable attention

The mathematical appendix adds sinusoidal time embeddings to patch features, sets the current-time embedding to zero, reuses normalized query/key/value projections, and expresses spatial/temporal attention before the usual value aggregation and feed-forward layer. All five displayed equation blocks, including the main policy factorization, are retained. The appendix’s set notation and attention composition are reproduced as written; the implementation narrative describes causal temporal attention interleaved with spatial attention. They should not be treated as a standalone proof of the stated initialization equivalence.

Equations 1–3
𝐳^p,tl−1=𝐳p,tl−1+e⁡(t),\hat{\mathbf{z}}^{l-1}_{p,t}=\mathbf{z}^{l-1}_{p,t}+e(t),
𝐪p,tl,a\displaystyle\mathbf{q}^{l,a}_{p,t} =WQl,a​LN​(𝐳^p,tl−1),\displaystyle=W^{l,a}_{Q}\text{LN}\big(\hat{\mathbf{z}}^{l-1}_{p,t}\big), (1)
𝐤p,tl,a\displaystyle\mathbf{k}^{l,a}_{p,t} =WKl,a​LN​(𝐳^p,tl−1),\displaystyle=W^{l,a}_{K}\text{LN}\big(\hat{\mathbf{z}}^{l-1}_{p,t}\big),
𝐯p,tl,a\displaystyle\mathbf{v}^{l,a}_{p,t} =WVl,a​LN​(𝐳^p,tl−1),\displaystyle=W^{l,a}_{V}\text{LN}\big(\hat{\mathbf{z}}^{l-1}_{p,t}\big),
αp,tl,a​(𝒮={1,…​N},𝒯={1,…,T})​[𝐳^]=SM​((𝐪p,tl,a)T⋅[𝐤0,0l,a​{𝐤p′,t′l,a}p′∈𝒮,t′∈𝒯]),\mathbf{\alpha}^{l,a}_{p,t}(\mathcal{S}=\{1,\dots N\},\mathcal{T}=\{1,\dots,T\})[\hat{\mathbf{z}}]=\text{SM}\Big((\mathbf{q}^{l,a}_{p,t})^{T}\cdot\big[\mathbf{k}^{l,a}_{0,0}\{\mathbf{k}^{l,a}_{p^{\prime},t^{\prime}}\}_{p^{\prime}\in\mathcal{S},t^{\prime}\in\mathcal{T}}\big]\Big), (2)
αp,tl,a​(𝒮={1,…​N},𝒯={})​[αp,tl,a​(𝒮={1,…​N},𝒯={1,…,T})​[𝐳^]].\mathbf{\alpha}^{l,a}_{p,t}(\mathcal{S}=\{1,\dots N\},\mathcal{T}=\{\})[\mathbf{\alpha}^{l,a}_{p,t}(\mathcal{S}=\{1,\dots N\},\mathcal{T}=\{1,\dots,T\})[\hat{\mathbf{z}}]]. (3)
Concept demonstration

Open image ↗