Overview
MEM combines compressed recent video with a running language summary so a robot can remember both detailed manipulation failures and long-horizon task progress.
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/a2/a252d639555204bb55486a21473ba107dbac0c07ca0037acedb1df2a13a6bc48.webp)
At a glance
Multi-Scale Embodied Memory equips π0.6 with two complementary memories: a video encoder for recent images and robot states, and a learned language summary of earlier semantic events. Physical experiments include kitchen tasks lasting up to fifteen minutes, correction-conditioned grasping and door opening, and tests of counting, hidden objects, timing, and dexterous manipulation. Evidence concerns memory within an episode; continual learning across weeks or months is proposed future work.
I Introduction
A recent occluded grasp and an ingredient already collected require different kinds of memory. Dense video can retain the geometry of the former but is expensive over many minutes; language can compactly record the latter but loses fine detail. MEM joins both in a hierarchical robot policy. Its central result is that this combination improves the tested long tasks while preserving useful short-term context and the base policy’s dexterous manipulation performance.
II Related Work
Prior policies retain state histories, image tokens, keyframes, point tracks, latent memories, or language summaries. Each trades detail against inference cost. MEM combines modalities rather than requiring one representation to retain everything. Its video encoder draws on space–time separable attention [3]; its experimental comparisons implement representative pooling and state-history approaches on the same policy backbone.
Memory can also encourage copying past actions instead of responding to the current scene. The authors hypothesize that diverse pretraining reduces this causal-confusion problem, but the reported ablations do not isolate data diversity as its cause.
III Multi-Scale Embodied Memory for VLAs
The architecture divides memory by the kind of information required. Recent camera frames and proprioception support occlusion handling and manipulation adjustment. A recursively updated text summary tracks completed subtasks and other relevant events beyond the video window. This avoids passing an entire long episode of image tokens to the action policy.
III-A Multi-Scale Embodied Memory ( MEM )
A high-level policy reads the task, recent observations, and previous language summary, then predicts both the next subtask and an updated summary. A low-level policy generates a continuous action chunk from the goal, subtask, and recent observation window. The text summary substitutes for dense distant observations; it is a learned compressed state, not a lossless episode record. The retained factorization specifies the interface between these levels.
Equation
III-B Language Memory for Long-Term Memory
Training summaries are produced by an off-the-shelf language model from annotated subtask sequences and success/failure indicators. Labels compress facts that remain relevant, such as the number and destination of bowls instead of every bowl’s appearance. The robot’s trained high-level policy generates its own updates at execution. Compression also reduces repeated-failure histories that differ from mostly successful training demonstrations, although discarded information and mistaken summaries can still limit recall.
III-C Video Encoder for Dense Short-Term Visual Memory
Every fourth layer of the image transformer adds causal temporal processing to spatial attention, reusing its query/key/value projections and adding fixed sinusoidal time encodings. Factorized attention costs O(Kn² + nK²) for K frames with n patches each, versus O(K²n²) for joint space–time attention. Only current-frame representations enter the language backbone, keeping its image-token count fixed as history grows.
The authors report no added learnable encoder parameters and initialize the current-frame positional offset to zero to preserve single-image behavior. Total video-encoding work still grows with history. The latency plot uses four camera streams on one H100 and compares against a 300 ms reference threshold; this is a specified hardware/model measurement, not a universal control-rate guarantee. Execution uses real-time action chunking [6] or its training-time variant [7].
III-D Integrating MEM into the π 0.6 VLA
The implementation starts from a Gemma3-4B vision–language model and uses an 860-million-parameter flow action expert alongside discrete FAST action-token training. Action-expert gradients do not enter the vision–language backbone. Up to four 448-by-448 camera streams and linearly projected state-history tokens supply observations. Pretraining mixes demonstrations, rollouts, human corrections, vision–language tasks, and video–language tasks, using six observations at one-second spacing. Post-training expands memory to as many as eighteen frames, described as up to fifty-four seconds. The paper does not provide a complete quantitative accounting of this large training mixture.
IV Experimental Evaluation
Experiments ask whether memory supports long kitchen tasks, whether recent failure context helps manipulation adjustment, and how the video representation compares with other memory designs. The long-task study uses ten rollouts per policy and task or recipe, with mean and standard error. Many graphs show normalized task progress rather than full-episode success, so those outcomes should not be conflated.
IV-A MEM Solves Tasks Requiring Long-Horizon Memory
Recipe setup retrieves specified ingredients and cookware from named locations; full kitchen cleanup combines storage, wiping, and washing. Training covers forty-two recipes; evaluation uses five seen recipes in unseen kitchens and with unseen objects. Both short video memory and long language memory contribute to progress. Removing either lowers performance on these tasks.
A concatenated subtask-history ablation also performs worse than learned compressed summaries. Repeated failures create histories uncommon in successful demonstrations; recording only relevant completed events can reduce that mismatch. This is supported by the ablation and examples, rather than a guarantee that the summary always retains the information required for future recovery.
IV-B In-Context Adaptation of Manipulation Strategies
The policy is finetuned on targeted corrective demonstrations for chopstick grasps at unusual table height and fridge doors with ambiguous opening direction. Failed attempts remain in the memory model’s training context; a memoryless policy receives the same correction data. Remembering a failed strategy improves subsequent adjustment. This is adaptation through observations during execution, learned from intervention data beforehand, not online gradient learning or untrained discovery of new manipulation skills.
IV-C Analysis Experiments
The comparison uses partial observability, counting, timing, wiping coverage, and dexterous tasks across several robot configurations. Pool Memory compresses separately encoded past images into an averaged memory token; Proprio Memory retains only projected state history. All use the π0.6 backbone, and this comparison omits the long-horizon language memory to focus on the observation representation. The pooling design follows ContextVLA [20]; the state-history design follows TA-VLA [53].
Video memory performs well across the tested memory capabilities. State history cannot directly remember which drawer a person used, and aggressive image pooling struggles with several longer visual histories. Introducing the video encoder only at task post-training is worse than using it during pretraining, even though both variants start from a base policy trained on the same broad robot corpus. Dexterous-task results remain broadly comparable to memoryless π0.6, with task-level variation. These controlled comparisons support the specific representations and training schedules tested, not every possible memory method.
V Conclusion
Combining detailed recent observations with compressed semantic history supports the demonstrated long tasks and failure-conditioned adjustments. Current evidence stays within individual episodes and the evaluated task/data distribution. Longer-lived memory, continual deployment learning, reliable summary-error correction, and retention of details beyond the video window remain open extensions; the paper does not demonstrate memory across days or years.
Acknowledgements
The acknowledgments credit robot operators, annotators, operations and hardware teams, and assistance with the video encoder, illustrations, filming, and editing.
-A Contributions
The contribution statement assigns video-memory design, language-memory design, evaluation, pretraining/data work, long-context infrastructure, inference optimization, project advice, and writing to the listed authors. These roles cover both the memory architecture and the substantial robot-data and evaluation effort.
-B Task Details
The appendix specifies initial conditions and scoring for the long tasks, correction tests, and analysis suite. These rubrics distinguish partial-progress points from binary success and define what information must be remembered.
-B 1 Long-horizon Tasks ( Section IV-A )
Long tasks test recipe-item retrieval and multi-step kitchen cleaning. The four original example prompts are retained in full, including object locations and required destinations.
Preparing the items for a recipe
Recipe prompts name all required items, their initial locations, and their target surfaces. One point is awarded per correctly retrieved and placed item; most recipes contain six or seven points. Evaluation therefore measures item-level progress as well as task duration, using seen recipes in new scenes. The examples cover mashed potatoes, fried rice, and pizza.
Cleaning the kitchen
Cleanup includes wiping and drying counters, refrigerating food, putting away dishes, and washing sink contents. Each completed subtask earns one point, with about eight subtasks per episode. The example prompt explicitly names relevant objects, receptacles, and dishes.
-B 2 In-context adaptation ( Section IV-B )
Both adjustment tests use corrective data and assess whether recent attempts help choose a better subsequent manipulation. Their success conditions are task-specific.
Chopstick Pick Up
Training places chopsticks at random locations with table heights in the upper half of the range and includes interventions after mis-grasps. Evaluation uses the lowest table setting. Picking up the chopstick and depositing it in the bin each earn one point; full success requires both.
Open Refrigerator
The fridge has no obvious visual cue for hinge direction. Training includes corrections of wrong-direction attempts. Evaluation counts success only if the door opens within four grasps, limiting the benefit of repeatedly trying random actions.
-B 3 Analysis Experiments ( Section IV-C )
The analysis tasks separately probe object-history recall, hidden scene state, counting, elapsed time, coverage, and dexterous control. Several are scored by partial completion, while others require the entire sequence to succeed.
Three-way swap mugs
Three mugs must visit the coffee machine one at a time without repeats or tipping. Each correct distinct mug earns one point, and previously used mugs return to random table locations, making current position insufficient to identify what remains.
Find object
A person hides an object in one of four drawers while the robot observes. Success requires opening the correct drawer, retrieving the object, and placing it on the table without opening any wrong drawer. Random drawer selection has a one-in-four chance.
Unpack groceries
A bag contains a variable number of items, invisible to the external camera. The policy must remember earlier wrist-camera views, retrieve every item, and signal completion by returning home. Success requires both complete unpacking and appropriate termination.
Scoop coffee
The robot removes the grinder lid, adds exactly two scoops of beans, and replaces the lid. The binary success condition checks the count and final lid placement, so a visually plausible final grinder state alone is insufficient.
Grilled cheese
The robot assembles, cooks, flips, and plates a sandwich. Scoring awards progress for these stages, including cooking between thirty seconds and three minutes per side. This broad accepted interval tests coarse timing, not precise temperature or cooking-quality control.
Window cleaning
The robot sprays cleaner, tears off a paper towel, wipes the entire booth window, and discards the towel. Success requires every stage and full coverage; unwiped regions count as failure.
Table bussing
Twelve tabletop objects are sorted into a bussing bin for tableware/utensils and a trash can for disposable containers and wrapping. Each correctly sorted item contributes one point.
Shirt folding
A flattened shirt must become a rectangular fold without protruding seams or excessive wrinkles. Success is judged on the final folded result.
Clean up counter
The robot opens a drawer, places every counter item inside, and closes it. All stages must finish for success.
Make bed
The robot straightens the blanket and places every pillow at the head of the bed. Both conditions are required.
Kitchen cleanup
This smaller cleanup task moves plates, bowls, and a cutting board from a counter into a sink. The stated success rubric requires all plates in the sink; it is distinct from the long-horizon washing/storage task.
Batch folding
Clothes start in a hamper. The robot retrieves, flattens, folds, and stacks shirts or shorts with existing folded clothes. Accepted folds are rectangular, without protruding seams or excessive wrinkling.
Box building
A flat cardboard cutout must be assembled into an intact box through coordinated two-arm manipulation. Full assembly without damage defines success.
-C Video encoder with Space-Time separable attention
The mathematical appendix adds sinusoidal time embeddings to patch features, sets the current-time embedding to zero, reuses normalized query/key/value projections, and expresses spatial/temporal attention before the usual value aggregation and feed-forward layer. All five displayed equation blocks, including the main policy factorization, are retained. The appendix’s set notation and attention composition are reproduced as written; the implementation narrative describes causal temporal attention interleaved with spatial attention. They should not be treated as a standalone proof of the stated initialization equivalence.
Equations 1–3
| (1) | ||||
| (2) |
| (3) |