Literature
Display
← All papers

Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation

Zirui Zheng · Jiaqian Yu · Xiongfeng Peng · Jun Shi · Mingyi Li · Chao Zhang · Weiming Li · Dong Wang · Huchuan Lu · Xu Jia

Preprint posted · CoRL 2026 (accepted) ↗

Robotics5 min text · CondensedOriginal paper ↗
Contents

Reading guide

This edition follows the latest 12-page v3 and its updated Mem-World title. All six figures, both exact tables, both equations, the complete algorithm and 39 references remain. The paper points to separate timing, 150-trajectory, parameter-ablation and task-detail supplements, but those are not in the available PDF/HTML and an accessible primary supplement was not recovered. Those extra results are therefore not inferred.

At a glance

Mem-World augments a multi-view action-conditioned video model with wrist-centered surfel memory. Planned wrist-camera views retrieve older observations that may reveal objects hidden in the current frame. On the reported setup, persistent video metrics improve; policy-evaluation correlation rises from about 0.85 to 0.97, and selected synthetic-success finetuning raises long-horizon real-robot success from 58% to 72%. These are improvements in one calibrated, adapted platform; visual persistence does not enforce physically valid contact or establish general simulator reliability.

1 Introduction

Wrist cameras repeatedly lose objects behind grippers or during large arm motions. Recent-only context and fixed-stride retrieval can omit the useful earlier view even when it exists. Mem-World uses scene geometry and temporal observation associations to retrieve context, then feeds generated wrist frames back into that memory. This addresses forgetting but also makes future retrieval depend on reconstruction and previous generated observations.

Original mathematical definitions and experimental protocol

Existing approaches only partially address this issue. Interactive World Simulator [30] mitigates limited temporal context by adjusting the conditioning window size, predicting future observations from the current observation together with adjacent NN historical frames; however, simply enlarging the context window does not essentially solve the problem. Ctrl-World [11] addresses history-frame retrieval by sampling NN conditioning frames from the entire history at a fixed stride and using joint-pose similarity to emphasize historical frames with similar joint configurations during future-observation prediction. However, joint-pose similarity is only an indirect proxy for contextual relevance. It fails to capture visibility constraints introduced by end-effector occlusions and cannot prioritize viewpoints that provide richer scene coverage, such as elevated wrist-camera observations. Similar memory retrieval mechanisms have been explored in navigation-oriented world models, often using camera field-of-view (FOV) overlap [34, 32] or static 3D anchors [18] to retrieve the relevant historical frames. However, manipulation violates the underlying assumptions: the scene is partially dynamic and the robot arm itself causes structured occlusions.

Original mathematical definitions and experimental protocol

To address the limitations, we propose Mem-World, a memory-augmented multi-view action-conditioned world model for persistent robot manipulation. At its core, Mem-World introduces the W-VMem, a 4D wrist-view-centered surfel-indexed view memory module inspired by VMem [18], which anchors previous wrist-view observations to the 4D surface elements they correspond to. Specifically, we first construct a global W-VMem from the initial multi-view frames using an off-the-shelf point map estimator [20]. During autoregressive generation of future observations, W-VMem is updated using only the wrist-view observations to capture the association between 4D surfels and the dynamic wrist view. Each surfel in W-VMem maintains its creation timestep, merged timestep, orientation, depth, and a binary flag indicating whether it belongs to a manipulated object. Before generating observations at subsequent timesteps, the memory reading module retrieves relevant historical frames by rendering the surfels at each timestep from future wrist-camera viewpoints and projecting them onto an image grid. Based on the rendered surfel maps and the stored surfel attributes, the module computes a surfel score for each historical timestep and selects observations corresponding to the top-KK timesteps as conditioning inputs. By grounding future predictions in these geometrically relevant contexts, Mem-World preserves persistent scene information over long-horizon rollouts.

2 Related Work

Action-conditioned video models provide a learned rollout environment; explicit reconstruction and implicit retrieved history offer different persistence mechanisms. Mem-World extends static surfel-indexed view memory to manipulation-centric temporal associations, building on a Ctrl-World backbone. Better memory is evaluated as part of that trained pipeline, while the shared-base retrieval ablation more directly compares context choices.

Complete original Figure 1: Pipeline of Mem-World. During rollout, a wrist-view 4D surfel memory maintains a geometry-aware representation of past observations, from which relevant history frames are retrieved. Given these retrieved frames and a sequence of future actions, the multi-view action-conditioned world model predicts subsequent observations. The predicted observations are then integrated back into the memory, enabling iterative rollout with an updated memory module.
Figure 1: Pipeline of Mem-World. During rollout, a wrist-view 4D surfel memory maintains a geometry-aware representation of past observations, from which relevant history frames are retrieved. Given these retrieved frames and a sequence of future actions, the multi-view action-conditioned world model predicts subsequent observations. The predicted observations are then integrated back into the memory, enabling iterative rollout with an updated memory module.

3 Method

The pipeline alternates action-conditioned prediction, geometry-aware context retrieval and wrist-only memory updates. Initial memory uses all three camera views. Later memory is reconstructed from generated wrist frames rather than fresh physical observations, so neither the surfel representation nor temporal indexing prevents a hallucination from being stored and reused.

3.1 Preliminary

A surfel stores position, normal, radius and observation associations. Earlier VMem retrieves historical views covering surfaces expected to be visible from a future camera pose. Mem-World adds timing/task relevance because robot arms and manipulated objects move. A temporal surfel association is not a complete tracked-object dynamics or material-state model.

Original mathematical definitions and experimental protocol

Surfel-indexed View Memory. For autoregressive video prediction conditioned on camera pose, VMem [18] constructs a Surfel-indexed view memory based on past observations (view), where each surfel sk=(pk,nk,rk,ℐk)s_{k}=(p_{k},n_{k},r_{k},\mathcal{I}_{k}) stores its 3D position pkp_{k}, surface normal nkn_{k}, radius rkr_{k}, and the indices ℐk\mathcal{I}_{k} of observations that have observed it. Given a future camera pose, VMem renders all visible surfels and aggregates their associated observation indices. It then retrieves historical observations that cover larger portions of the rendered surfels, using them as context for predicting the future observation. The predicted observations are then converted into surfels and merged into memory. This iterative geometry-aware retrieval and update process improves long-term consistency in static egocentric scenes, but it assumes that camera poses are directly available and does not explicitly distinguish temporally changing scene elements.

3.2 W-VMem for Persistent Manipulation World Modeling

Geometry and calibration. Joint-space actions pass through forward kinematics and fixed hand–eye/external-camera transforms to predict future wrist views. The setup assumes fixed external cameras and known calibration; planned camera motion need not equal actual future motion under contact or execution error.

Memory. The first three views initialize geometry; later wrist-only updates keep timestamps tied to wrist observations. A vision-language/segmentation pipeline labels task-relevant objects. Correct geometry, masks and associations are prerequisites, and a first multi-view snapshot does not guarantee every scene surface is observed.

Retrieval. Historical time slices are rendered separately from an average planned wrist pose. Equation 2 combines normal/view alignment, inverse depth, a logarithmic object-flag weight and bounded recency decay; non-maximum suppression selects nonredundant top-K history. Averaging a chunk’s pose is a heuristic and can miss visibility changes within the chunk. The paper does not fully specify its pose averaging, merge thresholds, NMS criterion or numeric K in the available source. H denotes action horizon in the rollout and temporal half-life in the score; those roles should not be silently equated. The full source definitions and algorithm remain selectable.

Original mathematical definitions and experimental protocol

A standard action-conditioned world model WW predicts future observations o^t+1:t+H\hat{o}_{t+1:t+H} conditioned on the current observation o^t\hat{o}_{t} and the action chunk at+1:t+Ha_{t+1:t+H}, Mem-World augments the conditions with additional historical observations. By extending Surfel-indexed View Memory to multi-view action-conditioned manipulation world models, W-VMem retrieves a set of relevant history frames ℋt={og1,…,ogK}\mathcal{H}_{t}=\{o_{g_{1}},\ldots,o_{g_{K}}\} at rollout timestep tt, where {gi}i=1K\{g_{i}\}_{i=1}^{K} are selected historical timesteps. The world model then predicts future multi-view observations as:

Complete original Equation (1)
Original selectable equation (1)
Equation 1
o^t+1:t+H∼W(⋅∣o^t,ℋt,at+1:t+H).\hat{o}_{t+1:t+H}\sim W(\cdot\mid\hat{o}_{t},\mathcal{H}_{t},a_{t+1:t+H}). (1)
Original mathematical definitions and experimental protocol

Time-aware Surfel Definition for Manipulation. VMem assumes a static scene and renders all surfels jointly, which is insufficient for robot manipulation where the robot arm and manipulated objects move over time. W-VMem augments each surfel with temporal and task-relevance attributes: sk=(pk,nk,rk,tk,mk)s_{k}=(p_{k},n_{k},r_{k},t_{k},m_{k}), where tkt_{k} denotes the timestep set at which the surfel is created and updated, enabling the representation to capture temporal dynamics. mk∈{0,1}m_{k}\in\{0,1\} is a binary flag indicating whether the surfel belongs to a manipulated object, allowing us to emphasize task-relevant regions. To obtain mkm_{k}, we use a vision-language model [27] to analyze the scene from the text instruction and initial observation, identifying task-relevant object categories. We then apply a segmentation model [36] to segment them in the image and assign the flag to associated surfels.

Complete original Algorithm1,Require,Ensure and all ten numbered lines.
Algorithm 1 Mem-World Rollout with W-VMem
Original selectable algorithm
Require: Initial multi-view observations ot=0o_{t=0}, action-conditioned world model WW, top-KK, action horizon HH, interaction step TT
Ensure: Generated multi-view rollout 𝒪\mathcal{O}
1: Initialize W-VMem 𝒮←InitSurfels​(o0)\mathcal{S}\leftarrow\textsc{InitSurfels}(o_{0}), 𝒪←𝒪∪{ot=0}\mathcal{O}\leftarrow\mathcal{O}\cup\{o_{t=0}\}
2: for t=0t=0 to T−1T-1 do
3:   Obtain action chunk 𝐚t+1:t+H\mathbf{a}_{t+1:t+H} from policy or replayed trajectory
4:   Estimate camera pose 𝐜¯w←AvgWristPose(at+1:t+H)\bar{\mathbf{c}}_{w}\leftarrow\textsc{AvgWristPose}(a_{t+1:t+H})
5:   Compute s​c​o​r​eτ←Score​(Render​(𝒮τ,𝐜¯w))score_{\tau}\leftarrow\textsc{Score}(\textsc{Render}(\mathcal{S}_{\tau},\bar{\mathbf{c}}_{w})) for each τ\tau
6:   Retrieve history frames ℋt←TopK-NMS​({s​c​o​r​eτ},K)\mathcal{H}_{t}\leftarrow\textsc{TopK-NMS}(\{score_{\tau}\},K)
7:   Generate o^t+1:t+H←W(o^t,ℋt,𝐚t+1:t+H)\hat{o}_{t+1:t+H}\leftarrow W(\hat{o}_{t},\mathcal{H}_{t},\mathbf{a}_{t+1:t+H})
8:   𝒪←𝒪∪{o^t+1,…,o^t+H}\mathcal{O}\leftarrow\mathcal{O}\cup\{\hat{o}_{t+1},\ldots,\hat{o}_{t+H}\}
9:   Update W-VMem 𝒮←UpdateSurfels(𝒮,o^t+1:t+Hw)\mathcal{S}\leftarrow\textsc{UpdateSurfels}(\mathcal{S},\hat{o}_{t+1:t+H}^{w})
10: end for
Original mathematical definitions and experimental protocol

Geometry-aware Memory Retrieval. After updating the W-VMem, the world model would receive a new action chunk from the policy model for future observation generation. To provide historical context for this process, W-VMem is read to retrieve the historical observations relevant to future states. Specifically, we first compute the average future wrist camera pose 𝐜¯w∈SE⁡(3)\bar{\mathbf{c}}_{w}\in\mathrm{SE}(3) based on the action chunk. Then we render the surfels 𝒮\mathcal{S} at each timestep individually from the 𝐜¯w\bar{\mathbf{c}}_{w}, instead of rendering the entire surfel set jointly like VMem. This design exploits wrist-view-dependent temporal dynamics, as surfels observed at different timesteps correspond to distinct interaction stages and should not be aggregated into a single rendering, which would otherwise blur temporal cues.

Complete original Equation (2)
Original selectable equation (2)
Equation 2
score​(s,t)=⟨𝐧s,𝐯¯w⟩1+ds⋅ln⁡(e+ms)⋅[λmin+(1−λmin)​2−T−tH],\text{score}(s,t)=\frac{\langle\mathbf{n}_{s},\bar{\mathbf{v}}_{w}\rangle}{1+d_{s}}\cdot\ln(e+m_{s})\cdot\left[\lambda_{\min}+\left(1-\lambda_{\min}\right)2^{-\frac{T-t}{H}}\right], (2)
Original mathematical definitions and experimental protocol

As shown in  Eq. 2, each rendered surfel at timestep tt is assigned a score based on geometric visibility, task relevance, and temporal recency. The geometric component depends on the depth dsd_{s}, normal 𝐧s\mathbf{n}_{s}, and its alignment with the average future wrist-camera viewing direction 𝐯¯w\bar{\mathbf{v}}_{w}. The task-relevance component is encoded by the is_manipulated flag msm_{s}, while the temporal decay term assigns higher weights to more recently observed surfels. TT denotes the latest timestep, HH denotes the temporal half-life, and λmin\lambda_{\min} denotes the minimum temporal decay factor. This prioritizes observations that are both geometrically informative and closely related to the manipulation process. To further avoid redundancy, we apply non-maximum suppression (NMS) over the candidate observations. It mitigates oversampling of repeatedly visited regions, such as cases where the wrist camera hovers above the manipulated object during a pick operation, and promotes broader scene coverage among the selected top-KK timestep observations og1,og2,…,ogKo_{g}^{1},o_{g}^{2},\ldots,o_{g}^{K}. The overall algorithm is listed in Algorithm 1.

4 Experiments

The physical platform is a Franka Panda with a 2-DoF gripper and three cameras. World-model training samples 11K DROID trajectories and adapts temporal attention in the Ctrl-World backbone for about two days on eight H100s, batch 32, with extra wrist-view loss weight. The retrieval module is training-free; the resulting world model is finetuned. Recency uses lambda_min=0.1 and half-life 0.3×T_max. Exact context count, wrist-loss weight and several memory settings are not provided here.

Original mathematical definitions and experimental protocol

Training Details. Our world model is built on top of Ctrl-World [11]. To enable geometry-aware context conditioning, we sample 11K trajectories and construct training clips by replacing the original temporally adjacent context with geometry-related context retrieved by the training-free W-VMem. the λmin\lambda_{\min} in Eq. 2 is set to 0.1, the temporal half-life HH is set to 0.3×Tm​a​x0.3\times T_{max}, the We fine-tune only the temporal attention layers of the world model on these samples using 8 H100 GPUs with a total batch size of 32, which takes approximately 2 days. Considering the importance of the wrist view, we upweight the loss term associated with wrist-view predictions.

4.1 World Model Consistency Evaluation

Thirty-four deliberately selected DROID trajectories stress occlusion and camera revisits over twenty-second autoregressive rollouts. Mem-World improves every retained metric against Ctrl-World, including wrist PSNR 17.34→19.21 and object similarity 0.476→0.524. The Cosmos baseline is adapted only for third-person prediction, so no unreported multi-view comparison is implied. SAM3 masks plus DINOv2 features measure object appearance, not contact physics. A common randomly-context-trained base yields the separate retrieval ablation table; its values should not be conflated with the main trained-model table. There are no retained confidence intervals or per-trajectory errors for these aggregate results.

Complete original Figure 2: Qualitative results on long-horizon rollouts. Mem-World exhibits persistent and temporally consistent world modeling: the lid remains recoverable after transient occlusion by the gripper, and the black snack is consistently regenerated after back-and-forth camera motion.
Figure 2: Qualitative results on long-horizon rollouts. Mem-World exhibits persistent and temporally consistent world modeling: the lid remains recoverable after transient occlusion by the gripper, and the black snack is consistently regenerated after back-and-forth camera motion.
Complete original Table 1: Quantitative world model consistency evaluation results.
Table 1: Quantitative world model consistency evaluation results.
Accessible table, all original values and citations
Evaluated Camera Method Computation-based Model-based
PSNR ↑\uparrow SSIM ↑\uparrow LPIPS ↓\downarrow Obj. Con. ↑\uparrow
Third-view Camera Cosmos Predict 2.5 22.80 0.819 0.089 0.579
Ctrl-World 23.17 0.828 0.076 0.573
Mem-World (Ours) 25.30 0.878 0.054 0.619
Wrist-view Camera Ctrl-World 17.34 0.623 0.281 0.476
Mem-World (Ours) 19.21 0.691 0.236 0.524
Complete original Table 2: Ablations on memory design.
Table 2: Ablations on memory design.
Accessible table, all original values and citations
Eval. Cam. Method Comp.-based Model-based
PSNR ↑\uparrow SSIM ↑\uparrow LPIPS ↓\downarrow Obj. Con. ↑\uparrow
Third View Short-term [30] 21.25 0.802 0.082 0.526
Stride [11, 15] 22.58 0.814 0.079 0.544
W-VMem (Ours) 24.78 0.869 0.062 0.597
Wrist View Short-term [30] 15.04 0.526 0.353 0.401
Stride [11, 15] 17.06 0.614 0.295 0.463
W-VMem (Ours) 18.97 0.680 0.248 0.502
Original mathematical definitions and experimental protocol

Ablation Study of Memory Retrieval Method. We train a base model conditioned on randomly sampled context to fairly evaluate different memory retrieval methods. Here we compare our W-VMem, the stride memory adopted by Ctrl-World [11] and EVAC [15], and short-term memory adopted by Interactive World Simulator  [30], which selects NN adjacent frames as context on replay-based evaluations. As shown in Table 2 and Fig. 3, W-VMem effectively retrieves relevant historical frames as context, allowing the world model to recover temporally occluded or previously observed scene content under back-and-forth camera motion, leading to more persistent predictions. Additional ablation study on the parameter of W-VMem can be found in supplementary materials.

Complete original Figure 3: Ablation on memory retrieval strategies. By retrieving geometrically relevant historical frames, W-VMem helps the world model preserve consistent object appearances across future observations. In contrast, Short-term Mem and Stride Mem provide less informative context, leading to object (spoon) distortion or disappearance during rollout.
Figure 3: Ablation on memory retrieval strategies. By retrieving geometrically relevant historical frames, W-VMem helps the world model preserve consistent object appearances across future observations. In contrast, Short-term Mem and Stride Mem provide less informative context, leading to object (spoon) distortion or disappearance during rollout.

4.2 World Model for Policy Evaluation

Five tasks and two policies produce ten policy–task success-rate pairs in Figure 4. The exact plot gives Pearson r=0.9704 versus 0.8476, with p displayed as 0.000 versus 0.002; the main text rounds these values. The headline 14.5% is the relative increase in correlation, not a 14.5-point success gain, and a rounded p=0.000 is not literally zero. High correlation is not calibrated success prediction: simulated success is generally lower, and shared tasks/policies make broad independence/generalization claims unwarranted.

The test setup is adapted with 50 real episodes per task: policies receive 20K finetuning steps and the world model 5K steps. This is not untouched zero-shot deployment. Physical/world-model evaluation trial denominators, repeated-seed uncertainty and the significance of the difference between the two dependent correlations are not supplied. Both start from matching initial observations, which controls selected scene conditions without proving accurate off-distribution rollouts.

Original mathematical definitions and experimental protocol

In this section, we evaluate whether Mem-World can serve as an effective simulator for evaluating the generalist robot policies, including π0\pi_{0}, π0.5\pi_{0.5}, and whether its predicted rollouts can accurately reflect their real-world performance.

Complete original Figure 4: Quantitative correlations between real-world and world-model.
Figure 4: Quantitative correlations between real-world and world-model.
Complete original Figure 5: Policy improvement. Post-training on synthetic data improves the average success rate to 72%.
Figure 5: Policy improvement. Post-training on synthetic data improves the average success rate to 72%.
Original mathematical definitions and experimental protocol

Comparison Between Real-World and World Model Rollouts. We initialize real-world executions and world-model rollouts, including Ctrl-World and Mem-World, from the same initial observations and evaluate each policy under identical task conditions. Since our setup is out-of-distribution for both policy and world models due to the custom environment and different 2DoF gripper, we collect 50 episodes per task for post-training. We fine-tune π0\pi_{0} and π0.5\pi_{0.5} for 20K steps on 4 H100 GPUs, and fine-tune the world model for 5K steps on 4 H100 GPUs to adapt to the visual domain shift. Policy evaluation is then conducted using the post-trained models. As shown in Fig. 4, policy success rates in Mem-World are highly correlated with their real-world success rates across the five tasks (r=0.97r=0.97, p<0.001p<0.001). Although the absolute success rates in Mem-World are generally lower than those in the real world, the gap is relatively uniform across tasks, with no obvious outliers, resulting in a strong linear correspondence. In contrast, Ctrl-World shows a weaker correlation (r=0.85r=0.85, p<0.01p<0.01), mainly due to its pronounced performance drop on long-horizon tasks. This deviation suggests that the stride-memory design struggles to preserve scene consistency during long-horizon rollouts, leading to poorer alignment with the real world. As shown in Fig. 6, in the pick 2 objects to 1 target task, Ctrl-World severely distorts the fork, preventing subsequent grasp completion.

Complete original Figure 6: Comparisons between π 0.5 \pi_{0.5} rollouts in the real-world, Ctrl-World, and Mem-World.
Figure 6: Comparisons between π0.5\pi_{0.5} rollouts in the real-world, Ctrl-World, and Mem-World.

4.3 World Model for Policy Training

The adapted pi0.5 policy generates 200 imagined trajectories per task; humans retain 20–30 successes for additional finetuning. Three task bars change 60→75%, 70→85% and 45→55%, giving means about 58.3→71.7%, displayed as 58→72%. Thus the rounded headline is about fourteen percentage points, rather than a 14% relative improvement. Human filtering and extra optimization are part of the treatment; there is no retained equal-budget real-data or alternate-generated-data comparison isolating the memory module’s contribution to this gain. Physical evaluation counts and training-seed variation are not stated.

Original mathematical definitions and experimental protocol

We now evaluate whether Mem-World can be used to generate synthetic post-training data for improving policy models. We use the π0.5\pi_{0.5} as a base policy to rollout with Mem-World in initial observation with unseen object arrangements. We generate 200 trajectories per task and retain 20–30 successful trajectories based on human success annotation, ensuring high-quality task-completion behaviors. we compare the post-trained π0.5\pi_{0.5} policy before and after additional fine-tuning on Mem-World-generated successful trajectories. On the three long-horizon tasks, synthetic post-training improves the average real-world success rate from 58% to 72%, as shown in Fig. 5.

5 Conclusion

Time-aware geometric retrieval improves visual persistence and supplies useful selected training videos in the tested setup. The authors explicitly acknowledge weak initialization and lack of physical constraints: a rendered spherical-object grasp can succeed without antipodal contact. A remembered plausible appearance is not a validated transition/contact simulator. Stronger multi-view grounding, contact/force signals and the referenced but unavailable supplements remain limits on interpretation and reproduction.

Concept demonstration

Open image ↗