Literature
Display
← All papers

SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation

Sunghwan Kim · Byeonghyun Pak · Kehan Long · Yulun Tian · Nikolay Atanasov

Preprint posted · CoRL 2026 (accepted) ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

Overview

SERF conditions robot action generation on a persistent feature map containing both movable environment objects and the articulated robot body.

Refer to caption
Figure 1: Top: A mobile manipulator performs a long-horizon task consisting of multiple subgoals. Bottom: A spatiotemporal feature map represents the evolving environment and the robot body in a shared latent space, visualized via PCA. The map is updated online from egocentric observations and proprioceptive state. Video results are available at the project website: https://existentialrobotics.org/serf/.

At a glance

SERF represents a scene and robot as neural points in a shared feature space. Object tracking moves environment points, forward kinematics moves robot points, and eight spatial tokens condition a vision–language–action policy. On three simulated household tasks, average subgoal progress rises from 44.0% for an image-only policy to 58.7%, with better shifted-scene performance and induced-drop recovery. Main experiments use prior maps, simulator instance labels, and simulator base poses; an onboard-estimation variant provides a limited feasibility check on one task.

1 Introduction

Long-horizon mobile manipulation requires remembering locations, updating object positions, and interpreting robot–environment relationships when targets leave the current view. SERF supplies persistent geometric features for these needs. Unlike a static scene map, it moves tracked object points; unlike environment-only memory, it embeds the robot’s current body configuration in the same space. The gains demonstrate the utility of this additional structured state under the mapping assumptions, rather than proving that current images alone contain equivalent information.

2 Problem Formulation

Inputs include head/wrist color and depth images, camera and robot base poses, joint state, and instance labels. Image patch features are back-projected into world coordinates. An offline reconstruction objective learns neural-point features and a decoder; behavior cloning then learns a policy conditioned on the resulting map, current observations, state, and task embedding. The prior reconstruction dataset is collected before execution. The formulation assumes calibrated geometry and a persistent coordinate frame, not localization-free operation from RGB alone.

Refer to caption
Figure 2: Per-patch VFM embeddings from robot observations are back-projected into 3D.
Refer to caption
Figure 3: Neural point features are interpolated and decoded to reconstruct per-patch VFM embeddings.

3 Spatiotemporal Feature Mapping

Each neural point stores position, latent feature, and object identity. A query gathers nearby points, interpolates their latent features, and decodes a vision-foundation-model embedding. Environment points come from voxelized back-projections. Robot points are sampled from link meshes in local coordinates and transformed through forward kinematics. Both sets share a decoder, with cosine reconstruction and category/part contrastive losses shaping the learned features.

At execution, latent features, identities, and decoder weights remain fixed; only positions change. Instance-level rigid transforms move environment points using tracked image keypoints lifted to three dimensions, registration, and closest-point refinement. Robot link transforms update body points from proprioception. Thus the online map update is geometric tracking, not continual feature learning or a general mechanism for adding arbitrary unseen semantic content.

The representation provides local robot–object geometry and global remembered scene structure, while rigid-instance motion and reliable identity association limit which changes it can represent. Offline environment and robot reconstruction samples are queried separately during feature learning, then combined for policy input.

Equation 1
F(x;𝒫τ)=∑i∈𝒩⁡(x,𝒫τ)wi(x;𝒫τ)fi,wi(x;𝒫τ)=softmax(−∥x−pi,τ∥/σ),σ>0.F(x;\mathcal{P}_{\tau})=\sum_{i\in\mathcal{N}(x;\mathcal{P}_{\tau})}w_{i}(x;\mathcal{P}_{\tau})\,f_{i},\quad w_{i}(x;\mathcal{P}_{\tau})=\mathrm{softmax}\bigl(-\|x-p_{i,\tau}\|/\sigma\bigr),\quad\sigma>0. (1)

4 Map-Conditioned VLA Policy

Task specifications filter the map to robot points and referenced objects, discarding structural/background points. A Point Transformer produces eight tokens: three around the base at one-, two-, and four-meter radii; one around each gripper within half a meter; and robot-only, environment-only, and global summaries. Relative positional encodings reduce dependence on absolute coordinates.

Tokens join image, state, and a learned task embedding in a π0.5-based policy. Flow matching trains its action expert; the vision/language backbone is frozen and action-expert low-rank adapters, tokenizer, and projection are trained. The base policy is the BEHAVIOR-adapted checkpoint [17]. Task embeddings and explicit target-instance filtering should be distinguished from unrestricted new-language goal understanding.

Refer to caption
Figure 4: Overview of map-conditioned VLA policy. A map tokenizer produces map tokens across multiple reference frames and spatial scales. The VLA model is conditioned on these map tokens, along with image observations, the task embedding, and proprioceptive state, to predict actions.

5 Long-Horizon Mobile Manipulation

Three BEHAVIOR-1K [18] tasks use two hundred demonstrations each and twenty evaluation configurations. Progress is the fraction of achieved symbolic subgoals, not the fraction of fully completed episodes. SERF reaches 63.5%, 60.1%, and 52.5% progress versus the task-finetuned image policy’s 40.7%, 43.0%, and 48.4%.

The static Seeing the Bigger Picture map [15] reaches 57.9%, 52.5%, and 51.6%. Dynamic environment-only SERF reaches 57.9%, 59.0%, and 49.4%, so dynamic updates help on average but do not improve every task. Adding robot points improves all three means. Environment-only variants have six rather than eight tokens, and main mapped variants receive privileged geometry/labels not used by image-only baselines.

Shifted goals, extra target objects, and target placements outside demonstrated routes are each tested on twenty configurations. Unvisited-region progress rises from 28.0% for the image policy to 51.0% for SERF. In twenty controlled object-drop trials, SERF recovers in nineteen and reports 20.5 seconds average recovery time. This evaluates a specific perturbation from matched post-drop states, not arbitrary failure recovery. The reported progress gains and direct example trajectories do not establish physical deployment performance.

Refer to caption
Figure 5: Qualitative comparison on long-horizon mobile manipulation. Map-conditioned SERF takes more direct trajectories than image-only PI0.5 (ft), reaches subgoals faster, and achieves higher task progress.
Method Task 21 Task 22 Task 26
PI0.5 (pre) [17] 42.9±\pm19.7 43.0±\pm23.5 44.1±\pm22.4
PI0.5 (ft) [17] 40.7±\pm18.8 43.0±\pm19.3 48.4±\pm21.1
SBP [15] 57.9±\pm17.8 52.5±\pm24.1 51.6±\pm11.7
SERF (env) 57.9±\pm15.3 59.0±\pm20.5 49.4±\pm13.4
SERF 63.5±\pm16.7 60.1±\pm19.1 52.5±\pm13.8

Table 1: Task progress (%) across BEHAVIOR-1K tasks. All methods are built on the same base policy, π0.5\pi_{0.5} [4]. All fine-tuned methods use the same training setup.
Original figure
Figure 6: Scene-configuration generalization under OOD settings. We report task progress (%) for each policy under test-time scene shifts.
Refer to caption
Figure 7: Failure recovery after object drop during transport. Left: SERF re-localizes and re-grasps the dropped object, whereas image-only PI0.5 (ft) fails to recover. Right: SERF outperforms all baselines in both recovery success rate and recovery time.

6 Conclusion

Persistent dynamic environment features and explicit robot-body features improve the tested simulated mobile-manipulation tasks. Results support using a spatially structured state alongside current images, while performance and transfer remain contingent on prior reconstruction, pose/instance estimates, and the tested task specifications.

Limitations

SERF learns a prior map before execution and uses privileged simulation identities and base poses in its main evaluation. Replacing these with estimates is tested only on one task. No physical mobile-manipulation evaluation is reported. Rigid object updates do not handle general articulation or deformation; features stay fixed during execution; and token regions are hand-designed. Proposed streaming feature initialization, part-level motion, nonrigid models, temporal prediction, and task-adaptive tokenization remain future work.

Acknowledgments

The acknowledgments identify United States National Science Foundation support and Korean defense-research funding.

Appendix

The appendix covers estimated inputs, map-data construction, feature representation, losses and tracking, policy training, token ablations, benchmark tasks, history baselines, progress curves, scene shifts, and additional rollout/map visualizations.

Appendix A Privileged Inputs and Onboard Estimates

On Task 22, category-prompted SAM 3 masks replace simulator instance labels and ORB-SLAM3 replaces simulator base poses at test time. The reported trajectory errors are 2.42 cm and 0.34 degrees. Progress falls from 60.1% with ground truth to 54.8% with estimates, still above the image-only baseline’s 43.0%. This supports feasibility of estimated inputs for that simulated task, while retaining the prior map and task-specific category prompts; it is not a full real-world end-to-end validation.

Refer to caption
Figure 8: Left: Text-prompted SAM 3 masks for the target object categories on Task 22. Right: ORB-SLAM3 base pose estimates compared with the ground truth and dead reckoning.
Method Task 22
PI0.5 (ft) [17] 43.0±\pm19.3
SERF (estimated labels/poses) 54.8±\pm22.9
SERF (GT labels/poses) 60.1±\pm19.1
Table 2: Task progress (%) on Task 22. SERF is evaluated with estimated or ground-truth (GT) instance labels and base poses.

Appendix B Map Dataset Generation

Environment samples pair world-frame coordinates, obtained by depth back-projection and camera pose, with image patch embeddings. Robot samples are rendered from multiple viewpoints at selected base, arm, and gripper configurations, then back-projected in the same way. State-conditioned supervision ties each local robot point to semantic appearance across configurations. The dataset therefore includes a designed pre-execution reconstruction process, not only observations accumulated during the evaluated rollout.

Equation 2
x⁡(ρ)=Rcam​Π−1​[ρ1]​Z​[ρ]+tcam.x(\rho)=R_{\text{cam}}\,\Pi^{-1}\begin{bmatrix}\rho\\ 1\end{bmatrix}Z[\rho]\,+\,t_{\text{cam}}. (2)
Refer to caption
Figure 9: Robot renderings from multiple viewpoints provide state-conditioned reconstruction samples.

Appendix C Map Representation Details

DINOv3 encodes 480-by-480 images into a 30-by-30 grid of 1,280-dimensional patch features. Environment and robot points use two-centimeter spatial resolution and 64-dimensional latent features. A residual multilayer perceptron decodes to DINO feature space. Queries interpolate six nearest points from a radius search with temperature 0.05. SAM 2 supplies part masks used in the representation learning setup.

Appendix D Contrastive Objectives

An inter-category loss groups same-category features across scenes and separates categories, treating the robot as another category. An intra-instance loss groups parts within an object and separates different parts, using segmentation for environment objects and rendered link labels for the robot. Both sample up to 16,384 points with category balancing; weights are 0.02 and 0.01, with contrastive temperature 0.1. The preserved principal-component plots visualize these learned structures but do not independently prove task-relevant feature disentanglement.

Equation 3
ℒinter=−log⁡exp⁡(sim⁡(fi,fj+)/σc)∑kexp⁡(sim⁡(fi,fk)/σc).\mathcal{L}_{\text{inter}}=-\log\frac{\exp\bigl(\mathrm{sim}(f_{i},f_{j}^{+})/\sigma_{\mathrm{c}}\bigr)}{\sum_{k}\exp\bigl(\mathrm{sim}(f_{i},f_{k})/\sigma_{\mathrm{c}}\bigr)}. (3)
Refer to caption
Figure 10: PCA projections show same-category features clustering across scenes and different-category features separating.
Equation 4
ℒintra=−log⁡exp⁡(sim⁡(fi,fj+)/σc)∑k∈partsexp⁡(sim⁡(fi,fk)/σc).\mathcal{L}_{\text{intra}}=-\log\frac{\exp\bigl(\mathrm{sim}(f_{i},f_{j}^{+})/\sigma_{\mathrm{c}}\bigr)}{\sum_{k\in\text{parts}}\exp\bigl(\mathrm{sim}(f_{i},f_{k})/\sigma_{\mathrm{c}}\bigr)}. (4)
Refer to caption
Figure 11: PCA projections show same-part features clustering together and different-part features separating within each instance.

Appendix E Map Updates and Tracking

Training uses recorded robot states and precomputed object motions; execution estimates object motion before each policy query. Shi–Tomasi corners are tracked with separate CoTracker3 buffers for head and wrist views, then lifted with depth/pose. Centroid movement below 1.5 cm skips registration. A discrepancy above 20 cm triggers centroid-initialized closest-point recovery, accepted only when final centroid error is at most 2 cm. These rules handle some occlusion/rapid-motion failures, but do not guarantee identity recovery or accurate motion when objects are unseen.

Appendix F VLA Policy Details

Each query samples 25,000 task-filtered neural points. A shared two-stage Point Transformer and eight branch heads produce 2,048-dimensional tokens. Each task is finetuned for twenty thousand steps at batch size sixteen, with fifteen flow/noise samples per example and a low cosine-scheduled learning rate. Inference uses twenty Euler integration steps to predict thirty actions. The first twenty-six are interpolated and resampled into twenty control commands. Map tokens are recomputed once per chunk, not at every low-level action, and complete end-to-end mapping/control latency is not reported.

Appendix G Map Token Ablation

On Task 22, full SERF scores 60.1% progress. Removing base tokens gives 56.5%, gripper tokens 54.0%, robot-only 54.0%, environment-only 58.5%, and global 55.1%. All group-removal means are lower, with substantial reported dispersion. This shows utility of the chosen groups within this configuration, without establishing a universal optimal token allocation.

Method Task 22
SERF w/o robot-base 56.5±\pm21.7
SERF w/o end-effector 54.0±\pm21.5
SERF w/o robot-only 54.0±\pm22.2
SERF w/o environment-only 58.5±\pm20.9
SERF w/o global 55.1±\pm20.6
SERF 60.1±\pm19.1
Table 3: Map token ablation. Each variant omits one map-token group. We report task progress (%).

Appendix H BEHAVIOR-1K Benchmark Details

Toy collection has seven targets in a 4.4-by-3.0-meter region; shoe organization has four items in 5.5 by 9.0 meters; gift baskets contain twenty target instances in 7.6 by 8.8 meters. Maximum rollouts are 38,372, 15,384, and 52,120 steps, twice average demonstration length. Symbolic conditions require toys in one bookcase, footwear off the floor and grouped by category, and a specified set of contents in each basket. Partial completion of these predicates determines the progress metric.

Task Target objects Workspace (x×yx{\times}y) Max steps
Task 21 2 dice, 2 teddy bears, 2 board games, 1 toy train 4.4×3.04.4{\times}3.0 m 38,372
Task 22 2 gym shoes, 2 sandals 5.5×9.05.5{\times}9.0 m 15,384
Task 26 4 baskets, 4 candles, 4 butter cookies, 4 Swiss cheeses, 4 bows 7.6×8.87.6{\times}8.8 m 52,120
Table 4: BEHAVIOR-1K task details.

Appendix I Comparison with Image-History Baseline

An image-history baseline adds eight tokens from eight past frames using a history module based on HAMLET [16]. Its progress is 50.0%, 49.5%, and 48.1%, improving over the image-only model on two tasks but remaining below SERF on all three. The comparison tests this particular fixed-stride history design, not all possible recurrent or retrieval-based memory systems.

Method Task 21 Task 22 Task 26
PI0.5 (ft) [17] 40.7±\pm18.8 43.0±\pm19.3 48.4±\pm21.1
PI0.5 (image history) [16] 50.0±\pm17.2 49.5±\pm19.4 48.1±\pm20.2
SBP [15] 57.9±\pm17.8 52.5±\pm24.1 51.6±\pm11.7
SERF 63.5±\pm16.7 60.1±\pm19.1 52.5±\pm13.8
Table 5: Comparison with an image-history baseline. We report task progress (%).

Appendix J Task Progress Curves

Progress curves for Tasks 21 and 22 average ten configurations, fewer than the twenty used for the main table. SERF has the largest area under the progress-versus-step curve and highest final progress. This supports earlier subgoal attainment in those evaluations; the area is an execution-step measure rather than a hardware wall-clock throughput benchmark.

Refer to caption
Refer to caption
Figure 12: Task progress over execution steps on Tasks 21 and 22. SERF achieves the highest AUC and final task progress on both tasks.

Appendix K Scene-Configuration Experiments

The moved-goal test relocates the toy bookcase without changing targets or language. The extra-object test adds two teddy bears, increasing targets from seven to nine and explicitly updating the symbolic task goal. The unvisited-region test moves two sandals outside demonstrated target locations within the same scene. Policies are trained on original configurations only, but SERF still has the prior/updated map and task-specified object identities; an unvisited demonstration region should not be read as an entirely unknown unmapped world.

Refer to caption
Figure 13: The goal bookcase is relocated from its original position.
Refer to caption
Figure 14: Additional teddy bears are added as target objects in the original scene.
Refer to caption
Figure 15: During evaluation, two sandals are placed in a region where no target objects appear in the training demonstrations.

Appendix L Additional Qualitative Results

Additional matched rollout comparisons show image-only policies stalling after targets leave view, while SERF often follows more direct paths and completes more subgoals. All three task sequences and their timing labels are retained. These illustrative trajectories accompany, rather than replace, the aggregate progress measurements.

Refer to caption
Refer to caption
Refer to caption
Figure 16: Qualitative comparisons on long-horizon mobile manipulation. The row pairs correspond to Task 21, Task 22, and Task 26, respectively, and compare PI0.5 (ft) with SERF. In these rollouts, SERF achieves higher task progress than PI0.5 (ft) across all three tasks.

Appendix M Additional Map Visualizations

The final figure pairs third-person execution images with corresponding evolving feature maps for all three tasks. The visualizations retain the robot and environment in their shared feature representation, showing tracked object movement and articulated robot updates across the rollout.

Refer to caption
Figure 17: SERF map visualizations. Row pairs correspond to Task 21, Task 22, and Task 26. For each task, the top row shows third-person observations of the robot during execution, and the bottom row shows the corresponding SERF feature map visualized with PCA.
Concept demonstration

Open image ↗