Overview
SERF conditions robot action generation on a persistent feature map containing both movable environment objects and the articulated robot body.
At a glance
SERF represents a scene and robot as neural points in a shared feature space. Object tracking moves environment points, forward kinematics moves robot points, and eight spatial tokens condition a vision–language–action policy. On three simulated household tasks, average subgoal progress rises from 44.0% for an image-only policy to 58.7%, with better shifted-scene performance and induced-drop recovery. Main experiments use prior maps, simulator instance labels, and simulator base poses; an onboard-estimation variant provides a limited feasibility check on one task.
1 Introduction
Long-horizon mobile manipulation requires remembering locations, updating object positions, and interpreting robot–environment relationships when targets leave the current view. SERF supplies persistent geometric features for these needs. Unlike a static scene map, it moves tracked object points; unlike environment-only memory, it embeds the robot’s current body configuration in the same space. The gains demonstrate the utility of this additional structured state under the mapping assumptions, rather than proving that current images alone contain equivalent information.
2 Problem Formulation
Inputs include head/wrist color and depth images, camera and robot base poses, joint state, and instance labels. Image patch features are back-projected into world coordinates. An offline reconstruction objective learns neural-point features and a decoder; behavior cloning then learns a policy conditioned on the resulting map, current observations, state, and task embedding. The prior reconstruction dataset is collected before execution. The formulation assumes calibrated geometry and a persistent coordinate frame, not localization-free operation from RGB alone.
3 Spatiotemporal Feature Mapping
Each neural point stores position, latent feature, and object identity. A query gathers nearby points, interpolates their latent features, and decodes a vision-foundation-model embedding. Environment points come from voxelized back-projections. Robot points are sampled from link meshes in local coordinates and transformed through forward kinematics. Both sets share a decoder, with cosine reconstruction and category/part contrastive losses shaping the learned features.
At execution, latent features, identities, and decoder weights remain fixed; only positions change. Instance-level rigid transforms move environment points using tracked image keypoints lifted to three dimensions, registration, and closest-point refinement. Robot link transforms update body points from proprioception. Thus the online map update is geometric tracking, not continual feature learning or a general mechanism for adding arbitrary unseen semantic content.
The representation provides local robot–object geometry and global remembered scene structure, while rigid-instance motion and reliable identity association limit which changes it can represent. Offline environment and robot reconstruction samples are queried separately during feature learning, then combined for policy input.
Equation 1
| (1) |
4 Map-Conditioned VLA Policy
Task specifications filter the map to robot points and referenced objects, discarding structural/background points. A Point Transformer produces eight tokens: three around the base at one-, two-, and four-meter radii; one around each gripper within half a meter; and robot-only, environment-only, and global summaries. Relative positional encodings reduce dependence on absolute coordinates.
Tokens join image, state, and a learned task embedding in a π0.5-based policy. Flow matching trains its action expert; the vision/language backbone is frozen and action-expert low-rank adapters, tokenizer, and projection are trained. The base policy is the BEHAVIOR-adapted checkpoint [17]. Task embeddings and explicit target-instance filtering should be distinguished from unrestricted new-language goal understanding.
5 Long-Horizon Mobile Manipulation
Three BEHAVIOR-1K [18] tasks use two hundred demonstrations each and twenty evaluation configurations. Progress is the fraction of achieved symbolic subgoals, not the fraction of fully completed episodes. SERF reaches 63.5%, 60.1%, and 52.5% progress versus the task-finetuned image policy’s 40.7%, 43.0%, and 48.4%.
The static Seeing the Bigger Picture map [15] reaches 57.9%, 52.5%, and 51.6%. Dynamic environment-only SERF reaches 57.9%, 59.0%, and 49.4%, so dynamic updates help on average but do not improve every task. Adding robot points improves all three means. Environment-only variants have six rather than eight tokens, and main mapped variants receive privileged geometry/labels not used by image-only baselines.
Shifted goals, extra target objects, and target placements outside demonstrated routes are each tested on twenty configurations. Unvisited-region progress rises from 28.0% for the image policy to 51.0% for SERF. In twenty controlled object-drop trials, SERF recovers in nineteen and reports 20.5 seconds average recovery time. This evaluates a specific perturbation from matched post-drop states, not arbitrary failure recovery. The reported progress gains and direct example trajectories do not establish physical deployment performance.
| Method | Task 21 | Task 22 | Task 26 |
| PI0.5 (pre) [17] | 42.919.7 | 43.023.5 | 44.122.4 |
| PI0.5 (ft) [17] | 40.718.8 | 43.019.3 | 48.421.1 |
| SBP [15] | 57.917.8 | 52.524.1 | 51.611.7 |
| SERF (env) | 57.915.3 | 59.020.5 | 49.413.4 |
| SERF | 63.516.7 | 60.119.1 | 52.513.8 |
6 Conclusion
Persistent dynamic environment features and explicit robot-body features improve the tested simulated mobile-manipulation tasks. Results support using a spatially structured state alongside current images, while performance and transfer remain contingent on prior reconstruction, pose/instance estimates, and the tested task specifications.
Limitations
SERF learns a prior map before execution and uses privileged simulation identities and base poses in its main evaluation. Replacing these with estimates is tested only on one task. No physical mobile-manipulation evaluation is reported. Rigid object updates do not handle general articulation or deformation; features stay fixed during execution; and token regions are hand-designed. Proposed streaming feature initialization, part-level motion, nonrigid models, temporal prediction, and task-adaptive tokenization remain future work.
Acknowledgments
The acknowledgments identify United States National Science Foundation support and Korean defense-research funding.
Appendix
The appendix covers estimated inputs, map-data construction, feature representation, losses and tracking, policy training, token ablations, benchmark tasks, history baselines, progress curves, scene shifts, and additional rollout/map visualizations.
Appendix A Privileged Inputs and Onboard Estimates
On Task 22, category-prompted SAM 3 masks replace simulator instance labels and ORB-SLAM3 replaces simulator base poses at test time. The reported trajectory errors are 2.42 cm and 0.34 degrees. Progress falls from 60.1% with ground truth to 54.8% with estimates, still above the image-only baseline’s 43.0%. This supports feasibility of estimated inputs for that simulated task, while retaining the prior map and task-specific category prompts; it is not a full real-world end-to-end validation.
Appendix B Map Dataset Generation
Environment samples pair world-frame coordinates, obtained by depth back-projection and camera pose, with image patch embeddings. Robot samples are rendered from multiple viewpoints at selected base, arm, and gripper configurations, then back-projected in the same way. State-conditioned supervision ties each local robot point to semantic appearance across configurations. The dataset therefore includes a designed pre-execution reconstruction process, not only observations accumulated during the evaluated rollout.
Equation 2
| (2) |
Appendix C Map Representation Details
DINOv3 encodes 480-by-480 images into a 30-by-30 grid of 1,280-dimensional patch features. Environment and robot points use two-centimeter spatial resolution and 64-dimensional latent features. A residual multilayer perceptron decodes to DINO feature space. Queries interpolate six nearest points from a radius search with temperature 0.05. SAM 2 supplies part masks used in the representation learning setup.
Appendix D Contrastive Objectives
An inter-category loss groups same-category features across scenes and separates categories, treating the robot as another category. An intra-instance loss groups parts within an object and separates different parts, using segmentation for environment objects and rendered link labels for the robot. Both sample up to 16,384 points with category balancing; weights are 0.02 and 0.01, with contrastive temperature 0.1. The preserved principal-component plots visualize these learned structures but do not independently prove task-relevant feature disentanglement.
Equation 3
| (3) |
Equation 4
| (4) |
Appendix E Map Updates and Tracking
Training uses recorded robot states and precomputed object motions; execution estimates object motion before each policy query. Shi–Tomasi corners are tracked with separate CoTracker3 buffers for head and wrist views, then lifted with depth/pose. Centroid movement below 1.5 cm skips registration. A discrepancy above 20 cm triggers centroid-initialized closest-point recovery, accepted only when final centroid error is at most 2 cm. These rules handle some occlusion/rapid-motion failures, but do not guarantee identity recovery or accurate motion when objects are unseen.
Appendix F VLA Policy Details
Each query samples 25,000 task-filtered neural points. A shared two-stage Point Transformer and eight branch heads produce 2,048-dimensional tokens. Each task is finetuned for twenty thousand steps at batch size sixteen, with fifteen flow/noise samples per example and a low cosine-scheduled learning rate. Inference uses twenty Euler integration steps to predict thirty actions. The first twenty-six are interpolated and resampled into twenty control commands. Map tokens are recomputed once per chunk, not at every low-level action, and complete end-to-end mapping/control latency is not reported.
Appendix G Map Token Ablation
On Task 22, full SERF scores 60.1% progress. Removing base tokens gives 56.5%, gripper tokens 54.0%, robot-only 54.0%, environment-only 58.5%, and global 55.1%. All group-removal means are lower, with substantial reported dispersion. This shows utility of the chosen groups within this configuration, without establishing a universal optimal token allocation.
Appendix H BEHAVIOR-1K Benchmark Details
Toy collection has seven targets in a 4.4-by-3.0-meter region; shoe organization has four items in 5.5 by 9.0 meters; gift baskets contain twenty target instances in 7.6 by 8.8 meters. Maximum rollouts are 38,372, 15,384, and 52,120 steps, twice average demonstration length. Symbolic conditions require toys in one bookcase, footwear off the floor and grouped by category, and a specified set of contents in each basket. Partial completion of these predicates determines the progress metric.
| Task | Target objects | Workspace () | Max steps |
| Task 21 | 2 dice, 2 teddy bears, 2 board games, 1 toy train | m | 38,372 |
| Task 22 | 2 gym shoes, 2 sandals | m | 15,384 |
| Task 26 | 4 baskets, 4 candles, 4 butter cookies, 4 Swiss cheeses, 4 bows | m | 52,120 |
Appendix I Comparison with Image-History Baseline
An image-history baseline adds eight tokens from eight past frames using a history module based on HAMLET [16]. Its progress is 50.0%, 49.5%, and 48.1%, improving over the image-only model on two tasks but remaining below SERF on all three. The comparison tests this particular fixed-stride history design, not all possible recurrent or retrieval-based memory systems.
Appendix J Task Progress Curves
Progress curves for Tasks 21 and 22 average ten configurations, fewer than the twenty used for the main table. SERF has the largest area under the progress-versus-step curve and highest final progress. This supports earlier subgoal attainment in those evaluations; the area is an execution-step measure rather than a hardware wall-clock throughput benchmark.


Appendix K Scene-Configuration Experiments
The moved-goal test relocates the toy bookcase without changing targets or language. The extra-object test adds two teddy bears, increasing targets from seven to nine and explicitly updating the symbolic task goal. The unvisited-region test moves two sandals outside demonstrated target locations within the same scene. Policies are trained on original configurations only, but SERF still has the prior/updated map and task-specified object identities; an unvisited demonstration region should not be read as an entirely unknown unmapped world.
Appendix L Additional Qualitative Results
Additional matched rollout comparisons show image-only policies stalling after targets leave view, while SERF often follows more direct paths and completes more subgoals. All three task sequences and their timing labels are retained. These illustrative trajectories accompany, rather than replace, the aggregate progress measurements.



Appendix M Additional Map Visualizations
The final figure pairs third-person execution images with corresponding evolving feature maps for all three tasks. The visualizations retain the robot and environment in their shared feature representation, showing tracked object movement and articulated robot updates across the rollout.