Reading guide
This edition covers all twelve pages, keeping both complete diagrams, all three tables, the metric definitions, three numbered losses and 34 references. The main contribution is to evaluate and improve the endpoint after coarse navigation: reaching a room, approaching an object’s navigable neighborhood, seeing it and detecting it are different outcomes. Read the training-only visibility supervision, fixed upstream scores and small real-robot test with those distinctions in mind.
At a glance
REALM takes over when an upstream language-guided navigator stops, using an adapted video-based policy to improve the final approach and viewpoint. REVERIE-AIM adds instance-centered endpoints and approximately 180,000 short-horizon training samples. Four backbone comparisons improve proximity, visibility and detection, although absolute grounding remains low. The stopping method is a supervised training loss, not an execution-time oracle that guarantees visibility. Hardware results are preliminary evidence from twelve episodes, with a different distance metric from simulation.
1 Introduction
Conventional three-meter arrival can count a stop where the named object is distant, hidden or behind the camera. The paper separates these failures with instance proximity, final-view detection and visibility metrics, then adds a refinement stage after navigation. The module only needs RGB and instructions itself; upstream navigators may still use depth, maps or other inputs. Compatibility with four backbones is useful evidence of modularity, while arbitrary wrong-room recovery and downstream manipulation readiness remain unproven.
2 Related Work
Continuous VLN, referring-object navigation and last-mile approaches address related parts of the problem. REALM emphasizes changing final-stage evaluation and finding the instance from free-form instructions, rather than assuming a known goal image or an already localized object. Its OWLv2 stage predicts a box instead of choosing solely among pre-annotated candidates. The paper’s comparison matrix describes the selected benchmarks, not a universal completeness or priority proof for every navigation dataset.
3 The Last-3-Meter Grounding Problem
The problem is to turn a coarse navigation endpoint into a viewpoint that supports instance-level use. Position near an annotated waypoint, position near the object’s navigable neighborhood, target visibility and successful detection are distinct axes. Neither final-view boxes nor short-range navigation alone certify reachability, contact feasibility or completion of an instruction such as cleaning the object.
3.1 Preliminaries
The upstream policy receives its own available observations and instruction, then stops at a position and heading. A detector must locate the specific referred object in that final view. REALM inserts its refinement before this final detection. The policy’s sensing configuration and the detector’s ability both influence success; the final refinement policy is not the entire navigation-and-grounding system.
Original mathematical definitions, training conditions and metric context
For the REVERIE-CE [15] task, at each step , the navigation policy receives an observation , which may consist of various sensory inputs depending on the system design, including egocentric or panoramic RGB-D views, RGB-only inputs, or other modality-specific features. The agent state at time is defined as , where denotes the 3D position and denotes the heading direction. Given a high-level natural language instruction , the agent is required to navigate in an unseen indoor environment and reach the goal location. After issues a stop command at step , the agent needs localise the referred target object within its final observation by predicting a bounding box :
3.2 Problem Formulation: The Last-3-Meter Grounding Gap
Legacy SR checks distance to a pre-annotated endpoint within three meters; it does not directly test the object or heading. The proposed gap is the difference between that arrival criterion and a state from which the specific object is near and visually groundable. The figure illustrates the failure modes, while the actual metrics use navigable sampling points and image boxes rather than a separate angular-error threshold or physical operability test.
Original mathematical definitions, training conditions and metric context
In standard VLN-CE evaluation, an episode is deemed successful if the agent’s stopping position lies within 3 meters of a pre-annotated endpoint :
Original mathematical definitions, training conditions and metric context
However, this paradigm fails in instance-level language instructions [19, 15] because is merely a neighbouring waypoint rather than the physical target. For practical embodied instruction following, the final state must satisfy two conditions: (i) Physical proximity to the actual target instance, and (ii) Visual groundability in the egocentric view with sufficient resolution.
3.3 Evaluation Metrics
The three metrics are reported separately over episodes. ONS measures proximity, GS measures detector overlap and OracleGS measures target visibility with an area threshold. They are not one combined ‘ready to interact’ score, and improving one does not automatically satisfy the others. Evaluation should retain their thresholds and denominators.
Object Navigation Success (ONS).
ONS is success within 0.1 or 0.5 meters geodesically of any sampled navigable point surrounding the specified object. This is distance to that point set, not necessarily ten centimeters from the physical object surface, nor a guarantee of manipulation reach. The object-instance identity matters, rather than any member of its category. Simulator point-set construction therefore helps define what counts as close.
Original selectable metric definition
Equation
Original mathematical definitions, training conditions and metric context
where is the set of navigable points sampled around the target instance and is geodesic distance. Unlike ObjectNav, we require proximity to the specified instance rather than any category match. We report m (standard ObjectNav) and m (for downstream positioning tolerance). Note that VLN-CE’s SR measures alignment with the annotated endpoint, whereas ONS directly measures proximity to the target object.
Grounding Success (GS).
GS tests predicted versus ground-truth box intersection-over-union at least 0.5 in the final view, unconditionally on navigation success. It can reveal detection failure even after a useful approach, and can count correct visual localization independently of the proximity threshold. The result includes the BERT phrase extractor and OWLv2 detector, so it is not a pure navigation metric.
Original mathematical definitions, training conditions and metric context
Grounding success is evaluated unconditionally, independent of whether navigation succeeded. Given the predicted bounding box produced by the detection module at the agent’s final viewpoint, and the ground-truth bounding box , we define:
Original selectable metric definition
Equation
Oracle Grounding Success (OracleGS).
OracleGS requires the true target to be visible and its box to cover at least 1% of the frame. This uses annotation-based visibility, not the detector. The added size threshold differs from GS: a correctly detected smaller box could satisfy GS while failing OracleGS, so OracleGS is a useful visibility diagnostic rather than an unconditional mathematical ceiling on GS. Comparing 34.67% legacy SR with 6.32% ONS shows at least about 81.8% of those SR successes cannot also meet ONS, even under maximal overlap; that is a proximity mismatch, not a direct measurement of the full joint grounding failure rate.
Original mathematical definitions, training conditions and metric context
An episode achieves oracle grounding success if the ground-truth bounding box of the target instance is visible in the agent’s current observation and occupies at least 1% of the camera frame area:
Original selectable metric definition
Equation
4 The REALM
The pipeline decouples long-range navigation, local refinement and phrase-conditioned box detection. An upstream stop triggers the adapted policy; the original instruction and new egocentric frames guide further moves until it stops again. This avoids editing each backbone, but requires an additional trained model and execution budget. The paper does not fully specify a recovery guarantee or refinement-step/latency budget when the handoff is poor.
Original mathematical definitions, training conditions and metric context
We propose REALM, a plug-and-play decoupled refinement module to solve the above last-3-meters grounding problem. The complete pipeline has three stages (Figure 2): (i) Navigation— follows and stops at ; (ii) Refinement— repositions and reorients the agent to a refined state where the target is both proximate and visible; (iii) Grounding—a detection module predicts the bounding box over the final observation. To maximize efficiency and flexibility, operates fully independently of the upstream navigation policy . This decoupled design avoids performance trade-offs between long-horizon path planning (tens of meters) and short-horizon precision approaching, while ensuring seamless plug-and-play compatibility across diverse VLN backbones (e.g., graph-based, video-based, or zero-shot paradigms).
4.1 Refinement Policy
UniNaVid predicts four actions from video and language over forward, left, right and stop. LoRA adapts all language-backbone linear layers while the vision encoder stays frozen. Cross-entropy is averaged only over action positions, with non-action tokens ignored. This is supervised adaptation on short-horizon data, even when the upstream backbone is labeled zero-shot or training-free; the whole augmented system is not training-free.
Original mathematical definitions, training conditions and metric context
We instantiate the refinement policy with UniNaVid [31], a video-based Vision-Language-Action (VLA) foundation model that maps egocentric video streams and language instructions to multi-step action sequences. We choose UniNaVid for two reasons: its video-level temporal encoding naturally suits the sequential observations arising in short-horizon refinement, and its pre-trained visual-linguistic representations provide a strong initialisation that reduces the data requirement for downstream adaptation. At each decision step , the model consumes the egocentric RGB sequence together with the instruction , and predicts the next four-step action sequence over the low-level action space .
Original mathematical definitions, training conditions and metric context
During training, non-action positions (e.g., instruction and visual tokens) are masked with the ignore index . Let denote the set of valid action token positions across a batch of size with token sequence length . The baseline imitation learning objective minimizing token-level cross-entropy is:
Original selectable equation
Equation 1
| (1) |
Original mathematical definitions, training conditions and metric context
where and represent the target tokens and predicted logits at batch index and token position , respectively.
Original mathematical definitions, training conditions and metric context
We apply LoRA [9] to all linear layers of the LLM backbone while keeping the vision encoder frozen. However, directly optimizing suffers from severe trajectory biases, causing the agent to stop prematurely. We address these failure modes via a visibility-aware stopping strategy.
Visibility-Aware Stop Penalty.
VSP uses simulator target-visibility labels during training. On invisible examples it penalizes a positive stop-logit margin over the strongest non-stop alternative, and averages over the invisible examples with a denominator clamped to one. It applies at the first valid decision in the four-action prediction, not all four positions as an independent guarantee. A zero/negative margin adds no penalty, and visible examples receive none. This changes learned stopping preferences; there is no described inference-time ground-truth gate that ensures the target is visible before stopping.
Original mathematical definitions, training conditions and metric context
Concretely, let be the logit vector at the first valid decision step (i.e., of the four-step sequence predicted at step ) for sample , with the action vocabulary, and let indicate target visibility from simulator annotations. We define the stop margin to quantify how strongly the model favors stopping over all alternative low-level actions: which is positive exactly when stop is the greedy action. To selectively punish incorrect termination without hindering correct stops, the penalty averages the positive part of this margin over all target-invisible samples:
Original selectable equation
Equation 2
| (2) |
Original mathematical definitions, training conditions and metric context
where , is the indicator function, and the denominator is clamped to at least for numerical stability. This leaves the agent free to stop once the target is perceived, while suppressing premature termination during exploration.
Total Training Objective.
The total objective adds the weighted visibility penalty to imitation loss. The retained formulas specify masking, normalization and margin behavior. Numerical penalty strength, LoRA rank, optimization schedule and inference budget are not fully provided in this twelve-page version, limiting exact reproduction. The ablation measures the full trained pipeline with and without VSP rather than proving an optimal stopping rule.
Original selectable equation
Equation 3
| (3) |
Original mathematical definitions, training conditions and metric context
where controls the penalty strength relative to the imitation signal.
4.2 Target Expression Extraction and Visual Grounding
GPT-4o-mini produces target-phrase annotations, and BERT-Large is adapted on 2,048 curated instruction–phrase pairs. At inference the extracted span queries OWLv2 on the refined final RGB frame. This removes distracting route/action words but introduces phrase-extraction and open-vocabulary detection failure modes. No separate phrase-extraction accuracy or detector-calibration study is reported; OracleGS versus GS helps diagnose visibility and detection collectively.
Original mathematical definitions, training conditions and metric context
After the refinement policy terminates at , the agent must localise the target object in its final observation. However, REVERIE-AIM instructions often contain both navigation cues and object descriptions, e.g., “go to the bathroom and clean the toilet seat”. Directly using the full instruction as an open-vocabulary detector query may introduce irrelevant spatial or action-related words. We therefore first extract a target phrase from the instruction and then use it for visual grounding.
Original mathematical definitions, training conditions and metric context
For target expression extraction, we use GPT-4o-mini to automatically annotate target noun phrases, and fine-tune a BERT-Large [7] extractive QA model on 2,048 curated instruction–phrase pairs. At inference time, the BERT model extracts the target span from the instruction , . For visual grounding, the extracted is passed to OWLv2 [18] as the text query to predict the final bounding box , , where is the agent’s final egocentric observation. This design removes irrelevant navigation cues from the grounding query.
5 The REVERIE-AIM Dataset
REVERIE-AIM reconstructs continuous trajectories, aligns endpoint annotations to specific object instances and generates local approach examples. It changes both training targets and evaluation criteria. The benchmark is built from reconstructed indoor scans and simulator annotations; it is not a new 180,000-episode real-robot dataset or proof of successful downstream interaction.
Accessible original table, all values and grouped headers
| Dataset | Continuous | Language | Instance-level | Object Proximity |
| Space | Goal | Target | Endpoint | |
| Vision-and-Language Navigation (Discrete) | ||||
| R2R [2] | ||||
| RxR [14] | ||||
| REVERIE [19] | ||||
| Vision-and-Language Navigation (Continuous) | ||||
| R2R-CE [13] | ||||
| RxR-CE [14, 13] | ||||
| REVERIE-CE [15] | ||||
| Object Navigation | ||||
| MP3D ObjNav [3] | ||||
| HM3D ObjNav [29] | ||||
| HM3D ImageNav [30] | ||||
| Ours | ||||
5.1 Continuous Trajectory Reconstruction
Discrete REVERIE graph trajectories are mapped and interpolated in Habitat into low-level movement/turn sequences. This supports continuous-space evaluation and collision-aware route reconstruction, while relying on the scan geometry, navigable mesh and transfer procedure. Interpolation does not establish that every route is physically executable on arbitrary robots or robust to real controller/perception error.
5.2 Object-Centric Goal Alignment
Shared Matterport3D instance identifiers connect REVERIE targets with ObjectNav sampling locations. The new endpoints refer to the object’s navigable or visually observable neighborhood rather than an arbitrary nearby waypoint. This improves annotation precision but does not turn the sampled position into the object surface or guarantee a useful final orientation. Instance mapping and sampling remain part of the benchmark definition.
5.3 Dataset Statistics
The split contains 3,691 training trajectories in 53 scenes/90 categories and 1,344 unseen-validation trajectories in 10 scenes/48 categories. Average valid endpoint-to-target geodesic distance changes 1.32→0.34 meters under the alignment procedure. These are benchmark annotation statistics, not the learned policy’s navigation error. The table retains the selected datasets’ continuous-space, language, instance-target and object-proximity properties.
Original mathematical definitions, training conditions and metric context
Our object-centric goal alignment directly bridges the region-to-object grounding gap. While standard REVERIE-CE endpoints terminate at a loose, region-level distance, REVERIE-AIM reduces the valid average minimum geodesic distance to the target from m to just m. This finer-grained spatial resolution enables more precise evaluation of both navigation and instance-level grounding.
5.4 Short-Horizon Training Data Generation
The local dataset augments starts near the reference route’s penultimate node, then records shortest-path action/video supervision toward an object-centered goal. Approximately180,000 resulting samples come from the training split. They train recovery from nearby pose/heading variation; they do not independently establish recovery from every upstream stopping error or a wrong-room handoff.
Task-specific Initial Sampling
Start positions use truncated Gaussian perturbations with standard deviation 1.5 meters and coordinate truncation within ±5 meters, plus headings over the full circle. The reference-path anchor supplies privileged near-goal training placement. The accepted navigable-start filtering and resulting difficulty distribution are not fully quantified, so the nominal perturbation range is not a uniform test of all errors within five meters.
Original mathematical definitions, training conditions and metric context
Spatial Perturbation. We perturb the anchor position using a truncated Gaussian distribution (, truncated within meter) to generate spatially diverse starting states around the target object.
Original mathematical definitions, training conditions and metric context
Viewpoint Augmentation. We randomly sample the initial heading angle within to simulate diverse target-approaching viewpoints.
Clip and Action Sequence Extraction
For valid starts, shortest paths to object-associated sampling locations generate video/action tuples over forward, left, right and stop. This is dense imitation supervision for the local phase, not observed human recovery behavior or outcome-labeled interaction. Shortest-path geometry and simulator visibility provide training information absent from the RGB-only deployed module.
Original mathematical definitions, training conditions and metric context
For each valid sampled starting point, we compute the shortest path to the target object’s surrounding sampling location, determined by the object-centric endpoint described in Section 5.2. We then extract the corresponding visual observation sequences (video clips) together with their continuous action labels. The action labels are a set of low-level actions: . In total, the resulting dataset contains approximately 180,000 fine-grained navigation samples from the training split. These samples provide dense supervision for learning precise short-horizon navigation behaviors, particularly during the final target-approaching stage.
6 Experiments
The main test combines four upstream backbones with six local movement variants on unseen-validation scenes, followed by twelve real episodes. Keep upstream legacy navigation scores separate from the post-refinement instance scores. The complete tables preserve all baselines and the human reference; no additional seed repeats, significance tests or timing results are inferred.
6.1 Experimental Setup
Comparisons include doing nothing, random movement, a CLIP-based local search, unadapted UniNaVid, imitation adaptation without VSP and the full loss. The setup spans different navigator families but reuses a particular adapted refinement architecture. Key control/action step sizes, rollout caps, training settings and runtime costs are not fully enumerated, which limits budget-matched interpretation of the extra local stage.
Upstream VLN Baselines
ETPNav-ZS, ETPNav-FT, UniNaVid-ZS and Smartway cover graph/video and zero-shot/fine-tuned/training-free upstream policies. Their labels describe those backbones. REALM adds supervised local adaptation and a trained phrase extractor, so the augmented zero-shot or training-free backbone should not be mistaken for a wholly untrained system.
Downstream Variants
The no-move baseline isolates raw upstream endpoints; random walk and CLIP search test simpler local actions; vanilla UniNaVid separates a pretrained local policy from adaptation. Removing VSP retains supervised training on the new data. These variants help attribute improvements, while unspecified action/time budgets and detector-query details limit claims that all alternatives receive identical compute or exploration opportunities.
Original mathematical definitions, training conditions and metric context
To verify the components and optimization of our refinement module, we compare six downstream variants: (1)w/o Refinement, which directly uses the upstream policy’s raw endpoints; (2) Random Walk, which performs random movements at the destination; (3) Heuristic CLIP-Search, which spins and moves toward the direction with the highest CLIP matching score; (4) Vanilla UniNaVid, the training-free base architecture executing raw upstream endpoints directly; (5) Ours w/o VSP, our model optimized on our data using standard cross-entropy loss; and (6) Ours (Full), our complete module optimized on our data by the full tailored loss function.
Standard VLN-CE Metrics
Trajectory length, navigation error, oracle success, legacy success and path-length-weighted success are printed once per backbone. They are fixed upstream reference numbers in Table 2, not newly measured post-refinement trajectory metrics. ONS, GS and OracleGS assess the later endpoint. The table therefore does not establish that refinement preserves legacy path efficiency or has negligible additional travel/runtime.
Accessible original table, all values and grouped headers
Backbone
Move Policy
ONS@0.1m
ONS@0.5m
GS
OracleGS
ETPNav-ZS
TL = 11.85, NE = 7.26, OSR = 35.86, SPL = 23.39, SR = 28.94
ETPNav-ZS
— (no move)
7.07
11.98
3.05
6.99
Random
5.21
10.42
1.93
4.61
Heuristic
7.51
12.57
4.02
10.19
Vanilla UniNavid
11.16
17.78
4.54
11.90
Ours w/o VSP
13.47
20.24
5.28
15.55
Ours (Full)
14.66
21.13
6.10
16.52
ETPNav-FT
TL = 13.86, NE = 6.71, OSR = 40.63, SPL = 27.68, SR = 34.67
ETPNav-FT
— (no move)
6.32
12.20
4.46
11.31
Random
5.65
12.95
2.75
8.78
Heuristic
6.77
13.24
5.88
12.05
Vanilla UniNavid
13.39
19.12
4.91
12.57
Ours w/o VSP
16.82
25.00
8.04
19.79
Ours (Full)
17.19
25.89
8.48
20.24
UniNavid-ZS
TL = 16.07, NE = 8.69, OSR = 34.40, SPL = 17.40, SR = 25.10
UniNavid-ZS
— (no move)
13.61
18.08
5.65
12.20
Random
9.75
16.89
2.83
8.78
Heuristic
12.35
18.15
5.13
10.42
Vanilla UniNavid
10.42
16.07
4.84
11.01
Ours w/o VSP
12.43
18.38
5.95
14.29
Ours (Full)
14.96
20.68
6.85
16.37
Smartway
TL = 20.00, NE = 8.79, OSR = 25.37, SPL = 4.81, SR = 8.21
Smartway
— (no move)
7.81
11.83
1.49
4.69
Random
5.21
10.42
1.34
4.61
Heuristic
7.22
11.98
2.90
8.04
Vanilla UniNavid
9.30
14.81
3.50
9.45
Ours w/o VSP
12.95
17.49
4.69
14.66
Ours (Full)
13.99
19.72
5.65
15.92
Human Eval
50.40
74.71
38.24
80.38
6.2 Results in Simulator
Full REALM leads the listed local variants on all four instance-metric columns within every backbone. ETPNav-FT improves ONS@0.1m 6.32→17.19% and GS 4.46→8.48%, gains 10.87 and 4.02 percentage points. ETPNav-ZS 7.07→14.66% is about 107.4% relative, but the resulting absolute rate is still 14.66%. On UniNaVid-ZS the no-refinement starting point 13.61% is already stronger than vanilla local UniNaVid’s 10.42%; full REALM reaches 14.96%, only 1.35 points above no refinement. Removing VSP there gives 12.43%, making the VSP ablation gain 2.53 points.
The human reference reaches 74.71% ONS@0.5m versus the best full model’s 25.89%, and human GS is 38.24%. Human protocol, seed variance and statistical tests are not detailed; ‘significantly’ in the prose should not imply a demonstrated significance test. The large remaining visibility/detection gaps limit claims of a solved interaction interface.
Original mathematical definitions, training conditions and metric context
As shown in Table 2, our full module (Ours (Full)) consistently and significantly outperforms all alternative move policies across all four backbones. For instance, on ETPNav-FT, it boosts ONS@0.1m from (no move) to and GS from to , demonstrating its superior capability in fine-grained object localization. The ablation variant (Ours w/o VSP) incurs consistent performance drops across all settings (e.g., ONS@0.1m falls by on UniNavid-ZS), validating the necessity of our proposed Visibility-Aware Stopping loss. Nevertheless, a substantial performance gap remains compared to Human Eval (e.g., vs. on ONS@0.5m), highlighting that this benchmark leaves massive room for future improvement.
6.3 Results in Real-world Environments
On Stretch with a RealSense camera and RTX 4090 workstation, twelve multi-room episodes compare UniNaVid with and without refinement. ONS@0.5m rises 1/12→4/12, GS 1/12→3/12 and OracleGS 2/12→3/12. Those small counts support a preliminary improvement, with no uncertainty or paired-transition analysis reported. The hardware ONS uses Euclidean distance to the nearest navigable object-associated position, unlike simulation’s geodesic metric. The prose calls it SR, but it is not the legacy three-meter waypoint SR. RGB-D camera hardware does not change the stated RGB-only refinement input, and no manipulation task or guaranteed visibility/safety outcome is measured.
7 Conclusion and Limitations
The study makes final-view object grounding a clearer target and shows a modular local refinement can improve several existing navigators. It leaves substantial absolute failures, especially relative to the human reference. Future work needs active viewpoint selection, reasoning under partial observability and reliable stopping, alongside clearer budgets and broader real-world evaluation. The reported system reduces the gap; it does not certify interaction readiness for arbitrary upstream endpoints.