Literature
Display
← All papers

From Region Arrival to Instance-Level Grounding in Vision-and-Language Navigation

Xiangyu Shi · Ruoxi Yang · Wei Tao · Jiwen Zhang · Yanyuan Qiao · Qi Wu

Preprint posted · CoRL 2026 (accepted) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Reading guide

This edition covers all twelve pages, keeping both complete diagrams, all three tables, the metric definitions, three numbered losses and 34 references. The main contribution is to evaluate and improve the endpoint after coarse navigation: reaching a room, approaching an object’s navigable neighborhood, seeing it and detecting it are different outcomes. Read the training-only visibility supervision, fixed upstream scores and small real-robot test with those distinctions in mind.

At a glance

REALM takes over when an upstream language-guided navigator stops, using an adapted video-based policy to improve the final approach and viewpoint. REVERIE-AIM adds instance-centered endpoints and approximately 180,000 short-horizon training samples. Four backbone comparisons improve proximity, visibility and detection, although absolute grounding remains low. The stopping method is a supervised training loss, not an execution-time oracle that guarantees visibility. Hardware results are preliminary evidence from twelve episodes, with a different distance metric from simulation.

1 Introduction

Conventional three-meter arrival can count a stop where the named object is distant, hidden or behind the camera. The paper separates these failures with instance proximity, final-view detection and visibility metrics, then adds a refinement stage after navigation. The module only needs RGB and instructions itself; upstream navigators may still use depth, maps or other inputs. Compatibility with four backbones is useful evidence of modularity, while arbitrary wrong-room recovery and downstream manipulation readiness remain unproven.

Complete original Figure 1: Existing evaluation rewards stopping near a target viewpoint, but object grounding requires a final viewpoint where the referred object is actually visible and localizable.
Figure 1: Existing evaluation rewards stopping near a target viewpoint, but object grounding requires a final viewpoint where the referred object is actually visible and localizable.

2 Related Work

Continuous VLN, referring-object navigation and last-mile approaches address related parts of the problem. REALM emphasizes changing final-stage evaluation and finding the instance from free-form instructions, rather than assuming a known goal image or an already localized object. Its OWLv2 stage predicts a box instead of choosing solely among pre-annotated candidates. The paper’s comparison matrix describes the selected benchmarks, not a universal completeness or priority proof for every navigation dataset.

3 The Last-3-Meter Grounding Problem

The problem is to turn a coarse navigation endpoint into a viewpoint that supports instance-level use. Position near an annotated waypoint, position near the object’s navigable neighborhood, target visibility and successful detection are distinct axes. Neither final-view boxes nor short-range navigation alone certify reachability, contact feasibility or completion of an instruction such as cleaning the object.

3.1 Preliminaries

The upstream policy receives its own available observations and instruction, then stops at a position and heading. A detector must locate the specific referred object in that final view. REALM inserts its refinement before this final detection. The policy’s sensing configuration and the detector’s ability both influence success; the final refinement policy is not the entire navigation-and-grounding system.

Original mathematical definitions, training conditions and metric context

For the REVERIE-CE [15] task, at each step tt, the navigation policy πnav\pi_{\text{nav}} receives an observation 𝒪t\mathcal{O}_{t}, which may consist of various sensory inputs depending on the system design, including egocentric or panoramic RGB-D views, RGB-only inputs, or other modality-specific features. The agent state at time tt is defined as st=(pt,θt)s_{t}=(p_{t},\theta_{t}), where pt∈ℝ3p_{t}\in\mathbb{R}^{3} denotes the 3D position and θt\theta_{t} denotes the heading direction. Given a high-level natural language instruction ℐ\mathcal{I}, the agent is required to navigate in an unseen indoor environment ℰ\mathcal{E} and reach the goal location. After πnav\pi_{\text{nav}} issues a stop command at step Tnav{T_{\text{nav}}}, the agent needs localise the referred target object o∗o^{*} within its final observation 𝒪Tnav\mathcal{O}_{T_{\text{nav}}} by predicting a bounding box b^\hat{b}: b^=Det⁡(𝒪Tnav,ℐ)\hat{b}=\mathrm{Det}(\mathcal{O}_{T_{\text{nav}}},\,\mathcal{I})

3.2 Problem Formulation: The Last-3-Meter Grounding Gap

Legacy SR checks distance to a pre-annotated endpoint within three meters; it does not directly test the object or heading. The proposed gap is the difference between that arrival criterion and a state from which the specific object is near and visually groundable. The figure illustrates the failure modes, while the actual metrics use navigable sampling points and image boxes rather than a separate angular-error threshold or physical operability test.

Original mathematical definitions, training conditions and metric context

In standard VLN-CE evaluation, an episode is deemed successful if the agent’s stopping position pTnavp_{T_{\text{nav}}} lies within 3 meters of a pre-annotated endpoint pepp_{\text{ep}}: Success Rate (SR)=[d(pTnav,pep)≤3m].\text{Success Rate (SR)}=\left[d(p_{T_{\text{nav}}},\,p_{\text{ep}})\leq 3\,\text{m}\right].

Original mathematical definitions, training conditions and metric context

However, this paradigm fails in instance-level language instructions [19, 15] because pepp_{\text{ep}} is merely a neighbouring waypoint rather than the physical target. For practical embodied instruction following, the final state sTn​a​v=(pTnav,θTnav)s_{T_{nav}}=(p_{T_{\text{nav}}},\theta_{T_{\text{nav}}}) must satisfy two conditions: (i) Physical proximity to the actual target instance, and (ii) Visual groundability in the egocentric view with sufficient resolution.

3.3 Evaluation Metrics

The three metrics are reported separately over episodes. ONS measures proximity, GS measures detector overlap and OracleGS measures target visibility with an area threshold. They are not one combined ‘ready to interact’ score, and improving one does not automatically satisfy the others. Evaluation should retain their thresholds and denominators.

Object Navigation Success (ONS).

ONS is success within 0.1 or 0.5 meters geodesically of any sampled navigable point surrounding the specified object. This is distance to that point set, not necessarily ten centimeters from the physical object surface, nor a guarantee of manipulation reach. The object-instance identity matters, rather than any member of its category. Simulator point-set construction therefore helps define what counts as close.

Complete original metric definition S3.Ex1
Original selectable metric definition
Equation
ONS@τ=[minq∈𝒮objd(pTnav,q)≤τ],\text{ONS}@\tau=\!\left[\min_{q\in\mathcal{S}_{\text{obj}}}d(p_{T_{\text{nav}}},\,q)\leq\tau\right],
Original mathematical definitions, training conditions and metric context

where 𝒮obj\mathcal{S}_{\text{obj}} is the set of navigable points sampled around the target instance and d⁡(⋅,⋅)d(\cdot,\cdot) is geodesic distance. Unlike ObjectNav, we require proximity to the specified instance rather than any category match. We report τ=0.1\tau{=}0.1 m (standard ObjectNav) and τ=0.5\tau{=}0.5 m (for downstream positioning tolerance). Note that VLN-CE’s SR measures alignment with the annotated endpoint, whereas ONS directly measures proximity to the target object.

Grounding Success (GS).

GS tests predicted versus ground-truth box intersection-over-union at least 0.5 in the final view, unconditionally on navigation success. It can reveal detection failure even after a useful approach, and can count correct visual localization independently of the proximity threshold. The result includes the BERT phrase extractor and OWLv2 detector, so it is not a pure navigation metric.

Original mathematical definitions, training conditions and metric context

Grounding success is evaluated unconditionally, independent of whether navigation succeeded. Given the predicted bounding box b^\hat{b} produced by the detection module at the agent’s final viewpoint, and the ground-truth bounding box b∗b^{*}, we define:

Complete original metric definition S3.Ex2
Original selectable metric definition
Equation
GS=[IoU(b^,b∗)≥0.5].\text{GS}=\!\left[\text{IoU}(\hat{b},\,b^{*})\geq 0.5\right].

Oracle Grounding Success (OracleGS).

OracleGS requires the true target to be visible and its box to cover at least 1% of the frame. This uses annotation-based visibility, not the detector. The added size threshold differs from GS: a correctly detected smaller box could satisfy GS while failing OracleGS, so OracleGS is a useful visibility diagnostic rather than an unconditional mathematical ceiling on GS. Comparing 34.67% legacy SR with 6.32% ONS shows at least about 81.8% of those SR successes cannot also meet ONS, even under maximal overlap; that is a proximity mismatch, not a direct measurement of the full joint grounding failure rate.

Original mathematical definitions, training conditions and metric context

An episode achieves oracle grounding success if the ground-truth bounding box b∗b^{*} of the target instance is visible in the agent’s current observation 𝒪T\mathcal{O}_{T} and occupies at least 1% of the camera frame area:

Complete original metric definition S3.Ex3
Original selectable metric definition
Equation
OracleGS=[Area​(b∗)Area​(𝒪T)≥0.01].\text{OracleGS}=\left[\frac{\text{Area}(b^{*})}{\text{Area}(\mathcal{O}_{T})}\geq 0.01\right].

4 The REALM

The pipeline decouples long-range navigation, local refinement and phrase-conditioned box detection. An upstream stop triggers the adapted policy; the original instruction and new egocentric frames guide further moves until it stops again. This avoids editing each backbone, but requires an additional trained model and execution budget. The paper does not fully specify a recovery guarantee or refinement-step/latency budget when the handoff is poor.

Complete original Figure 2: An overview of the REALM framework. The pipeline comprises three decoupled stages: (i) Navigation : π nav \pi_{\mathrm{nav}} follows instruction ℐ \mathcal{I} (e.g., “Go to the living room and water the flowers on the table.” ) and stops at s T nav s_{T_{\text{nav}}} ; (ii) Refinement : π ref \pi_{\mathrm{ref}} repositions the agent to s T ref s_{T_{\text{ref}}} where the target is proximate and visible; (iii) Grounding : a BERT-based extractor yields a target phrase r ^ \hat{r} , passed to OWLv2 for bounding box prediction. The refinement module is plug-and-play across arbitrary VLN backbones.
Figure 2: An overview of the REALM framework. The pipeline comprises three decoupled stages: (i) Navigation: πnav\pi_{\mathrm{nav}} follows instruction ℐ\mathcal{I} (e.g., “Go to the living room and water the flowers on the table.”) and stops at sTnavs_{T_{\text{nav}}}; (ii) Refinement: πref\pi_{\mathrm{ref}} repositions the agent to sTrefs_{T_{\text{ref}}} where the target is proximate and visible; (iii) Grounding: a BERT-based extractor yields a target phrase r^\hat{r}, passed to OWLv2 for bounding box prediction. The refinement module is plug-and-play across arbitrary VLN backbones.
Original mathematical definitions, training conditions and metric context

We propose REALM, a plug-and-play decoupled refinement module to solve the above last-3-meters grounding problem. The complete pipeline has three stages (Figure 2): (i) Navigation—πnav\pi_{\mathrm{nav}} follows ℐ\mathcal{I} and stops at sTnavs_{T_{\text{nav}}}; (ii) Refinement—πref\pi_{\mathrm{ref}} repositions and reorients the agent to a refined state sTrefs_{T_{\text{ref}}} where the target is both proximate and visible; (iii) Grounding—a detection module predicts the bounding box over the final observation. To maximize efficiency and flexibility, πref\pi_{\mathrm{ref}} operates fully independently of the upstream navigation policy πnav\pi_{\mathrm{nav}}. This decoupled design avoids performance trade-offs between long-horizon path planning (tens of meters) and short-horizon precision approaching, while ensuring seamless plug-and-play compatibility across diverse VLN backbones (e.g., graph-based, video-based, or zero-shot paradigms).

4.1 Refinement Policy

UniNaVid predicts four actions from video and language over forward, left, right and stop. LoRA adapts all language-backbone linear layers while the vision encoder stays frozen. Cross-entropy is averaged only over action positions, with non-action tokens ignored. This is supervised adaptation on short-horizon data, even when the upstream backbone is labeled zero-shot or training-free; the whole augmented system is not training-free.

Original mathematical definitions, training conditions and metric context

We instantiate the refinement policy πref\pi_{\text{ref}} with UniNaVid [31], a video-based Vision-Language-Action (VLA) foundation model that maps egocentric video streams and language instructions to multi-step action sequences. We choose UniNaVid for two reasons: its video-level temporal encoding naturally suits the sequential observations arising in short-horizon refinement, and its pre-trained visual-linguistic representations provide a strong initialisation that reduces the data requirement for downstream adaptation. At each decision step tt, the model consumes the egocentric RGB sequence 𝒪t\mathcal{O}_{t} together with the instruction ℐ\mathcal{I}, and predicts the next four-step action sequence at,1,at,2,at,3,at,4a_{t,1},a_{t,2},a_{t,3},a_{t,4} over the low-level action space 𝒜={forward,turn_left,turn_right,stop}\mathcal{A}=\{\texttt{forward},\,\texttt{turn\_left},\,\texttt{turn\_right},\,\texttt{stop}\}.

Original mathematical definitions, training conditions and metric context

During training, non-action positions (e.g., instruction and visual tokens) are masked with the ignore index −100-100. Let ℳ={(b,n):yb,n≠−100}\mathcal{M}=\{(b,n):y_{b,n}\neq-100\} denote the set of valid action token positions across a batch of size BB with token sequence length NN. The baseline imitation learning objective minimizing token-level cross-entropy is:

Complete original equation 1
Original selectable equation
Equation 1
ℒbase=1|ℳ|​∑(b,n)∈ℳℓCE​(𝐳^b,n,yb,n),\mathcal{L}_{\mathrm{base}}=\frac{1}{|\mathcal{M}|}\sum_{(b,n)\in\mathcal{M}}\ell_{\mathrm{CE}}\!\left(\hat{\mathbf{z}}_{b,n},\,y_{b,n}\right), (1)
Original mathematical definitions, training conditions and metric context

where yb,ny_{b,n} and 𝐳^b,n\hat{\mathbf{z}}_{b,n} represent the target tokens and predicted logits at batch index bb and token position nn, respectively.

Original mathematical definitions, training conditions and metric context

We apply LoRA [9] to all linear layers of the LLM backbone while keeping the vision encoder frozen. However, directly optimizing ℒbase\mathcal{L}_{\mathrm{base}} suffers from severe trajectory biases, causing the agent to stop prematurely. We address these failure modes via a visibility-aware stopping strategy.

Visibility-Aware Stop Penalty.

VSP uses simulator target-visibility labels during training. On invisible examples it penalizes a positive stop-logit margin over the strongest non-stop alternative, and averages over the invisible examples with a denominator clamped to one. It applies at the first valid decision in the four-action prediction, not all four positions as an independent guarantee. A zero/negative margin adds no penalty, and visible examples receive none. This changes learned stopping preferences; there is no described inference-time ground-truth gate that ensures the target is visible before stopping.

Original mathematical definitions, training conditions and metric context

Concretely, let 𝐳^b∈ℝ|𝒱|\hat{\mathbf{z}}_{b}\in\mathbb{R}^{|\mathcal{V}|} be the logit vector at the first valid decision step (i.e., at,1a_{t,1} of the four-step sequence at,1,at,2,at,3,at,4a_{t,1},a_{t,2},a_{t,3},a_{t,4} predicted at step tt) for sample bb, with 𝒱\mathcal{V} the action vocabulary, and let vb∈{0,1}v_{b}\in\{0,1\} indicate target visibility from simulator annotations. We define the stop margin to quantify how strongly the model favors stopping over all alternative low-level actions: mb=z^b,stop−maxk≠stop⁡z^b,k,m_{b}\;=\;\hat{z}_{b,\,\texttt{stop}}\;-\;\max_{k\neq\texttt{stop}}\,\hat{z}_{b,\,k}\,, which is positive exactly when stop is the greedy action. To selectively punish incorrect termination without hindering correct stops, the penalty averages the positive part of this margin over all target-invisible samples:

Complete original equation 2
Original selectable equation
Equation 2
ℒvsp=1Ninv∑b=1BReLU(mb)⋅𝕀[vb=0],\mathcal{L}_{\mathrm{vsp}}\;=\;\frac{1}{N_{\mathrm{inv}}}\sum_{b=1}^{B}\,\operatorname{ReLU}(m_{b})\;\cdot\;\mathbb{I}[v_{b}=0]\,, (2)
Original mathematical definitions, training conditions and metric context

where Ninv=max(∑b𝕀[vb=0], 1)N_{\mathrm{inv}}=\max\!\bigl(\sum_{b}\mathbb{I}[v_{b}{=}0],\;1\bigr), 𝕀⁡[⋅]\mathbb{I}[\cdot] is the indicator function, and the denominator is clamped to at least 11 for numerical stability. This leaves the agent free to stop once the target is perceived, while suppressing premature termination during exploration.

Total Training Objective.

The total objective adds the weighted visibility penalty to imitation loss. The retained formulas specify masking, normalization and margin behavior. Numerical penalty strength, LoRA rank, optimization schedule and inference budget are not fully provided in this twelve-page version, limiting exact reproduction. The ablation measures the full trained pipeline with and without VSP rather than proving an optimal stopping rule.

Complete original equation 3
Original selectable equation
Equation 3
ℒtotal=ℒbase+λvsp​ℒvsp,\mathcal{L}_{\mathrm{total}}\;=\;\mathcal{L}_{\mathrm{base}}\;+\;\lambda_{\mathrm{vsp}}\,\mathcal{L}_{\mathrm{vsp}}, (3)
Original mathematical definitions, training conditions and metric context

where λvsp\lambda_{\mathrm{vsp}} controls the penalty strength relative to the imitation signal.

4.2 Target Expression Extraction and Visual Grounding

GPT-4o-mini produces target-phrase annotations, and BERT-Large is adapted on 2,048 curated instruction–phrase pairs. At inference the extracted span queries OWLv2 on the refined final RGB frame. This removes distracting route/action words but introduces phrase-extraction and open-vocabulary detection failure modes. No separate phrase-extraction accuracy or detector-calibration study is reported; OracleGS versus GS helps diagnose visibility and detection collectively.

Original mathematical definitions, training conditions and metric context

After the refinement policy terminates at sTrefs_{T_{\text{ref}}}, the agent must localise the target object in its final observation. However, REVERIE-AIM instructions often contain both navigation cues and object descriptions, e.g., “go to the bathroom and clean the toilet seat”. Directly using the full instruction as an open-vocabulary detector query may introduce irrelevant spatial or action-related words. We therefore first extract a target phrase from the instruction and then use it for visual grounding.

Original mathematical definitions, training conditions and metric context

For target expression extraction, we use GPT-4o-mini to automatically annotate target noun phrases, and fine-tune a BERT-Large [7] extractive QA model on 2,048 curated instruction–phrase pairs. At inference time, the BERT model extracts the target span r^\hat{r} from the instruction ℐ\mathcal{I}, r^=BERTQ​A​(ℐ)\hat{r}=\mathrm{BERT}_{QA}(\mathcal{I}). For visual grounding, the extracted r^\hat{r} is passed to OWLv2 [18] as the text query to predict the final bounding box b^\hat{b}, b^=OWLV​2​(𝒪Tref,r^)\hat{b}=\mathrm{OWL}_{V2}(\mathcal{O}_{T_{\text{ref}}},\hat{r}), where OTrefO_{T_{\text{ref}}} is the agent’s final egocentric observation. This design removes irrelevant navigation cues from the grounding query.

5 The REVERIE-AIM Dataset

REVERIE-AIM reconstructs continuous trajectories, aligns endpoint annotations to specific object instances and generates local approach examples. It changes both training targets and evaluation criteria. The benchmark is built from reconstructed indoor scans and simulator annotations; it is not a new 180,000-episode real-robot dataset or proof of successful downstream interaction.

Complete original Table 1: Comparison with existing embodied navigation datasets. Our dataset is the first to combine all four desirable properties.
Table 1: Comparison with existing embodied navigation datasets. Our dataset is the first to combine all four desirable properties.
Accessible original table, all values and grouped headers
Dataset Continuous Language Instance-level Object Proximity
Space Goal Target Endpoint
Vision-and-Language Navigation (Discrete)
R2R [2] ×\bm{\times} ✓\checkmark ×\bm{\times} ×\bm{\times}
RxR [14] ×\bm{\times} ✓\checkmark ×\bm{\times} ×\bm{\times}
REVERIE [19] ×\bm{\times} ✓\checkmark ✓\checkmark ×\bm{\times}
Vision-and-Language Navigation (Continuous)
R2R-CE [13] ✓\checkmark ✓\checkmark ×\bm{\times} ×\bm{\times}
RxR-CE [14, 13] ✓\checkmark ✓\checkmark ×\bm{\times} ×\bm{\times}
REVERIE-CE [15] ✓\checkmark ✓\checkmark ✓\checkmark ×\bm{\times}
Object Navigation
MP3D ObjNav [3] ✓\checkmark ×\bm{\times} ×\bm{\times} ✓\checkmark
HM3D ObjNav [29] ✓\checkmark ×\bm{\times} ×\bm{\times} ✓\checkmark
HM3D ImageNav [30] ✓\checkmark ×\bm{\times} ✓\checkmark ✓\checkmark
Ours ✓\checkmark ✓\checkmark ✓\checkmark ✓\checkmark

5.1 Continuous Trajectory Reconstruction

Discrete REVERIE graph trajectories are mapped and interpolated in Habitat into low-level movement/turn sequences. This supports continuous-space evaluation and collision-aware route reconstruction, while relying on the scan geometry, navigable mesh and transfer procedure. Interpolation does not establish that every route is physically executable on arbitrary robots or robust to real controller/perception error.

5.2 Object-Centric Goal Alignment

Shared Matterport3D instance identifiers connect REVERIE targets with ObjectNav sampling locations. The new endpoints refer to the object’s navigable or visually observable neighborhood rather than an arbitrary nearby waypoint. This improves annotation precision but does not turn the sampled position into the object surface or guarantee a useful final orientation. Instance mapping and sampling remain part of the benchmark definition.

5.3 Dataset Statistics

The split contains 3,691 training trajectories in 53 scenes/90 categories and 1,344 unseen-validation trajectories in 10 scenes/48 categories. Average valid endpoint-to-target geodesic distance changes 1.32→0.34 meters under the alignment procedure. These are benchmark annotation statistics, not the learned policy’s navigation error. The table retains the selected datasets’ continuous-space, language, instance-target and object-proximity properties.

Original mathematical definitions, training conditions and metric context

Our object-centric goal alignment directly bridges the region-to-object grounding gap. While standard REVERIE-CE endpoints terminate at a loose, region-level distance, REVERIE-AIM reduces the valid average minimum geodesic distance to the target from 1.321.32 m to just 0.340.34 m. This finer-grained spatial resolution enables more precise evaluation of both navigation and instance-level grounding.

5.4 Short-Horizon Training Data Generation

The local dataset augments starts near the reference route’s penultimate node, then records shortest-path action/video supervision toward an object-centered goal. Approximately180,000 resulting samples come from the training split. They train recovery from nearby pose/heading variation; they do not independently establish recovery from every upstream stopping error or a wrong-room handoff.

Task-specific Initial Sampling

Start positions use truncated Gaussian perturbations with standard deviation 1.5 meters and coordinate truncation within ±5 meters, plus headings over the full circle. The reference-path anchor supplies privileged near-goal training placement. The accepted navigable-start filtering and resulting difficulty distribution are not fully quantified, so the nominal perturbation range is not a uniform test of all errors within five meters.

Original mathematical definitions, training conditions and metric context

Spatial Perturbation. We perturb the anchor position using a truncated Gaussian distribution (σ=1.5\sigma=1.5, truncated within [−5,5][-5,5] meter) to generate spatially diverse starting states around the target object.

Original mathematical definitions, training conditions and metric context

Viewpoint Augmentation. We randomly sample the initial heading angle within [−π,π][-\pi,\pi] to simulate diverse target-approaching viewpoints.

Clip and Action Sequence Extraction

For valid starts, shortest paths to object-associated sampling locations generate video/action tuples over forward, left, right and stop. This is dense imitation supervision for the local phase, not observed human recovery behavior or outcome-labeled interaction. Shortest-path geometry and simulator visibility provide training information absent from the RGB-only deployed module.

Original mathematical definitions, training conditions and metric context

For each valid sampled starting point, we compute the shortest path to the target object’s surrounding sampling location, determined by the object-centric endpoint described in Section 5.2. We then extract the corresponding visual observation sequences (video clips) together with their continuous action labels. The action labels are a set of low-level actions: {forward,turn_left,turn_right,stop}\{\texttt{forward},\,\texttt{turn\_left},\,\texttt{turn\_right},\,\texttt{stop}\}. In total, the resulting dataset contains approximately 180,000 fine-grained navigation samples from the training split. These samples provide dense supervision for learning precise short-horizon navigation behaviors, particularly during the final target-approaching stage.

6 Experiments

The main test combines four upstream backbones with six local movement variants on unseen-validation scenes, followed by twelve real episodes. Keep upstream legacy navigation scores separate from the post-refinement instance scores. The complete tables preserve all baselines and the human reference; no additional seed repeats, significance tests or timing results are inferred.

6.1 Experimental Setup

Comparisons include doing nothing, random movement, a CLIP-based local search, unadapted UniNaVid, imitation adaptation without VSP and the full loss. The setup spans different navigator families but reuses a particular adapted refinement architecture. Key control/action step sizes, rollout caps, training settings and runtime costs are not fully enumerated, which limits budget-matched interpretation of the extra local stage.

Upstream VLN Baselines

ETPNav-ZS, ETPNav-FT, UniNaVid-ZS and Smartway cover graph/video and zero-shot/fine-tuned/training-free upstream policies. Their labels describe those backbones. REALM adds supervised local adaptation and a trained phrase extractor, so the augmented zero-shot or training-free backbone should not be mistaken for a wholly untrained system.

Downstream Variants

The no-move baseline isolates raw upstream endpoints; random walk and CLIP search test simpler local actions; vanilla UniNaVid separates a pretrained local policy from adaptation. Removing VSP retains supervised training on the new data. These variants help attribute improvements, while unspecified action/time budgets and detector-query details limit claims that all alternatives receive identical compute or exploration opportunities.

Original mathematical definitions, training conditions and metric context

To verify the components and optimization of our refinement module, we compare six downstream variants: (1)w/o Refinement, which directly uses the upstream policy’s raw endpoints; (2) Random Walk, which performs random movements at the destination; (3) Heuristic CLIP-Search, which spins 360∘360^{\circ} and moves toward the direction with the highest CLIP matching score; (4) Vanilla UniNaVid, the training-free base architecture executing raw upstream endpoints directly; (5) Ours w/o VSP, our model optimized on our data using standard cross-entropy loss; and (6) Ours (Full), our complete module optimized on our data by the full tailored loss function.

Standard VLN-CE Metrics

Trajectory length, navigation error, oracle success, legacy success and path-length-weighted success are printed once per backbone. They are fixed upstream reference numbers in Table 2, not newly measured post-refinement trajectory metrics. ONS, GS and OracleGS assess the later endpoint. The table therefore does not establish that refinement preserves legacy path efficiency or has negligible additional travel/runtime.

Complete original Table 2: Plug-in fine-grained navigation module performance on REVERIE-AIM (val-unseen).
Standard VLN metrics are fixed per backbone and shown in parentheses beside each backbone name. Bold = best in group; underline = second best.
Table 2: Plug-in fine-grained navigation module performance on REVERIE-AIM (val-unseen). Standard VLN metrics are fixed per backbone and shown in parentheses beside each backbone name. Bold = best in group; underline = second best.
Accessible original table, all values and grouped headers

Backbone Move Policy ONS@0.1m↑\uparrow ONS@0.5m↑\uparrow GS↑\uparrow OracleGS↑\uparrow
ETPNav-ZS  TL = 11.85,  NE↓\downarrow = 7.26,  OSR = 35.86,  SPL = 23.39,  SR = 28.94
ETPNav-ZS — (no move) 7.07 11.98 3.05 6.99
Random 5.21 10.42 1.93 4.61
Heuristic 7.51 12.57 4.02 10.19
Vanilla UniNavid 11.16 17.78 4.54 11.90
Ours w/o VSP 13.47 20.24 5.28 15.55
Ours (Full) 14.66 21.13 6.10 16.52
ETPNav-FT  TL = 13.86,  NE↓\downarrow = 6.71,  OSR = 40.63,  SPL = 27.68,  SR = 34.67
ETPNav-FT — (no move) 6.32 12.20 4.46 11.31
Random 5.65 12.95 2.75 8.78
Heuristic 6.77 13.24 5.88 12.05
Vanilla UniNavid 13.39 19.12 4.91 12.57
Ours w/o VSP 16.82 25.00 8.04 19.79
Ours (Full) 17.19 25.89 8.48 20.24
UniNavid-ZS  TL = 16.07,  NE↓\downarrow = 8.69,  OSR = 34.40,  SPL = 17.40,  SR = 25.10
UniNavid-ZS — (no move) 13.61 18.08 5.65 12.20
Random 9.75 16.89 2.83 8.78
Heuristic 12.35 18.15 5.13 10.42
Vanilla UniNavid 10.42 16.07 4.84 11.01
Ours w/o VSP 12.43 18.38 5.95 14.29
Ours (Full) 14.96 20.68 6.85 16.37
Smartway  TL = 20.00,  NE↓\downarrow = 8.79,  OSR = 25.37,  SPL = 4.81,  SR = 8.21
Smartway — (no move) 7.81 11.83 1.49 4.69
Random 5.21 10.42 1.34 4.61
Heuristic 7.22 11.98 2.90 8.04
Vanilla UniNavid 9.30 14.81 3.50 9.45
Ours w/o VSP 12.95 17.49 4.69 14.66
Ours (Full) 13.99 19.72 5.65 15.92
Human Eval 50.40 74.71 38.24 80.38

6.2 Results in Simulator

Full REALM leads the listed local variants on all four instance-metric columns within every backbone. ETPNav-FT improves ONS@0.1m 6.32→17.19% and GS 4.46→8.48%, gains 10.87 and 4.02 percentage points. ETPNav-ZS 7.07→14.66% is about 107.4% relative, but the resulting absolute rate is still 14.66%. On UniNaVid-ZS the no-refinement starting point 13.61% is already stronger than vanilla local UniNaVid’s 10.42%; full REALM reaches 14.96%, only 1.35 points above no refinement. Removing VSP there gives 12.43%, making the VSP ablation gain 2.53 points.

The human reference reaches 74.71% ONS@0.5m versus the best full model’s 25.89%, and human GS is 38.24%. Human protocol, seed variance and statistical tests are not detailed; ‘significantly’ in the prose should not imply a demonstrated significance test. The large remaining visibility/detection gaps limit claims of a solved interaction interface.

Original mathematical definitions, training conditions and metric context

As shown in Table 2, our full module (Ours (Full)) consistently and significantly outperforms all alternative move policies across all four backbones. For instance, on ETPNav-FT, it boosts ONS@0.1m from 6.32%6.32\% (no move) to 17.19%\mathbf{17.19\%} and GS from 4.46%4.46\% to 8.48%\mathbf{8.48\%}, demonstrating its superior capability in fine-grained object localization. The ablation variant (Ours w/o VSP) incurs consistent performance drops across all settings (e.g., ONS@0.1m falls by 2.53%2.53\% on UniNavid-ZS), validating the necessity of our proposed Visibility-Aware Stopping loss. Nevertheless, a substantial performance gap remains compared to Human Eval (e.g., 25.89%25.89\% vs. 74.71%74.71\% on ONS@0.5m), highlighting that this benchmark leaves massive room for future improvement.

6.3 Results in Real-world Environments

On Stretch with a RealSense camera and RTX 4090 workstation, twelve multi-room episodes compare UniNaVid with and without refinement. ONS@0.5m rises 1/12→4/12, GS 1/12→3/12 and OracleGS 2/12→3/12. Those small counts support a preliminary improvement, with no uncertainty or paired-transition analysis reported. The hardware ONS uses Euclidean distance to the nearest navigable object-associated position, unlike simulation’s geodesic metric. The prose calls it SR, but it is not the legacy three-meter waypoint SR. RGB-D camera hardware does not change the stated RGB-only refinement input, and no manipulation task or guaranteed visibility/safety outcome is measured.

Complete original Table 3: Comparison on Real-world robot evaluation.
Table 3: Comparison on Real-world robot evaluation.
Accessible original table, all values and grouped headers
Method ONS@0.5m↑\uparrow GS↑\uparrow OracleGS↑\uparrow
UniNavid 8.33 8.33 16.67
UniNavid + Ours 33.33 25.00 25.00

7 Conclusion and Limitations

The study makes final-view object grounding a clearer target and shows a modular local refinement can improve several existing navigators. It leaves substantial absolute failures, especially relative to the human reference. Future work needs active viewpoint selection, reasoning under partial observability and reliable stopping, alongside clearer budgets and broader real-world evaluation. The reported system reduces the gap; it does not certify interaction readiness for arbitrary upstream endpoints.

Concept demonstration

Open image ↗