At a glance
This study probes four vision-language navigation policies by perturbing images, instructions, and memory, then tests internal activation interventions. Input sensitivity partly aligns with human-relevant cues but does not reliably predict closed-loop robustness. Semantic navigation directions remain accessible after backbone fine-tuning, and a simulation-derived reasoning vector can improve real-world recovery when an operator triggers it. The hardware result concerns a sequential assisted recovery protocol, not an autonomous failure detector or a retrained policy.
I Introduction
Navigation success alone does not reveal which inputs a model uses or whether its internal progress tracking works as intended. The paper combines attribution, input corruption, and activation steering to investigate those questions. Its interventions establish sensitivity to specific changes; interpreting that sensitivity as semantic grounding, a shortcut, or a complete explanation requires additional assumptions.
II Related Work
The analysis connects VLM-based navigation, multimodal attribution, and inference-time steering of learned representations.
II-A Vision-and-Language Navigation
The evaluated systems include CorrectNav [11], AwareVLN [6], DualVLN, and NaVILA [14]. They differ in visual history, explicit reasoning, backbone, camera configuration, and action decoding. These architectural differences motivate comparing sensitivity patterns rather than assuming identical decision mechanisms.
II-B Interpretability and Grounding Analysis
Gradient and perturbation attribution methods estimate input relevance from output changes, while activation visualizations inspect internal representations. The study adapts integrated gradients [16] and token activation maps [19] alongside a behavioral intervention score. Human annotations supply a reference for relevant regions, not a complete ground-truth account of how navigation must be solved.
II-C Steering
Steering methods modify hidden states or selected neuron contributions while keeping weights fixed. Prior work identifies semantic or behavior-associated directions but also shows limited generalization. This paper tests whether analogous interventions remain useful after a VLM becomes a navigation policy.
III Overview
At each decision, the policy combines a tokenized instruction with a sampled history of egocentric images. Its prompt and visual encoding are model-specific. Autoregressive outputs are decoded into forward, left, right, and stop actions, sometimes through an additional action decoder. Different models produce different action horizons, which affects the scale of behavioral perturbation scores.
Equations 1–3
| (1) |
| (2) |
| (3) |
IV Methodology
The methodology separates input attribution from internal steering. Neither changes model weights, but their measurements answer different questions: what changes the predicted action, and whether selected hidden-state changes can bias behavior.
IV-A Interpretability
Three attribution methods are compared: direct behavioral interventions, integrated gradients of output likelihood, and token activation maps.
IV-A 1 Interventional Significance Score (ISS) [ 10 ]
The Interventional Significance Score corrupts selected input groups—blurring visual content or deleting/replacing words—and measures the accumulated positional and angular divergence of predicted action trajectories. Token scores average interventions containing that token. The score measures sensitivity under the chosen corruption, not whether the changed action is better, worse, or semantically justified.
Equation 4
| (4) |
IV-A 2 Integrated Gradients (IG) [ 16 ]
Integrated gradients interpolates between corrupted and original token embeddings while keeping the generated output sequence fixed. It accumulates gradients of that sequence’s average log likelihood. This explains support for a fixed prediction and is not equivalent to measuring a newly decoded trajectory after every intervention.
Equations 5–7
| (5) |
| (6) |
| (7) |
IV-A 3 Token Activation Map (TAM) [ 19 ]
Token activation maps project hidden representations onto output-token directions, apply rectification, and remove activation attributed to preceding autoregressive context. Averaging over generated action tokens produces a map. Concrete object words may yield interpretable localization, while abstract actions need not correspond to one image region.
Equation 8
| (8) |
IV-B Steering
Internal interventions either modify contributions of vocabulary-associated feed-forward value vectors or add a direction derived from contrasting recorded behaviors.
IV-B 1 Value-Vector-Guided Semantic Steering [ 35 ]
Feed-forward down-projection columns are associated with their highest-scoring vocabulary tokens. Neurons linked to a desired semantic cluster are enhanced and those linked to an opposing cluster are suppressed by clamping their inputs, with the retained equation specifying the sign convention. This changes selected vector contributions without updating their weights or requiring those words to appear in the policy’s output vocabulary.
Equations 9–11
| (9) |
| (10) |
| (11) |
IV-B 2 Activation Steering [ 27 ]
Activation steering computes a layer-wise difference between mean hidden states associated with two behaviors. Adding a scaled version at selected token positions biases the model toward the positive behavior. Effectiveness depends on the contrastive examples, layer, token position, and intervention strength.
Equations 12–13
| (12) |
| (13) |
V Experiments
Experiments study visual and textual attribution, memory corruption, semantic motion steering, explicit reasoning under a camera shift, and operator-triggered real-world recovery.
V-A VLN Baselines
All four policies are evaluated on subsets of unseen-validation R2R trajectories in VLN-CE, using each model’s nominal camera and motion increments. Their published benchmark scores differ from the subset results reported here. Open-loop intervention comparisons follow common ground-truth trajectories; closed-loop runs assess whether the resulting actions actually complete navigation.
V-B Sensitivity to Visual Semantics
On 200 manually annotated decision points, ISS generally aligns its strongest visual attributions with relevant regions better than integrated gradients. CorrectNav’s top-1% precision is 0.66 for ISS versus 0.52 for gradients; NaVILA’s is 0.48 versus 0.47. At broader thresholds the advantage is not universal, as Table I shows. Annotation overlap supports partial alignment, but low overlap alone does not prove spurious reasoning: annotations are incomplete and perturbations can create unusual inputs. Token activation maps localize concrete concepts more clearly than “move forward.”
(b) Token Activation Map (TAM).
Equation 14
| (14) |
| Model | Method | P@1 | P@10 | P@30 | P@50 |
|---|---|---|---|---|---|
| CorrectNav | ISS | 0.66 | 0.52 | 0.41 | 0.36 |
| IG | 0.52 | 0.46 | 0.4 | 0.37 | |
| AwareVLN | ISS | 0.60 | 0.55 | 0.44 | 0.40 |
| IG | 0.46 | 0.43 | 0.40 | 0.38 | |
| DualVLN | ISS | 0.55 | 0.49 | 0.43 | 0.39 |
| IG | 0.42 | 0.42 | 0.41 | 0.40 | |
| NaVILA | ISS | 0.48 | 0.42 | 0.37 | 0.34 |
| IG | 0.47 | 0.44 | 0.40 | 0.37 | |
Highest value across policies in bold; highest value per policy underlined.
V-C Sensitivity to Visual Memory
Blurring progressively older memory frames changes predicted actions differently across models. CorrectNav and DualVLN show stronger growth in sensitivity with trajectory progress and corruption, while AwareVLN and NaVILA rely more on recent observations in these tests. Scores are normalized within each method because action horizons differ. Proposed explanations involving bottlenecks, temporal tags, and reasoning state are architectural hypotheses, not separately isolated mechanisms.
V-D Robustness to Visual Memory
Closed-loop memory-corruption results differ from open-loop sensitivity. At 90% corruption, CorrectNav retains 35.82% success versus its 55.97% uncorrupted result; DualVLN retains 21.64%, AwareVLN 15.67%, and NaVILA 14.18%. AwareVLN’s relatively low sensitivity does not translate into the best robustness. Other observations and language may compensate for altered actions, showing why action change and task failure should not be conflated.
| Model | 0% | 10% | 20% | 30% | 40% | 50% | 60% | 70% | 80% | 90% |
|---|---|---|---|---|---|---|---|---|---|---|
| CorrectNav | 55.97 | 54.48 | 52.99 | 48.51 | 51.49 | 50.00 | 47.01 | 47.76 | 44.03 | 35.82 |
| DualVLN | 48.51 | 47.76 | 47.76 | 42.54 | 50.75 | 44.03 | 33.58 | 30.60 | 32.84 | 21.64 |
| AwareVLN | 58.21 | 58.21 | 57.46 | 52.24 | 48.51 | 49.25 | 35.82 | 32.84 | 32.09 | 15.67 |
| NaVILA | 50.00 | 46.27 | 48.51 | 42.54 | 41.79 | 48.51 | 37.31 | 32.84 | 23.13 | 14.18 |
V-E Sensitivity to Textual Semantics
Deleting groups of one, three, or five instruction words on 100 decision samples tests language sensitivity against manually annotated active subtasks. CorrectNav has the highest reported precision. AwareVLN has many zero-score cases, while NaVILA is sensitive in nearly all cases but has lower annotation overlap. Repeated landmarks and nearby context complicate interpretation: a sensitive word outside the active annotation may still be relevant, and no action change may reflect redundancy rather than neglect.
| Model | P@1 | P@3 | P@5 | P@10 | P@15 | ZS(%) |
|---|---|---|---|---|---|---|
| CorrectNav | 50.93 | 51.54 | 50.00 | 44.84 | 37.95 | 32.41 |
| AwareVLN | 41.67 | 43.21 | 42.96 | 39.35 | 36.06 | 48.15 |
| DualVLN | 40.74 | 43.83 | 42.04 | 36.84 | 34.05 | 20.37 |
| NaVILA | 34.26 | 32.10 | 32.41 | 29.58 | 29.45 | 2.78 |
V-F Sensitivity to Textual Memory
Replacing cumulative instruction prefixes with antonyms introduces conflicting content rather than simply removing information. CorrectNav and DualVLN show sensitivity boundaries that shift with navigation progress; AwareVLN and NaVILA exhibit different, less consistent patterns. The study finds no common textual-progress pattern across all models. Because corruption reaches different subtasks as it grows, these heatmaps reflect both progress and the intervention schedule.
V-G Semantic Concepts from VLMs in VLNs
Comparing vocabulary associations of Qwen2.5-VL and DualVLN value vectors suggests that verbal direction concepts survive fine-tuning for symbolic navigation output. Clamping concept-associated neurons over five episodes biases motion-token frequencies: forward-versus-turn steering increases forward output, while left-versus-right steering increases left output. This demonstrates influence on behavior, not improved navigation success or a complete localization of each concept.
| Steering / Motion | Forward | Right | Left |
|---|---|---|---|
| forward + turn - | +3.2% | -0.8% | -19.3% |
| left + right - | +0.8% | -3.0% | +7.8% |
V-H Activation Reasoning Under Visual Distribution Shift
A lower-height fisheye camera reduces AwareVLN’s explicit-reasoning frequency from 12±2% to 3±1% at matched simulator states. A mean action-to-reasoning activation direction increases that frequency on 306 shifted-camera decision contexts: 20%, 55%, 90%, and 100% at strengths 0.5, 1, 2, and 4. Random-vector controls do not help. These measurements show controllable reasoning emission; reasoning frequency alone does not establish correctness or task benefit.
V-I Real World Deployment of Activation Steering
The real-world study describes over 40 trials across eight scenes and twelve instructions. The unmodified policy runs first; an operator identifies an unrecovered wrong choice or loop, then applies the simulation-derived reasoning direction at strength 3 to the first output token. No new-domain weight training is performed.
The paper reports success increasing from 37.2% before intervention to 74.4% after this recovery procedure, corresponding to 59.3% of initial failures recovered. These figures evaluate sequential operator-triggered assistance with additional execution, not two independent randomized policy conditions. Automatic failure detection and intervention timing remain future work.
| Condition | Reasoning Frequency | |||
|---|---|---|---|---|
| Baseline Visual Distribution | ||||
| Visual distribution shift | ||||
| Random vector steering | ||||
| Mean vector steering | ||||
VI Conclusions
Input sensitivity, human-semantic alignment, and closed-loop robustness are distinct properties. The study provides evidence that multiple modalities influence navigation and that retained semantic and reasoning representations can be steered, but it does not establish a universal causal explanation of policy behavior. The most practical result is simulation-to-real transfer of a recovery intervention under human triggering. Better annotations, broader controlled evaluations, and automatic strength/timing selection are open directions.
Acknowledgments
The authors acknowledge Erlangen and Jülich computing resources, Bavarian funding, the Körber European Science Prize, and German federal support through the Robotics Institute Germany.