Literature
Display
← All papers

Multimodal Spatial Language Maps for Robot Navigation and Manipulation

Chenguang Huang · Oier Mees · Andy Zeng · Wolfram Burgard

Preprint posted · The International Journal of Robotics Research (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

The framework stores pretrained visual, language, and audio features at mapped spatial locations. VLMaps support open-vocabulary object and spatial-goal navigation; AVLMaps add image localization, region recognition, and remembered audio, combining their spatial scores to resolve ambiguous goals. Language models generate calls to a fixed navigation library. Experiments demonstrate navigation and tabletop goal reaching without task-specific model fine-tuning, but require prior mapping and do not localize live sound sources.

1 Introduction

The central idea is persistent spatial memory: foundation-model descriptors become useful for navigation when attached to geometry rather than used only to score the current image. VLMaps attach visual-language features to a reconstructed map; AVLMaps add several complementary localization modules and expose them through a common heatmap interface. The same representation can support different obstacle definitions for different robot bodies.

This journal-style extension combines the earlier visual and audio-visual mapping work, tests alternative pretrained models, and adds tabletop reaching. “Zero-shot” refers to using the pretrained models without navigation-specific fine-tuning; it does not mean that the system operates without exploration, pose estimates, manually specified APIs, or prompting examples.

Related source: the original VLMaps paper.

Refer to caption
Figure 1: AVLMaps provide an open-vocabulary 3D map representation for storing cross-modal information from audio, visual, and language cues. When combined with large language models, AVLMaps consumes multimodal prompts from audio, vision, and language to solve zero-shot spatial goal navigation by effectively leveraging complementary information sources to disambiguate goals.

2 Related Work

Earlier semantic maps often use fixed labels, while open-vocabulary visual maps inherit broader concepts from pretrained models. Language-navigation systems still vary in whether they learn direct actions or retrieve landmarks from maps. Audio-navigation research commonly follows an active sound without recognizing its semantic category. AVLMaps instead stores audio semantics at past observation poses and combines them with visual concepts. Its distinction from audio queries into a visual-only map is that audio observations themselves become part of the persistent representation.

Refer to caption
Figure 2: The creation and language-conditioned indexing of a VLMap. A VLMap is created by fusing pretrained visual-language features into the reconstruction of the environment to enable visual-spatial-language-based reasoning. By providing a list of open-vocabulary labels, we retrieve segmentation masks for semantic classes required by downstream applications.

3 Method

The method progresses from a visual-language map to a modular multimodal spatial memory. Geometry supports explicit spatial offsets and body-specific planning; separate image, object, region, and audio modules produce comparable spatial heatmaps. A language model selects and composes supplied operations rather than learning a new low-level controller.

3.1 Building a Visual-Language Map

Registered depth and camera poses back-project pixel features into a common frame. LSeg produces dense descriptors in CLIP space; descriptors from multiple observations are averaged at each map location. The initial VLMap formulation projects 3D points onto a top-down grid, so different heights can contribute to the same cell. AVLMaps later retains full 3D voxels. Reliable odometry and RGB-depth registration are prerequisites, and averaging does not itself handle moving objects.

Equation 1
pm​a​px=⌊H¯2+PWxs+0.5⌋,pm​a​py=⌊W¯2−PWzs+0.5⌋p^{x}_{map}=\Bigl\lfloor\frac{\bar{H}}{2}+\frac{P^{x}_{W}}{s}+0.5\Bigr\rfloor,~p^{y}_{map}=\Bigl\lfloor\frac{\bar{W}}{2}-\frac{P^{z}_{W}}{s}+0.5\Bigr\rfloor (1)

3.2 Localizing Open-Vocabulary Landmarks

User-specified category descriptions are encoded with the CLIP text encoder and compared with stored map features. Each cell receives the highest-scoring category among that query list, producing open-vocabulary segmentation on demand. The “other” category helps define the comparison set, but these relative similarities are not calibrated probabilities or a guarantee of recognizing every unseen object.

Refer to caption
Figure 3: The overview of building customized obstacle maps for different robot embodiments. By specifying different obstacle categories in natural language for different embodiments, different obstacle maps can be built to ensure the most efficient path planning for different embodiments.

3.3 Generating Open-Vocabulary Obstacle Maps

Height-filtered geometry gives a candidate obstacle mask; language-derived semantic masks select which categories matter for a particular robot. Their intersection suppresses predicted obstacles on geometric floor regions. A drone and ground robot can therefore share features but plan with different category lists. The demonstrated drone case uses a fixed flight-height map, not unrestricted 3D planning or a complete learned model of embodiment clearance.

Equation 2
𝒪i​j={1,t1≤PWy≤t2​a​n​d​pm​a​px=i​a​n​d​pm​a​py=j0,otherwise\mathcal{O}_{ij}=\begin{cases}1,&t_{1}\leq P^{y}_{W}\leq t_{2}\ and\ p^{x}_{map}=i\ and\ p^{y}_{map}=j\\ 0,&\text{otherwise}\end{cases} (2)

3.4 Zero-Shot Spatial Goal Navigation from Language

Few-shot instruction/code examples teach the language model to call navigation primitives, sequence subgoals, loop, and compute offsets. Functions such as moving left of an object first localize that object, then apply predefined geometric rules; a conventional planner executes the resulting goal. Spatial understanding is therefore shared between generated code and a manually implemented API. Figures 4 and 6 preserve the full prompt and example outputs, including indentation and highlighted code.

Original instruction preserving all source text, code indentation, and highlights.
Original figure 4 preserving all source text, code indentation, and highlights.
Figure 4: The full context prompt (prompt in gray) VLMap used for achieving spatial goal navigation tasks in the experiments.
Refer to caption
Figure 5: VLMaps enable a robot to perform complex zero-shot spatial goal navigation tasks given natural language commands, without additional data collection or model finetuning.
APIs Functions
move_to(pos) move to a position on the map.
move_to_left(object_name) move to the left side of the nearest front object.
move_to_right(object_name) move to the right side of the nearest front object.
with_pos_on_left(object_name) turn until the object is on the robot’s left side.
with_pos_on_right(object_name) turn until the object is on the robot’s right side.
move_in_between(object_a, object_b) move in between two objects.
face(object_name) turn until the robot’s front is pointing to the object.
turn(angle) turn right a certain angle. If the angle value is negative, turn left.
turn_absolute(angle) turn to absolute angle. 0 is north, 90 is east, -90 is west, 180 is south.
move_north(object_name) move to the north side of the object.
move_south(object_name) move to the south side of the object.
move_east(object_name) move to the east side of the object.
move_west(object_name) move to the west side of the object.
move_forward(dist) move forward “dist” meters.
Table 1: Navigation API library for spatial goal navigation in VLMaps.
Original figure 6 preserving all source text, code indentation, and highlights.
Figure 6: The query and the generated results from the LLM for spatial goal navigation tasks. During the query, the context prompt in Fig. 4 and the input task commands are prompted to the LLM together. The input task commands are in green and generated outputs are highlighted
Refer to caption
Figure 7: System overview. AVLMaps are constructed from RGB-D, audio, and odometry inputs, converting raw data into visual localization features, visual-language features, and audio-language features. During inference time, each module’s output is unified with cross-modal reasoning, allowing users to query spatial location with multimodal information.

3.5 Building an Audio Visual Language Map

Four modules share a 3D heatmap interface. Image localization retrieves a reference using NetVLAD, matches SuperPoint features with SuperGLUE, and solves a 2D–3D pose problem. Object localization averages dense visual-language descriptors in 3D voxels. Region localization uses a separate sparse map of whole-image CLIP features, preserving broad concepts such as kitchen that segmentation-tuned encoders may represent poorly.

Audio is segmented by silence, embedded with an audio-language model, and stored alongside the robot poses during that segment. A text sound query retrieves these remembered observations. It predicts where the robot heard the sound, not the acoustic source location. Distance-decaying heatmaps spread the image, object, region, or audio scores spatially; audio/image heatmaps can also define forbidden areas. These scores are heuristic spatial relevancies despite the paper’s probability terminology. The original audio equations 9–10 repeat a distance term and leave an index ambiguous in equation 10; their notation is retained without silently changing the implementation specification.

Related source: SuperGLUE.

Equations 3–4
ℋv​(𝐩)=max⁡(1.0−ϵ⋅d​i​s​tx​y​(𝐩,𝐩v),0)\mathcal{H}_{v}(\mathbf{p})=\mathrm{max}(1.0-\epsilon\cdot dist_{xy}(\mathbf{p},\mathbf{p}_{v}),0) (3)
d​i​s​tx​y​(𝐩,𝐪)=(px−qx)2+(py−qy)2dist_{xy}(\mathbf{p},\mathbf{q})=\sqrt{(p_{x}-q_{x})^{2}+(p_{y}-q_{y})^{2}} (4)
Refer to caption
Figure 8: The overview of the visual localization module. We follow the hierarchical localization scheme by using NetVLAD and SuperGLUE to localize the query image’s location before generating a heatmap as the indexing result.
Equations 5–6
dm​i​n(𝐩)=min{dist(𝐩,𝐩o​i)∣i=1,…,N}d_{min}(\mathbf{p})=\mathrm{min}\{dist(\mathbf{p},\mathbf{p}_{oi})\mid i=1,\ldots,N\} (5)
ℋo​(𝐩)=max⁡(1.0−ϵ⋅dm​i​n​(𝐩),0)\mathcal{H}_{o}(\mathbf{p})=\mathrm{max}(1.0-\epsilon\cdot d_{min}(\mathbf{p}),0) (6)
Refer to caption
Figure 9: The overview of the object localization module. Similar to Sec. 3.1, during mapping, the RGB images in the exploration video are input to a vision-language model, LSeg (Li et al., 2021), to generate pixel-level features. Corresponding pixels are back-projected with depth images and transformed to locations in the global coordinate frame, where the features are associated with. During inference, the query text is encoded by LSeg’s text encoder, and a dot product with the point-level embeddings generates a score for each point. A heatmap is then created based on point scores and distances.
Refer to caption
Figure 10: The overview of the audio localization module. During mapping, the exploration video’s audio is segmented by silence and encoded using AudioCLIP’s audio encoder. The resulting embeddings are linked to poses based on their timestamps. During inference, the query text is encoded by AudioCLIP’s text encoder, and a dot product with the pose embeddings generates a score for each pose. A heatmap is then created based on pose scores and distances.
Equations 7–10
ℋ¯a​(𝐩)=maxi=1,…,M⁡{sa​i−ϵ⋅d​i​s​tx​y​(𝐩,𝐩a​i)}\bar{\mathcal{H}}_{a}(\mathbf{p})=\max_{i=1,\ldots,M}\{s_{ai}-\epsilon\cdot dist_{xy}(\mathbf{p},\mathbf{p}_{ai})\} (7)
ℋa​(𝐩)=max⁡(ℋ¯a​(𝐩),0)\mathcal{H}_{a}(\mathbf{p})=\max(\bar{\mathcal{H}}_{a}(\mathbf{p}),0) (8)
ℋ¯s​(𝐩)=maxi=1,…,K⁡{ss​i−ϵ⋅d​i​s​tx​y​(𝐩,𝐩s​i)}\bar{\mathcal{H}}_{s}(\mathbf{p})=\max_{i=1,\ldots,K}\{s_{si}-\epsilon\cdot dist_{xy}(\mathbf{p},\mathbf{p}_{si})\} (9)
ℋs​(𝐩)=max⁡(ℋ¯s​(𝐩)−ϵ⋅d​i​s​tx​y​(𝐩,𝐩s​i),0)\mathcal{H}_{s}(\mathbf{p})=\max(\bar{\mathcal{H}}_{s}(\mathbf{p})-\epsilon\cdot dist_{xy}(\mathbf{p},\mathbf{p}_{si}),0) (10)

3.6 Cross-Modality Reasoning

Cross-modal reasoning multiplies heatmaps elementwise and chooses the strongest resulting location. The primary target uses a sharper decay, while auxiliary concepts use broader fields to select a nearby instance. Reported rates are 0.1 and 0.01 per five-centimeter voxel. This simple product assumes conditional independence and can amplify errors when one module is wrong. Equal-width maps can favor an intermediate location when their support overlaps; they do not guarantee a meaningful midpoint for every configuration.

Equation 11
ℋt​a​r​g​e​t=ℋ1⊙…⊙ℋL\mathcal{H}_{\mathit{t}arget}=\mathcal{H}_{1}\odot\ldots\odot\mathcal{H}_{L} (11)
Refer to caption
Figure 11: The key idea of cross-modal reasoning is converting the prediction from different modalities into heatmaps, and then fusing them with element-wise multiplication, effectively using complementary multimodal information to resolve ambiguous prompts.
APIs Functions
load_image(path) load an image as an array from the path.
move_to(pos) move to a position on the map.
get_major_map(img=None, obj=None, sound=None) generate a heatmap with high heat decay rate for a specific modality (image, object description, or sound description).
get_map(img=None, obj=None, sound=None) generate a heatmap with low heat decay rate for a specific modality (image, object description, or sound description).
get_max_pose_3d(heatmap) retrieve the coordinate in a 3d voxel heatmap which has the maximal heat value).
Table 2: Navigation API library for multimodal goal navigation in AVLMaps.

3.7 Multimodal Goal Navigation from Language

The language model generates code using image loading, sharp or broad heatmap construction, map multiplication, maximum-location retrieval, and movement primitives. Spatial goals can be arithmetic combinations of localized positions. The model used in the reported experiments is text-davinci-003, with the complete context examples preserved in Figure 12 and an output example in Figure 13.

A 3D target is projected to the navigation grid and converted into primitive actions by a planner. The method text lists 0.25-meter forward steps, whereas the multimodal simulation experiment later specifies 0.1-meter steps; those source settings are kept distinct. This is prompted program composition over a fixed tool library rather than end-to-end learning of robot dynamics.

Original figure 12 preserving all source text, code indentation, and highlights.
Figure 12: The full context prompt (prompt in gray) AVLMaps used for achieving all navigation tasks in the experiments.
Original figure 13 preserving all source text, code indentation, and highlights.
Figure 13: The query and the generated results from the LLM. During the query, the context prompt in Fig. 12 and the input task commands are prompted to the LLM together. The input task commands are in green and generated outputs are highlighted

4 Experiments

Evaluation separates spatial-goal navigation, cross-embodiment obstacle maps, multimodal navigation, ambiguous-goal indexing, model substitutions, and two physical robot settings. Individual-goal success, complete-sequence success, localization recall, and path efficiency measure different aspects and should not be interchanged.

4.1 Zero-Shot Spatial Goal Navigation from Language

The spatial-navigation study maps ten Matterport3D scenes using 12,096 RGB-D frames, then evaluates 21 manually specified four-subgoal trajectories in seven scenes. Reaching within one meter counts as success. VLMaps achieves 62% for the first subgoal but only 10% for all four in sequence; the ground-truth semantic-map reference reaches 76% and 29%. The tested learned-feature baselines fail all four-goal sequences.

All map-based methods receive the same code-generation procedure, while LM-Nav keeps its own parsing approach. The result supports better spatial localization in this small benchmark while exposing substantial compounded failures even with perfect semantics.

Tasks No. Subgoals in a Row
1 2 3 4
LM-Nav (Shah et al., 2023) 5 5 0 0
CoW (Gadre et al., 2023) 33 5 0 0
CLIP Map 19 0 0 0
VLMaps (ours) 62 33 14 10
GT Map 76 48 33 29
Table 3: The success rate (%) of zero-shot spatial goal navigation with language.
Refer to caption
Figure 14: VLMaps enable different embodiments to define their own obstacle maps for navigation. The left image shows the top-down view of an environment. The middle columns show the observations of agents during navigation. The images on the right demonstrate the obstacles maps generated for different embodiments and the corresponding navigation paths.
Tasks No. Subgoals in a Row Indep.
1 2 3 4 Subgoals
SR SPL SR SPL SR SPL SR SPL SR
LoCoBot (ground map) 53 49.0 28 17.8 14 6.7 6 2.5 52.3
Drone (ground map) 53 41.8 28 15.5 14 5.3 6 2.0 53.3
Drone (drone map) 56 45.4 30 16.3 17 7.0 7 2.5 55.0
Table 4: The success rate (%) and SPL of multi-embodiment object goal navigation with language.

4.2 Cross-Embodiment Navigation

Ten AI2THOR rooms are mapped from 1,826 frames and evaluated on more than 100 subgoal sequences. Changing the obstacle categories lets a drone plan over furniture that blocks a LoCoBot. For the first subgoal, drone path-weighted success rises from 41.8 with a ground map to 45.4 with its tailored map, with success changing from 53% to 56%. Complete four-goal success remains only 7% for the tailored drone. This tests reuse of one map at a 1.7-meter flight level, with manually chosen obstacle lists.

4.3 Multimodal Navigation Simulation Setup

Multimodal mapping data is manually collected in ten simulated scenes and augmented with randomly inserted ESC-50 validation-fold sounds. The stored frame positions are the audio targets; no acoustic propagation or source triangulation is evaluated. Navigation stops within one meter of a target. Additional datasets pair objects with region labels, nearby images, or repeated sounds to create ambiguity. This protocol evaluates semantic retrieval and spatial association under controlled audio placement.

4.4 Multimodal Goal Navigation

Sound navigation evaluates 200 four-goal sequences. For the 30-category mixture, independent sound-goal success is 77.5%, while completing four goals in a row reaches 33%; these are different quantities. A separate 200-sequence test requests one image goal followed by two object goals: first-goal success is 71.5%, complete three-goal success 25%, and independent-goal success 47.4%. Multimodal querying works, but errors compound over longer sequences.

Tasks No. Subgoals in a Row Independent
1 2 3 4 Subgoals
Domestic Sound (10 categories) 59.5 33.0 15.5 7.0 62.5
+ Human Sound (20 categories) 69.5 47.0 36.5 23.0 72.38
+ Animal Sound (30 categories) 74.5 58.5 45.5 33.0 77.5
Table 5: The success rate (%) of sound goal navigation with AVLMaps.
Tasks No. Subgoals in a Row Independent
1 2 3 Subgoals
AVLMaps (Ours) 71.5 40.5 25.0 47.4
Table 6: The success rate (%) of multimodal goal navigation with AVLMaps. The agent is required to navigate to one visual goal, and two object goals in sequence.

4.5 Cross-Modal Goal Indexing

Region-object indexing uses 100 tasks; sound-object indexing uses 119. Combining a sparse region map with VLMaps improves half-meter recall from 5.56% to 15.56%, still below the ground-truth-region reference of 37.78%. Object-conditioned AudioCLIP improves half-meter sound recall from 26.05% to 53.78%. Adding image localization improves ambiguous-object half-meter recall from 7.55% to 62.26%.

These exact comparisons explain the headline improvement of roughly 50 percentage points in one ambiguous setting; it is not a uniform 50% improvement for every modality or tolerance. Even improved methods retain nonzero average localization error.

Method Recall@1 (%) Average min. distance (m)
<0.5m <1m <1.5m <2m
baseline (VLMaps)
5.56 7.78 13.33 17.78 8.22
+ ConceptFusion
12.22 13.33 16.67 21.11 7.60
+ CLIP sparse (Ours)
15.56 24.44 31.11 35.56 6.17
+ GT region map
37.78 44.44 55.56 61.11 2.62
Table 7: The recall (%) of area-object cross-modal indexing.
Method Recall@1 (%) Average min. distance (m)
<0.5m <1m <1.5m <2m
baseline (wav2clip)
8.40 10.08 10.92 14.29 8.52
baseline (AudioCLIP)
26.05 35.29 36.97 42.01 5.04
VLMaps + wav2clip
24.37 30.25 33.61 38.66 6.27
VLMaps + AudioCLIP
(Ours)
53.78 65.55 67.23 70.59 2.74
Table 8: The recall (%) of object-sound cross-modal indexing.
Method Recall@1 (%) Average min. distance (m)
<0.5m <1m <1.5m <2m
VLMaps w/o
visual module
7.55 9.43 11.32 11.94 11.22
VLMaps w/
visual module (Ours)
62.26 66.67 70.44 72.32 3.11
Table 9: The recall (%) of visual-object cross-modal indexing.
Refer to caption
Figure 15: Real-world navigation experiments are conducted in a room with multiple ambiguous goals such as tables, chairs, backpacks, and paper boxes (left). The robot setup is also shown in the left image. We leverage dense SLAM techniques to build a 3D reconstruction of the scene from RGB-D camera data into which we anchor features from multiple foundation models (right). We artificially insert sounds with different semantics at locations shown in the image. Different sounds are played when the robot moves to these locations during mapping. Sounds are sampled from the ESC-50 dataset.

4.6 Multimodal Ambiguous Goal Navigation

The 119 two-goal sequences require an ambiguous sound target followed by an ambiguous object target. AudioCLIP-based AVLMaps achieves 46.2% sound-goal and 55.5% object-goal success, versus 22.0% and 53.4% for the wav2clip multimodal alternative. Both goals are completed in 28.6% of sequences versus 12.7%. The stated gains of 24.2 and 2.1 are percentage-point differences; the large benefit is concentrated in the audio component.

Related source: AudioCLIP.

Tasks No. Subgoals in a Row Sound Object
1 2 Goals Goals
VLMaps - - - 27.1
AudioCLIP - - 16.9 -
VLMaps + wav2clip 22.0 12.7 22.0 53.4
VLMaps + AudioCLIP (Ours)
46.2 28.6 46.2 55.5
Table 10: The success rate (%) of multimodal ambiguous goal navigation with AVLMaps. The agent is required to navigate to one ambiguous sound goal and one ambiguous object goal sequentially.
ALM VLM No. Subgoals in a Row Sound Object
1 2 3 4
AudioCLIP LSeg 71.0 59.0 29.0 17.0 71.0 41.8
CLAP LSeg 81.0 69.0 36.5 19.0 81.0 52.0
CLAP SAM+CLIP 80.0 68.5 29.5 12.0 80.0 40.5
CLAP OVSeg 81.0 69.0 37.0 21.0 81.0 53.5
Table 11: The success rate (%) of multimodal goal navigation with different foundation models. The agent is required to navigate to one sound goal, one visual goal, and two object goals in sequence. The right-most three columns indicate the independent success rate of navigating to specific types of goals.

4.7 Scaling Experiment

Replacing AudioCLIP with CLAP while retaining LSeg raises sound-goal success from 71% to 81%, but complete four-goal success rises only from 17% to 19%. With CLAP fixed, OVSeg reaches 21% complete success and SAM+CLIP falls to 12%, so a newer component does not automatically improve the system. The authors attribute the weak visual substitutions to mask quality, masked-image feature behavior, and scene-specific tuning. These experiments test particular module configurations, not a general scaling law.

4.8 Real World Experiment for Mobile Robot

A Ridgeback base with a Panda arm and RGB-D camera supplies physical navigation observations; LiDAR localization and arm kinematics provide poses. Audio categories are inserted into the mapping recording and mixed with measured background noise. Attempted live loudspeaker recording was largely masked by robot vibration, so the reported experiment is not fully live acoustic-source navigation.

A noise gate uses a −10 dB threshold, 250 ms attack, 1000 ms hold, and 170 ms release, followed by silence segmentation. Offline maps and ROS planning yield 10 successes in 20 multimodal tasks and six in ten spatial tasks. Object confusions and ambiguous audio scores remain failure modes. These small physical tests demonstrate integrated navigation under the stated audio construction rather than general noise-robust hearing.

Refer to caption
Figure 16: Audio denoising for real-world experiment. We pre-process the audio by applying a noise gate. Simply applying a threshold for filtering out low-level noise might lead to frequent fluctuation of the audio level, leading to fragmentation of the target audio (see the left). A noise gate contains “attack”, “hold”, and “release” phases, which introduce smooth transitions and prevent cutting off short audio signals (see the right).
Refer to caption
Figure 17: Visualization of example heatmaps in AVLMaps for multimodal goal reasoning for ambiguous object goals. The first column shows the positions of ambiguous objects (green bounding boxes) and the location of a sound (the icon of a speaker) or an image (the icon of a camera), while the second column shows the close-up view of ambiguous objects in the scene. The third column shows the predicted 3D heatmap for the object. The fourth column shows the heatmap for the extra modality, and the fifth column shows the fused heatmap after cross-modal reasoning. Sounds are artificially inserted into the scene for benchmarking and evaluation. The locations of sounds are not sound-source locations but the places where the sounds were heard. The heatmap is shown in the JET color scheme (red means the highest score and blue means the lowest score).
Refer to caption
Figure 18: Visualization of tabletop goal-reaching experiments. We set up two tabletop scenes and inserted sounds to different locations in the observation data (left). We show the indexing heatmap results of sound goals, visual goals, object goals, and ambiguous goal reaching results on the right. In addition, for visual goals, object goals, and ambiguous goals, we show the ground truth target locations (green cubes or green boxes) as well as the reached positions (red cubes). The heatmap is shown in the JET color scheme (red means the highest score and blue means the lowest score).

4.9 Real World Experiment for Table-Top Goal Reaching

A fixed Panda arm maps two tabletop scenes using calibrated wrist RGB-D observations and inserted ESC-50 audio. CLAP and OVSeg provide audio and object features; the image-localization module is unchanged. Success means ending within 10 cm of the target, not grasping or manipulating the object.

The arm succeeds on all 20 image goals and 12 sound goals, nine of 13 object goals, and nine of ten ambiguous multimodal goals. The experiment demonstrates more precise goal reaching and map reuse across embodiments, with limited scenes and artificially associated sound locations.

5 Limitations and Discussions

The map remembers past observations and does not react reliably to new, disappearing, or moving sounds. Monaural audio is attached to the listener’s pose, so the representation lacks acoustic source localization. Moving objects during mapping leave geometric and semantic artifacts, and later object relocation makes the stored map stale; neither short-term nor long-term dynamics is handled by the current system.

Adding another modality would require its own feature storage and retrieval module, sharp and broad heatmap APIs, and prompting examples. Odor, temperature, magnetic fields, and other inputs are proposed extensions rather than demonstrated capabilities. Integrating transient sound response and map updates remains future work.

6 Conclusion

The framework shows how pretrained semantic features, explicit geometry, and prompted API composition can support spatial and multimodal navigation. Its strongest evidence is improved ambiguous-goal retrieval and reusable maps; its constraints are prior exploration, module errors, coarse audio localization, dynamic scenes, and declining complete-sequence success. The tabletop results add target reaching, while broader manipulation remains outside the demonstrated task.

Concept demonstration

Open image ↗