Literature
Display
← All papers

CAVER: Curious Audiovisual Exploring Robot

Luca Macesanu · Boueny Folefack · Samik Singh · Ruchira Ray · Ben Abbatematteo · Roberto Martín-Martín

· ICRA 2026 ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

CAVER builds a scene-specific audiovisual memory by tapping objects with a spring-loaded tool. It stores multiscale visual features together with impact-sound descriptors, uses visual novelty to choose new interactions, and retrieves recorded sounds or candidate object parts by nearest-neighbor search. Experiments demonstrate more efficient sound prediction than random/cycling sampling, a reported 87% material-classification headline, musical-note inference of 64–68%, and manipulated-object inference of 42%. These are bounded tabletop tasks, not general audio-only skill learning.

I Introduction

The central problem is learning how an object sounds when touched, rather than recognizing appearance alone. CAVER joins a repeatable impact tool, visual/audio representations and an exploration policy that visits unfamiliar regions before exploiting its collected memory.

The edition follows the complete eight-page arXiv v2 dated 5 March 2026. The “66% melody imitation” headline summarizes the 64% xylophone-key and 68% drum-hit inference results; the paper does not report complete-song timing or sequence accuracy. Its learning avoids training a large paired generative model, but still uses pretrained vision models and synthetic paired data for feature-distance calibration.

Complete original PDF figure S1.F1.
Fig. 1: Curiously Building and Exploiting an Audiovisual Representation with CAVER. CAVER incrementally builds a KNN-based audiovisual representation capable of audio-to-visual and visual-to-audio prediction. To explore an environment, CAVER considers many candidate interaction points (dots), and ranks their uncertainty by comparing their visual features to those of prior samples. CAVER then collects an audio sample of the most uncertain point (red) using our novel impact tool. The audio and visual features are added as a pair to the audiovisual representation in an intrinsically motivated process that results in efficient interactive exploration.
Complete original PDF figure S1.F2.
Fig. 2: Overview of CAVER’s audiovisual representation and curious exploration. CAVER curiously and efficiently learns correlations between object visual appearance and acoustic properties. Given candidate hitting points, CAVER uses a KNN model with fine tuned features from foundation vision models to predict corresponding impact sounds. To select informative interaction points, CAVER selects the most uncertain candidate hitting location using distance in visual feature space between a candidate (red) and all prior samples (green) as a proxy for uncertainty. After sampling the most uncertain candidate, the corresponding visual and audio embeddings are paired as two sides of the Audio-Visual representation. This can best be thought of as a bi-directional mapping where audio features can be used to predict visual features, visual features can be used to predict audio features, and concatenating the embeddings gives an informative multimodal representation of that sample point.

II Related Work

Multimodal learning. Audio can complement vision and touch in robot control, as in ManiWAV. ObjectFolder supplies multisensory objects, while earlier sound-generation systems often rely on curated observations or detailed simulation.

Interactive perception. Physical actions reveal otherwise hidden properties, following the broader interactive-perception framework. CAVER specifically acquires contact sounds with its own tool. Curiosity. Related exploration work motivates seeking unfamiliar experiences; here the concrete score is distance from previous visual samples. Sound prediction. CAVER retrieves an existing impact recording from a single image/location, rather than synthesizing arbitrary new waveforms from video.

III CAVER: Curious Audiovisual Exploring Robot

The system assumes visually distinct regions often sound different and that local, environment-specific recordings help capture acoustic variation. Its memory stores interaction locations, visual features and audio descriptors. The same nearest-neighbor structure supports exploration, visual-to-audio prediction and audio-to-object retrieval; ambiguity remains when similar appearances hide different materials.

III-A Impact Tool

A parallel gripper actuates a cam–follower, loading a 3D-printed spring and releasing a metal rod for a short strike. The spring thickness is selected qualitatively at 2 mm, and a nearby directional microphone records at 44.1 kHz. Surface normals guide contact orientation. The authors report no measurable deterioration after 1,800 samples, but do not provide a general durability or contact-force benchmark. The tool standardizes excitation within these experiments.

III-B CAVER Audiovisual Representation

For each interaction point, concatenate object/part mask features from GroundedSAM with ResNet features from the object crop, part crop and a local patch. Store the normalized, clipped impact’s cepstral descriptors alongside those visual features. This retains both local appearance and broader object context.

Mel-cepstral distortion compares sound descriptors. They emphasize spectral shape rather than raw amplitude, helping compare timbre across strikes; normalization does not prove invariance to every contact force, shape, support or background condition.

Equation 1
𝐟⁡(x)=[𝐟objSAM,𝐟partSAM,𝐟objRN,𝐟partRN,𝐟patchRN]\mathbf{f}(x)=\big[\mathbf{f}^{\text{SAM}}_{\text{obj}},\,\mathbf{f}^{\text{SAM}}_{\text{part}},\,\mathbf{f}^{\text{RN}}_{\text{obj}},\,\mathbf{f}^{\text{RN}}_{\text{part}},\,\mathbf{f}^{\text{RN}}_{\text{patch}}\big] (1)

Aligning Audio and Visual Distances

Fit one scalar weight per visual-feature component by linear regression on synthetic ObjectFolder pairs, making weighted visual distances predict audio MCD. The explicit adaptation is this feature-distance weighting. Although Figure 2 describes “fine tuned features,” the method does not detail end-to-end backbone fine-tuning. Calibration uses prior paired data before online robot exploration.

Equation 2
dv​(xi,xj)=∑kwk​‖𝐟k​(xi)−𝐟k​(xj)‖2,d_{v}(x_{i},x_{j})=\sum_{k}w_{k}\,\big\|\mathbf{f}_{k}(x_{i})-\mathbf{f}_{k}(x_{j})\big\|_{2}, (2)

III-C Uncertainty-guided Exploration

Prompt GroundedSAM with a ground-truth object list, project masks into the depth point cloud, and score each candidate by its minimum weighted visual distance to any stored sample. Select the candidate with maximum distance, subject to cycling through detected objects before revisiting them. This combines object coverage with farthest-first sampling; “uncertainty” is a novelty proxy, not a calibrated prediction-error probability or an explicit information-gain estimate.

Equation 3
U⁡(x)=mins∈𝒮⁡dv​(x,s).U(x)=\min_{s\in\mathcal{S}}d_{v}\big(x,s\big). (3)

III-D Exploiting the Representation for Downstream Tasks

Sound prediction: retrieve the recorded audio paired with a visually nearby interaction. Material prediction: train supervised classifiers on manually labeled visual, audio or joint features. Audio-guided imitation: match a demonstration sound to candidate impacts and infer the corresponding object part. These applications reuse the collected memory, but material classification additionally requires human labels and task-specific learned heads.

Complete original PDF figure S3.F3.
Fig. 3: Robotic Impact Tool. To generate consistent impact sounds, we designed a 3D-printed, spring-loaded impact tool that attaches to the robot’s gripper. When the gripper closes, a cam-follower retracts the rod, building tension in the spring until the cam slips and the stored energy in the spring drives the metal rod forward, impacting the object. A directional microphone is mounted near the scene camera to record the resulting sound at 44.1 kHz.

IV Experimental Evaluation

A Franka Research 3 uses a fixed RGB-D camera, depth-derived surface normals, RRT planning and OctoMap geometry. Audio thresholds reject failed impacts. Each recording is peak-aligned, cropped from 10 ms before to 100 ms after the peak, amplitude-normalized and resampled to 22.05 kHz. Spectral envelopes use 5 ms frames and a 512-point FFT, then 13th-order mel-cepstral coefficients with α=0.65 are flattened.

End-to-end collection averages 52.9 seconds per successful sample, about 68/hour, including failed attempts and replanning. Reported environment means are 53.7±3.7 seconds in the garage, 56.2±8.5 in the kitchen and 48.6±3.1 in the playroom. This describes acquisition throughput, not neural inference latency.

Complete original PDF figure S4.F4. (a) Our experiments take place across 3 environments: kitchen , garage , and playroom . (b) Audio prediction error as a function of samples collected for different audiovisual exploration approaches for the kitchen , garage , and playroom .
Fig. 4: Curious Exploration for Audio Prediction. The top row (a) visualizes the objects that comprise each environment. The bottom row (b) shows the average audio prediction error on a held-out test set, measured as the mel-cepstral distortion, as a function of the number of interaction points sampled by the robot. Vertical lines indicate scenes within an environment in which a subset of the objects appear. Each environment is plotted individually (left to right: kitchen, garage, and playroom). CAVER consistently achieves higher prediction accuracy more quickly than the naive exploration baselines via its curious exploration and audiovisual representation.
Complete original PDF figure S4.F5.
Fig. 5: Material Classification Results. We measure the efficacy of CAVER’s unified audiovisual embeddings for material classification compared to visual embeddings alone, audio embeddings alone, and a random baseline. The plots show the class-balanced material classification accuracy on a held-out test set as a function of the number of interaction points sampled by the robot, aggregated over 20 runs. Performance is evaluated in the kitchen, garage, and playroom environments, respectively. Incorporating both audio and visual inputs as in CAVER’s unified embedding is necessary to achieve strong material classification performance given ambiguous stimuli.

IV-A Can CAVER enable a robot to efficiently gather data for training an audio prediction model?

Each of three tabletop environments contains five successive scenes, including recurring objects. A manually collected test set covers object parts; test points for a scene enter the evaluation when that scene starts. Compare novelty-plus-cycling against uniform random point sampling and cycling with random points.

Figure 4 generally shows lower MCD with CAVER, especially after previously seen objects recur. The test pool changes at scene boundaries, so the curve is not learning on one fixed test distribution. Visually similar plastic and ceramic white mugs remain ambiguous: visual distance alone cannot reliably retrieve distinct sounds for indistinguishable-looking instances.

Complete original PDF figure S4.F6. (a) CAVER replicating a song on the drums and xylophone. (b) CAVER imitating previously unheard object interactions.
Fig. 6: Audio Imitation with CAVER. CAVER’s audiovisual representation enables imitation of solely auditory demonstrations. In (a), CAVER plays a song by ear after exploring the instruments autonomously. In (b), the robot replicates unseen manipulation actions by predicting what objects sound like together (ie, combining the audio from a wood cup sample and a ceramic plate sample) and comparing them to the observed sound. The ability to reason about multi-object interaction sounds gives CAVER the flexibility to be applied to diverse downstream tasks.

IV-B Can CAVER enable efficient learning of object material prediction given audio data?

Predict ceramic, glass, metal, plastic, rubber or wood using visual MLP features, audio CNN features, or both. The audio classifier uses MFCC, chroma and mel-spectrogram features, rather than only the simple MCD retrieval representation. Figure 5 reports class-balanced held-out accuracy over 20 randomized training-data orders and generally shows the joint model performing best across the three environments. The text reports 87%, while the playroom curve approaches 100%; this headline and the environment-specific curves should not be treated as one identical aggregate metric.

These 20 runs are order randomizations of collected data, not 20 independently repeated robot explorations. The paper does not fully specify split sizes, trial counts for every task or the meaning of the shaded bands. The headline accuracy should not be treated as uniform six-class performance on arbitrary unseen household objects.

IV-C Can CAVER ’s audiovisual representation enable an agent to reproduce musical sounds?

After exploring a xylophone or drum setup, compare predicted candidate sounds to human-played test notes by MCD and select the closest match. Correct-note inference is 64% for xylophone keys and 68% for drum hits. Humans receive a few minutes of familiarization, but a numerical human music score and sample count are not given in the manuscript. Figure 6 shows demonstrations; the quantitative endpoint is identifying the target note/hit, not a complete musical-performance score.

IV-D Can CAVER enable activity recognition by inferring objects that are interacted with based on sound alone?

A human places an object on a plate; the placement object is assumed known. CAVER superimposes predicted sounds of candidate picked objects with the known placement object’s sound, then selects the lowest-MCD match. Manipulated-object inference reaches 42%, versus a reported average human 27%.

This is candidate-object selection for a specified pick-and-place setting after prior object exploration. It does not identify arbitrary action sequences or recover a manipulation policy from sound alone. Human participant/trial counts and uncertainty are not supplied, limiting assessment of the reported comparison.

V Conclusion and Limitations

CAVER demonstrates practical active acquisition of impact-sound memory and several uses for it. Limitations include no continuous-contact sounds such as dragging, separate exploration and inference phases, no dedicated background-noise filtering, discarded failure states, and a required ground-truth object list for segmentation. Visual aliasing and environmental acoustics further constrain transfer. Open-ended noisy homes and broader comparisons remain future work.

VI Acknowledgements

The authors acknowledge David Harwath’s audio-representation guidance and partial funding from the University of Texas at Austin’s Good Systems initiative.

Concept demonstration

Open image ↗