At a glance
CAVER builds a scene-specific audiovisual memory by tapping objects with a spring-loaded tool. It stores multiscale visual features together with impact-sound descriptors, uses visual novelty to choose new interactions, and retrieves recorded sounds or candidate object parts by nearest-neighbor search. Experiments demonstrate more efficient sound prediction than random/cycling sampling, a reported 87% material-classification headline, musical-note inference of 64–68%, and manipulated-object inference of 42%. These are bounded tabletop tasks, not general audio-only skill learning.
I Introduction
The central problem is learning how an object sounds when touched, rather than recognizing appearance alone. CAVER joins a repeatable impact tool, visual/audio representations and an exploration policy that visits unfamiliar regions before exploiting its collected memory.
The edition follows the complete eight-page arXiv v2 dated 5 March 2026. The “66% melody imitation” headline summarizes the 64% xylophone-key and 68% drum-hit inference results; the paper does not report complete-song timing or sequence accuracy. Its learning avoids training a large paired generative model, but still uses pretrained vision models and synthetic paired data for feature-distance calibration.
II Related Work
Multimodal learning. Audio can complement vision and touch in robot control, as in ManiWAV. ObjectFolder supplies multisensory objects, while earlier sound-generation systems often rely on curated observations or detailed simulation.
Interactive perception. Physical actions reveal otherwise hidden properties, following the broader interactive-perception framework. CAVER specifically acquires contact sounds with its own tool. Curiosity. Related exploration work motivates seeking unfamiliar experiences; here the concrete score is distance from previous visual samples. Sound prediction. CAVER retrieves an existing impact recording from a single image/location, rather than synthesizing arbitrary new waveforms from video.
III CAVER: Curious Audiovisual Exploring Robot
The system assumes visually distinct regions often sound different and that local, environment-specific recordings help capture acoustic variation. Its memory stores interaction locations, visual features and audio descriptors. The same nearest-neighbor structure supports exploration, visual-to-audio prediction and audio-to-object retrieval; ambiguity remains when similar appearances hide different materials.
III-A Impact Tool
A parallel gripper actuates a cam–follower, loading a 3D-printed spring and releasing a metal rod for a short strike. The spring thickness is selected qualitatively at 2 mm, and a nearby directional microphone records at 44.1 kHz. Surface normals guide contact orientation. The authors report no measurable deterioration after 1,800 samples, but do not provide a general durability or contact-force benchmark. The tool standardizes excitation within these experiments.
III-B CAVER Audiovisual Representation
For each interaction point, concatenate object/part mask features from GroundedSAM with ResNet features from the object crop, part crop and a local patch. Store the normalized, clipped impact’s cepstral descriptors alongside those visual features. This retains both local appearance and broader object context.
Mel-cepstral distortion compares sound descriptors. They emphasize spectral shape rather than raw amplitude, helping compare timbre across strikes; normalization does not prove invariance to every contact force, shape, support or background condition.
Equation 1
| (1) |
Aligning Audio and Visual Distances
Fit one scalar weight per visual-feature component by linear regression on synthetic ObjectFolder pairs, making weighted visual distances predict audio MCD. The explicit adaptation is this feature-distance weighting. Although Figure 2 describes “fine tuned features,” the method does not detail end-to-end backbone fine-tuning. Calibration uses prior paired data before online robot exploration.
Equation 2
| (2) |
III-C Uncertainty-guided Exploration
Prompt GroundedSAM with a ground-truth object list, project masks into the depth point cloud, and score each candidate by its minimum weighted visual distance to any stored sample. Select the candidate with maximum distance, subject to cycling through detected objects before revisiting them. This combines object coverage with farthest-first sampling; “uncertainty” is a novelty proxy, not a calibrated prediction-error probability or an explicit information-gain estimate.
Equation 3
| (3) |
III-D Exploiting the Representation for Downstream Tasks
Sound prediction: retrieve the recorded audio paired with a visually nearby interaction. Material prediction: train supervised classifiers on manually labeled visual, audio or joint features. Audio-guided imitation: match a demonstration sound to candidate impacts and infer the corresponding object part. These applications reuse the collected memory, but material classification additionally requires human labels and task-specific learned heads.
IV Experimental Evaluation
A Franka Research 3 uses a fixed RGB-D camera, depth-derived surface normals, RRT planning and OctoMap geometry. Audio thresholds reject failed impacts. Each recording is peak-aligned, cropped from 10 ms before to 100 ms after the peak, amplitude-normalized and resampled to 22.05 kHz. Spectral envelopes use 5 ms frames and a 512-point FFT, then 13th-order mel-cepstral coefficients with α=0.65 are flattened.
End-to-end collection averages 52.9 seconds per successful sample, about 68/hour, including failed attempts and replanning. Reported environment means are 53.7±3.7 seconds in the garage, 56.2±8.5 in the kitchen and 48.6±3.1 in the playroom. This describes acquisition throughput, not neural inference latency.
IV-A Can CAVER enable a robot to efficiently gather data for training an audio prediction model?
Each of three tabletop environments contains five successive scenes, including recurring objects. A manually collected test set covers object parts; test points for a scene enter the evaluation when that scene starts. Compare novelty-plus-cycling against uniform random point sampling and cycling with random points.
Figure 4 generally shows lower MCD with CAVER, especially after previously seen objects recur. The test pool changes at scene boundaries, so the curve is not learning on one fixed test distribution. Visually similar plastic and ceramic white mugs remain ambiguous: visual distance alone cannot reliably retrieve distinct sounds for indistinguishable-looking instances.
IV-B Can CAVER enable efficient learning of object material prediction given audio data?
Predict ceramic, glass, metal, plastic, rubber or wood using visual MLP features, audio CNN features, or both. The audio classifier uses MFCC, chroma and mel-spectrogram features, rather than only the simple MCD retrieval representation. Figure 5 reports class-balanced held-out accuracy over 20 randomized training-data orders and generally shows the joint model performing best across the three environments. The text reports 87%, while the playroom curve approaches 100%; this headline and the environment-specific curves should not be treated as one identical aggregate metric.
These 20 runs are order randomizations of collected data, not 20 independently repeated robot explorations. The paper does not fully specify split sizes, trial counts for every task or the meaning of the shaded bands. The headline accuracy should not be treated as uniform six-class performance on arbitrary unseen household objects.
IV-C Can CAVER ’s audiovisual representation enable an agent to reproduce musical sounds?
After exploring a xylophone or drum setup, compare predicted candidate sounds to human-played test notes by MCD and select the closest match. Correct-note inference is 64% for xylophone keys and 68% for drum hits. Humans receive a few minutes of familiarization, but a numerical human music score and sample count are not given in the manuscript. Figure 6 shows demonstrations; the quantitative endpoint is identifying the target note/hit, not a complete musical-performance score.
IV-D Can CAVER enable activity recognition by inferring objects that are interacted with based on sound alone?
A human places an object on a plate; the placement object is assumed known. CAVER superimposes predicted sounds of candidate picked objects with the known placement object’s sound, then selects the lowest-MCD match. Manipulated-object inference reaches 42%, versus a reported average human 27%.
This is candidate-object selection for a specified pick-and-place setting after prior object exploration. It does not identify arbitrary action sequences or recover a manipulation policy from sound alone. Human participant/trial counts and uncertainty are not supplied, limiting assessment of the reported comparison.
V Conclusion and Limitations
CAVER demonstrates practical active acquisition of impact-sound memory and several uses for it. Limitations include no continuous-contact sounds such as dragging, separate exploration and inference phases, no dedicated background-noise filtering, discarded failure states, and a required ground-truth object list for segmentation. Visual aliasing and environmental acoustics further constrain transfer. Open-ended noisy homes and broader comparisons remain future work.
VI Acknowledgements
The authors acknowledge David Harwath’s audio-representation guidance and partial funding from the University of Texas at Austin’s Good Systems initiative.