Literature
Display
← All papers

Supervising Sound Localization by In-the-wild Egomotion

Anna Min · Ziyang Chen · Hang Zhao · Andrew Owens

· CVPR 2025 Spotlight ↗

Contents

Overview

The complete teaser illustrates camera motion relative to multiple possible sound sources. Its unknown-source setting motivates weak consistency supervision rather than requiring the dominant sound to appear visibly in the image.

Refer to caption
Figure 1: Supervising sound localization using egomotion from natural video. We use camera motion as a supervisory signal for stereo sound localization. Our model learns to predict changes in sound direction that correlate with changes in visually indicated camera motion. In contrast to prior self-supervised sound localization methods, our model is trained solely on in-the-wild natural videos, which often contain complex camera motions and noisy mixtures of different component sounds.

At a glance

The method trains an audio-only azimuth model using directions of camera rotation/translation estimated from synchronized video. Interaural intensity difference adds complementary left/right supervision, avoiding manual sound-angle labels for the main training objective.

StereoWalks supplies internet walking-tour and controlled recording data. The full model reaches 34° mean angular error on the tested iPhone subset, versus 71.4° for the GTRot comparison. Results depend on device, scene and evaluation convention, with important limits from overlapping sounds and front/back ambiguity.

1 Introduction

Natural stereo recordings vary in microphone geometry, channel response and mixtures of sound sources. Simulation or a single hand-chosen audio cue may transfer poorly to that diversity.

The proposed training uses moments when camera-induced angular change dominates source motion. Off-the-shelf visual geometry provides motion-direction constraints, and audio intensity cues help orient the predicted angles. The method avoids requiring a visible sound-emitting object, but still assumes useful synchronization and motion consistency [13].

Refer to caption
Figure 2: The StereoWalks dataset. Example videos from different subsets of our dataset. We focus on in-the-wild audio-visual data sourced from the internet. Our dataset contains YouTube videos recorded with stereo microphones and iPhones, as well as lab-collected in-ear binaural and stereo data. We also show the estimated distribution of different sound source categories in each subset, to help visualize the contents of each dataset.

2 Related Work

The paper connects spatial audio learning, audio-visual correspondence and visual camera-motion estimation. Its distinction is weak sound-direction supervision from natural-video egomotion rather than jointly learning a visual pose model from simulated scenes.

Audio-visual spatial perception

Prior work relates spatial audio to images, scene geometry, source separation and camera motion. Here pretrained visual matching/geometry is used as a teacher; the deployed direction predictor takes stereo audio [13].

Spatial audio datasets

Existing spatial-audio datasets often use specialized multichannel rigs, controlled rooms or synthetic acoustics. StereoWalks adds internet recordings with camera motion and human-labeled evaluation subsets, alongside controlled stereo/binaural recordings.

Audio-visual learning.

Related audio-visual learning uses temporal correspondence, semantic objects or sound separation. This method uses geometric consistency over motion, which can supervise sources outside the field of view.

Camera motion estimation

Feature matching and camera calibration estimate relative pose. SuperGlue provides correspondences and Perspective Fields estimates field of view; the audio model is trained separately rather than jointly estimating visual geometry [46] [29].

3 Method

Two short audio segments from a moving-camera video should have sound directions compatible with visual motion. Binary motion-direction labels constrain the probability assigned to pairs of predicted angles.

Complete original Table 1 : Dataset comparison. We provide details on dataset length, the proportion of visible sound sources, camera motion types, and sound source types, with each clip representing 5 seconds. Visibility is represented by two numbers: the first indicates the number of sound sources visible for over 4 seconds within the 5-second clip, and the second denotes the number of sound sources visible for less than 4 seconds. IID Binaural acc denotes the accuracy of IID predictions respected by the actual left-right labels.
Table 1: Dataset comparison. We provide details on dataset length, the proportion of visible sound sources, camera motion types, and sound source types, with each clip representing 5 seconds. Visibility is represented by two numbers: the first indicates the number of sound sources visible for over 4 seconds within the 5-second clip, and the second denotes the number of sound sources visible for less than 4 seconds. IID Binaural acc denotes the accuracy of IID predictions respected by the actual left-right labels.
Accessible table and original citation links
Dataset Scene subset Split Size Visibility
(%)
Motion Type IID binaural
acc (%)
Clips (k) Duration (hr) Camera Sound sources
StereoWalks (ours) YT-Stereo Train 14.4 20.0 20 / 50 Rotation&Translation Unknown –
Val 00.1 00.2 10 / 60 Mainly Rotation Moving 57.5
Test 00.1 00.2 10 / 60 Mainly Rotation Moving 57.5
Stereo-Fountain Raw 05.0 07.0 10 / 30 Mainly Stationary ✗ 76.2
Binaural-Fountain Raw 01.4 02.0 10 / 30 Mainly Stationary ✗ 98.0
L/R Binaural [11] Raw 01.8 03.0 – ✗ ✗ 75.4
Simulated HM3D-SS 2.0 (ours) Raw 21.6 30.0 – Rotation&Translation Rendered 97.4

3.1 The StereoWalks dataset

StereoWalks collects roughly 20 training hours from stereo walking tours, with an iPhone-focused subset identified using titles/captions. Evaluation clips receive perceived sound-angle and category labels.

Controlled recordings compare an iPhone 13 Pro near a fountain with an in-ear Sennheiser AMBEO headset. Those subsets test device/head-related cues. Reported collection counts differ between the main and supplementary tables; both original tables remain.

3.2 Learning audio localization from egomotion

The model predicts a distribution over 32 azimuth categories from stereo spectrograms. The core formulation restricts angles to the frontal half-plane, excluding elevation and distance.

A single ordinary stereo segment can be ambiguous between front and back. The main evaluation reflects back angles to the front; full-circle localization is examined separately with supervised device comparisons.

Refer to caption
Figure 3: Method overview. We use a camera’s egomotion to supervise binaural sound localization models. We obtain the rotation and translation of the camera using traditional methods from multi-view geometry. We then train an audio model to predict sound directions that are consistent with visual camera motions, using a dataset that includes in-the-wild walking tours.

Camera rotation estimation.

Estimated clockwise/counterclockwise camera rotation supplies a binary pseudo label. Only the direction is needed, reducing dependence on precise calibrated rotation magnitude.

Camera front-back translation estimation.

Estimated forward/backward camera translation constrains relative angular changes when source motion is sufficiently slow. The constraint becomes less informative for distant sources, while moving or competing sources can violate it.

Supervising audio localization using sight.

Rotation and translation define compatible pairs of azimuth bins. The losses sum products of the two predicted distributions over those allowed pairs, then apply the stated cross-entropy objectives. Equal-angle cases are excluded.

The prose calls this a sum of individual cross-entropies, whereas the displayed expressions use a masked probability sum inside the loss. Both source formulas are retained; the precise scalar/binary normalization is not fully spelled out.

Equations 1–2
ℒ𝚛𝚘𝚝=L𝚌𝚎​(∑(i,j)∈Rf​(𝐬1)i​f​(𝐬2)j,dr),\mathcal{L}_{\mathtt{rot}}=L_{\mathtt{ce}}\left(\sum_{(i,j)\in R}f({\mathbf{s}}_{1})_{i}f({\mathbf{s}}_{2})_{j},d_{r}\right), (1)
ℒ𝚝𝚛𝚊𝚗𝚜=L𝚌𝚎​(∑(i,j)∈Tf​(𝐬1)i​f​(𝐬2)j,dt),\mathcal{L}_{\mathtt{trans}}=L_{\mathtt{ce}}\left(\sum_{(i,j)\in T}f({\mathbf{s}}_{1})_{i}f({\mathbf{s}}_{2})_{j},d_{t}\right), (2)

Binaural cues

Interaural intensity difference estimates which side is louder. A complementary loss favors direction bins consistent with that left/right pseudo label.

It is a noisy physical cue rather than manually annotated ground truth. Its reported accuracy varies from 57.5% on internet iPhone clips to about 98% on the in-ear fountain set.

Equation 3
ℒ𝚋𝚒𝚗=L𝚌𝚎​(∑j∈Bf​(𝐬i)j,bt),\mathcal{L}_{\mathtt{bin}}=L_{\mathtt{ce}}\left(\sum_{j\in B}f({\mathbf{s}}_{i})_{j},b_{t}\right), (3)

Overall loss.

The full objective weights rotation/translation by 0.9/0.1 and gives the intensity-cue loss weight 1. The egomotion-only variant sets the intensity term to zero. All four numbered equations remain below.

Equation 4
ℒ=λ1​L𝚛𝚘𝚝+(1−λ1)​L𝚝𝚛𝚊𝚗𝚜+λ2​L𝚋𝚒𝚗,\mathcal{L}=\lambda_{1}L_{\mathtt{rot}}+(1-\lambda_{1})L_{\mathtt{trans}}+\lambda_{2}L_{\mathtt{bin}}, (4)
Complete original Table 2 : Comparison with state-of-the-art methods and other self-supervised methods on YT-Stereo and In-the-wild dataset. Sim. denote the simulated dataset.
Table 2: Comparison with state-of-the-art methods and other self-supervised methods on YT-Stereo and In-the-wild dataset. Sim. denote the simulated dataset.
Accessible table and original citation links
Model Training Set Test Set
YT-Stereo Sim. YT-Stereo-iPhone Stereo-Fountain Simulated
MAE (°)↓\downarrow 2clf (%)↑\uparrow 8clf (%)↑\uparrow MAE (°)↓\downarrow 2clf (%)↑\uparrow 8clf (%)↑\uparrow MAE (°)↓\downarrow 2clf (%)↑\uparrow 8clf (%)↑\uparrow
Chance 55.3 46.7 12.7 62.1 52.0 16.0 39.4 49.0 15.0
IID – direct – 57.5 – – 97.0 – – 97.4 –
GTRot [14] ✓ 71.4 54.0 7.1 57.2 97.0 18.4 40.2 48.7 18.7
Ours – IID only [14] ✓ 37.3 54.7 28.0 30.1 97.0 46.0 88.3 48.7 6.8
Ours – Simulated ✓ 73.4 61.7 13.3 28.5 85.7 22.3 9.8 98.3 63.4
Ours – Full ✓ 34.0 61.7 33.3 29.3 97.3 46.0 57.2 48.7 10.7
In-the-wild audio [11] 2clf (%)↑\uparrow
Chance 50.0
IID – direct 75.4
GCC-PHAT 77.2
GTRot [14] 84.9
MonoCLR [11] 87.4
StereoCRW [11] 87.2
Ours – IIDonly 87.7
Ours – full 87.5

4 Experiments

Evaluation combines internet audio, controlled device recordings and simulated motion/sound mixtures. Angular error and coarse left/right classification measure different aspects of performance.

4.1 Datasets

Real datasets provide natural recording variation; SoundSpaces simulations isolate camera motion, source overlap and intermittency. The simulation data are an analysis/training baseline, while the main proposed model can learn from natural videos.

Real-world dataset

Evaluation curation selects clips with reliable camera rotation, substantial intensity changes and agreement between the two directions, then annotates sources/categories. The test set therefore emphasizes conditions in which the supervisory assumptions are easier to satisfy.

The main text describes five-second clips with three-second overlap, while the supplement says clips every three seconds. That stride/overlap discrepancy is retained rather than resolved by assumption.

Simulated dataset

HM3D-SS2.0 renders binaural room responses in scene-disjoint train/validation/test environments and convolves them with LibriSpeech. Camera/source motion and competing/intermittent sources extend the earlier simulation [6] [38].

The supplement reports 81/9/10 scenes,50,000 audio-visual pairs and 200,000 viewpoints, with about 30 rendered hours for the size-limited comparisons. Discrete source positions approximate motion.

4.2 Implementation details

ResNet-18 maps magnitude/phase stereo spectrograms of size 256×256×4 to 32 azimuth logits. Training uses an 80 GB A100, batch 300 and 100 epochs; later details specify AdamW at 10⁻⁴, cosine annealing and early stopping.

Table 2 first trains on YT-Stereo and part of the iPhone subset, then finetunes for one epoch on each target dataset. These adapted comparisons should not be read as completely zero-shot cross-dataset transfer. Each reported experiment is run once with a shared seed.

Complete original Table 3: 360-Degree Sound Localization Prediction Results for YT-Stereo-iPhone and Stereo-Fountain Datasets. The YT-Stereo-iPhone dataset includes rotations and translations with mostly moving sound sources, while the Stereo-Fountain dataset has mostly stationary cameras and sound sources.
Table 3: 360-Degree Sound Localization Prediction Results for YT-Stereo-iPhone and Stereo-Fountain Datasets. The YT-Stereo-iPhone dataset includes rotations and translations with mostly moving sound sources, while the Stereo-Fountain dataset has mostly stationary cameras and sound sources.
Accessible table and original citation links
Model YT-Stereo-iPhone   Stereo-Fountain  
MAE (°) ↓\downarrow 2clf (%) ↑\uparrow 8clf (%) ↑\uparrow MAE (°) ↓\downarrow 2clf (%) ↑\uparrow 8clf (%) ↑\uparrow
Chance 55.3 46.7 12.7 62.1 52.0 16.0
IID-direct – 57.5 – – 97.0 –
GTRot [14] 71.4 54.0 7.1 55.1 97.1 18.7
Ours – Simulated 73.4 61.7 13.3 28.5 85.7 22.3
Ours – IID only [14] 37.3 54.7 28.0 29.6 97.3 46.0
Ours – R&B 34.5 55.4 33.4 28.5 97.3 46.0
Ours – T&B 37.4 53.4 26.8 – – –
Ours – Full 34.0 61.7 31.7 28.5 97.3 46.0
Supervised – – – 33.8 98.0 49.5
Supervised – RTF – – – 27.1 98.0 49.5

4.3 Evaluation

Metrics are mean angular error, two-way left/right accuracy and eight-bin classification. Probability-weighted bin midpoints form the angle estimate.

The stated reflection convention removes front/back ambiguity for the main comparison. Despite Table 3’s 360-degree heading, its results should not be taken as evidence of fully resolved front/back direction without reconciling that evaluation description.

Baselines and ablations

Comparisons include direct intensity cues, GTRot’s stronger rotation supervision, simulated training, supervised angles and supervised reflection-to-front. Ablations retain intensity alone, rotation plus intensity, translation plus intensity, or egomotion alone.

The older in-the-wild binaural benchmark also includes time-delay/contrastive methods. Different training sets and information make these complete-system comparisons rather than uniformly identical supervision budgets [12] [13].

4.4 Results

On internet iPhone clips, the full model improves angular error from 37.3° for intensity-only to 34.0°, and left/right accuracy from 54.7% to 61.7%. Coarse localization remains difficult. GTRot’s 71.4° error is much worse in this setting.

On the older binaural benchmark, full-model 87.5% accuracy is comparable to MonoCLR87.4% and StereoCRW87.2%, while intensity-only scores 87.7%. Visual supervision is therefore not uniformly best across every dataset/metric.

Real-trained and simulation-trained models transfer poorly in opposite directions. The complete tables retain that domain gap, target-specific finetuning and differences between the two real-data evaluation protocols.

Complete original Table 4 : Overlapping sounds make in-the-wild sound localization challenging: Experiments on three simulated datasets where the camera makes small translations in all settings. To better align with the real-world experiments, as in settings (1)(2)(3), the dataset is restricted to 30 hours, while in (4), the dataset is not limited. “Intermittent” refers to sound sources that are silent for half the time. “Overlap” refers to the presence of multiple sound sources.
Table 4: Overlapping sounds make in-the-wild sound localization challenging: Experiments on three simulated datasets where the camera makes small translations in all settings. To better align with the real-world experiments, as in settings (1)(2)(3), the dataset is restricted to 30 hours, while in (4), the dataset is not limited. “Intermittent” refers to sound sources that are silent for half the time. “Overlap” refers to the presence of multiple sound sources.
Accessible table and original citation links
Model (1) One Source (2) Overlap (3) Overlap Intermittent (4) More Simulated Data of (3)
MAE(°)↓\downarrow 2clf(%)↑\uparrow 8clf(%)↑\uparrow MAE(°)↓\downarrow 2clf(%)↑\uparrow 8clf(%)↑\uparrow MAE(°)↓\downarrow 2clf(%)↑\uparrow 8clf(%)↑\uparrow MAE(°)↓\downarrow 2clf(%)↑\uparrow 8clf(%)↑\uparrow
Chance 39.4 49.0 15.0 39.4 49.0 15.0 39.4 49.0 15.0 39.4 49.0 15.0
IID - 97.4 - - 80.1 - - 80.8 - - 80.8 -
GTRot [14] 4.3 97.5 84.7 22.5 83.5 33.5 26.9 81.5 32.5 19.7 87.2 41.0
Ours – IIDonly [14] 22.2 99.0 35.7 25.6 78.8 26.1 28.0 78.0 23.7 26.7 79.5 25.7
Ours – Full 9.8 98.3 63.4 23.2 87.2 36.2 24.1 82.1 35.7 23.7 82.4 41.2
Supervised 2.8 98.7 89.8 10.8 91.8 70.2 8.4 94.1 74.6 5.7 96.0 80.2
Complete original Table 5 : Relationship between ego-translation and relative motion of sources. Experiments were conducted with various types of motion and distances from the camera using the simulated dataset HMSS-3D 2.0 made by SoundSpaces 2.0 [ 7 ] . We report Mean Absolute Error in the unit of degrees (°), and accuracy of 2clf and 8clf in the unit of percentage (%).
Table 5: Relationship between ego-translation and relative motion of sources. Experiments were conducted with various types of motion and distances from the camera using the simulated dataset HMSS-3D 2.0 made by SoundSpaces 2.0 [7]. We report Mean Absolute Error in the unit of degrees (°), and accuracy of 2clf and 8clf in the unit of percentage (%).
Accessible table and original citation links
Model (1) Rotation Only (2) Translation Only (3) Rotation&Translation (4) Distant Rotation (5) Distant R&T (6) Overlap R&T
MAE↓\downarrow 2clf↑\uparrow 8clf↑\uparrow MAE↓\downarrow 2clf↑\uparrow 8clf↑\uparrow MAE↓\downarrow 2clf↑\uparrow 8clf↑\uparrow MAE↓\downarrow 2clf↑\uparrow 8clf↑\uparrow MAE↓\downarrow 2clf↑\uparrow 8clf↑\uparrow MAE↓\downarrow 2clf↑\uparrow 8clf↑\uparrow
Chance 39.4 49.0 15.0 39.4 49.0 15.0 39.4 49.0 15.0 39.9 49.2 15.7 39.8 49.1 15.8 39.0 49.5 15.2
IID - 97.4 - - 97.4 - - 97.4 - - 95.6 - - 95.1 - - 73.5 -
GTRot [14] 4.3 97.5 84.7 - - - 4.4 97.5 84.5 4.5 97.1 82.0 4.7 96.7 86.9 28.1 79.6 30.1
Ours – IIDonly [14] 22.2 99.0 35.7 22.7 99.0 35.7 22.0 99.1 35.9 23.1 97.8 35.2 23.0 98.1 35.0 32.5 75.0 25.7
Ours – R&B 9.8 98.3 63.4 - - - 10.1 98.0 60.1 10.7 98.1 59.1 11.7 98.6 58.7 25.6 81.4 34.6
Ours – T&B - - - 23.2 96.0 35.7 23.0 95.7 35.0 - - - 22.7 96.7 35.1 30.1 72.5 22.3
Ours – R&T - - - - - - 14.7 96.3 48.3 - - - 22.5 95.0 45.7 26.7 79.0 29.2
Ours – Full 9.8 98.3 63.4 23.2 96.0 35.7 11.7 98.0 58.7 11.2 98.1 56.8 11.9 97.6 55.3 28.6 76.0 28.5
Supervised 2.8 98.7 89.6 2.6 99.1 89.9 2.6 98.9 89.5 2.7 98.8 90.0 2.5 99.0 90.4 12.1 86.0 65.4

4.5 Analysis

Controlled experiments test whether motion-direction supervision helps under competing sound, how translation changes with distance, and whether in-ear recordings improve front/back discrimination.

Overlapping sound assumption

Simulations add a secondary source at 0.7 relative loudness, optionally making both sources independently silent half the time. The first three settings use about 30 hours; a fourth adds more rendered data.

GTRot is strongest among the self-supervised comparisons for a single source. In continuous overlap, it also has slightly lower angular error than the full model,22.5° versus 23.2°, while the full model wins coarse classification. Intermittency favors the proposed error metric; with more data GTRot again gives lower error,19.7° versus 23.7°. Supervised labels remain substantially stronger.

Rotation and translation

Rotation is useful for distant sources; translation changes are weaker at larger range. Removing translation on internet tours gives similar angular error,34.5° versus 34.0°, though other metrics differ.

In near-source rotation-plus-translation simulation, egomotion-only error 14.7° approaches the full model’s 11.7°. In overlapping motion, rotation-plus-intensity gives 25.6° versus full 28.6°. These results support condition-dependent cues rather than universal benefit from adding every loss.

Complete original Table 6 : Evaluation of sound localization on the Stereo-Fountain and Binaural-Fountain datasets, showing improved accuracy in front/back localization with binaural recordings.
Table 6: Evaluation of sound localization on the Stereo-Fountain and Binaural-Fountain datasets, showing improved accuracy in front/back localization with binaural recordings.
Accessible table and original citation links
Dataset MAE ↓\downarrow Left/Right Acc ↑\uparrow Front/Back Acc ↑\uparrow
Stereo-Fountain 33.8 98.0% 51.0%
Binaural-Fountain 27.8 99.0% 69.3%

Front-Back ambiguity

A separate supervised 360-degree experiment compares recordings beside the same fountain. Front/back accuracy is 51.0% with the iPhone and 69.3% with the in-ear headset; angular error is 33.8° versus 27.8°.

This controlled device comparison supports extra head/ear cues, while still leaving many front/back mistakes. It is not a full-circle result obtained solely by the main weakly supervised method.

Refer to caption
Figure 4: Visualizations of results in the internet walking tour videos (the YT-Stereo subset). We show our predictions and ground-truth annotations with angles and audio class labels.

5 Conclusion

Visual motion directions can improve audio localization learned from natural video, especially relative to simulation-only transfer. The contribution combines a weak-supervision recipe with a dataset and controlled analysis of acoustic ambiguity.

Limitations

Low-quality stereo, overlapping events, source motion and biased clip filtering remain limitations. Selecting stable, clear events can omit brief sounds and does not establish performance over arbitrary walking-tour audio.

Future work

The authors propose richer visual/contextual information for harder real-world sound localization. Robust multi-source, device-independent and full-circle perception remain open beyond these experiments.

Acknowledgements

The work was completed during Anna Min’s internship at the University of Michigan. The original acknowledgment credits research feedback and preliminary YouTube recordings.

A.1 Dataset

The supplement retains collection, filtering, annotation and recording details beyond the main dataset summary. Both the original expanded data table and full annotation questionnaire remain.

A.1.1 Internet data collection

Internet collection searches walking-tour videos and removes portrait or mono recordings. Useful supervision assumes source angular motion is slower than camera-induced angular change.

The reported 20-hour training pool follows curation, although a later paragraph calls training unfiltered. That wording does not fully specify which processing stages are shared between training and the more selective evaluation set.

Complete original angle-distribution histogram
Figure 5: The angle distribution of YT-Stereo-iPhone validation set. The angles range from -180 to 180 without filtering.

A.1.2 Dataset Filtering

The supplement details motion/intensity screening and controlled device recordings. Random clip splits within recordings differ from testing unseen recording sessions, environments or speakers.

YT-Stereo-iPhone

The authors select 100 validation clips from 1,600 filtered candidates. A preliminary motion stage uses a 10° rotation threshold; later evaluation curation requires 15°, intensity/motion-direction agreement and angle-distribution balancing.

Figure 5’s caption states angles from−180°to 180°, but the displayed histogram axis spans roughly−100°to 100°. The original chart/caption remain unchanged. Video-level disjointness and possible leakage through overlapping clips are not fully specified.

Complete original Table 7 : Dataset comparison. We provide details on dataset length, the proportion of visible sound sources, camera motion types, and sound source types, with each clip representing 5 seconds. Visibility is represented by two numbers: the first indicates the number of sound sources visible for over 4 seconds within the 5-second clip, and the second denotes the number of sound sources visible for less than 4 seconds. IID Binaural acc denotes the accuracy of IID predictions respected by the actual left-right labels.
Table 7: Dataset comparison. We provide details on dataset length, the proportion of visible sound sources, camera motion types, and sound source types, with each clip representing 5 seconds. Visibility is represented by two numbers: the first indicates the number of sound sources visible for over 4 seconds within the 5-second clip, and the second denotes the number of sound sources visible for less than 4 seconds. IID Binaural acc denotes the accuracy of IID predictions respected by the actual left-right labels.
Accessible table and original citation links
Dataset Split Size Visibility ( % ) Motion Type IID binaural acc ( % )
Clips Duration Camera Sound sources
In the wild dataset YT-
stereo
Raw 13,000k 8.0k hrs – Mainly rotation –
Train 14.6k 20hrs - Unknown Unknown
YT-
Stereo-iPhone
Raw 95k 80hrs – – –
Train 14.6k 20hrs 20/50 Mainly rotation Unknown Unknown
Val 0.1k 0.2hrs 10/60 Moving 57.5
Test 0.1k 0.2hrs 10/60 Moving
L/R Binaural[11] Raw 1.8k 3hrs - ✗ ✗ 75.4
Real-world lab-made Stereo-
Fountain
Raw 1.4k 2hrs 10/30 Mainly Stationary ✗ 97.0
Stereo-
Music
Raw 3.6k 5hrs 10/30 ✗ ✗ 70.0
Binaural-
Fountain
Raw 1.4k 2hrs 10/30 Mainly Stationary ✗ 98.0
RWAVS [32] Raw 2.7k 3.8hrs Unknown Rotation&Translation ✗ 71.9
Simulated Dataset Raw 21.6k 30hrs - Rotation&Translation Rendered 97.4

Stereo-Fountain

Fountain stereo collection uses an iPhone 13 Pro, a visually aligned laser pointer and 15 angles around the fountain, about ten minutes per position. One author annotates the recordings, which are randomly split.

Main Table 1 reports 5,000 clips/seven hours and 76.2% intensity accuracy, while supplementary Table 7 reports 1,400 clips/two hours and 97.0%. The source does not reconcile those particular collection summaries.

Binaural-Fountain

The in-ear fountain set follows the same 15-position collection procedure using the AMBEO headset and an iPhone recorder. The seated author’s head/ears affect the acoustic signal; recordings are annotated and randomly split.

Stereo-Music

Stereo-Music records a stationary iPhone and a phone playing music in one room. All data are mixed before random train/validation/test division, limiting conclusions about unseen rooms or recording sessions.

RWAVS [ 32 ]

RWAVS uses binaural 3 Dio microphones, a TASCAM recorder and a GoPro Max moved together through indoor/outdoor scenes. It comes from the cited audio-visual synthesis work, rather than being a newly collected StereoWalks subset [32].

Refer to caption
Figure 6: Examples from the Stereo-Music subset. We record sound using a stereo microphone in a scene containing music.
Refer to caption
Figure 7: The examples from the RWAVS dataset were recorded using binaural microphones in various scenarios where the sound source was a loudspeaker. Above, we present images from the original paper [32].
Complete original Table 8 : Summary of classification accuracy for RWAVS and Stereo-Music datasets.
Table 8: Summary of classification accuracy for RWAVS and Stereo-Music datasets.
Accessible table and original citation links
Dataset Classification Task Accuracy (%)
RWAVS [32] supervised 4-classification 50
Stereo-Music supervised 4-classification 28

Simulated dataset

Simulation approximates moving sound using different discrete source positions. Room responses have reported mean RT60 of 0.4 seconds, and speech samples provide the dry source audio.

The scene-disjoint 81/9/10 partition is stronger separation than the within-recording random splits described for several real datasets.

A.1.3 Label the dataset

One author and another participant annotate perceived sound direction/categories. Instructions use 16 full-circle classes plus unknown/skip options, distinct from the model’s 32 output categories.

The complete original instructions and all five questions are retained verbatim below, including the positive-left, negative-right angle convention. No inter-annotator agreement or repeated-label uncertainty analysis is reported.

Original five-question sound-annotation instructions

You will need to label relatively clear audio events whether in the scene or out of the scene. Please use the best headphones you can have. We provide some examples to help you label the other data.

1) Please provide the quality of the video’s audio for scoring.

2) Provide the video, with a 5-second audio clip, where are the audio sources located, totaling 16 classes. Or indicate if it is unknown. You only need to annotate the ones that you can distinguish where the sound is. As 90 denotes the left, -90 denotes the right. 0 denotes in front of you. 180/-180 denotes the back of you.

3) What are the categories of the audio sources? (If repeated), please list all of them. Existing categories include car/male speech/female speech/speech/sea/animals

4) What is the direction of movement of the audio sources? If consistent with question 3, no need to fill it.

5) If the category cannot be indicated by numbers, there’s no need to label it. If the quality is too low, just skip it.

A.1.4 Additional scenes

Additional recordings examine different devices and indoor music. Supplementary Table 7 retains all raw/train/evaluation counts rather than silently harmonizing its differences from Table 1.

Music data out of distribution.

Each of 11 RWAVS scenes is randomly split 80/10/10. Even supervised four-way classification reaches only 50% on RWAVS and 28% on Stereo-Music, versus 25% chance.

These weak indoor-music results temper broad claims about generalization across sound types, even when angle labels are available.

A.2 Experiment Implementations

The implementation supplement specifies pose sampling, optimizer settings and single-run training. It does not add a separately evaluated end-to-end visual/audio deployment system.

A.2.1 Estimate camera motion

Pose estimation samples five frames per second in a five-second clip. Relative transforms are also estimated across three- and six-frame intervals; cleaning accumulates three-frame estimates and proportionally fills gaps.

The learned audio loss uses binary motion directions, rather than requiring calibrated metric translation magnitudes or exact rotation angles.

A.2.2 Training details

AdamW uses learning rate 10⁻⁴ with cosine annealing and early stopping. The full and egomotion-only variants share rotation/translation mixing 0.9, with intensity weights 1 and 0 respectively.

The paper reports about two hours per trained model and explicitly runs each experiment once with the same seed. Reported differences have no multi-seed uncertainty estimate.

Concept demonstration

Open image ↗