Overview
The complete teaser illustrates camera motion relative to multiple possible sound sources. Its unknown-source setting motivates weak consistency supervision rather than requiring the dominant sound to appear visibly in the image.
At a glance
The method trains an audio-only azimuth model using directions of camera rotation/translation estimated from synchronized video. Interaural intensity difference adds complementary left/right supervision, avoiding manual sound-angle labels for the main training objective.
StereoWalks supplies internet walking-tour and controlled recording data. The full model reaches 34° mean angular error on the tested iPhone subset, versus 71.4° for the GTRot comparison. Results depend on device, scene and evaluation convention, with important limits from overlapping sounds and front/back ambiguity.
1 Introduction
Natural stereo recordings vary in microphone geometry, channel response and mixtures of sound sources. Simulation or a single hand-chosen audio cue may transfer poorly to that diversity.
The proposed training uses moments when camera-induced angular change dominates source motion. Off-the-shelf visual geometry provides motion-direction constraints, and audio intensity cues help orient the predicted angles. The method avoids requiring a visible sound-emitting object, but still assumes useful synchronization and motion consistency [13].
2 Related Work
The paper connects spatial audio learning, audio-visual correspondence and visual camera-motion estimation. Its distinction is weak sound-direction supervision from natural-video egomotion rather than jointly learning a visual pose model from simulated scenes.
Audio-visual spatial perception
Prior work relates spatial audio to images, scene geometry, source separation and camera motion. Here pretrained visual matching/geometry is used as a teacher; the deployed direction predictor takes stereo audio [13].
Spatial audio datasets
Existing spatial-audio datasets often use specialized multichannel rigs, controlled rooms or synthetic acoustics. StereoWalks adds internet recordings with camera motion and human-labeled evaluation subsets, alongside controlled stereo/binaural recordings.
Audio-visual learning.
Related audio-visual learning uses temporal correspondence, semantic objects or sound separation. This method uses geometric consistency over motion, which can supervise sources outside the field of view.
Camera motion estimation
Feature matching and camera calibration estimate relative pose. SuperGlue provides correspondences and Perspective Fields estimates field of view; the audio model is trained separately rather than jointly estimating visual geometry [46] [29].
3 Method
Two short audio segments from a moving-camera video should have sound directions compatible with visual motion. Binary motion-direction labels constrain the probability assigned to pairs of predicted angles.
Accessible table and original citation links
| Dataset | Scene subset | Split | Size |
Visibility (%) |
Motion Type |
IID binaural acc (%) | ||
| Clips (k) | Duration (hr) | Camera | Sound sources | |||||
| StereoWalks (ours) | YT-Stereo | Train | 14.4 | 20.0 | 20 / 50 | Rotation&Translation | Unknown | – |
| Val | 00.1 | 00.2 | 10 / 60 | Mainly Rotation | Moving | 57.5 | ||
| Test | 00.1 | 00.2 | 10 / 60 | Mainly Rotation | Moving | 57.5 | ||
| Stereo-Fountain | Raw | 05.0 | 07.0 | 10 / 30 | Mainly Stationary | ✗ | 76.2 | |
| Binaural-Fountain | Raw | 01.4 | 02.0 | 10 / 30 | Mainly Stationary | ✗ | 98.0 | |
| L/R Binaural [11] | Raw | 01.8 | 03.0 | – | ✗ | ✗ | 75.4 | |
| Simulated | HM3D-SS 2.0 (ours) | Raw | 21.6 | 30.0 | – | Rotation&Translation | Rendered | 97.4 |
3.1 The StereoWalks dataset
StereoWalks collects roughly 20 training hours from stereo walking tours, with an iPhone-focused subset identified using titles/captions. Evaluation clips receive perceived sound-angle and category labels.
Controlled recordings compare an iPhone 13 Pro near a fountain with an in-ear Sennheiser AMBEO headset. Those subsets test device/head-related cues. Reported collection counts differ between the main and supplementary tables; both original tables remain.
3.2 Learning audio localization from egomotion
The model predicts a distribution over 32 azimuth categories from stereo spectrograms. The core formulation restricts angles to the frontal half-plane, excluding elevation and distance.
A single ordinary stereo segment can be ambiguous between front and back. The main evaluation reflects back angles to the front; full-circle localization is examined separately with supervised device comparisons.
Camera rotation estimation.
Estimated clockwise/counterclockwise camera rotation supplies a binary pseudo label. Only the direction is needed, reducing dependence on precise calibrated rotation magnitude.
Camera front-back translation estimation.
Estimated forward/backward camera translation constrains relative angular changes when source motion is sufficiently slow. The constraint becomes less informative for distant sources, while moving or competing sources can violate it.
Supervising audio localization using sight.
Rotation and translation define compatible pairs of azimuth bins. The losses sum products of the two predicted distributions over those allowed pairs, then apply the stated cross-entropy objectives. Equal-angle cases are excluded.
The prose calls this a sum of individual cross-entropies, whereas the displayed expressions use a masked probability sum inside the loss. Both source formulas are retained; the precise scalar/binary normalization is not fully spelled out.
Equations 1–2
| (1) |
| (2) |
Binaural cues
Interaural intensity difference estimates which side is louder. A complementary loss favors direction bins consistent with that left/right pseudo label.
It is a noisy physical cue rather than manually annotated ground truth. Its reported accuracy varies from 57.5% on internet iPhone clips to about 98% on the in-ear fountain set.
Equation 3
| (3) |
Overall loss.
The full objective weights rotation/translation by 0.9/0.1 and gives the intensity-cue loss weight 1. The egomotion-only variant sets the intensity term to zero. All four numbered equations remain below.
Equation 4
| (4) |
Accessible table and original citation links
| Model | Training Set | Test Set | |||||||||
| YT-Stereo | Sim. | YT-Stereo-iPhone | Stereo-Fountain | Simulated | |||||||
| MAE (°) | 2clf (%) | 8clf (%) | MAE (°) | 2clf (%) | 8clf (%) | MAE (°) | 2clf (%) | 8clf (%) | |||
| Chance | 55.3 | 46.7 | 12.7 | 62.1 | 52.0 | 16.0 | 39.4 | 49.0 | 15.0 | ||
| IID – direct | – | 57.5 | – | – | 97.0 | – | – | 97.4 | – | ||
| GTRot [14] | ✓ | 71.4 | 54.0 | 7.1 | 57.2 | 97.0 | 18.4 | 40.2 | 48.7 | 18.7 | |
| Ours – IID only [14] | ✓ | 37.3 | 54.7 | 28.0 | 30.1 | 97.0 | 46.0 | 88.3 | 48.7 | 6.8 | |
| Ours – Simulated | ✓ | 73.4 | 61.7 | 13.3 | 28.5 | 85.7 | 22.3 | 9.8 | 98.3 | 63.4 | |
| Ours – Full | ✓ | 34.0 | 61.7 | 33.3 | 29.3 | 97.3 | 46.0 | 57.2 | 48.7 | 10.7 | |
4 Experiments
Evaluation combines internet audio, controlled device recordings and simulated motion/sound mixtures. Angular error and coarse left/right classification measure different aspects of performance.
4.1 Datasets
Real datasets provide natural recording variation; SoundSpaces simulations isolate camera motion, source overlap and intermittency. The simulation data are an analysis/training baseline, while the main proposed model can learn from natural videos.
Real-world dataset
Evaluation curation selects clips with reliable camera rotation, substantial intensity changes and agreement between the two directions, then annotates sources/categories. The test set therefore emphasizes conditions in which the supervisory assumptions are easier to satisfy.
The main text describes five-second clips with three-second overlap, while the supplement says clips every three seconds. That stride/overlap discrepancy is retained rather than resolved by assumption.
Simulated dataset
HM3D-SS2.0 renders binaural room responses in scene-disjoint train/validation/test environments and convolves them with LibriSpeech. Camera/source motion and competing/intermittent sources extend the earlier simulation [6] [38].
The supplement reports 81/9/10 scenes,50,000 audio-visual pairs and 200,000 viewpoints, with about 30 rendered hours for the size-limited comparisons. Discrete source positions approximate motion.
4.2 Implementation details
ResNet-18 maps magnitude/phase stereo spectrograms of size 256×256×4 to 32 azimuth logits. Training uses an 80 GB A100, batch 300 and 100 epochs; later details specify AdamW at 10⁻⁴, cosine annealing and early stopping.
Table 2 first trains on YT-Stereo and part of the iPhone subset, then finetunes for one epoch on each target dataset. These adapted comparisons should not be read as completely zero-shot cross-dataset transfer. Each reported experiment is run once with a shared seed.
Accessible table and original citation links
| Model | YT-Stereo-iPhone | Stereo-Fountain | |||||
| MAE (°) | 2clf (%) | 8clf (%) | MAE (°) | 2clf (%) | 8clf (%) | ||
| Chance | 55.3 | 46.7 | 12.7 | 62.1 | 52.0 | 16.0 | |
| IID-direct | – | 57.5 | – | – | 97.0 | – | |
| GTRot [14] | 71.4 | 54.0 | 7.1 | 55.1 | 97.1 | 18.7 | |
| Ours – Simulated | 73.4 | 61.7 | 13.3 | 28.5 | 85.7 | 22.3 | |
| Ours – IID only [14] | 37.3 | 54.7 | 28.0 | 29.6 | 97.3 | 46.0 | |
| Ours – R&B | 34.5 | 55.4 | 33.4 | 28.5 | 97.3 | 46.0 | |
| Ours – T&B | 37.4 | 53.4 | 26.8 | – | – | – | |
| Ours – Full | 34.0 | 61.7 | 31.7 | 28.5 | 97.3 | 46.0 | |
| Supervised | – | – | – | 33.8 | 98.0 | 49.5 | |
| Supervised – RTF | – | – | – | 27.1 | 98.0 | 49.5 | |
4.3 Evaluation
Metrics are mean angular error, two-way left/right accuracy and eight-bin classification. Probability-weighted bin midpoints form the angle estimate.
The stated reflection convention removes front/back ambiguity for the main comparison. Despite Table 3’s 360-degree heading, its results should not be taken as evidence of fully resolved front/back direction without reconciling that evaluation description.
Baselines and ablations
Comparisons include direct intensity cues, GTRot’s stronger rotation supervision, simulated training, supervised angles and supervised reflection-to-front. Ablations retain intensity alone, rotation plus intensity, translation plus intensity, or egomotion alone.
The older in-the-wild binaural benchmark also includes time-delay/contrastive methods. Different training sets and information make these complete-system comparisons rather than uniformly identical supervision budgets [12] [13].
4.4 Results
On internet iPhone clips, the full model improves angular error from 37.3° for intensity-only to 34.0°, and left/right accuracy from 54.7% to 61.7%. Coarse localization remains difficult. GTRot’s 71.4° error is much worse in this setting.
On the older binaural benchmark, full-model 87.5% accuracy is comparable to MonoCLR87.4% and StereoCRW87.2%, while intensity-only scores 87.7%. Visual supervision is therefore not uniformly best across every dataset/metric.
Real-trained and simulation-trained models transfer poorly in opposite directions. The complete tables retain that domain gap, target-specific finetuning and differences between the two real-data evaluation protocols.
Accessible table and original citation links
| Model | (1) One Source | (2) Overlap | (3) Overlap Intermittent | (4) More Simulated Data of (3) | ||||||||
| MAE(°) | 2clf(%) | 8clf(%) | MAE(°) | 2clf(%) | 8clf(%) | MAE(°) | 2clf(%) | 8clf(%) | MAE(°) | 2clf(%) | 8clf(%) | |
| Chance | 39.4 | 49.0 | 15.0 | 39.4 | 49.0 | 15.0 | 39.4 | 49.0 | 15.0 | 39.4 | 49.0 | 15.0 |
| IID | - | 97.4 | - | - | 80.1 | - | - | 80.8 | - | - | 80.8 | - |
| GTRot [14] | 4.3 | 97.5 | 84.7 | 22.5 | 83.5 | 33.5 | 26.9 | 81.5 | 32.5 | 19.7 | 87.2 | 41.0 |
| Ours – IIDonly [14] | 22.2 | 99.0 | 35.7 | 25.6 | 78.8 | 26.1 | 28.0 | 78.0 | 23.7 | 26.7 | 79.5 | 25.7 |
| Ours – Full | 9.8 | 98.3 | 63.4 | 23.2 | 87.2 | 36.2 | 24.1 | 82.1 | 35.7 | 23.7 | 82.4 | 41.2 |
| Supervised | 2.8 | 98.7 | 89.8 | 10.8 | 91.8 | 70.2 | 8.4 | 94.1 | 74.6 | 5.7 | 96.0 | 80.2 |
Accessible table and original citation links
| Model | (1) Rotation Only | (2) Translation Only | (3) Rotation&Translation | (4) Distant Rotation | (5) Distant R&T | (6) Overlap R&T | ||||||||||||
| MAE | 2clf | 8clf | MAE | 2clf | 8clf | MAE | 2clf | 8clf | MAE | 2clf | 8clf | MAE | 2clf | 8clf | MAE | 2clf | 8clf | |
| Chance | 39.4 | 49.0 | 15.0 | 39.4 | 49.0 | 15.0 | 39.4 | 49.0 | 15.0 | 39.9 | 49.2 | 15.7 | 39.8 | 49.1 | 15.8 | 39.0 | 49.5 | 15.2 |
| IID | - | 97.4 | - | - | 97.4 | - | - | 97.4 | - | - | 95.6 | - | - | 95.1 | - | - | 73.5 | - |
| GTRot [14] | 4.3 | 97.5 | 84.7 | - | - | - | 4.4 | 97.5 | 84.5 | 4.5 | 97.1 | 82.0 | 4.7 | 96.7 | 86.9 | 28.1 | 79.6 | 30.1 |
| Ours – IIDonly [14] | 22.2 | 99.0 | 35.7 | 22.7 | 99.0 | 35.7 | 22.0 | 99.1 | 35.9 | 23.1 | 97.8 | 35.2 | 23.0 | 98.1 | 35.0 | 32.5 | 75.0 | 25.7 |
| Ours – R&B | 9.8 | 98.3 | 63.4 | - | - | - | 10.1 | 98.0 | 60.1 | 10.7 | 98.1 | 59.1 | 11.7 | 98.6 | 58.7 | 25.6 | 81.4 | 34.6 |
| Ours – T&B | - | - | - | 23.2 | 96.0 | 35.7 | 23.0 | 95.7 | 35.0 | - | - | - | 22.7 | 96.7 | 35.1 | 30.1 | 72.5 | 22.3 |
| Ours – R&T | - | - | - | - | - | - | 14.7 | 96.3 | 48.3 | - | - | - | 22.5 | 95.0 | 45.7 | 26.7 | 79.0 | 29.2 |
| Ours – Full | 9.8 | 98.3 | 63.4 | 23.2 | 96.0 | 35.7 | 11.7 | 98.0 | 58.7 | 11.2 | 98.1 | 56.8 | 11.9 | 97.6 | 55.3 | 28.6 | 76.0 | 28.5 |
| Supervised | 2.8 | 98.7 | 89.6 | 2.6 | 99.1 | 89.9 | 2.6 | 98.9 | 89.5 | 2.7 | 98.8 | 90.0 | 2.5 | 99.0 | 90.4 | 12.1 | 86.0 | 65.4 |
4.5 Analysis
Controlled experiments test whether motion-direction supervision helps under competing sound, how translation changes with distance, and whether in-ear recordings improve front/back discrimination.
Overlapping sound assumption
Simulations add a secondary source at 0.7 relative loudness, optionally making both sources independently silent half the time. The first three settings use about 30 hours; a fourth adds more rendered data.
GTRot is strongest among the self-supervised comparisons for a single source. In continuous overlap, it also has slightly lower angular error than the full model,22.5° versus 23.2°, while the full model wins coarse classification. Intermittency favors the proposed error metric; with more data GTRot again gives lower error,19.7° versus 23.7°. Supervised labels remain substantially stronger.
Rotation and translation
Rotation is useful for distant sources; translation changes are weaker at larger range. Removing translation on internet tours gives similar angular error,34.5° versus 34.0°, though other metrics differ.
In near-source rotation-plus-translation simulation, egomotion-only error 14.7° approaches the full model’s 11.7°. In overlapping motion, rotation-plus-intensity gives 25.6° versus full 28.6°. These results support condition-dependent cues rather than universal benefit from adding every loss.
Accessible table and original citation links
| Dataset | MAE | Left/Right Acc | Front/Back Acc |
| Stereo-Fountain | 33.8 | 98.0% | 51.0% |
| Binaural-Fountain | 27.8 | 99.0% | 69.3% |
Front-Back ambiguity
A separate supervised 360-degree experiment compares recordings beside the same fountain. Front/back accuracy is 51.0% with the iPhone and 69.3% with the in-ear headset; angular error is 33.8° versus 27.8°.
This controlled device comparison supports extra head/ear cues, while still leaving many front/back mistakes. It is not a full-circle result obtained solely by the main weakly supervised method.
5 Conclusion
Visual motion directions can improve audio localization learned from natural video, especially relative to simulation-only transfer. The contribution combines a weak-supervision recipe with a dataset and controlled analysis of acoustic ambiguity.
Limitations
Low-quality stereo, overlapping events, source motion and biased clip filtering remain limitations. Selecting stable, clear events can omit brief sounds and does not establish performance over arbitrary walking-tour audio.
Future work
The authors propose richer visual/contextual information for harder real-world sound localization. Robust multi-source, device-independent and full-circle perception remain open beyond these experiments.
Acknowledgements
The work was completed during Anna Min’s internship at the University of Michigan. The original acknowledgment credits research feedback and preliminary YouTube recordings.
A.1 Dataset
The supplement retains collection, filtering, annotation and recording details beyond the main dataset summary. Both the original expanded data table and full annotation questionnaire remain.
A.1.1 Internet data collection
Internet collection searches walking-tour videos and removes portrait or mono recordings. Useful supervision assumes source angular motion is slower than camera-induced angular change.
The reported 20-hour training pool follows curation, although a later paragraph calls training unfiltered. That wording does not fully specify which processing stages are shared between training and the more selective evaluation set.

A.1.2 Dataset Filtering
The supplement details motion/intensity screening and controlled device recordings. Random clip splits within recordings differ from testing unseen recording sessions, environments or speakers.
YT-Stereo-iPhone
The authors select 100 validation clips from 1,600 filtered candidates. A preliminary motion stage uses a 10° rotation threshold; later evaluation curation requires 15°, intensity/motion-direction agreement and angle-distribution balancing.
Figure 5’s caption states angles from−180°to 180°, but the displayed histogram axis spans roughly−100°to 100°. The original chart/caption remain unchanged. Video-level disjointness and possible leakage through overlapping clips are not fully specified.
Accessible table and original citation links
| Dataset | Split | Size | Visibility ( % ) | Motion Type | IID binaural acc ( % ) | |||
| Clips | Duration | Camera | Sound sources | |||||
| In the wild dataset |
YT- stereo |
Raw | 13,000k | 8.0k hrs | – | Mainly rotation | – | |
| Train | 14.6k | 20hrs | - | Unknown | Unknown | |||
|
YT- Stereo-iPhone |
Raw | 95k | 80hrs | – | – | – | ||
| Train | 14.6k | 20hrs | 20/50 | Mainly rotation | Unknown | Unknown | ||
| Val | 0.1k | 0.2hrs | 10/60 | Moving | 57.5 | |||
| Test | 0.1k | 0.2hrs | 10/60 | Moving | ||||
| L/R Binaural[11] | Raw | 1.8k | 3hrs | - | ✗ | ✗ | 75.4 | |
| Real-world lab-made |
Stereo- Fountain |
Raw | 1.4k | 2hrs | 10/30 | Mainly Stationary | ✗ | 97.0 |
|
Stereo- Music |
Raw | 3.6k | 5hrs | 10/30 | ✗ | ✗ | 70.0 | |
|
Binaural- Fountain |
Raw | 1.4k | 2hrs | 10/30 | Mainly Stationary | ✗ | 98.0 | |
| RWAVS [32] | Raw | 2.7k | 3.8hrs | Unknown | Rotation&Translation | ✗ | 71.9 | |
| Simulated Dataset | Raw | 21.6k | 30hrs | - | Rotation&Translation | Rendered | 97.4 | |
Stereo-Fountain
Fountain stereo collection uses an iPhone 13 Pro, a visually aligned laser pointer and 15 angles around the fountain, about ten minutes per position. One author annotates the recordings, which are randomly split.
Main Table 1 reports 5,000 clips/seven hours and 76.2% intensity accuracy, while supplementary Table 7 reports 1,400 clips/two hours and 97.0%. The source does not reconcile those particular collection summaries.
Binaural-Fountain
The in-ear fountain set follows the same 15-position collection procedure using the AMBEO headset and an iPhone recorder. The seated author’s head/ears affect the acoustic signal; recordings are annotated and randomly split.
Stereo-Music
Stereo-Music records a stationary iPhone and a phone playing music in one room. All data are mixed before random train/validation/test division, limiting conclusions about unseen rooms or recording sessions.
RWAVS [ 32 ]
RWAVS uses binaural 3 Dio microphones, a TASCAM recorder and a GoPro Max moved together through indoor/outdoor scenes. It comes from the cited audio-visual synthesis work, rather than being a newly collected StereoWalks subset [32].
Simulated dataset
Simulation approximates moving sound using different discrete source positions. Room responses have reported mean RT60 of 0.4 seconds, and speech samples provide the dry source audio.
The scene-disjoint 81/9/10 partition is stronger separation than the within-recording random splits described for several real datasets.
A.1.3 Label the dataset
One author and another participant annotate perceived sound direction/categories. Instructions use 16 full-circle classes plus unknown/skip options, distinct from the model’s 32 output categories.
The complete original instructions and all five questions are retained verbatim below, including the positive-left, negative-right angle convention. No inter-annotator agreement or repeated-label uncertainty analysis is reported.
You will need to label relatively clear audio events whether in the scene or out of the scene. Please use the best headphones you can have. We provide some examples to help you label the other data.
1) Please provide the quality of the video’s audio for scoring.
2) Provide the video, with a 5-second audio clip, where are the audio sources located, totaling 16 classes. Or indicate if it is unknown. You only need to annotate the ones that you can distinguish where the sound is. As 90 denotes the left, -90 denotes the right. 0 denotes in front of you. 180/-180 denotes the back of you.
3) What are the categories of the audio sources? (If repeated), please list all of them. Existing categories include car/male speech/female speech/speech/sea/animals
4) What is the direction of movement of the audio sources? If consistent with question 3, no need to fill it.
5) If the category cannot be indicated by numbers, there’s no need to label it. If the quality is too low, just skip it.
A.1.4 Additional scenes
Additional recordings examine different devices and indoor music. Supplementary Table 7 retains all raw/train/evaluation counts rather than silently harmonizing its differences from Table 1.
Music data out of distribution.
Each of 11 RWAVS scenes is randomly split 80/10/10. Even supervised four-way classification reaches only 50% on RWAVS and 28% on Stereo-Music, versus 25% chance.
These weak indoor-music results temper broad claims about generalization across sound types, even when angle labels are available.
A.2 Experiment Implementations
The implementation supplement specifies pose sampling, optimizer settings and single-run training. It does not add a separately evaluated end-to-end visual/audio deployment system.
A.2.1 Estimate camera motion
Pose estimation samples five frames per second in a five-second clip. Relative transforms are also estimated across three- and six-frame intervals; cleaning accumulates three-frame estimates and proportionally fills gaps.
The learned audio loss uses binary motion directions, rather than requiring calibrated metric translation magnitudes or exact rotation angles.
A.2.2 Training details
AdamW uses learning rate 10⁻⁴ with cosine annealing and early stopping. The full and egomotion-only variants share rotation/translation mixing 0.9, with intensity weights 1 and 0 respectively.
The paper reports about two hours per trained model and explicitly runs each experiment once with the same seed. Reported differences have no multi-seed uncertainty estimate.