At a glance
This work pairs English and Spanish movie/television dialogue and conditions a unit-to-waveform synthesizer on global voice style and local prosody. It aims to retain emotion, emphasis, intonation, and rhythm while translating speech. Reported listener ratings favor the expressive system, while ASR-based translation scores remain slightly below the vanilla baseline. The paper provides a compact system sketch, but inconsistencies in data filtering, system labels, and translation-model description limit reproducibility and interpretation.
1 Introduction
A text-mediated speech-recognition/translation/synthesis pipeline can discard delivery cues. Discrete speech units offer another intermediate representation, yet can also lose pitch and emotion. Building on direct unit-based translation [5] and earlier expressive cascades [3], the authors pair dubbed dialogue and add reference-audio conditioning to synthesis. Their contribution concerns expressivity beyond linguistic content; the experiments should not be read as establishing superior translation accuracy or a general solution for arbitrary language pairs.
2 Movie Dataset
The data pipeline uses multilingual audio tracks and subtitles to find dialogue pairs with approximately aligned timing and emotional delivery. The implemented language pair is English–Spanish; expansion to other languages is proposed.
2.1 Dataset Source
The authors describe roughly 300 hours of paired material from Money Heist seasons 1–5, Elite seasons 1–4, 59 Disney films, Dragon Ball Z, 24 James Bond films, ten superhero films, Shrek 1–4, Harry Potter 1–8, and Poltergeist 1–3. Dubbing motivates approximate emotion/timing correspondence, but different voice actors need not share speaker identity. The paper does not provide a quantitative annotation study establishing emotion equivalence for all pairs or clearly separate raw collection hours from the final filtered training hours.
2.2 Dataset Construction
Subtitle SRT files become CSV records. Rules use speaker and pause cues to remove inconsistent entries, and consecutive sentences assigned to the same speaker are merged using the cited speaker-representation method [17]. This supplies dialogue segments for subsequent audio filtering; detailed train/test partitioning and speaker/title disjointness are not reported.
2.2.1 Filter by ASR Accuracy and Duration
RNNoise denoises the clips before Azure ASR is compared with subtitles. Audio is sampled at 16 kHz, and the text says clips shorter than three seconds or longer than fifteen seconds are excluded. The ASR filter is internally inconsistent: the prose selects the “top 80%” with word error rate greater than 0.6, while Figure 2 highlights error rates below 40%. The figure also reports 12,610 utterances, a 0.833–244.250 s range, and 5.096 s average, extending outside the stated duration filter. It is unclear which processing stage these statistics describe; neither threshold is silently treated as a resolved recipe.
2.2.2 Filter by Speakers of the Utterances
English and Spanish segments are matched within a show. The stated filter favors cross-language audio cosine similarity below 0.5, and retains lists with at least five remaining segments. The authors motivate diversity, but do not fully specify the grouping or how this low-similarity criterion validates emotional correspondence. These thresholds should be distinguished from evidence that the paired clips express identical emotions.
3 Method
The system combines discrete-unit translation, pseudo speaker identities, and expressive unit-to-waveform synthesis. The synthesis module receives translated units and a reference waveform, adding global style and local acoustic information that the units alone may omit.
3.1 Obtain Discrete Units
HuBERT [18] produces features at 20 ms intervals. K-means maps them to a vocabulary of 1,000 acoustic units, following the prior direct-translation system. The representation models use English, Spanish, and French VoxPopuli audio, although this study focuses on English/Spanish outputs. Consecutive repeated units are collapsed to shorten sequences, then expanded during waveform synthesis. The paper does not spell out the duration-recovery mechanism or a complete unit-translation training recipe.
3.2 Get Speaker ID
Utterance-level speaker embeddings and cosine similarity produce pseudo speaker labels. These labels condition or supervise voice characteristics during mixed-corpus training. They are inferred identities rather than verified same-speaker labels across languages; dubbed counterparts may represent the same character using different actors.
3.3 Unit2Wav Synthesis
A unit-based HiFi-GAN [21] synthesizes speech, using a reference encoder for global style and local reference features for fine-grained prosody [23]. Figure 3 retains both training and inference paths, including waveform, mel, feature, and adversarial supervision. The authors first use clean monolingual audio, then mixed data, to mitigate movie noise. They add speaker prediction from reference features, pitch prediction, and voiced/unvoiced prediction. Both translation directions are mentioned as a way to reduce language-related tone bias.
The text does not fully define loss weights or explain how the speaker-prediction objective separates timbre from other reference information. Fine-grained cross-language alignment and the relative effects of global style, pitch, speaker conditioning, and training data are not isolated through ablations. The architecture is therefore a useful design outline rather than a complete independently reproducible specification.
4 Experiments
4.1 Experimental Setup
4.1.1 Dataset Setup
Additional material comes from VoxPopuli, Libri-Light, VCTK, and LJ Speech; the paper describes their aggregate scale as over 400,000 hours without giving per-corpus usage. Unpaired monolingual examples use the same audio in place of translated audio. The reported evaluation sets English as target and Spanish as reference, and evaluates only the movie dataset. Each mixed-training utterance receives equal weight, so corpus sizes can materially affect the training distribution.
4.1.2 Human Evaluation Protocol
Ten participants rate pairs of baseline and proposed outputs on emotion, emphasis, intonation, and rhythm, following the cited expressive-translation protocol [3]. Scores are averaged across listeners. The paper does not report the number of evaluated clips, rating-scale anchors, listener language proficiency, randomization/blinding details, uncertainty intervals, or significance tests. Naturalness is described as a separate study, but no separate quantitative naturalness table is supplied.
4.1.3 Model Setup and Training Details
The translation component is described as a pretrained Spanish-to-English Fairseq S2T model, although the main method claims translation through units without intermediate text. The connection between that checkpoint and the claimed textless path is not clarified. The waveform model trains from scratch with learning rate 2 × 10⁻⁴, decay 0.999, inverse-square-root scheduling with warmup, Adam β values (0.8, 0.99), and 0.1 dropout in reference/prosody encoders. The vanilla unit-synthesis baseline uses LJ Speech. Different training data and conditioning complicate attributing improvements to individual architecture changes.
4.2 Main Results
4.2.1 Preserve Emotion
Table 1 reports emotion/emphasis/intonation/rhythm scores of 2.03/2.68/2.46/2.30 for Vanilla Unit-TTS and 3.58/3.26/3.17/3.56 for the row labeled “Holistic Cascade.” The surrounding text presents these as improvements from the proposed method, but the row and caption retain cascade terminology; this labeling ambiguity is unresolved. The relative increases are approximately 22% for emphasis, 29% for intonation, and 55% for rhythm. These are changes in reported ratings, not demonstrated gains in objectively measured emotion-recognition accuracy. No uncertainty estimates justify the text’s claims of statistical superiority.
| System | Emotion | Emphasis | Intonation | Rhythm |
|---|---|---|---|---|
| Vanilla Unit-TTS | 2.03 | 2.68 | 2.46 | 2.30 |
| Holistic Cascade | 3.58 | 3.26 | 3.17 | 3.56 |
4.2.2 Translation Quality
Azure ASR transcribes synthesized outputs, and BLEU compares those transcripts with English references. With translated inputs, Table 2 reports 28.3 for the proposed system versus 29.2 for Vanilla Unit-TTS. With ground-truth inputs, scores are 74.6 and 81.0; the recorded ground-truth audio scores 78.2. Thus 74.6 is the ground-truth-input synthesis result, not the end-to-end translation score. The baseline scoring above recorded audio also shows that this ASR-mediated measure reflects recognizability and synthesis characteristics, not translation quality alone. Movie noise is offered as an explanation for the gap, but is not experimentally isolated.
5 Conclusion
The paper explores paired dubbed dialogue and combined global/local reference conditioning for expressive unit-based speech synthesis. The reported trend favors expression preservation while translation BLEU is slightly lower than the vanilla baseline. Main limitations are incomplete dataset/split details, inconsistent filter and system labels, an unclear S2T-to-unit interface, unmatched training conditions, missing component ablations, and limited human-evaluation reporting. Real-time performance is asserted without latency measurements, and additional language pairs remain future work.