Literature
Display
← All papers

When End-to-End is Overkill: Rethinking Cascaded Speech-to-Text Translation

Anna Min · Chenxu Hu · Yi Ren · Hang Zhao

Preprint posted · arXiv (preprint) ↗

Contents

At a glance

A cascaded speech translator need not discard every recognition hypothesis except the highest-scoring one. This paper aligns several ASR candidates, combines their translation attention, and optionally adds discrete self-supervised speech units. On English-to-Chinese GigaST, the complete system reaches 38.1 BLEU versus 36.8 for its conventional cascade and 38.0 for the reported end-to-end SSL-Transformer. The study supports this configuration on one benchmark; it does not establish that cascades generally outperform end-to-end translation.

I Introduction

Cascaded systems can reuse separately trained speech recognition and translation models, but incorrect recognized words propagate downstream. The proposal preserves alternative words from the recognition beam instead of forcing an early single-transcript decision. Attention averaging itself adds no parameters; the optional acoustic branch introduces a trainable speech-unit vocabulary, so the no-new-parameters claim should not be read as applying unchanged to every component.

Refer to caption
Fig. 1: The correct words are scattered among various candidates, while the former cascaded system directly selected the first candidate, resulting in similar pronunciation errors in the ASR output that are further propagated through the translation model, causing cascading losses.

II Analysis

The diagnostic asks whether the top recognition candidate contains all useful words and whether its translation is necessarily best. Lexical coverage and translation quality provide different answers.

II-A Preliminary experiments

Across the top 20 recognition results, cumulative overlap with reference words rises from 92.0% for one candidate to 95.5% for twenty, while average candidate quality falls. Table II attributes the best translated candidate to rank one in only 45.35% of cases. The surrounding prose inconsistently identifies that rank with lowest word error and says it always performs best; the table supports only the weaker conclusion that lower-ranked candidates frequently translate better. Reported oracle-style comparisons diagnose available information, not deployable selection performance.

TABLE I: Lexical overlap between ASR candidates and GT
nn candidates Average Cumulative
1 92.0% 92.0%
5 90.0% 94.3%
10 89.4% 95.0%
20 89.5% 95.5%
TABLE II: The proportion of the ASR result index of the best candidate based on the BLEU score of the translation result
Best BLEU idx 1 2 3 4 5
percentage(%) 45.35 16.73 14.07 12.12 11.74

II-B Source of cascade loss

Acoustically similar strings can have very different meanings. Errors such as substituting lecture for ledger illustrate how a recognition model can commit to a plausible sound sequence that is wrong in context. The authors hypothesize that a translation model can use richer semantic patterns to choose among the surviving alternatives; the examples do not quantify how much of total cascade loss this mechanism explains.

Refer to caption
Fig. 2: Overview of our proposed MC-sslS system

III Method

The method combines aligned transcript alternatives within mBART and optionally fuses discrete speech representations. Figure 2 retains the complete architecture.

III-A Multi-candidate from ASR

WeNet [20] produces the top 20 candidates ranked by cumulative beam-search log probability. The main translation configuration uses five candidates. Keeping alternatives exposes words that would be discarded by a conventional top-one cascade.

III-A 1 Aligned by common substrings

Candidates are aligned progressively using common word sequences. Shared spans line up, differing spans are padded with unknown-token markers, and earlier candidates receive the same padding whenever the growing alignment changes. The final equal-length sequences allow attention values at corresponding positions to be combined.

Refer to caption
Fig. 3: Above is how the 3r​d3^{rd} process is calculated. After finding the longest common subsequences, candidates are aligned and padded. The orange circles denote “unk” tokens.

III-A 2 Average attention among candidates

The mBART-based model shares parameters across candidates. During training it averages candidate attention once after the decoder stack, before final normalization. During generation it combines corresponding beam states at that same point for each token. The preserved equations specify the averaging. This mechanism can reuse translation-model weights, but its multiple inputs and beams are not evidence of unchanged inference cost.

Equations 1–2
A′​(Q)=1ncandidate​∑i=1ncandidateAc​a​n​d​i​d​a​t​ei​(Q)\displaystyle\text{A}^{\prime}(Q)=\frac{1}{n_{\text{candidate}}}\sum_{i=1}^{n_{\text{candidate}}}\text{A}_{candidate_{i}}(Q) (1)
Ab​e​a​mk​(Q)=1ncandidate​∑i=1ncandidateAc​a​n​d​i​d​a​t​ei​_​b​e​a​mk​(Q)\displaystyle\text{A}_{beam_{k}}(Q)=\frac{1}{n_{\text{candidate}}}\sum_{i=1}^{n_{\text{candidate}}}\text{A}_{candidate_{i}\_{beam_{k}}}(Q) (2)

III-B Acoustic and linguistic features fusion

The acoustic extension extracts layer-11 HuBERT features [24], quantizes them into 1,000 discrete units, and collapses consecutive repeats. New trainable unit embeddings feed an encoder whose outputs interact with decoder cross-attention. This supplies speech information lost during transcription. The method discussion refers to English-trained quantization, whereas the experimental setup specifies multilingual mHuBERT and clustering trained on English, Spanish, and French; that source distinction is retained.

IV Experiments & Results

The experiments compare a conventional cascade, increasingly augmented variants, and reported end-to-end baselines on English-to-Chinese speech translation.

IV-A Data

GigaST [21] contains approximately 7.5 million English-to-Chinese pairs. Training translations are machine-generated from GigaSpeech transcripts, while test translations are human-produced. This data construction is relevant when comparing a cascade that reuses text-translation components with direct speech-translation models.

IV-B Model setup

The setup combines a trained WeNet recognizer, fine-tuned mBART, and optionally pretrained discrete speech units.

IV-B 1 Cascaded system

After filtering, 7.4 million pairs are used. The ASR encoder is trained from scratch rather than initialized from a self-supervised speech model. The optional speech-unit branch uses multilingual HuBERT and 1,000-way clustering trained on unlabeled VoxPopuli speech; the downstream input here is English.

IV-B 2 Multi-candidate mBART

mBART is fine-tuned on eight A100 GPUs using the referenced fairseq configuration. Comparisons include the original cascade, SpeechTransformer, and SSL-Transformer. The reported fine-tuning time therefore depends on substantial accelerator resources and pretrained components.

TABLE III: Giga-ST main results: “MC” denotes the method of multi-candidate in Section 3.1.2, “Alignment” denotes the method in Section 3.1.1, “sslS” denotes the method of fusing acoustic and linguistic features in Section 3.2
Settings Models BLEU score
(1) Machine translation 40.2
End-to-End Model
(2) SpeechTransformer 36.3
(3) SSL-Transformer 38.0
Cascaded Model
(4) Wenet + mBART 36.8
(5) (4) + MC 36.9
(6) (4) + MC + Alignment 37.8
(7) (4) + MC + Alignment + sslS 38.1

IV-C Results

Table III preserves every reported system and ablation score, including the text-only translation reference.

IV-C 1 Comparison with baselines

The plain WeNet-plus-mBART cascade scores 36.8 BLEU. Aligned multi-candidate attention raises this to 37.8, and the speech-unit branch reaches 38.1. The listed end-to-end systems score 36.3 and 38.0, while text-only translation reaches 40.2. Thus the full cascade slightly exceeds the stronger reported end-to-end baseline but retains a gap to reference-transcript translation. One epoch yields 37.3 BLEU in a reported hour on eight GPUs; no uncertainty or multi-seed estimate is supplied.

IV-C 2 Ablation

Adding multiple candidates without alignment changes BLEU only from 36.8 to 36.9. Alignment produces the larger gain to 37.8, showing that averaging corresponding lexical positions matters in this implementation.

IV-C 3 Case study

Examples involving Golgi apparatus and distributed ledger illustrate recovery from incorrect top-one transcripts when a suitable alternative survives among the five candidates. These selected cases explain the mechanism; they do not establish correction rates for arbitrary recognition errors.

V Conclusion

Aligned recognition alternatives reduce information loss at the ASR-to-translation boundary, and the acoustic branch supplies additional evidence. The results remain limited to the reported language direction, dataset, model choices, and baseline settings. They support reusing strong components rather than a general claim that end-to-end translation is unnecessary.

Concept demonstration

Open image ↗