Literature
Display
← All papers

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Dario Amodei · Rishita Anubhai · Eric Battenberg · Carl Case · Jared Casper · Bryan Catanzaro · Jingdong Chen · Mike Chrzanowski · Adam Coates · Greg Diamos · Erich Elsen · Jesse Engel · Linxi Fan · Christopher Fougner · Tony Han · Awni Hannun · Billy Jun · Patrick LeGresley · Libby Lin · Sharan Narang · Andrew Ng · Sherjil Ozair · Ryan Prenger · Jonathan Raiman · Sanjeev Satheesh · David Seetapun · Shubho Sengupta · Yi Wang · Zhiqian Wang · Chong Wang · Bo Xiao · Dani Yogatama · Jun Zhan · Zhenyao Zhu

· ICML 2016 ↗

Contents

At a glance

Deep Speech 2 combines convolutional and recurrent networks with CTC training for English and Mandarin transcription. Better optimization, more labeled data, and faster GPU systems improve recognition and make large experiments practical. It matches or exceeds the tested human workers on some benchmarks, but remains worse on several accents and noisy conditions.

1 Introduction

The contribution spans models, data, and systems: deeper speech networks, about 12,000 English and 9,400 Mandarin training hours, and synchronized multi-GPU training that finishes in days. A separate streaming deployment configuration trades a small accuracy loss for low compute latency. “End-to-end” here still includes an external language model during decoding.

2 Related Work

The work builds on neural acoustic models, recurrent and convolutional recognition, CTC, and earlier Deep Speech. Unlike pipelines that require frame-level alignment labels or phonetic pretraining, its acoustic model trains from audio–transcript pairs. Data augmentation and recognition-assisted dataset cleaning also draw on prior work.

3 Model Architecture

Deeper networks require better optimization and more computation. The paper studies sequence-wise batch normalization, a short-first training curriculum, frequency convolutions, temporal strides, and streaming-compatible recurrence. Ablations separate several of these effects.

3.1 Preliminaries

A power-normalized spectrogram passes through convolutional, recurrent, and fully connected layers to predict characters plus a blank symbol. CTC sums over possible alignments between audio frames and transcription. English is evaluated with WER, Mandarin with CER; most reported recognition results include language-model decoding.

Original figure
Figure 1:Architecture of the DS2 system used to train on both English and Mandarin speech. We explore variants of this architecture by varying the number of convolutional layers from 1 to 3 and the number of recurrent or GRU layers from 1 to 7.
Equations 1–5
ht,il=f(wil∘ht−c:t+cl−1)h_{t,i}^{l}=f(w_{i}^{l}\circ h^{l-1}_{t-c:t+c}) (1)
h→tl\displaystyle\overrightarrow{h}_{t}^{l} =g⁡(htl−1,h→t−1l)\displaystyle=g(h_{t}^{l-1},\overrightarrow{h}_{t-1}^{l}) (2)
h←tl\displaystyle\overleftarrow{h}_{t}^{l} =g⁡(htl−1,h←t+1l)\displaystyle=g(h_{t}^{l-1},\overleftarrow{h}_{t+1}^{l})
h→tl=f⁡(Wl​htl−1+U→l​h→t−1l+bl)\overrightarrow{h}_{t}^{l}=f(W^{l}h_{t}^{l-1}+\overrightarrow{U}^{l}\overrightarrow{h}_{t-1}^{l}+b^{l}) (3)
htl=f⁡(Wl​htl−1+bl)h^{l}_{t}=f(W^{l}h^{l-1}_{t}+b^{l}) (4)
p⁡(ℓt=k|x)=exp⁡(wkL⋅htL−1)∑jexp⁡(wjL⋅htL−1)p(\ell_{t}=k|x)=\frac{\exp(w^{L}_{k}\cdot h^{L-1}_{t})}{\sum_{j}\exp(w^{L}_{j}\cdot h^{L-1}_{t})} (5)

3.2 Batch Normalization for Deep RNNs

Batch normalization acts on input-to-hidden transformations using statistics pooled over sequences and the minibatch, leaving recurrent contributions outside that normalization. It improves deeper networks but can hurt the shallowest one. Deployment uses running training statistics rather than recomputing normalization from one utterance.

Equations 6–8
ℬ⁡(x)=γ​x−E⁡[x](Var⁡[x]+ϵ)1/2+β.\mathcal{B}(x)=\gamma\frac{x-\mathrm{E}[x]}{\left(\mathrm{Var}[x]+\epsilon\right)^{1/2}}+\beta. (6)
h→tl=f⁡(ℬ⁡(Wl​htl−1+U→l​h→t−1l)).\overrightarrow{h}^{l}_{t}=f(\mathcal{B}(W^{l}h^{l-1}_{t}+\overrightarrow{U}^{l}\overrightarrow{h}^{l}_{t-1})). (7)
h→tl=f⁡(ℬ⁡(Wl​htl−1)+U→l​h→t−1l).\overrightarrow{h}^{l}_{t}=f(\mathcal{B}(W^{l}h^{l-1}_{t})+\overrightarrow{U}^{l}\overrightarrow{h}^{l}_{t-1}). (8)
Architecture Hidden Units Train Dev
Baseline BatchNorm Baseline BatchNorm
1 RNN, 5 total 2400 10.55 11.99 13.55 14.40
3 RNN, 5 total 1880 9.55 8.29 11.61 10.56
5 RNN, 7 total 1510 8.59 7.61 10.77 9.78
7 RNN, 9 total 1280 8.76 7.68 10.83 9.52
Table 1:Comparison of WER on a training and development set for various depths of RNN, with and without BatchNorm. The number of parameters is kept constant as the depth increases, thus the number of hidden units per layer decreases. All networks have 38 million parameters. The architecture “M RNN, N total” implies 1 layer of 1D convolution at the input, M consecutive bidirectional RNN layers, and the rest as fully-connected layers with N total layers in the network.
Original figure
Figure 2:Training curves of two models trained with and without BatchNorm. We start the plot after the first epoch of training as the curve is more difficult to interpret due to the SortaGrad curriculum method mentioned in Section3.3

3.3 SortaGrad

SortaGrad processes shorter utterances first only during the first epoch, then returns to randomized minibatches. It reduces early optimization instability and complements batch normalization. Utterance length is a difficulty heuristic, not an externally labeled curriculum.

Equation 9
ℒ(x,y;θ)=−log∑ℓ∈Align​(x,y)∏tTpctc(ℓt|x;θ).\mathcal{L}(x,y;\theta)=-\log\sum_{\ell\in\textrm{Align}(x,y)}\prod_{t}^{T}p_{\textrm{ctc}}(\ell_{t}|x;\theta). (9)
Train Dev
Baseline BatchNorm Baseline BatchNorm
Not Sorted 10.71 8.04 11.96 9.78
Sorted 8.76 7.68 10.83 9.52
Table 2:Comparison of WER on a training and development set with and without SortaGrad, and with and without batch normalization.

3.4 Comparison of simple RNNs and GRUs

At roughly equal parameter counts, GRUs outperform simple recurrent units across the tested depths. With larger models and a fixed computational budget, deeper simple RNNs become competitive or better. The paper’s GRU variant moves the reset operation after the recurrent matrix multiplication to combine computations efficiently.

Equation 10
zt\displaystyle z_{t} =σ⁡(Wz​xt+Uz​ht−1+bz)\displaystyle=\sigma(W_{z}x_{t}+U_{z}h_{t-1}+b_{z}) (10)
rt\displaystyle r_{t} =σ⁡(Wr​xt+Ur​ht−1+br)\displaystyle=\sigma(W_{r}x_{t}+U_{r}h_{t-1}+b_{r})
h~t\displaystyle\tilde{h}_{t} =f⁡(Wh​xt+rt∘Uh​ht−1+bh)\displaystyle=f(W_{h}x_{t}+r_{t}\circ U_{h}h_{t-1}+b_{h})
ht\displaystyle h_{t} =(1−zt)​ht−1+zt​h~t\displaystyle=(1-z_{t})h_{t-1}+z_{t}\tilde{h}_{t}
Architecture Simple RNN GRU
5 layers, 1 Recurrent 14.40 10.53
5 layers, 3 Recurrent 10.56 8.00
7 layers, 5 Recurrent 9.78 7.79
9 layers, 7 Recurrent 9.52 8.19
Table 3:Comparison of development set WER for networks with either simple RNN or GRU, for various depths. All models have batch normalization, one layer of 1D-invariant convolution, and approximately 38 million parameters.

3.5 Frequency Convolutions

Convolutions over both frequency and time improve robustness to spectral variation, especially in noisy audio. Moving from one temporal convolution to three two-dimensional convolutions reduces noisy-development WER by 23.9% relative in the reported ablation. Convolutions come before recurrent layers so frequency structure remains available.

Architecture Channels Filter dimension Stride Regular Dev Noisy Dev
1-layer 1D 1280 11 2 9.52 19.36
2-layer 1D 640, 640 5, 5 1, 2 9.67 19.21
3-layer 1D 512, 512, 512 5, 5, 5 1, 1, 2 9.20 20.22
1-layer 2D 32 41x11 2x2 8.94 16.22
2-layer 2D 32, 32 41x11, 21x11 2x2, 2x1 9.06 15.71
3-layer 2D 32, 32, 96 41x11, 21x11, 21x11 2x2, 2x1, 2x1 8.61 14.74
Table 4:Comparison of WER for various arrangements of convolutional layers. In all cases, the convolutions are followed by 7 recurrent layers and 1 fully connected layer. For 2D-invariant convolutions the first dimension is frequency and the second dimension is time. All models have BatchNorm, SortaGrad, and 35 million parameters.

3.6 Striding

Larger temporal strides shorten recurrent sequences and reduce memory and work. English character outputs become too dense for aggressive strides, so the model predicts non-overlapping character bigrams, with single-character leftovers and spaces. This shortens the output sequence while preserving a deterministic text mapping.

Dev no LM Dev LM
Stride Unigrams Bigrams Unigrams Bigrams
2 14.93 14.56 9.52 9.66
3 15.01 15.60 9.65 10.06
4 18.86 14.84 11.92 9.93
Table 5:Comparison of WER with different amounts of striding for unigram and bigram outputs on a model with 1 layer of 1D-invariant convolution, 7 recurrent layers, and 1 fully connected layer. All models have BatchNorm, SortaGrad, and 35 million parameters. The models are compared on a development set with and without the use of a 5-gram language model.

3.7 Row Convolution and Unidirectional Models

Bidirectional recurrence needs the whole utterance. For streaming, forward-only recurrent layers feed a row convolution that uses a bounded future window. This separates most incremental computation from the limited lookahead required near the output.

Original figure
Figure 3:Row convolution architecture with future context size of 2
Equation 11
rt,i=∑j=1τ+1Wi,j​ht+j−1,i,for​ 1≤i≤d.\displaystyle r_{t,i}=\sum_{j=1}^{\tau+1}W_{i,j}h_{t+j-1,i},{\rm\ for\ }1\leq i\leq d. (11)

3.8 Language Model

A five-gram language model supplements the acoustic network: word-level for English, character-level for Mandarin. Beam search combines acoustic log probability, language-model log probability, and an insertion term. Deeper networks need less relative help from the external model, but the reported gains remain substantial.

Equation 12
Q⁡(y)=log⁡(pctc​(y|x))+α​log⁡(plm​(y))+β​word_count​(y)Q(y)=\log(p_{\textrm{ctc}}(y|x))+\alpha\log(p_{\textrm{lm}}(y))+\beta\>\textrm{word\_count}(y) (12)
Language Architecture Dev no LM Dev LM
English 5-layer, 1 RNN 27.79 14.39
English 9-layer, 7 RNN 14.93 9.52
Mandarin 5-layer, 1 RNN 9.80 7.13
Mandarin 9-layer, 7 RNN 7.55 5.81
Table 6:Comparison of WER for English and CER for Mandarin with and without a language model. These are simple RNN models with only one layer of 1D invariant convolution.

3.9 Adaptation to Mandarin

The Mandarin system directly predicts about 6,000 characters, including Roman letters, without a separate pronunciation or tone model. Decoding uses a character language model and insertion term. Its lower character rate also makes temporal striding easier than in English.

4 System Optimizations

The custom training system combines optimized recurrent computation, communication, CTC, and allocation. These engineering changes make architecture and data experiments feasible at the reported scale.

4.1 Scalability and Data-Parallelism

Synchronous data-parallel training averages gradients across GPUs. A ring all-reduce implementation reduces unnecessary memory copies and improves communication, especially within an eight-GPU node. The scaling experiment keeps examples per GPU fixed, so global minibatch size grows; it is a weak-scaling comparison, not constant-batch scaling.

Original figure
Figure 4:Scaling comparison of two networks—a 5 layer model with 3 recurrent layers containing 2560 hidden units in each layer and a 9 layer model with 7 recurrent layers containing 1760 hidden units in each layer. The times shown are to train 1 epoch. The 5 layer model trains faster because it uses larger matrices and is more computationally efficient.
GPU OpenMPI Our Performance
all-reduce all-reduce Gain
4 55359.1 2587.4 21.4
8 48881.6 2470.9 19.8
16 21562.6 1393.7 15.5
32 8191.8 1339.6 6.1
64 1395.2 611.0 2.3
128 1602.1 422.6 3.8
Table 7:Comparison of two different all-reduce implementations. All times are in seconds. Performance gain is the ratio of OpenMPI all-reduce time to our all-reduce time.

4.2 GPU implementation of CTC loss function

Moving CTC loss and gradient computation onto GPUs avoids large CPU transfers and speeds that component by 28.9 times for English and 12.5 times for Mandarin in the reported tests. End-to-end training improves by a smaller 10–20%, since CTC is only one component.

Language Architecture CPU CTC Time GPU CTC Time Speedup
English 5-layer, 3 RNN 5888.12 203.56 28.9
Mandarin 5-layer, 3 RNN 1688.01 135.05 12.5
Table 8:Comparison of time spent in seconds in computing the CTC loss function and gradient in one epoch for two different implementations. Speedup is the ratio of CPU CTC time to GPU CTC time.

4.3 Memory allocation

A custom allocator subdivides preallocated blocks to avoid repeated operating-system allocation overhead. Activations dominate memory use; exceptional long utterances can spill into mapped host memory at reduced bandwidth. This reduces failures without making host access as fast as local GPU memory.

5 Training Data

English training combines public and internal corpora for 11,940 hours and eight million utterances. Mandarin uses 9,400 hours and eleven million utterances, including read and spontaneous speech. Table 9 retains the English corpus breakdown.

Dataset Speech Type Hours
WSJ read 80
Switchboard conversational 300
Fisher conversational 2000
LibriSpeech read 960
Baidu read 5000
Baidu mixed 3600
Total 11940
Table 9:Summary of the datasets used to train DS2 in English. The Wall Street Journal (WSJ), Switchboard and Fisher[13] corpora are all published by the Linguistic Data Consortium. The LibriSpeech dataset[46] is available free on-line. The other datasets are internal Baidu corpora.

5.1 Dataset Construction

An existing bidirectional CTC recognizer aligns long audio with noisy transcripts; stretches of blank predictions provide segmentation boundaries. A classifier filters likely bad alignments using recognition costs and length ratios. On the described English subset, filtering lowers transcript WER from 17% to 5% while retaining over half the examples.

Equation 13
ℓ∗=arg​maxℓ∈Align​(x,y)∏tTpctc(ℓt|x;θ).\ell^{*}=\argmax_{\ell\in\text{Align}(x,y)}\prod_{t}^{T}p_{\text{ctc}}(\ell_{t}|x;\theta). (13)

5.2 Data Augmentation

Noise is added to 40% of randomly selected training utterances, using a large pool of audio sources. Too little augmentation weakens noise robustness; too much makes optimization harder.

5.3 Scaling Data

Increasing training data improves regular and noisy development WER across the tested subsets. The authors describe an approximate power-law trend and hypothesize that diversity of speakers and contexts matters alongside total hours. The experiment varies randomly sampled data volume, so it does not directly isolate that diversity hypothesis.

Fraction of Data Hours Regular Dev Noisy Dev
1% 120 29.23 50.97
10% 1200 13.80 22.99
20% 2400 11.65 20.41
50% 6000 9.51 15.90
100% 12000 8.46 13.59
Table 10:Comparison of English WER for Regular and Noisy development sets on increasing training dataset size. The architecture is a 9-layer model with 2 layers of 2D-invariant convolution and 7 recurrent layers with 68M parameters.

6 Results

Models train with momentum, gradient clipping, and scheduled learning-rate decay; the best held-out development checkpoint is evaluated. English and Mandarin use different beam widths and metrics. Results span public and internal datasets rather than one uniform test distribution.

6.1 English

The strongest English model has three frequency–time convolution layers and seven bidirectional recurrent layers. Human comparisons use the better of two crowd-worker transcriptions per clip, with repeated listening allowed. These workers and short-clip conditions define the comparison; the numbers are not a universal human-performance ceiling.

6.1.1 Model Size

Recognition improves up to the tested 100-million-parameter size. The selected simple-RNN model reduces WER on an internal mixed-condition test from 24.01% to 13.59%. Table 11’s caption incorrectly says all models have approximately 35 million parameters despite listing 18–100 million; its explicit row values and accompanying discussion describe the actual size comparison.

Model size Model type Regular Dev Noisy Dev
18×10618\times 10^{6} GRU 10.59 21.38
38×10638\times 10^{6} GRU 9.06 17.07
70×10670\times 10^{6} GRU 8.54 15.98
70×10670\times 10^{6} RNN 8.44 15.09
100×106100\times 10^{6} GRU 7.78 14.17
100×106100\times 10^{6} RNN 7.73 13.06
Table 11:Comparing the effect of model size on the WER of the English speech system on both the regular and noisy development sets. We vary the number of hidden units in all but the convolutional layers. The GRU model has 3 layers of bidirectional GRUs with 1 layer of 2D-invariant convolution. The RNN model has 7 layers of bidirectional simple recurrence with 3 layers of 2D-invariant convolution. Both models output bigrams with a temporal stride of 3. All models contain approximately 35 million parameters and are trained with BatchNorm and SortaGrad.
Test set DS1 DS2
Baidu Test 24.01 13.59
Table 12:Comparison of DS1 and DS2 WER on an internal test set of 3,300 examples. The test set contains a wide variety of speech including accents, low signal-to-noise speech, spontaneous and conversational speech.

6.1.2 Read Speech

On clean read speech, the model beats the measured workers on three of four test sets. It remains slightly worse on LibriSpeech test-other, at 13.25% versus 12.69% WER. These results concern read speech and do not remove the harder accent and noise gaps below.

Read Speech
Test set DS1 DS2 Human
WSJ eval’92 4.94 3.60 5.03
WSJ eval’93 6.94 4.98 8.08
LibriSpeech test-clean 7.89 5.33 5.83
LibriSpeech test-other 21.74 13.25 12.69
Table 13:Comparison of WER for two speech systems and human level performance on read speech.

6.1.3 Accented Speech

VoxForge tests cover American–Canadian, Commonwealth, European, and Indian accents. Deep Speech 2 substantially improves its predecessor but stays worse than the measured human results in every row, with the Indian-accent comparison close. Training breadth remains a limitation.

Accented Speech
Test set DS1 DS2 Human
VoxForge American-Canadian 15.01 7.55 4.85
VoxForge Commonwealth 28.46 13.56 8.15
VoxForge European 31.20 17.55 12.76
VoxForge Indian 45.35 22.44 22.15
Table 14:Comparing WER of the DS1 system to the DS2 system on accented speech.

6.1.4 Noisy Speech

Single-channel CHiME evaluation shows major improvement over the earlier system but substantial remaining human gaps: real-noise WER is 21.79% versus 11.84%, and simulated-noise WER 45.05% versus 31.33%. Multi-channel enhancement is not used in these comparisons.

Noisy Speech
Test set DS1 DS2 Human
CHiME eval clean 6.30 3.34 3.46
CHiME eval real 67.94 21.79 11.84
CHiME eval sim 80.27 45.05 31.33
Table 15:Comparison of DS1 and DS2 system on noisy speech. “CHiME eval clean” is a noise-free baseline. The “CHiME eval real” dataset is collected in real noisy environments and the “CHiME eval sim” dataset has similar noise synthetically added to clean speech. Note that we use only one of the six channels to test each utterance.

6.2 Mandarin

Deeper normalized recurrent models with frequency convolutions reduce internal Mandarin test CER from 15.41% to 7.93%. Separate human comparisons use small samples of 100 and 250 short queries; their results support those settings, not a broad claim about all Mandarin speech.

Architecture Dev Test
5-layer, 1 RNN 7.13 15.41
5-layer, 3 RNN 6.49 11.85
5-layer, 3 RNN + BatchNorm 6.22 9.39
9-layer, 7 RNN + BatchNorm + 2D conv 5.81 7.93
Table 16:Comparison of the improvements in DeepSpeech with architectural improvements. The development and test sets are Baidu internal corpora. All the models in the table have about 80 million parameters each

7 Deployment

Deployment replaces full-utterance operations with adaptive input normalization, forward recurrence, bounded lookahead, and cheaper decoding. Reported latency starts at the end of the utterance and excludes Internet transport; it is not time to the first partial transcript.

7.1 Batch Dispatch

Batch Dispatch eagerly groups ready streams whenever the previous batch finishes. This gains some matrix-multiplication efficiency without waiting to fill a large batch. At ten concurrent streams, the section reports 44 ms median and roughly 70 ms 98th-percentile compute latency; the introduction gives 67 ms for the latter.

Original figure
Figure 5:Probability that a request is processed in a batch of given size
Original figure
Figure 6:Median and 98 percentile latencies as a function of server load

7.2 Deployment Optimized Matrix Multiply Kernels

Half-precision inference reduces weight traffic with no measurable recognition loss in these tests. Custom matrix kernels target the very small batches typical of interactive serving, a different regime from large training batches.

Original figure
Figure 7:Comparison of kernels that computeA​x=bAx=b whereAA is a matrix with dimension2560×25602560\times 2560, andxx is a matrix with dimension2560×Batch size2560\times\textrm{Batch size}, whereBatch size∈[1,10]\textrm{Batch size}\in[1,10]. All matrices are in half-precision format.

7.3 Beam Search

Mandarin beam search considers only enough likely next characters to cover cumulative probability 0.99, capped at forty. This sharply reduces language-model lookups with a reported 0.1–0.3% relative CER cost.

7.4 Results

The deployed Mandarin system reaches 6.10% CER versus 5.81% for the research system on 2,000 held-out utterances. Its five forward recurrent layers, row convolution, half precision, batching, and pruned decoding jointly achieve the stated latency–accuracy tradeoff.

8 Conclusion

Larger datasets and models become useful when optimization and hardware execution scale with them. The paper demonstrates this across two languages and a deployable recognizer, while its own tests still expose sizable accent and noise limitations.

Acknowledgments

The authors acknowledge Baidu’s speech group, Nervana Systems, and NVIDIA for data, numerical kernels, and discussions.

Appendix A Scalability improvements

The appendix details the hardware topology and GPU loss computation underlying the training-speed results.

A.1 Node and cluster architecture

Each node combines two CPUs, eight Titan X GPUs, local data storage, and substantial host memory. GPU placement within PCI root complexes enables faster transfers, and InfiniBand carries gradients between nodes. Replicated local datasets keep input traffic away from the inter-node network.

Original figure
Figure 8:Schematic of our training node where PLX indicates a PCI switch and the dotted box includes all devices that are connected by the same PCI root complex.

A.2 GPU Implementation of CTC Loss Function

The GPU CTC implementation works in log-probability space and computes whole dynamic-programming columns in parallel, including some invalid entries whose contributions vanish when forward and backward values combine. Sorting labels once groups repeated characters for efficient gradient reductions. Columns remain sequentially dependent, while work within each column is parallel. Figure 9 retains the valid, invalid, and discarded-value layout.

Original figure
Figure 9:Forward and backward pass for GPU implementation of CTC. Gray circles contain valid values, circle withI contain−∞-\infty and circle withG contain garbage values that are finite.ℬ\mathcal{B} stand for the blank character that the CTC algorithm adds to the input utterance label. Column labels on top show different time-steps going from1 toT.
Concept demonstration

Open image ↗