Literature
Display
← All papers

Soft Clustering Anchors for Self-Supervised Speech Representation Learning in Joint Embedding Prediction Architectures

Georgios Ioannides · Adrian Kieback · Judah Goldfeder · Linsey Pang · Aman Chadha · Aaron Elkins · Yann LeCun · Ravid Shwartz-Ziv

Preprint posted · arXiv (preprint) ↗

Speech and audio6 min text · CondensedOriginal paper ↗
Contents

Overview

Intuition: a moving teacher and student can agree on useless constant speech features. This method gives them an additional acoustic reference: a fixed Gaussian mixture fitted to log-mel frames. The anchor supplies soft cluster probabilities rather than a single hard label. This January 2026 precursor is distinct from the later S-JEPA system, which changes both the target objective and how its mixture is updated.

At a glance

GMM-Anchored JEPA combines masked feature prediction with soft acoustic-cluster supervision. The cluster model is fitted once and remains fixed; its loss weight decays but retains a nonzero floor. Matched-architecture experiments compare speech recognition, emotion recognition and slot filling. The claim is improved stability and performance against these controlled baselines, not absolute leadership over every published speech model.

1 Introduction

Speech JEPA can collapse in the tested setup when the moving targets have no stable acoustic grounding. A frozen mixture over log-mel features supplies that reference without repeated offline reclustering. Keeping probabilities over several components preserves ambiguity near cluster boundaries. The contribution is best judged by whether this extra supervision prevents collapse and improves useful frozen features at matched compute.

Refer to caption
Figure 1: GMM-Anchored JEPA: one-time clustering replaces iterative re-training. Phase 1: A GMM is fitted once on log-mel features. Phase 2: The encoder trains with two objectives: predicting masked EMA teacher latents (JEPA loss) and matching frozen GMM posteriors (cluster loss). λ⁡(t)\lambda(t) decays from 1.0 to 0.01.

2 Related Work

The paper contrasts contrastive speech learning, hard-cluster prediction and moving-teacher distillation. Their anti-collapse mechanisms and target sources differ. A fixed acoustic mixture is neither a semantic transcript nor a guarantee that its clusters correspond to phonemes.

3 Method

The method separates one-time acoustic clustering from joint encoder training. The decoder-free representation objective remains accompanied by an explicit fixed-target anchor.

3.1 Framework Overview

A fixed Gaussian mixture produces soft assignments from log-mel frames. During encoder training, one branch predicts moving-teacher features and another matches those fixed cluster probabilities. This combines adaptable latent targets with a stationary reference.

3.2 Phase 1: GMM Fitting

Diagonal-covariance mixtures keep fitting tractable and assign probability mass to several clusters for ambiguous frames. Because the input log-mel space does not change with encoder training, the anchor avoids a feedback loop in which both representation and cluster target drift together.

Equation 1
qk​(𝐦)=πk​𝒩​(𝐦,𝝁k,𝝈k2)∑jπj​𝒩​(𝐦,𝝁j,𝝈j2).q_{k}(\mathbf{m})=\frac{\pi_{k}\mathcal{N}(\mathbf{m};\boldsymbol{\mu}_{k},\boldsymbol{\sigma}_{k}^{2})}{\sum_{j}\pi_{j}\mathcal{N}(\mathbf{m};\boldsymbol{\mu}_{j},\boldsymbol{\sigma}_{j}^{2})}. (1)

3.3 Phase 2: Joint Training

The objective adds masked feature regression to a cluster-distribution KL loss. The anchor starts strong and decays to a small positive value rather than disappearing. An ablation later shows that removing it entirely degrades this trained system, so the method is not an anchor-free JEPA after warmup.

Equations 2–4
ℒJEPA=1|ℳ|​∑t∈ℳ‖hψ​(z~student)t−zteacher,t‖2.\mathcal{L}_{\text{JEPA}}=\frac{1}{|\mathcal{M}|}\sum_{t\in\mathcal{M}}\|h_{\psi}(\tilde{z}_{\text{student}})_{t}-z_{\text{teacher},t}\|^{2}. (2)
ℒcluster=1|ℳ|∑t∈ℳKL(qgmm,t∥pcluster,t).\mathcal{L}_{\text{cluster}}=\frac{1}{|\mathcal{M}|}\sum_{t\in\mathcal{M}}\text{KL}(q_{\text{gmm},t}\|p_{\text{cluster},t}). (3)
ℒtotal=ℒJEPA+λ⁡(t)⋅ℒcluster,\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{JEPA}}+\lambda(t)\cdot\mathcal{L}_{\text{cluster}}, (4)

3.4 Denoising Augmentation

The online encoder sees noise and mixed utterances while the teacher sees clean audio. Energy-based mixing defines the perturbation strength. This adds an invariance demand beyond simply predicting a missing interval.

Equations 5–6
𝐱aug=𝐱clean+α⋅𝐧,α=Eclean10SNR/10⋅Enoise,\mathbf{x}_{\text{aug}}=\mathbf{x}_{\text{clean}}+\alpha\cdot\mathbf{n},\quad\alpha=\sqrt{\frac{E_{\text{clean}}}{10^{\text{SNR}/10}\cdot E_{\text{noise}}}}, (5)
𝐱mix[t1:t2]=𝐱1[t1:t2]+β⋅𝐱2[s1:s2],\mathbf{x}_{\text{mix}}[t_{1}:t_{2}]=\mathbf{x}_{1}[t_{1}:t_{2}]+\beta\cdot\mathbf{x}_{2}[s_{1}:s_{2}], (6)
Refer to caption
(a) Pure JEPA
Refer to caption
(b) WavLM-style
Refer to caption
(c) GMM-JEPA-T
Refer to caption
(d) GMM-JEPA
Figure 2: GMM-JEPA learns well-separated clusters; baselines collapse or overlap. UMAP of frame-level embeddings colored by predicted cluster. (a) Pure JEPA collapses. (b) WavLM-style overlaps. (c,d) GMM-JEPA variants form distinct regions.
Refer to caption
Figure 3: GMM-JEPA uses all clusters uniformly; baselines collapse to few. Frame counts per cluster (log scale, sorted by rank). Flat = high entropy. Baselines drop steeply; GMM-JEPA stays flat.
Refer to caption
(a) Pure JEPA
Refer to caption
(b) WavLM-style
Refer to caption
(c) GMM-JEPA-T
Refer to caption
(d) GMM-JEPA
Figure 4: GMM-JEPA variants maintain stable, high-confidence clusters over time. Top: cluster ID per frame. Bottom: confidence (1 −- normalized entropy). (a) Pure JEPA flickers rapidly with near-zero confidence, indicating degenerate representations. (b) WavLM-style shows moderate confidence (0.4–0.6) with frequent cluster switching. (c) GMM-JEPA-T maintains high confidence (0.7–0.9) with moderate cluster and sparse transitions. (d) GMM-JEPA also shows moderate clusters with temporally coherent spans and moderate-to-high confidence (0.5–0.8).

3.5 Training Algorithm

The complete algorithm specifies teacher averaging, block masks and the anchor schedule. After training, the representation encoder is retained; the target branch and auxiliary supervision support training rather than ordinary feature extraction.

Algorithm 1 GMM-Anchored JEPA Training
1:  Phase 1: Fit GMM on log-mel features
2:  Phase 2:
3:  Init: fϕf_{\phi}, fϕ′←fϕf_{\phi^{\prime}}\leftarrow f_{\phi}, hψh_{\psi}, cluster head, 𝐭mask\mathbf{t}_{\text{mask}}
4:  for each minibatch {𝐱}\{\mathbf{x}\} do
5:   𝐱aug,𝐱clean←Augment​(𝐱)\mathbf{x}_{\text{aug}},\mathbf{x}_{\text{clean}}\leftarrow\text{Augment}(\mathbf{x})
6:   zonline←fϕ​(𝐱aug)z_{\text{online}}\leftarrow f_{\phi}(\mathbf{x}_{\text{aug}})
7:   ztarget←fϕ′​(𝐱clean)z_{\text{target}}\leftarrow f_{\phi^{\prime}}(\mathbf{x}_{\text{clean}}) {No grad}
8:   qgmm←GMM.posterior​(LogMel​(𝐱clean))q_{\text{gmm}}\leftarrow\text{GMM.posterior}(\text{LogMel}(\mathbf{x}_{\text{clean}}))
9:   m←SampleBlockMask​()m\leftarrow\text{SampleBlockMask}()
10:   z~←m⊙zonline+(1−m)⊙𝐭mask\tilde{z}\leftarrow m\odot z_{\text{online}}+(1-m)\odot\mathbf{t}_{\text{mask}}
11:   zpred←hψ​(z~)z_{\text{pred}}\leftarrow h_{\psi}(\tilde{z})
12:   pcluster←softmax​(ClusterHead​(zonline))p_{\text{cluster}}\leftarrow\text{softmax}(\text{ClusterHead}(z_{\text{online}}))
13:   ℒJEPA←MSE​(zpred​[ℳ],ztarget​[ℳ])\mathcal{L}_{\text{JEPA}}\leftarrow\text{MSE}(z_{\text{pred}}[\mathcal{M}],z_{\text{target}}[\mathcal{M}])
14:   ℒcluster←KL(qgmm[ℳ]∥pcluster[ℳ])\mathcal{L}_{\text{cluster}}\leftarrow\text{KL}(q_{\text{gmm}}[\mathcal{M}]\|p_{\text{cluster}}[\mathcal{M}])
15:   λ←λstart+(λend−λstart)⋅t/Tmax\lambda\leftarrow\lambda_{\text{start}}+(\lambda_{\text{end}}-\lambda_{\text{start}})\cdot t/T_{\text{max}}
16:   ℒ←ℒJEPA+λ⋅ℒcluster\mathcal{L}\leftarrow\mathcal{L}_{\text{JEPA}}+\lambda\cdot\mathcal{L}_{\text{cluster}}
17:   Update ϕ,ψ\phi,\psi via ∇ℒ\nabla\mathcal{L}
18:   ϕ′←τ​ϕ′+(1−τ)​ϕ\phi^{\prime}\leftarrow\tau\phi^{\prime}+(1-\tau)\phi {EMA update}
19:  end for

3.6 Architecture

The main encoder combines strided audio convolutions, density gates and a Conformer stack at 20 ms frames. A transformer variant checks whether the anchor’s benefit depends on that backbone. The controlled WavLM-style baseline uses the same architecture, unlike an unrestricted comparison with published WavLM checkpoints.

4 Experiments

Models are trained on about 50,000 hours of English audio and evaluated using frozen encoders with task-specific heads. The matched WavLM-style comparison isolates objective changes better than comparisons across unrelated pretrained systems. Recognition uses the specified labeled downstream data and greedy decoding; those supervised heads are not part of pretraining.

Table 1: GMM-JEPA reduces WER by 14% relative. ASR on LibriSpeech dev-clean. Pure JEPA collapses (100% WER).
Model WER (%) ↓\downarrow CER (%) ↓\downarrow Δ\Delta WER
Pure JEPA 100.00 93.11 –
WavLM-style 33.22 11.28 baseline
GMM-JEPA 29.18 9.62 −-12.2%
GMM-JEPA-T 28.68 9.44 −-13.7%
Table 2: GMM-JEPA outperforms WavLM-style by 5.6% absolute. Slot filling on SNIPS.
Model Type F1 ↑\uparrow Edit F1 ↑\uparrow Δ\Delta Type
Pure JEPA 5.0 2.2 –
WavLM-style 59.1 33.7 baseline
GMM-JEPA-T 59.2 32.8 ++0.1
GMM-JEPA 64.7 36.0 ++5.6
Table 3: GMM-JEPA improves emotion recognition by 2% absolute. SER accuracy (%) consistent across IEMOCAP folds.
Model F1 F2 F3 F4 F5 Avg Δ\Delta
Pure JEPA 48.88 46.12 49.89 48.67 46.97 48.11 –
WavLM-style 64.83 64.19 65.30 67.11 65.85 65.46 base
GMM-JEPA 66.07 67.85 66.67 68.56 67.37 67.30 ++1.8
GMM-JEPA-T 67.19 67.52 67.58 68.11 68.41 67.76 ++2.3

5 Analysis

Cluster-use entropy measures whether many categories are used, while adjacent consistency measures temporal smoothness. A high value of either alone is insufficient: uniform random categories can have high entropy, and a constant output can be perfectly smooth. The combination of diagnostics and downstream tasks is therefore more informative than a single visualization.

Equations 7–8
H=−∑k=1Kpklogpk,pk=|{t:ct=k}|Ttotal,H=-\sum_{k=1}^{K}p_{k}\log p_{k},\quad p_{k}=\frac{|\{t:c_{t}=k\}|}{T_{\text{total}}}, (7)
Consistency=1T−1∑t=1T−1𝟏[ct=ct+1].\text{Consistency}=\frac{1}{T-1}\sum_{t=1}^{T-1}\mathbf{1}[c_{t}=c_{t+1}]. (8)
Table 4: GMM-JEPA achieves 98% entropy vs. 31% for WavLM-style. Cluster analysis metrics with K=1024K=1024.
Model Entropy ↑\uparrow Used ↑\uparrow Adj. Cons.
Pure JEPA 45% 516 0.205
WavLM-style 31% 978 0.750
GMM-JEPA 85% 1007 0.395
GMM-JEPA-T 98% 1013 0.299

5.1 Embedding Quality Analysis

The study combines cluster agreement, linear and nonlinear probes, and label alignment. Some features favor speaker or gender information while others favor phonetic distinctions. Probe success measures recoverability under a trained readout, not a transcript already present in the unsupervised cluster IDs.

Equation 9
NMI​(U,V)=2⋅I⁡(U,V)H⁡(U)+H⁡(V),\text{NMI}(U,V)=\frac{2\cdot I(U;V)}{H(U)+H(V)}, (9)
Refer to caption
(a) Diffuse clusters
Refer to caption
(b) 55.9% entropy
Refer to caption
(c) Unstable assignments
Figure 5: Removing residual GMM supervision (λend=0\lambda_{\text{end}}=0) causes collapse. Without ongoing anchoring, representations deteriorate. (a) UMAP shows overlapping clusters. (b) Only 506/1024 clusters used. (c) Confidence drops, flickering increases.
Table 5: All anchored models saturate on speaker/gender; Pure JEPA fails. Linear probe accuracy (%) on paralinguistic categories.
Model Speaker ↑\uparrow Gender ↑\uparrow
Pure JEPA 18.7±\pm1.8 69.2±\pm1.5
WavLM-style 99.6±\pm0.3 99.4±\pm0.2
GMM-JEPA-T 99.7±\pm0.2 99.7±\pm0.2
GMM-JEPA 99.8±\pm0.2 99.6±\pm0.3
Table 6: GMM-JEPA improves phoneme classification by +5.4% (linear) and +6.7% (MLP). Phoneme probe accuracy (%).
Model Linear ↑\uparrow MLP ↑\uparrow
Pure JEPA 7.4±\pm0.0 9.0±\pm0.1
WavLM-style 47.6±\pm0.2 57.5±\pm0.2
GMM-JEPA-T 51.4±\pm0.1 59.4±\pm0.4
GMM-JEPA 53.0±\pm0.1 64.2±\pm0.2
Table 7: Pure JEPA shares no structure with other models (NMI<<0.11). NMI between embedding clusters.
Pure WavLM GMM GMM-T
Pure JEPA 1.00 0.08 0.11 0.06
WavLM-style 0.08 1.00 0.47 0.57
GMM-JEPA 0.11 0.47 1.00 0.47
GMM-JEPA-T 0.06 0.57 0.47 1.00
Table 8: GMM-JEPA captures speaker/gender; GMM-JEPA-T captures phonemes. Cluster-label alignment (NMI).
Model Phoneme ↑\uparrow Speaker ↑\uparrow Gender ↑\uparrow
Pure JEPA 0.03 0.14 0.02
WavLM-style 0.20 0.27 0.10
GMM-JEPA 0.19 0.48 0.15
GMM-JEPA-T 0.24 0.22 0.10

5.2 Ablation: GMM Anchoring

When the anchor coefficient decays to zero, cluster utilization and downstream results worsen. This supports retaining the residual fixed-target loss in the tested recipe. It does not establish that every JEPA speech architecture requires this particular mixture or schedule.

Table 9: Anchoring is essential: λend=0\lambda_{\text{end}}=0 degrades WER to 40.9%. Effect of removing GMM supervision after decay.
λend\lambda_{\text{end}} Ent.↑\uparrow Used↑\uparrow WER ↓\downarrow SER ↑\uparrow SF-T ↑\uparrow SF-E ↑\uparrow
0.00 57.7 530 40.9 63.8 62.3 34.2
0.01 84.7 1011 29.2 67.3 64.7 36.0

6 Discussion

The discussion interprets the acoustic anchor as a practical stabilizer. Limits include the fixed feature space, mixture-model assumptions, English data and the need to test larger scales and other acoustic domains. The controlled baseline results should remain separate from claims about absolute state of the art.

7 Conclusion

The method demonstrates that a one-time soft acoustic clustering can support useful latent prediction without repeated offline reclustering. Its gains concern the evaluated backbones and tasks. The later S-JEPA formulation is a continuation of this line, not merely another title for this exact objective.

Impact Statement

The impact statement concerns speech-model research broadly. The reported representation metrics do not themselves assess privacy, demographic fairness or deployment reliability.

Appendix A Architecture Details

The appendix fixes the architecture shared by the main model and its controlled baseline. These details are needed to distinguish an objective comparison from a backbone comparison.

A.1 Encoder Backbone

The encoder uses strided convolutional blocks before the Conformer. This establishes frame resolution and local receptive fields before attention.

Equations 10–14
h0=Conv1d1→C0k=7,p=3​(𝐱).h_{0}=\text{Conv1d}^{k=7,\,p=3}_{1\to C_{0}}(\mathbf{x}). (10)
x\displaystyle x =x+12​FFN1​(x),\displaystyle=x+\frac{1}{2}\text{FFN}_{1}(x), (11)
x\displaystyle x =x+MHSA​(x),\displaystyle=x+\text{MHSA}(x), (12)
x\displaystyle x =x+ConvModule​(x),\displaystyle=x+\text{ConvModule}(x), (13)
x\displaystyle x =x+12​FFN2​(x).\displaystyle=x+\frac{1}{2}\text{FFN}_{2}(x). (14)

A.2 Snake-Beta Activation

Snake-Beta supplies a learnable periodic component to the nonlinear activation. Its presence is a shared architectural choice rather than the isolated GMM-anchoring contribution.

Equation 15
Snakeα​(x)=x+sin2⁡(α​x)α,α=softplus​(a)+0.01,\text{Snake}_{\alpha}(x)=x+\frac{\sin^{2}(\alpha x)}{\alpha},\quad\alpha=\text{softplus}(a)+0.01, (15)

A.3 Gated Relative Position Bias

Gated relative-position bias adjusts attention scores according to distance and learned gates. The complete equations preserve how positional information enters the Conformer.

Equations 16–20
bucket​(i−j)={|i−j|if ​|i−j|<B4B4+B4⋅log⁡(|i−j|/B4)log⁡(Dmax/B4)otherwise\text{bucket}(i-j)=\begin{cases}|i-j|&\text{if }|i-j|<\frac{B}{4}\\ \frac{B}{4}+\frac{B}{4}\cdot\frac{\log(|i-j|/\frac{B}{4})}{\log(D_{\max}/\frac{B}{4})}&\text{otherwise}\end{cases} (16)
gupdate\displaystyle g_{\text{update}} =σ⁡(𝐪⋅𝐮),\displaystyle=\sigma(\mathbf{q}\cdot\mathbf{u}), (17)
greset\displaystyle g_{\text{reset}} =σ⁡(𝐪⋅𝐰),\displaystyle=\sigma(\mathbf{q}\cdot\mathbf{w}), (18)
r~i−j\displaystyle\tilde{r}_{i-j} =s⋅greset,i⋅di−j,\displaystyle=s\cdot g_{\text{reset},i}\cdot d_{i-j}, (19)
ri−j\displaystyle r_{i-j} =di−j+gupdate,i⋅di−j+(1−gupdate,i)⋅r~i−j,\displaystyle=d_{i-j}+g_{\text{update},i}\cdot d_{i-j}+(1-g_{\text{update},i})\cdot\tilde{r}_{i-j}, (20)

A.4 Layer Aggregation

Cross-attention aggregates intermediate layers into a representation. Learned layer combination can preserve information at several depths; it should not be confused with selecting a transcript-aligned acoustic unit.

Equations 21–25
𝐐\displaystyle\mathbf{Q} =WQ⋅pool​(z(L)),\displaystyle=W_{Q}\cdot\text{pool}(z^{(L)}), (21)
𝐊\displaystyle\mathbf{K} =WK⋅stack​(pool​(z(l))),\displaystyle=W_{K}\cdot\text{stack}(\text{pool}(z^{(l)})), (22)
𝐕\displaystyle\mathbf{V} =WV⋅stack​(pool​(z(l))),\displaystyle=W_{V}\cdot\text{stack}(\text{pool}(z^{(l)})), (23)
weights =softmax​(𝐐𝐊⊤/d),\displaystyle=\text{softmax}(\mathbf{Q}\mathbf{K}^{\top}/\sqrt{d}), (24)
zagg\displaystyle z_{\text{agg}} =∑lweightsl⋅z(l),\displaystyle=\sum_{l}\text{weights}_{l}\cdot z^{(l)}, (25)

A.5 Density Adaptive Attention (DAAM)

The density-adaptive gate normalizes temporal features and combines Gaussian-mixture responses before modulation. This gate is distinct from the frozen acoustic GMM that supplies the auxiliary training target.

Equations 26–29
x¯=x−μxσx+ϵ.\bar{x}=\frac{x-\mu_{x}}{\sigma_{x}+\epsilon}. (26)
log⁡wi\displaystyle\log w_{i} =−(x¯−δi)22​ci2−12​log⁡(2​π​ci2),\displaystyle=-\frac{(\bar{x}-\delta_{i})^{2}}{2c_{i}^{2}}-\frac{1}{2}\log(2\pi c_{i}^{2}), (27)
log⁡w\displaystyle\log w =∑i=1Nglog⁡wi.\displaystyle=\sum_{i=1}^{N_{g}}\log w_{i}. (28)
out=x⊙softmax​(log⁡w)+x.\text{out}=x\odot\text{softmax}(\log w)+x. (29)

A.6 Cluster Prediction Head

The cluster head maps encoder features to a distribution over the fixed mixture components. It learns to match soft probabilities rather than force each frame into a single label.

Equations 30–32
h0\displaystyle h_{0} =GELU​(LayerNorm​(LinearC→H​(z))),\displaystyle=\text{GELU}(\text{LayerNorm}(\text{Linear}_{C\to H}(z))), (30)
hl\displaystyle h_{l} =hl−1+Blockl(hl−1),l=1,…,Lhead,\displaystyle=h_{l-1}+\text{Block}_{l}(h_{l-1}),\quad l=1,\ldots,L_{\text{head}}, (31)
logits =LinearH→K​(LayerNorm​(hLhead)),\displaystyle=\text{Linear}_{H\to K}(\text{LayerNorm}(h_{L_{\text{head}}})), (32)

A.7 Predictor Architecture

The predictor estimates teacher features at hidden positions from context. It supports the latent-regression part of the objective, separately from cluster-distribution supervision.

A.8 Target Encoder and EMA Update

The averaged target encoder supplies slowly changing latent targets. Its averaging coefficient controls lag and does not by itself supply the stationary acoustic reference.

Equation 33
ϕ′←τ​ϕ′+(1−τ)​ϕ,τ=0.996.\phi^{\prime}\leftarrow\tau\phi^{\prime}+(1-\tau)\phi,\quad\tau=0.996. (33)

Appendix B Training Hyperparameters

The full hyperparameter table records masking, schedules, optimization and architecture settings for the reported runs. These values describe the experiment; they are not claimed as universally optimal speech settings.

Table 10: Training hyperparameters.
Conformer Transformer
Encoder Architecture
Parameters 31.1M 41.3M
CNN layers 3 7
Channels [32,64,128,256] [256]×\times7
Strides [8,8,5] [5,2,2,2,2,2,2]
Latent dim 512 512
Encoder layers 4 10
Attention heads 32 8
FFN mult / dim 4×\times512 2048
Conv kernel 31 –
GMM Fitting
Components KK 1024
Mel bins 80
Initialization K-means++
Optimization Mini-batch SGD
Learning rate 10−2→10−410^{-2}\to 10^{-4} (cosine)
Training frames 9.6B
K-Means Fitting (WavLM-style baseline)
Components KK 1024
Feature type Log-mel (80 bins)
Initialization K-means++
Update method Lloyd’s algorithm
Lloyd iterations 20
Training frames 9.6B
Training
Optimizer AdamW [22]
Learning rate 10−5→10−4→10−510^{-5}\to 10^{-4}\to 10^{-5}
LR schedule 10%10\% linear warmup / decay
Weight decay 10−310^{-3}
Micro Batch size 96 (Conformer) 192 (Transf.)
Gradient clipping 1.0
GPUs 8 ×\times B200
EMA τ\tau 0.996
λ\lambda decay 1.0→0.011.0\to 0.01
Mask ratio [0.4,0.65][0.4,0.65]
Span length [10,25][10,25] frames
Augmentation
Noise SNR [−5,20][-5,20] dB
Mix ratio [−5,5][-5,5] dB
Noise/mix prob. 0.25 each

Appendix C GMM Details

The mixture appendix specifies how acoustic features become probabilistic cluster targets. It is the fixed anchor’s implementation contract.

C.1 Posterior Computation

Posterior probabilities combine each component’s likelihood with its mixture weight and normalize across components. Diagonal covariance and variance floors constrain the statistical model and numerical behavior.

Equation 34
qk​(𝐦)=softmaxk​(log⁡πk−D2​log⁡(2​π)−12​∑d=1D[log⁡σk,d2+(md−μk,d)2σk,d2]).q_{k}(\mathbf{m})=\text{softmax}_{k}\left(\log\pi_{k}-\frac{D}{2}\log(2\pi)-\frac{1}{2}\sum_{d=1}^{D}\left[\log\sigma_{k,d}^{2}+\frac{(m_{d}-\mu_{k,d})^{2}}{\sigma_{k,d}^{2}}\right]\right). (34)

C.2 Chunked Soft Assignment

Chunking the assignment calculation bounds memory without changing the target probability distribution. The preserved algorithm retains how chunks are accumulated.

Algorithm 2 Chunked Soft Assignment
1:  Input: Features X∈ℝN×DX\in\mathbb{R}^{N\times D}, GMM parameters, chunk sizes
2:  Output: Posteriors Q∈ℝN×KQ\in\mathbb{R}^{N\times K}
3:  for n=0n=0 to NN step batch_size do
4:   for k=0k=0 to KK step chunk_k do
5:    Δ←X[n:n′]−μ[k:k′]\Delta\leftarrow X[n:n^{\prime}]-\mu[k:k^{\prime}] {[B,K′,D][B,K^{\prime},D]}
6:    mahal←∑dΔd2/σk,d2\text{mahal}\leftarrow\sum_{d}\Delta_{d}^{2}/\sigma_{k,d}^{2}
7:    log⁡pn,k←−D2​log⁡(2​π)−12​∑dlog⁡σk,d2−12​mahal\log p_{n,k}\leftarrow-\frac{D}{2}\log(2\pi)-\frac{1}{2}\sum_{d}\log\sigma_{k,d}^{2}-\frac{1}{2}\text{mahal}
8:   end for
9:   Q[n:n′]←softmax(logπ+logp)Q[n:n^{\prime}]\leftarrow\text{softmax}(\log\pi+\log p)
10:  end for

Appendix D Augmentation Details

Noise and overlap augmentation are defined by waveform energy so their strength has a consistent acoustic interpretation.

D.1 Energy-Based SNR Mixing

The noise scale is chosen from a target signal-to-noise ratio using clean and noise energies. The exact sampling range remains in the source equations.

Equations 35–37
Ex\displaystyle E_{x} =1|x|​∑ixi2,En=1|n|​∑ini2,\displaystyle=\frac{1}{|x|}\sum_{i}x_{i}^{2},\quad E_{n}=\frac{1}{|n|}\sum_{i}n_{i}^{2}, (35)
α\displaystyle\alpha =Ex10SNR/10⋅En,\displaystyle=\sqrt{\frac{E_{x}}{10^{\text{SNR}/10}\cdot E_{n}}}, (36)
xaug\displaystyle x_{\text{aug}} =x+α⋅n,\displaystyle=x+\alpha\cdot n, (37)

D.2 Utterance Mixing

A segment from another utterance is scaled and mixed into the primary waveform. This is acoustic augmentation, not an independent semantic label for the mixed speech.

D.3 Augmentor Buffer

The augmentor buffer supplies reusable noise or utterance segments. Its role is practical data mixing; the attached fold-level result table is retained in its original location.

Table 11: SER accuracy (%) per fold for λend=0\lambda_{\text{end}}=0 ablation.
λend\lambda_{\text{end}} F1 F2 F3 F4 F5 Avg
0.00 63.48 63.41 63.70 63.22 65.15 63.79
0.01 66.07 67.85 66.67 68.56 67.37 67.30

Appendix E Visualization Details

UMAP projection settings document the visual comparison. Such projections illustrate cluster organization but cannot establish high-dimensional separation or downstream quality without the quantitative tests.

Table 12: UMAP hyperparameters.
Parameter Value
n_neighbors 15
min_dist 0.1
metric Euclidean
n_components 2
random_state 42
n_samples 10,000
Concept demonstration

Open image ↗