Literature
Display
← All papers

S-JEPA : Soft Clustering Anchors for Self-Supervised Speech Representation Learning

Georgios Ioannides · Adrian Kieback · Judah Goldfeder · Linsey Pang · Aman Chadha · Aaron Elkins · Yann LeCun · Ravid Shwartz-Ziv

Preprint posted · arXiv (preprint) ↗

Speech and audio6 min text · CondensedOriginal paper ↗
Contents

At a glance

S-JEPA predicts soft acoustic-cluster distributions at masked positions. It begins with a fixed MFCC mixture, then updates a mixture online from averaged encoder features while choosing the representation layer adaptively. A 51.8 million-parameter encoder performs strongly among the compared smaller speech models. The method differs from the January precursor: its target is categorical KL prediction rather than a combined latent-regression and fixed-anchor loss.

1 Introduction

Hard cluster labels discard uncertainty when a sound frame lies between acoustic categories. S-JEPA keeps probabilities over possible categories and trains a predictor to reproduce them. The engineering aim is useful compact speech features without a repeated stop-and-recluster pipeline. Soft targets are acoustic information, not human-supplied phoneme or word labels.

Refer to caption
Figure 1: S-JEPA: a JEPA-style encoder–predictor pair matches soft GMM posteriors at masked positions, with the GMM fit in Phase 1 (frozen, over MFCC features) and updated online in Phase 2 (over EMA encoder features). The single training signal is KL divergence at masked positions; the predictor, cluster head, and GMM are discarded after pre-training.

2 Related Work

The lineage combines masked speech prediction, soft clustering and teacher averaging. The paper explicitly connects to the earlier GMM-anchored precursor but changes its objective and training procedure. Its relationship to JEPA is the encoder–predictor masked-prediction pattern; the target itself is a categorical distribution.

3 Method

The method needs three pieces: the encoder–predictor architecture, a probability-matching objective and a continuous two-phase target-update recipe.

3.1 Architecture

The encoder processes waveform frames, a predictor fills masked positions, and a small head outputs cluster probabilities. An EMA encoder is maintained for the second phase’s target features. Training-only predictor and cluster components are discarded for ordinary encoder inference.

Equations 1–2
utvis\displaystyle u^{\text{vis}}_{t} =gω​(zt)​(visible positions; not supervised)\displaystyle=g_{\omega}(z_{t})\;\;\text{\small(visible positions; not supervised)} (1)
utmask\displaystyle u^{\text{mask}}_{t} =gω​(z^t)​(masked positions; supervised in Equation 4)\displaystyle=g_{\omega}(\hat{z}_{t})\;\;\text{\small(masked positions; supervised in Equation~\ref{eq:loss})} (2)

3.2 Soft Targets and KL Loss

The target is a soft posterior over GMM components. KL divergence matches the predicted distribution to that target, preserving ambiguity a hard argmax label would remove. Which positions contribute to the loss changes during training, so the final masked-only configuration should not be described as the entire training history.

Equations 3–4
qk​(m)=πk​𝒩​(m,μk,σk2)∑jπj​𝒩​(m,μj,σj2).q_{k}(m)=\frac{\pi_{k}\mathcal{N}(m;\mu_{k},\sigma^{2}_{k})}{\sum_{j}\pi_{j}\mathcal{N}(m;\mu_{j},\sigma^{2}_{j})}. (3)
ℒ=1|S|∑t∈SKL(qt∥pt),\mathcal{L}=\frac{1}{|S|}\sum_{t\in S}\mathrm{KL}\!\left(q_{t}\,\|\,p_{t}\right), (4)

3.3 Two-Phase Training

Phase 1 fits a fixed mixture to MFCC features. Phase 2 initializes a mixture over EMA encoder features, updates it online and chooses its input layer using effective-rank measurements. A periodically switched teacher-averaging rate changes how rapidly the targets follow the student. These components form a recipe, not independently established causes of the measured gain.

Refer to caption
(a) Phase 1. Layer 2 settles as the most expressive layer, motivating its use as the initial GMM input layer for Phase 2.
Refer to caption
(b) Phase 2. Under the online GMM, layer 4 dominates from the first measured checkpoint and the L4–L2 gap widens.
Figure 2: Per-layer effective rank across both phases of training. The horizontal axis shows training step
Equation 5
αt∈{αfast,αslow},αfast=0.999,αslow=0.9999,\alpha_{t}\in\{\alpha_{\text{fast}},\alpha_{\text{slow}}\},\quad\alpha_{\text{fast}}=0.999,\quad\alpha_{\text{slow}}=0.9999, (5)

3.4 Training Algorithm

The algorithm makes the phase transition, masked-position updates and target-averaging schedule explicit. No gradients flow through the GMM updates; mixture statistics follow their own update rule.

4 Experiments

Pretraining uses approximately 83,000 hours of English speech. Evaluation freezes the 51.8 million-parameter encoder and trains smaller task-specific heads under SUPERB protocols. Parameter comparisons count inference encoders, not every teacher, predictor or pretraining computation used by alternative methods.

Original figure
Figure 3: S-JEPA advances the Pareto frontier on SUPERB below 90M parameters. Parameter count vs. performance on (a) ASR (WER on LibriSpeech test-clean, ↓\downarrow) and (b) emotion recognition (accuracy, ↑\uparrow). Dashed line: Pareto frontier across all evaluated SSL methods. Shaded region: S-JEPA’s improvement over the prior best in the 51.8M–90M range. S-JEPA reaches 12.10% WER (vs. 13.02% for DeCoAR 2.0 at 89.8M) and 64.83% ER accuracy (matching HuBERT-Base at 55% of its parameters), trained directly without a teacher.
Table 1: Frozen-encoder evaluation on the SUPERB benchmark: ASR (WER on LibriSpeech test-clean), emotion recognition (ER, accuracy), and slot filling (SF, F1 and CER). All entries use the standard SUPERB protocol with greedy CTC decoding for ASR. Baseline numbers are from the public SUPERB leaderboard. Parameter counts are inference-time encoder only. Best result within the sub-90M block in bold.
Method # Params ASR ER SF SF
WER (↓\downarrow) Acc. (↑\uparrow) F1 (↑\uparrow) CER (↓\downarrow)
Sub-90M baselines
modified CPC [39] 001.8M 20.18 60.96 71.19 49.91
APC [11] 004.1M 21.28 59.33 70.46 50.89
VQ-APC [12] 004.6M 21.20 59.66 68.53 52.91
PASE+ [33] 007.8M 25.11 57.86 62.14 60.17
NPC [27] 019.4M 20.20 59.08 72.79 48.44
TERA [29] 021.3M 18.17 56.27 67.50 54.17
DistilHuBERT [8] 023.5M 13.37 63.02 82.57 35.59
wav2vec [35] 032.5M 15.86 59.79 76.37 43.71
vq-wav2vec [2] 034.2M 17.71 58.24 77.68 41.54
Mockingjay [28] 085.1M 22.82 50.28 61.59 58.89
DeCoAR 2.0 [26] 089.8M 13.02 62.47 83.28 34.73
S-JEPA (ours) 051.8M 012.10 64.83 83.05 33.17
Base-class (≈\approx95M)
HuBERT Base [20] 094.7M 06.42 64.92 88.53 25.20
WavLM Base [9] 094.7M 06.21 65.94 89.38 22.86
WavLM Base+ [9] 094.7M 05.59 68.65 90.58 21.20
wav2vec 2.0 Base [3] 095.0M 06.43 63.43 88.30 24.77
Large-class (reference, ≥\geq316M)
HuBERT Large [20] 316.6M 03.62 67.62 89.81 21.76
WavLM Large [9] 316.6M 03.44 70.62 92.21 18.36
wav2vec 2.0 Large [3] 316.6M 03.75 65.64 87.11 27.31

4.1 Frozen-Encoder Results on SUPERB

Greedy CTC recognition reaches 12.10% WER in the reported setup; the lower 8.50% result uses language-model rescoring and is a separate setting. The model has the best WER among the compared sub 90 million-parameter systems and strong emotion recognition. Slot-filling F1 is close to, but slightly below, DeCoAR2.0; the narrative’s broad dominance wording should not hide that table entry.

4.2 Probing Frozen Representations

Frozen-feature probes test speaker, gender, chapter and phoneme information. A linear probe favors a different phoneme model than a nonlinear probe, showing that readout capacity changes what information appears accessible. Random splits within the evaluation corpus and task-specific supervised labels define the scope of the comparison.

Table 2: S-JEPA leads on Speaker, Gender, and Chapter ID, and benefits more from a nonlinear probe on phoneme classification. Probing accuracy on LibriSpeech [32] across four tasks. Mean ±\pm standard deviation across 3 splits. Chapter ID is the most discriminative comparison: chapters within a book are typically read by the same speaker, so the cue is subtle prosodic and recording variation rather than speaker identity.
Task (Accuracy ↑\uparrow) WavLM HuBERT S-JEPA
Speaker ID 91.1 ±\pm 1.0 99.1 ±\pm 0.3 99.7 ±\pm 0.3
Gender Classification 96.3 ±\pm 0.5 99.5 ±\pm 0.3 99.6 ±\pm 0.3
Chapter ID 59.5 ±\pm 1.0 88.2 ±\pm 1.0 93.2 ±\pm 1.0
Phoneme (linear) 86.1 ±\pm 0.2 84.8 ±\pm 0.1 82.8 ±\pm 0.1
Phoneme (MLP) 87.5 ±\pm 0.1 86.5 ±\pm 0.1 88.0 ±\pm 0.1
Refer to caption
(a) One held-out utterance. From top: log-mel spectrogram; word-level Whisper alignments; argmax cluster ID per 2020 ms frame (K=500K\!=\!500); the predictor’s top-three cluster posteriors.
Original figure
(b) Per-frame predictor entropy across ∼\sim153K frames (N=500N\!=\!500 dev-clean utterances). Reference verticals at 0.50.5, 11, and 22 bits.
Figure 4: The soft-target objective produces structured boundary uncertainty, not diffuse fuzziness. 4(a) On a single utterance, rank-1 posteriors sit near 1.01.0 within phonetically stable regions and drop to the 0.30.3–0.450.45 range near word boundaries with rank-2 rising to compete. 4(b) The same pattern at the population level: a bimodal entropy distribution with a confident regime below 0.30.3 bits and a clear secondary mode at ∼1\sim 1bit (the entropy of a perfect two-way tie); 36.6%36.6\% of frames carry more than 11 bit.

5 Analysis

The analysis studies one illustrative utterance and an aggregate distribution of frame-level uncertainty. These answer different questions: what a particular prediction looks like and how often uncertain predictions occur.

5.1 Per-utterance Interpretability

The plotted word timings come from an external Whisper system used only for visual reference. They do not supervise S-JEPA or supply its quantitative scores. Rapid changes in cluster ID also show why the unsupervised categories should not be treated as one-to-one phoneme labels.

5.2 Aggregate Distribution of Predictor Uncertainty

Entropy across held-out frames has a confident regime and a secondary regime near a two-way tie. The reported 36.6% of frames above one bit supports preserving inter-cluster uncertainty. This is evidence about the predicted distributions; the paper leaves a matched hard-target ablation as future work.

6 Discussion

The compact model and structured uncertainty are the strongest findings. Several recipe components remain insufficiently isolated: teacher switching has no controlled alternative comparison, and augmentation and visible-position loss were changed together. English-only training limits claims about other languages and non-speech audio.

7 Conclusion

S-JEPA is a soft-target masked speech learner with online target adaptation. It supports a favorable parameter–performance tradeoff in the reported comparisons. Its mixture targets are categorical probabilities, and the remaining ablations matter when attributing gains to softness, online updates or the rest of the recipe.

Appendix A Architecture Details

The architecture largely follows HuBERT-style processing with a shallower six-layer encoder. Predictor and cluster head are training-only additions.

A.1 CNN Feature Extractor

A seven-layer convolutional frontend maps 16 kHz audio to 20 ms frames. Its stride and gradient scaling set the temporal resolution and learning balance before attention.

A.2 Transformer Stack

Six transformer layers use the stated width, heads, feed-forward size and regularization. The architecture saves encoder parameters relative to deeper base models; it does not remove transformer computation.

A.3 Encoder Mask Handling

Masked features are zeroed after the frontend/projection and before positional encoding and transformer processing. The predictor later inserts learned mask tokens. This placement is part of the implementation, not interchangeable with masking only at the final loss.

A.4 Predictor and Cluster Head

A small transformer predictor supplies hidden-position features to the cluster head. Only the specified positions receive supervision, even when the shared head is also evaluated on visible positions for analysis.

Equation
Linear⁡(768→768)→GELU→Linear⁡(768→768)→GELU→Linear⁡(768→K),\mathrm{Linear}(768\to 768)\to\mathrm{GELU}\to\mathrm{Linear}(768\to 768)\to\mathrm{GELU}\to\mathrm{Linear}(768\to K),

A.5 EMA Encoder

The averaged encoder is unused for Phase 1 targets and supplies target features in Phase 2. Its decay schedule affects how stationary those targets are while the online mixture adapts.

A.6 Masking

Temporal blocks hide contiguous frame spans. The target fraction and predictor-mask handling specify the prediction task’s difficulty.

Appendix B Training algorithm

The full algorithm preserves the sequence of GMM initialization, gradient updates, target averaging and layer selection. It is necessary for reproducing the continuous two-phase recipe.

Algorithm 1 S-JEPA training. Phase 1 uses a fixed MFCC GMM; Phase 2 uses the online encoder-feature GMM with adaptive layer selection and switched EMA decay.
1:  Init: fϕf_{\phi}, hψh_{\psi} (with mask token + pos. embeddings), gωg_{\omega}, EMA encoder f¯\bar{f}, GMM, active layer ℓ⋆\ell^{\star}, EMA decay αt←αfast\alpha_{t}\leftarrow\alpha_{\text{fast}}
2:  for each minibatch {𝐱}\{\mathbf{x}\} do
3:    𝐱aug,𝐱clean←Augment​(𝐱)\mathbf{x}_{\text{aug}},\mathbf{x}_{\text{clean}}\leftarrow\text{Augment}(\mathbf{x})
4:    𝐦←\mathbf{m}\leftarrow MFCC(𝐱clean)(\mathbf{x}_{\text{clean}}) (Phase 1) or f¯​(𝐱clean)\bar{f}(\mathbf{x}_{\text{clean}}) at ℓ⋆\ell^{\star} (Phase 2) {no_grad}
5:    𝐪←GMM.posterior​(𝐦)\mathbf{q}\leftarrow\text{GMM.posterior}(\mathbf{m})
6:    if Phase 2 then
7:     GMM ←\leftarrow EMA-update from 𝐦\mathbf{m}, 𝐪\mathbf{q} {online; no gradient}
8:     if layer-check step then
9:      Compute erank⁡(Z(ℓ))\mathrm{erank}(Z^{(\ell)}) for all ℓ\ell; EMA-update per-layer scores; ℓ⋆←arg⁡maxℓ⁡erankℓema\ell^{\star}\leftarrow\arg\max_{\ell}\mathrm{erank}^{\text{ema}}_{\ell}
10:     end if
11:    end if
12:    𝑚𝑎𝑠𝑘←SampleBlockMask​()\mathit{mask}\leftarrow\text{SampleBlockMask}(); z←fϕ​(𝐱aug,𝑚𝑎𝑠𝑘)z\leftarrow f_{\phi}(\mathbf{x}_{\text{aug}},\mathit{mask}); z^←hψ​(z,𝑚𝑎𝑠𝑘)\hat{z}\leftarrow h_{\psi}(z,\mathit{mask})
13:    𝐩t←softmax​(gω​(z^t))\mathbf{p}_{t}\leftarrow\text{softmax}(g_{\omega}(\hat{z}_{t})) for t∈ℳt\in\mathcal{M}
14:    ℒ←1|ℳ|∑t∈ℳKL(𝐪t∥𝐩t)\mathcal{L}\leftarrow\tfrac{1}{|\mathcal{M}|}\sum_{t\in\mathcal{M}}\text{KL}(\mathbf{q}_{t}\|\mathbf{p}_{t}); update ϕ,ψ,ω\phi,\psi,\omega via ∇ℒ\nabla\mathcal{L}
15:    if Phase 2 then
16:     Flip αt∈{αfast,αslow}\alpha_{t}\in\{\alpha_{\text{fast}},\alpha_{\text{slow}}\} per switching schedule (Equation 5);  ϕ¯←αt​ϕ¯+(1−αt)​ϕ\bar{\phi}\leftarrow\alpha_{t}\bar{\phi}+(1-\alpha_{t})\phi
17:    end if
18:  end for
19:  Return: fϕf_{\phi} {predictor, cluster head, GMM discarded}

Appendix C Phase 2 Transition: Masked-Only Loss and Augmentation Off

Partway through Phase 2, visible-position loss was removed and augmentation was disabled together, guided by development-set WER. Because the changes were simultaneous, their individual effects are not identified. The manuscript explicitly acknowledges this limitation.

Appendix D Training Hyperparameters

The hyperparameter table records the launch settings used for reported checkpoints. These settings include phase-specific masking, losses and schedules rather than one uniform training configuration.

Table 3: Training hyperparameters used in our experiments. AdamW [30] is the optimizer for both phases.
Phase 1 (MFCC GMM) Phase 2 (online GMM)
Encoder backbone (shared across phases)
Hidden dimension 768 768
Transformer layers 6 6
Attention heads 12 12
FFN dimension 3072 3072
CNN feature-extractor channels 512 512
CNN total stride / frame rate 320 / 50 Hz 320 / 50 Hz
Conv positional encoding (kernel, groups) (128, 16) (128, 16)
Dropout / attention / activation 0.1 / 0.1 / 0.1 0.1 / 0.1 / 0.1
LayerDrop 0.05 0.05
Feature-extractor gradient multiplier 0.1 0.1
GMM
KK (number of components) 100 500
Feature space 39-dim MFCC+Δ\Delta+Δ​Δ\Delta\Delta 768-dim EMA encoder features
Fitting k-means init (5 it.) + EM (20 it.) online EMA of suff. stats
Online param EMA decay — (frozen) 0.999
Layer-check cadence (auto-layer) — (n/a) every ∼10,000\sim 10{,}000 steps
Optimization
Optimizer AdamW AdamW
β1,β2\beta_{1},\beta_{2} (0.9,0.99)(0.9,0.99) (0.9,0.99)(0.9,0.99)
Weight decay 10−310^{-3} 10−310^{-3}
Learning rate ×10−41\!\times\!10^{-4} ×10−52.5\!\times\!10^{-5}
Batch size (global) 192 192
GPUs 8 8
Max input duration 15 s 15 s
EMA encoder f¯\bar{f} (periodically switched decay, Phase 2 only)
αslow\alpha_{\text{slow}} — (n/a) 0.9999
αfast\alpha_{\text{fast}} — (n/a) 0.999
Switching cadence — (n/a) every ∼20,000\sim 20{,}000 steps (avg.)
Masking
Mask ratio 0.65 0.65
Mask span length (frames) 10 10
Loss configuration
KL applied at masked + visible masked + visible (early), masked only (after)
Augmentation
pnoisep_{\text{noise}} 0.25 0.25 (early), 0.0 (after)
pmixp_{\text{mix}} 0.25 0.25 (early), 0.0 (after)
SNR range (noise mix), dB (−5,20)(-5,20) (−5,20)(-5,20) when active
Energy ratio range (utterance mix), dB (−5,5)(-5,5) (−5,5)(-5,5) when active

Appendix E GMM Details

The mixture implementation supplies probabilistic targets and updates without backpropagating through the mixture parameters.

E.1 Posterior Computation

Normalized Gaussian likelihoods produce soft assignments, with variance floors for stability. A posterior can spread mass across several plausible components.

Equation 6
qk​(𝐦)=softmaxk​(log⁡πk−D2​log⁡(2​π)−12​∑d=1D[log⁡σk,d2+(md−μk,d)2σk,d2])q_{k}(\mathbf{m})=\text{softmax}_{k}\left(\log\pi_{k}-\frac{D}{2}\log(2\pi)-\frac{1}{2}\sum_{d=1}^{D}\left[\log\sigma_{k,d}^{2}+\frac{(m_{d}-\mu_{k,d})^{2}}{\sigma_{k,d}^{2}}\right]\right) (6)

E.2 Phase 1 GMM Fitting

Phase 1 gathers MFCC frames using bounded-memory reservoir sampling, initializes with mini-batch clustering and refines by expectation-maximization. The resulting mixture stays fixed during that phase.

E.3 Online GMM Updates (Phase 2)

Phase 2 updates component statistics from EMA-encoder features with responsibility-weighted moving averages. This replaces repeated full-corpus offline reclustering but introduces target-update dynamics of its own.

Equation 7
θ←α​θ+(1−α)​θ^,θ∈{π,𝝁,𝝈2}\theta\leftarrow\alpha\theta+(1-\alpha)\hat{\theta},\quad\theta\in\{\pi,\boldsymbol{\mu},\boldsymbol{\sigma}^{2}\} (7)

E.4 Chunked Soft Assignment

The soft-assignment calculation is split into chunks to bound peak memory. The complete source algorithm preserves the normalization and accumulation steps.

Algorithm 2 Chunked Soft Assignment
1:  Input: Features X∈ℝN×DX\in\mathbb{R}^{N\times D}, GMM parameters {πk,μk,σk2}\{\pi_{k},\mu_{k},\sigma_{k}^{2}\}, chunk sizes
2:  Output: Posteriors Q∈ℝN×KQ\in\mathbb{R}^{N\times K}
3:  for n=0n=0 to NN step batch_size do
4:   for k=0k=0 to KK step chunk_k do
5:    Δ←X[n:n′]−μ[k:k′]\Delta\leftarrow X[n:n^{\prime}]-\mu[k:k^{\prime}]
6:    mahal←∑dΔd2/σk,d2\text{mahal}\leftarrow\sum_{d}\Delta_{d}^{2}/\sigma_{k,d}^{2}
7:    log⁡pn,k←−D2​log⁡(2​π)−12​∑dlog⁡σk,d2−12​mahal\log p_{n,k}\leftarrow-\frac{D}{2}\log(2\pi)-\frac{1}{2}\sum_{d}\log\sigma_{k,d}^{2}-\frac{1}{2}\text{mahal}
8:   end for
9:   Q[n:n′]←softmax(logπ+logp)Q[n:n^{\prime}]\leftarrow\text{softmax}(\log\pi+\log p)
10:  end for

Appendix F Periodically Switched EMA Decay

Teacher decay alternates between faster and slower tracking intervals. The appendix explains the corresponding timescales and interaction with mixture updates. The schedule is a heuristic whose independent benefit was not isolated by a controlled comparison.

Appendix G Effective Rank and Adaptive Layer Selection

Effective rank is computed from normalized singular values of centered layer features, then smoothed before selecting the mixture-input layer. It measures feature-spectrum spread, not direct semantic accuracy.

Appendix H Probing Protocol

Linear and nonlinear probes are trained on frozen features using the stated splits, scaling and labels. Different readout families can reveal different recoverability of phoneme information. These probes use supervised downstream labels even though encoder pretraining is self-supervised.

Appendix I Augmentation Details

The online encoder can receive denoising augmentation while target features come from clean audio. The final second-phase recipe turns augmentation off.

I.1 Energy-Based SNR Mixing

Energy-based scaling sets a target signal-to-noise ratio before mixing clean audio with noise. The same noise amplitude would otherwise affect quiet and loud utterances differently.

Equations 8–10
Ex\displaystyle E_{x} =1|x|​∑ixi2,En=1|n|​∑ini2,\displaystyle=\frac{1}{|x|}\sum_{i}x_{i}^{2},\quad E_{n}=\frac{1}{|n|}\sum_{i}n_{i}^{2}, (8)
α\displaystyle\alpha =Ex10SNR/10⋅En,\displaystyle=\sqrt{\frac{E_{x}}{10^{\text{SNR}/10}\cdot E_{n}}}, (9)
xaug\displaystyle x_{\text{aug}} =x+α⋅n\displaystyle=x+\alpha\cdot n (10)

I.2 Utterance Mixing

Another utterance segment is mixed at a sampled energy ratio and overlap position. This creates an acoustic denoising task rather than a new transcript target.

Equation 11
𝐱mix[t1:t2]=𝐱1[t1:t2]+β⋅𝐱2[s1:s2],\mathbf{x}_{\text{mix}}[t_{1}:t_{2}]=\mathbf{x}_{1}[t_{1}:t_{2}]+\beta\cdot\mathbf{x}_{2}[s_{1}:s_{2}], (11)

Appendix J SUPERB Evaluation Protocol

SUPERB trains small task-specific heads on a frozen encoder. Greedy decoding is the principal ASR comparison; optional language-model rescoring is reported separately. Matching this protocol is necessary for interpreting the baseline table.

Concept demonstration

Open image ↗