Literature
Display
← All papers

UltraBERT: Beyond the Limit of BERT Pre-Training

Jungwoo Park · Taewhoo Lee · Jaewoo Kang

Preprint posted · arXiv (preprint) ↗

Language models12 min text · CondensedOriginal paper ↗
Contents

Overview

Original figure

At a glance

UltraBERT revisits bidirectional-encoder pretraining after training long enough for downstream performance to saturate. In the tested settings, low masking rates favor classification, high rates favor retrieval, and logit distillation helps primarily at low masking rates.

The resulting 150M/380M encoders mix 5% and 50% masked sequences with separate language-model heads, distilling only the 5% examples. Both support 32K context. The 380M model reaches 91.0 GLUE and 47.7 BEIR under the paper's task-specific fine-tuning protocols.

These are broad improvements across the selected benchmarks, with individual exceptions. The observed saturation is empirical for the studied architectures, data, objectives, and evaluations; it is not a universal limit on encoders.

1 Introduction

Efficient bidirectional encoders remain useful for retrieval, classification, and reward modeling [17] [18]. Recent models borrow architectural improvements from causal language models, but masking and distillation choices often inherit conclusions from much smaller training budgets.

This study trains models up to 8.8T tokens, separates convergence speed from later downstream quality, and finds opposing masking preferences for classification and retrieval. Separate prediction heads and selective distillation reconcile much of that trade-off. A shared pretrained backbone is then fine-tuned separately for downstream tasks; the results do not describe one unchanged checkpoint handling all tasks.

Original figure
Figure 1: Classification and retrieval performance of encoders in two size groups, (a) base (≤\leq200M) and (b) medium (≤\leq500M). The classification average is the mean of GLUE, SQuAD 2.0, MultiCoNER v2, Few-NERD, and RewardBench, and the retrieval average is the mean of BEIR, RAR-b (averaged over its Code and Math subsets), and MLDR. All scores are taken from Table 3.

2 Related Work

Modernized encoders

Modern encoders add rotary positions, gated feed-forward layers, mixed local/global attention, and larger token budgets to the BERT family [69] [87] [90] [91]. This work focuses on pretraining objectives rather than another architectural redesign.

Masking ratio and pre-training objectives

Previous masking studies often use 10B–100B tokens, mixing faster convergence with final quality [97] [100] [101]. RTD improves some classification results but commonly harms retrieval [67] [68]. The paper compares objectives after longer training and across corruption rates.

Encoders for retrieval

Retrieval bi-encoders usually start with MLM and then receive contrastive fine-tuning [70] [71] [74]. Pretraining determines which representation properties this later training can exploit.

Knowledge distillation for language models

Distillation can match token logits, hidden states, or attention structure [77] [78] [79]. Cascade distillation initializes smaller models by pruning a parent, then transfers knowledge from a stronger teacher [83] [85]. Its interaction with masking is the question tested here.

3 Exploring the Pre-Training Recipe for Stronger Encoders

Controlled studies use 150M, 380M, and 1.3B encoders with shared tokenizer/data mixture, rotary positions, and alternating local/global attention. Classification evaluates fine-tuned GLUE and SQuAD 2.0; retrieval fine-tunes on MS MARCO and evaluates BEIR nDCG@10.

3.1 Training Token Scaling and Downstream Saturation

At 30% masking, 150M and 1.3B encoders improve pretraining loss through 8.8T tokens while fine-tuned classification largely plateaus around 1–2T. BEIR shows little improvement beyond the first 275B-token checkpoint.

On six multiple-choice benchmarks, base encoders and the SmolLM2 decoder both improve in five-shot evaluation. After instruction tuning, the encoder plateaus while the decoder continues improving. Thus lower MLM loss does not reliably predict these fine-tuned outcomes. Different data, objectives, architectures, or evaluations could yield different scaling behavior.

Original figure
Figure 2: Scaling the training token budget. Pre-training loss and downstream performance of 150M and 1.3B encoders. In (a), solid lines are the training loss (left axis) and dashed lines are the validation loss on WikiText-103 (Merity et al., 2017) (right axis). (b) GLUE average score, (c) SQuAD 2.0 F1, and (d) BEIR average nDCG@10 are obtained.
Original figure
Figure 3: Accuracy before and after instruction tuning. (a) Pre-training loss of the 1.3B encoder. (b) Its MCQA average accuracy as a 5-shot base model (solid) and after instruction tuning (dash-dotted). (c) The same comparison for a 1.7B decoder; the horizontal line is the random baseline.

3.2 Masking Ratios and Classification Tasks

Across all three model sizes, lower masking rates yield stronger classification once slower convergence is accommodated. Training budgets are 1T tokens at 30%/50%, 2T at 15%, and 5T at 3%/6%. This reverses conclusions favoring higher rates for large models under smaller budgets.

Original figure
Figure 4: (a) GLUE average score and (b) SQuAD 2.0 F1 against the pretraining masking ratio, for encoders of 150M, 380M and 1.3B scales. Both classification tasks degrade consistently as the masking ratio increases.

Impact of contextual corruption

A controlled 150M comparison adjusts the RTD generator's masking rate to approximately match actual token replacement to MLM corruption. Its classification advantage peaks near 15% corruption and shrinks near 3% and 50%. CoLA is excluded because its plausibility task particularly aligns with RTD.

The proposed explanation is that plausible generator replacements preserve more context than MASK tokens. At low corruption that distinction matters less; at high corruption the generator lacks context. This interpretation is supported by the comparison, rather than a formal causal identification.

Original figure
Figure 5: MLM and RTD at matched corruption levels, for 150M encoders. (a) Average of GLUE without CoLA and SQuAD 2.0 F1 score. (b) The gap between interpolated curves. The dashed vertical line marks 15%.

3.3 Masking Ratios and Retrieval Tasks

Retrieval favors 50% masking at every tested size. Higher masking also correlates with greater effective rank of pooled embeddings, but the study does not establish that rank causes the improvement.

Adding KoLeo regularization at 6% masking raises rank and improves BEIR, yet harms GLUE and remains below 50%-masking retrieval. Artificially spreading embeddings therefore does not eliminate the classification/retrieval trade-off.

Original figure
Figure 6: Masking ratios and retrieval. (a) BEIR average nDCG@10 against the masking ratio for 150M, 380M, and 1.3B encoders. (b) Effective rank of the 150M model against the masking ratio. (c) GLUE average score and (d) BEIR average nDCG@10 of the 150M encoder pre-trained at 6% masking with (dashed) and without (solid) the KoLeo regularizer.

3.4 Mixing Masking Ratios with LM Head Decoupling

Low- and high-mask models have similar input embedding geometry but different prediction-head geometry and spectra. The authors hypothesize that a shared head sends conflicting gradients and test separate heads for 5%/50% examples, mixed 4:1 because low masking converges more slowly.

For 150M, separate heads reach 89.4 GLUE/45.1 BEIR, close to the best single-rate results of 89.5/45.0. Sharing the head yields 88.9/44.7. Uniformly sampling 5%–50% behaves similarly to its average rate; decaying high to low masking does not match the best combination. These ablations support decoupling without proving a complete gradient-interference mechanism.

Refer to caption
Figure 7: Geometry of input embeddings and LM heads. First three principal components of the input embedding matrix (a, b) and the LM head (c, d) of 150M encoders pre-trained at 6% and 50% masking. Each point is one vocabulary entry, colored by its frequency percentile in the corpus.
Original figure
Figure 8: Relative singular values of (a) the input embedding matrix and (b) the LM head, for 150M encoders pre-trained at 6% (solid) and 50% (dashed) masking. Each spectrum is divided by its own leading singular value.
Masking Ratio GLUE BEIR
Single Ratio
6% 89.5 43.5
15% 88.9 43.8
30% 88.3 44.2
50% 87.5 45.0
Decaying Ratio
50%→6%50\%\rightarrow 6\% 88.8 43.9
50%→15%50\%\rightarrow 15\% 88.6 44.1
Mixing Ratios
𝒰⁡(5%,50%)\mathcal{U}(5\%,50\%) 88.0 44.2
{5%,50%}\{5\%,50\%\} 88.9 44.7
+ Decoupling 89.4 45.1
Table 1: Comparison of masking ratio mixing for the 150M encoder. Decoupling denotes separate LM heads for each masking ratio.
Refer to caption
Figure 9: LM head geometry under masking ratio mixing. First three principal components of the LM head, with each point a vocabulary entry colored by its frequency percentile. (a) An encoder pre-trained at a single 30% masking ratio, (b) the shared head of 𝒰⁡(5%,50%)\mathcal{U}(5\%,50\%), and (c, d) the decoupled heads of {5%,50%}\{5\%,50\%\} for 5% and 50% masking, respectively.

3.5 Revisiting Pruning and Distillation

The study prunes 1.3B models to 380M and distills token logits for 500B tokens. At 15% masking, GLUE rises from 90.2 to 90.8 while BEIR remains unchanged. At 50%, BEIR instead drops from 47.2 to 46.5 relative to training from scratch.

This motivates distilling only low-mask sequences. The proposed explanation of weak high-corruption teacher signals remains a hypothesis. Attention-matching distillation is excluded because quadratic sequence-length cost conflicts with the long-context training design.

Model GLUE BEIR
15% Masking
1.3B 91.8 47.4
380M 90.2 46.0
380M (Distilled) 90.8 46.0
50% Masking
1.3B 90.7 48.3
380M 89.6 47.2
380M (Distilled) 89.6 46.5
Table 2: Logit-based distillation from 1.3B to 380M at 15% and 50% ratios.

4 UltraBERT: Beyond the Limit of BERT Pre-Training

UltraBERT prunes a 1.3B parent to 380M and then to 150M; both students distill from the original 1.3B teacher. Batches mix 5%/50% masking at 4:1 with separate heads. Low-mask examples use teacher–student KL loss; high-mask examples use ordinary MLM.

Each student trains for 2T tokens, extending to 32K context in the last 100B. These token counts exclude the cost of developing/training the teacher and preceding cascade stages.

Original figure
Figure 10: Overview of UltraBERT. Starting from the 1.3B encoder as a teacher, we prune it to 380M and then further to 150M. Each batch mixes 5% and 50% masked sequences at a 4:1 ratio, and each ratio has its own LM head. The 5% head is trained with the KL distillation loss from the 1.3B teacher, while the 50% head is trained with the standard MLM loss.

4.1 Evaluation Setup

Classification covers GLUE, SQuAD 2.0, English MultiCoNER v2, supervised Few-NERD, and RewardBench. Each model/task receives a fine-tuning grid search. Retrieval uses MS MARCO fine-tuning, BEIR-based checkpoint selection, then code/math RAR-b evaluation; further English MLDR training supports long-document evaluation.

Baselines include BERT, RoBERTa, MosaicBERT, NomicBERT, GTE-en-MLM, ModernBERT, Ettin, NeoBERT, and LFM2.5-Encoder. Some classification numbers come from their papers, while remaining results use the authors' implementations.

4.2 Results

Among compared models in each size group, UltraBERT leads nearly all aggregate benchmarks. Exceptions are base-size MultiCoNER v2 and medium-size MLDR. Both sizes lead LongEmbed; code/math retrieval gains over the runner-up are 7.3/8.3 points at base size and 8.4/7.7 at medium size.

The headline 380M scores are 91.0 GLUE and 47.7 BEIR. Task-specific fine-tuning, BEIR-based selection, heterogeneous baseline score sources, and differing context limits matter when interpreting these comparisons. The claim is broad benchmark performance, not dominance on every individual task.

Classification Retrieval
Model # Params GLUE SQuAD 2.0 Multi- CoNER v2 Few- NERD Reward Bench BEIR RAR-b Code RAR-b Math MLDR Long Embed
Base (≤\leq200M)
BERT 110M 84.7 76.3 53.8 66.6 59.4 36.7 6.6 27.5 29.0 29.3
RoBERTa 125M 86.4 83.7 56.5 67.1 63.2 39.6 11.7 33.4 28.7 33.8
MosaicBERT 137M 85.4 79.4 57.8 65.7 62.5 41.4 12.7 30.9 29.6 40.5
NomicBERT 137M 84.0 82.7 60.1 66.9 64.1 42.0 13.8 30.2 29.9 45.0
GTE-en-MLM 137M 85.6 82.7 59.2 66.4 63.4 42.4 12.0 33.6 37.1 58.5
ModernBERT 149M 88.4 86.7 57.4 66.7 68.0 42.6 29.6 46.1 37.7 63.2
Ettin Encoder 149M 88.9 87.4 57.4 66.5 70.3 42.4 31.8 47.1 35.9 60.1
UltraBERT 152M 89.5 89.2 57.9 67.4 73.0 45.2 39.1 55.4 43.5 72.0
Medium (≤\leq500M)
BERT 334M 85.2 81.8 57.7 67.8 62.6 38.6 12.2 33.0 29.8 30.0
RoBERTa 355M 88.9 89.4 62.3 68.1 68.5 42.8 20.3 37.8 29.5 32.2
GTE-en-MLM 434M 87.6 87.2 63.0 67.1 62.2 39.4 13.5 32.0 40.8 57.0
NeoBERT 222M 89.0 86.6 57.4 66.4 57.6 42.9 17.3 33.4 21.7 32.4
LFM2.5-Encoder 230M 88.2 86.1 59.6 67.6 67.5 46.7 30.9 50.2 43.7 70.0
LFM2.5-Encoder 355M 89.2 87.2 62.7 67.9 68.5 47.3 38.6 50.2 44.5 71.7
ModernBERT 395M 90.4 89.8 60.9 67.6 72.8 45.2 34.0 46.1 39.9 63.5
Ettin Encoder 395M 90.8 90.4 62.3 67.8 76.7 45.9 38.5 54.0 40.9 64.3
UltraBERT 380M 91.0 90.9 63.5 68.3 78.8 47.7 47.0 61.7 44.1 74.9
Table 3: Comparison with existing encoders on classification and retrieval benchmarks. Parameter counts exclude the LM head. Best scores in each group are in bold and second best are underlined.

5 Conclusion

The experiments identify pretraining choices that jointly improve classification and retrieval: mixed masking with separate heads and low-mask-only distillation. The evidence supports this recipe at the tested sizes and budgets; it does not establish a universal token-scaling ceiling or remove the cost of teacher training.

AI use statement

The authors disclose generative-AI help formalizing individual proof steps in Appendix D, editing prose, converting logs into tables, and finding/summarizing literature. They report author-led conceptual design, experiments, implementation, and interpretation, and verification of assisted proofs, numbers, and prior-work claims.

Ethics statement

Training uses publicly released datasets under their respective licenses, without new human-subject data collection. The authors note that web-scale biases can carry into the encoders and require use-specific evaluation.

Reproducibility statement

The paper supplies training/pruning details, dataset mixture, fine-tuning grids, a geometric derivation, and full benchmark results. It announces release of evaluation code, weights, and the pretraining mixture.

Acknowledgments

The authors acknowledge Korean research and healthcare-technology funding and Google TPU Research Cloud support for most pretraining.

Appendix A Training Setup

A.1 Architecture and Optimization

All models have 24 layers with global attention every third layer and local attention otherwise. Widths are 768/1,024/2,048, with SwiGLU intermediate widths 1,024/3,072/5,632. Prediction heads are untied from input embeddings; the SmolLM2 tokenizer has 49,152 entries.

AdamW uses β=(0.9,0.95), ε=10⁻⁸, weight decay 0.1, and gradient clipping 1.0. A 10B-token warmup precedes a stable phase and final 10% decay; even intermediate evaluated checkpoints receive a branched decay to zero. Peak learning rate is 3×10⁻⁴ for controlled studies and 10⁻⁴ for distillation/final training.

The 4M-token batch initially contains 2,048 sequences of length 2,048. Final 100B-token context extension raises global RoPE base from 640K to 2.56M. Training uses 256-chip TPU slices with data, sharded, and later tensor parallelism.

A.2 Pruning and Distillation

Cascade pruning retains all 24 layers and removes width using activation-based importance from 2,000 calibration documents. Hidden-channel scores aggregate across layers; attention-head and FFN-neuron scores remain layer-specific.

When head dimension halves, query/key pruning preserves every other rotary-frequency pair and rescales queries. Shared random orthogonal projections reduce value/output dimensions while preserving their product in expectation. Both pruned models then distill directly from the original 1.3B teacher on low-mask examples.

A.3 Dataset Mixture

The mixture allocates 66.9% to web, 14.0% code, 7.1% academic, 5.8% math, 3.7% reference, and 2.5% instruction data. Table 4 retains per-dataset unique and training-token counts.

Instruction examples help the untuned encoder follow multiple-choice formats. Preliminary changing-mixture schedules barely alter fine-tuned outcomes, so final training keeps one mixture. The suggested distinction between token-distribution changes and intrinsic linguistic knowledge is an interpretation, not an established separation.

Category Dataset Unique (B) Epochs Trained (B) Ratio (%)
Web (66.9%) DCLM (Li et al., 2024) 756.4 1.17 881.7 44.1
FineWeb-Edu (Penedo et al., 2024) 195.4 2.33 455.5 22.8
Code (14.0%) Stack-Edu (Ben Allal et al., 2025) 120.3 2.33 280.5 14.0
Academic (7.1%) peS2o (Soldaini and Lo, 2023) 60.7 2.33 141.6 7.1
Math (5.8%) MegaMath (Zhou et al., 2025) 20.9 2.80 58.4 2.9
FineMath (Ben Allal et al., 2025) 10.9 2.80 30.5 1.5
InfiWebMath (Ben Allal et al., 2025) 9.6 2.80 26.9 1.3
Reference (3.7%) Cosmopedia v2 (Ben Allal et al., 2024) 28.1 2.33 65.6 3.3
Wikipedia (Soldaini et al., 2024) 3.9 2.33 9.2 0.5
Instruction (2.5%) FLAN (Longpre et al., 2023) 19.0 2.33 44.4 2.2
StackExchange (Weber et al., 2024) 1.5 2.33 3.4 0.2
Natural Reasoning (Yuan et al., 2025) 1.0 2.33 2.4 0.1
Total 1,227.8 1.63 2,000.0 100.0
Table 4: Pre-training data mixture. Unique denotes the number of tokens in one epoch of each dataset, and Trained the number of tokens seen within the 2T-token budget of UltraBERT, both counted with our tokenizer. Ratios in parentheses are per category.

Appendix B Evaluation Protocols

Evaluation combines published baseline scores where available with new fine-tuning runs. Protocol choices and selection criteria are part of the reported results.

B.1 Classification Tasks

Classification fine-tuning uses AdamW, 5% warmup, linear decay, mixed precision, and task/model-specific grid search. Six learning rates from 5×10⁻⁶ to 8×10⁻⁵ combine with the length, batch, and epoch grid in Table 5. Development performance selects checkpoints/configurations; labeled test scores are reported for entity-recognition tasks.

GLUE

GLUE reports eight development tasks excluding WNLI, with 128-token inputs. Metrics are Matthews correlation for CoLA, Pearson correlation for STS-B, matched MNLI accuracy, and accuracy elsewhere. RTE/MRPC/STS-B initialize from the best MNLI checkpoint. Small-task results take the median of four seeds after selecting each seed's best grid score.

SQuAD 2.0

SQuAD 2.0 uses 384-token question/context windows with stride 128. Batch sizes 16/32/64 and one to three epochs are searched; the selected development F1 is reported.

Named Entity Recognition

English MultiCoNER v2 uses 33 entity types and three seeds for development-based selection; supervised Few-NERD uses 66 types. Leading spaces align pretokenized words with byte-level BPE. Test metrics are entity-type macro-F1 and entity-level micro-F1 respectively.

RewardBench

Reward models train with Bradley–Terry loss on 135,981 single-turn preference pairs, holding out 6,800 for selection. Plain USER:/ASSISTANT: templates are truncated to 1,024 tokens. The four reported section scores use official subset weighting and are averaged.

Task Max Length Batch Size Epochs Metric
MNLI, QQP, QNLI, SST-2 128 64 {1, 2, 3} Accuracy
CoLA, STS-B, MRPC, RTE 128 {16, 32, 64} {2, 5, 10} MCC / Pearson / Acc.
SQuAD 2.0 384 {16, 32, 64} {1, 2, 3} F1
MultiCoNER v2 128 16 {2, 5, 10} Macro-F1
Few-NERD 128 64 {1, 2, 3} Micro-F1
RewardBench 1024 64 {1, 2, 3} Section average
Table 5: Fine-tuning grids. The learning rate is swept over the 6 values in Section B.1 for every task.

B.2 Retrieval Tasks

Retrieval fine-tunes mean-pooled single-vector encoders using cached multiple-negatives ranking loss, mixed precision, and eight GPUs with 512 examples per device. Evaluation uses the MTEB library.

BEIR and RAR-b

MS MARCO training uses 1.25M hard-negative triplets, one epoch, and 256-token inputs. Ten learning rates are searched, and the checkpoint with the highest BEIR average is selected. Thus BEIR is a selection criterion as well as a reported benchmark.

The score averages nDCG@10 over 15 datasets, using MS MARCO development data and averaging CQADupstack's 12 forums. The same selected checkpoint transfers to RAR-b code/math retrieval with 301K/389K candidates, providing a separate domain-transfer evaluation.

MLDR and LongEmbed

The BEIR-selected model receives further training on 80K English MLDR triplets, with training context capped at 16K and evaluation at 32K. Each model remains bounded by its supported context length.

LongEmbed averages four real-task nDCG@10 scores and two synthetic recall@1 scores across lengths 256–32K. Different native context limits therefore directly affect the comparison.

B.3 Zero-Shot and Few-Shot MCQA

Six benchmarks compare encoder and decoder multiple-choice performance: ARC, BoolQ, CommonsenseQA, MMLU, OpenBookQA, and SIQA. Public test sets are used where available, validation otherwise. MMLU averages 57 subjects; ARC averages Easy/Challenge, then all six benchmark scores are averaged.

Prompting and Scoring

Prompts use Question: with options and Answer:, prepending Context: when supplied. Five-shot examples are the first five from the designated training/development split. The encoder reads option-letter logits at one appended MASK; the decoder reads next-token logits.

BoolQ sums Yes/yes/True/true versus No/no/False/false probabilities. Both models use the same prompt and left truncation to 2,048 tokens.

Instruction Tuning

Base models use five-shot evaluation; instruction-tuned models use zero-shot. Filtering 545K FLAN examples to answers of at most five tokens leaves 480K, with 1% held out. Encoders append one MASK per answer token; decoders use causal answer-token loss. Both train three epochs at learning rate 5×10⁻⁵, batch 128, and 256-token maximum length.

Appendix C Additional Analysis on Replaced Token Detection

C.1 CoLA as a Confounder

After 100B tokens of continued RTD training from a 50%-masking model, CoLA jumps 9.9 points while the other GLUE tasks move only 0.5 on average. The authors attribute the disproportionate gain to alignment between token plausibility and sentence acceptability, motivating CoLA's exclusion from the controlled objective comparison.

Model GLUE w/o CoLA CoLA
50% MLM 91.2 61.6
+ RTD 100B 91.7 71.5
Table 6: Effect of continued RTD training. Starting from the 150M pre-trained with 50% masking, we continue training with RTD for 100B tokens.

C.2 Comparison with RTD-Trained Encoders

RTD-trained DeBERTaV3 is omitted from the main comparison but evaluated here. Its base/large classification averages are 74.2/78.1 versus UltraBERT's 75.4/78.5; retrieval averages are 20.7/25.8 versus 45.3/48.7. These averages use the figure's benchmark grouping and differ from individual GLUE/BEIR scores.

Appendix D Why KoLeo Increases Effective Rank

The derivation connects entropy pressure, approximate isotropy, and a flatter representation spectrum. It explains a tendency of KoLeo regularization rather than proving improved retrieval or guaranteed finite-training convergence.

Setup

For normalized vectors, let Σ be the population second moment and U the uniform sphere distribution. KoLeo approximates negative entropy. If KL(P∥U) is small, bounded quadratic test functions plus Pinsker's inequality imply ||Σ−I/d||₂≤√(KL/2).

The spectral theorem identifies this operator norm with the largest eigenvalue deviation from 1/d. Thus approaching uniformity flattens population eigenvalues. When empirical second moments concentrate, singular values flatten too, raising effective rank toward its maximum.

The chain assumes that the entropy estimator's optimization meaningfully approaches the population uniform distribution and that sampling supports concentration. It does not imply every optimization step increases rank, nor that maximal rank guarantees useful retrieval. All displayed derivations are retained below.

Equations
∥Σ−1dId∥2≤KL(P∥U)/2.\bigl\lVert\Sigma-\tfrac{1}{d}I_{d}\bigr\rVert_{2}\;\leq\;\sqrt{\mathrm{KL}(P\,\|\,U)/2}.
𝔼P​[gv]=𝔼P​[v⊤​z​z⊤​v]=v⊤​Σ​v,𝔼U​[gv]=v⊤​(1d​Id)​v=1d,\mathbb{E}_{P}[g_{v}]=\mathbb{E}_{P}\bigl[v^{\top}zz^{\top}v\bigr]=v^{\top}\Sigma v,\qquad\mathbb{E}_{U}[g_{v}]=v^{\top}\bigl(\tfrac{1}{d}I_{d}\bigr)v=\tfrac{1}{d},
|v⊤​(Σ−1d​Id)​v|=|𝔼P​[gv]−𝔼U​[gv]|.\bigl|v^{\top}\bigl(\Sigma-\tfrac{1}{d}I_{d}\bigr)v\bigr|=\bigl|\mathbb{E}_{P}[g_{v}]-\mathbb{E}_{U}[g_{v}]\bigr|.
|𝔼P​[gv]−𝔼U​[gv]|=|∫g~v​d​(P−U)|≤12​∥P−U∥1=TV⁡(P,U),\bigl|\mathbb{E}_{P}[g_{v}]-\mathbb{E}_{U}[g_{v}]\bigr|=\Bigl|\int\tilde{g}_{v}\,\mathrm{d}(P-U)\Bigr|\leq\tfrac{1}{2}\,\lVert P-U\rVert_{1}=\mathrm{TV}(P,U),
∥Σ−1dId∥2=maxk|λk−1d|.\bigl\lVert\Sigma-\tfrac{1}{d}I_{d}\bigr\rVert_{2}\;=\;\max_{k}\bigl|\lambda_{k}-\tfrac{1}{d}\bigr|.
(Σ−1d​Id)​uk=λk​uk−1d​uk=(λk−1d)​uk,\bigl(\Sigma-\tfrac{1}{d}I_{d}\bigr)u_{k}=\lambda_{k}u_{k}-\tfrac{1}{d}u_{k}=\bigl(\lambda_{k}-\tfrac{1}{d}\bigr)u_{k},
∥(Σ−1dId)v∥2=∥∑kck(λk−1d)uk∥2=∑kck2(λk−1d)2≤maxk(λk−1d)2,\Bigl\lVert\bigl(\Sigma-\tfrac{1}{d}I_{d}\bigr)v\Bigr\rVert^{2}=\Bigl\lVert\sum_{k}c_{k}\bigl(\lambda_{k}-\tfrac{1}{d}\bigr)u_{k}\Bigr\rVert^{2}=\sum_{k}c_{k}^{2}\bigl(\lambda_{k}-\tfrac{1}{d}\bigr)^{2}\leq\max_{k}\bigl(\lambda_{k}-\tfrac{1}{d}\bigr)^{2},
∥Σ−1dId∥2=max∥v∥=1∥(Σ−1dId)v∥=maxk|λk−1d|.\bigl\lVert\Sigma-\tfrac{1}{d}I_{d}\bigr\rVert_{2}=\max_{\lVert v\rVert=1}\Bigl\lVert\bigl(\Sigma-\tfrac{1}{d}I_{d}\bigr)v\Bigr\rVert=\max_{k}\bigl|\lambda_{k}-\tfrac{1}{d}\bigr|.
maxk⁡|λk−1d|≤KL(P∥U)/2.\max_{k}\bigl|\lambda_{k}-\tfrac{1}{d}\bigr|\;\leq\;\sqrt{\mathrm{KL}(P\,\|\,U)/2}.

Appendix E Detailed Evaluation Results

E.1 Reported Baseline Scores

Most baseline GLUE scores and selected SQuAD results come from prior papers; other classification and retrieval results are newly evaluated. Reproducing Ettin GLUE within 0.2 points supports the protocol's comparability, while some cross-source variation remains.

Model # Params MNLI QNLI QQP RTE SST-2 MRPC CoLA STS-B Avg.
Ettin Encoder 149M 89.2 94.0 92.4 87.7 95.8 92.6 66.9 92.2 88.9
(Our evaluation) 88.9 93.8 92.1 88.4 95.4 92.2 67.0 92.5 88.8
Ettin Encoder 395M 91.3 95.2 93.0 92.8 96.7 93.6 71.3 92.7 90.8
(Our evaluation) 90.9 95.3 93.0 91.7 96.7 93.1 71.1 92.9 90.6
Table 7: GLUE scores of Ettin Encoder as reported by Weller et al. (2026) and as re-evaluated with our fine-tuning protocol.

E.2 Full GLUE Results

Per-task GLUE results show that aggregate leadership does not mean universal task leadership. Exceptions include QQP/CoLA at base size and QQP/SST-2/MRPC/CoLA at medium size. Most deficits are within one point, except CoLA.

E.3 Full RewardBench Results

UltraBERT leads three RewardBench sections at base size and Chat Hard/Safety at medium size. Ettin remains slightly ahead on medium Chat/Reasoning. The largest margin is 4.9 points on medium Chat Hard.

E.4 Full BEIR Results

Detailed BEIR results retain all 15 datasets and selected learning rates. UltraBERT leads 10 datasets at base size and eight at medium size; LFM2.5-Encoder-350M leads several scientific and question-duplication datasets.

E.5 Full LongEmbed Results

UltraBERT leads every base-size LongEmbed task and all but Needle/2WikiMultihopQA at medium size. Figure 11 reports 99% Passkey accuracy through 32K at both sizes, while shorter-context baselines degrade beyond their supported windows. These synthetic retrieval results demonstrate long-context access under this test, not unrestricted long-context understanding.

Refer to caption
Figure 11: Passkey retrieval accuracy on LongEmbed as a function of document length, for encoders in two size groups, (a) base (≤\leq200M parameters) and (b) medium (≤\leq500M parameters). Each cell is the recall@1 over the 50 queries at that length, from 256 to 32,768 tokens.
Model # Params MNLI QNLI QQP RTE SST-2 MRPC CoLA STS-B Avg.
Base (≤\leq200M)
BERT 110M 85.4 91.6 91.4 78.2 93.1 89.5 59.0 89.4 84.7
RoBERTa 125M 87.6 92.8 91.9 78.7 94.8 90.2 63.6 91.2 86.4
MosaicBERT 137M 85.6 91.4 92.0 83.0 93.5 89.0 58.2 90.3 85.4
NomicBERT 137M 86.0 92.0 92.0 82.0 93.0 88.0 50.0 90.0 84.0
GTE-en-MLM 137M 86.7 91.9 88.8 84.8 93.4 92.1 57.0 90.2 85.6
ModernBERT 149M 89.1 93.9 92.1 87.4 96.0 92.2 65.1 91.8 88.4
Ettin Encoder 149M 89.2 94.0 92.4 87.7 95.8 92.6 66.9 92.2 88.9
UltraBERT 152M 90.0 94.5 92.2 91.0 96.3 93.6 65.2 93.1 89.5
Medium (≤\leq500M)
BERT 334M 86.3 92.8 90.9 83.8 93.3 87.8 56.2 90.6 85.2
RoBERTa 355M 90.2 94.7 92.2 86.6 96.4 90.9 68.0 92.4 88.9
GTE-en-MLM 434M 89.2 93.9 89.2 88.1 95.1 93.5 60.4 91.4 87.6
NeoBERT 222M 89.0 93.7 90.7 91.3 95.6 93.4 66.2 91.8 89.0
LFM2.5-Encoder 230M 88.4 93.7 92.4 86.6 94.6 91.9 65.5 92.4 88.2
LFM2.5-Encoder 355M 89.5 94.4 92.6 89.2 96.3 91.8 66.8 92.8 89.2
ModernBERT 395M 90.8 95.2 92.7 92.1 97.1 91.7 71.4 92.8 90.4
Ettin Encoder 395M 91.3 95.2 93.0 92.8 96.7 93.6 71.3 92.7 90.8
UltraBERT 380M 91.3 95.4 92.6 93.9 96.4 93.4 71.2 93.4 91.0
Table 8: Detailed GLUE results on the development set. Best scores in each group are in bold and second best are underlined.
Model # Params Chat Chat Hard Safety Reasoning Avg.
Base (≤\leq200M)
BERT 110M 74.0 52.9 58.6 51.9 59.4
RoBERTa 125M 77.7 55.7 58.6 60.8 63.2
MosaicBERT 137M 75.7 51.1 58.2 65.0 62.5
NomicBERT 137M 74.9 53.9 66.4 61.4 64.1
GTE-en-MLM 137M 77.7 55.7 66.5 53.7 63.4
ModernBERT 149M 77.7 59.2 60.7 74.4 68.0
Ettin Encoder 149M 80.7 61.2 63.1 76.2 70.3
UltraBERT 152M 85.8 63.4 66.4 76.5 73.0
Medium (≤\leq500M)
BERT 334M 72.1 52.6 60.9 64.9 62.6
RoBERTa 355M 80.4 62.3 65.3 66.0 68.5
GTE-en-MLM 434M 78.2 49.8 61.4 59.6 62.2
NeoBERT 222M 61.5 53.7 54.5 60.9 57.6
LFM2.5-Encoder 230M 82.1 59.2 61.9 66.7 67.5
LFM2.5-Encoder 355M 82.4 57.9 63.5 70.4 68.5
ModernBERT 395M 84.6 64.3 65.1 77.0 72.8
Ettin Encoder 395M 88.3 66.4 70.7 81.6 76.7
UltraBERT 380M 88.0 71.3 74.9 81.2 78.8
Table 9: Detailed RewardBench results. Best scores in each group are in bold and second best are underlined.
Model # Params LR NF Corpus SciFact TREC- COVID FiQA ArguAna Climate FEVER DBPedia FEVER Hotpot QA MS MARCO NQ Quora SciDocs Touche 2020 CQA Dupstack Avg.
Base (≤\leq200M)
BERT 110M 3e-4 22.7 49.3 54.7 21.7 40.9 18.9 24.0 62.6 42.0 34.1 37.0 82.9 9.7 22.9 27.1 36.7
RoBERTa 125M 3e-4 24.3 53.1 59.1 26.8 46.5 20.6 26.1 64.3 44.5 37.9 41.5 85.3 12.2 22.4 29.9 39.6
MosaicBERT 137M 3e-4 27.5 53.7 58.9 28.9 45.0 21.5 29.6 67.1 48.9 40.4 44.1 86.0 13.6 23.6 32.3 41.4
NomicBERT 137M 3e-4 26.7 55.5 67.8 26.1 44.6 22.3 31.7 67.1 50.4 39.8 46.2 85.1 13.6 21.9 30.6 42.0
GTE-en-MLM 137M 1e-4 29.3 56.2 56.2 31.1 50.0 25.5 30.5 68.4 51.0 38.7 43.1 86.1 15.0 20.5 34.0 42.4
ModernBERT 149M 5e-4 26.5 58.3 73.7 29.8 46.3 23.1 28.1 67.0 48.2 38.6 44.3 86.6 13.3 21.9 33.2 42.6
Ettin Encoder 149M 5e-4 25.4 58.8 73.9 30.0 48.3 22.7 26.7 64.4 47.5 38.1 43.5 86.7 13.3 23.1 33.8 42.4
UltraBERT 152M 5e-5 28.4 64.3 75.8 33.5 50.2 23.9 30.6 69.6 52.5 40.1 47.8 87.2 15.2 23.5 36.0 45.2
Medium (≤\leq500M)
BERT 334M 1e-4 24.6 52.5 55.2 25.1 46.3 21.4 23.1 65.7 44.5 32.8 39.0 84.3 13.0 20.7 30.5 38.6
RoBERTa 355M 1e-4 27.9 55.4 60.3 33.0 51.3 24.9 28.6 70.0 49.8 38.6 46.6 86.9 14.4 20.1 34.2 42.8
GTE-en-MLM 434M 1e-4 30.3 58.5 53.4 33.2 48.9 19.6 25.1 49.8 42.0 35.4 42.3 83.7 15.4 19.3 34.8 39.4
NeoBERT 222M 3e-4 30.0 56.3 71.3 28.7 47.8 19.9 31.6 67.5 50.7 40.1 44.3 86.2 15.0 22.4 31.9 42.9
LFM2.5-Encoder 230M 5e-4 30.6 66.1 78.8 33.0 52.3 26.1 32.2 69.2 55.0 40.8 50.0 87.3 16.2 25.9 37.4 46.7
LFM2.5-Encoder 355M 3e-4 32.1 68.5 80.3 34.9 53.9 25.7 32.8 69.8 56.5 40.3 50.4 87.6 16.8 21.9 38.4 47.3
ModernBERT 395M 3e-4 29.2 62.4 77.7 34.4 51.1 24.1 28.9 68.7 51.2 40.7 49.2 87.2 15.5 22.1 36.0 45.2
Ettin Encoder 395M 5e-4 29.6 65.2 79.4 33.3 50.8 24.2 31.7 67.1 51.2 41.6 49.5 87.3 15.6 25.1 36.5 45.9
UltraBERT 380M 3e-5 31.7 66.4 77.0 38.1 53.9 25.7 33.9 73.2 56.5 41.1 51.6 87.7 18.0 23.3 38.0 47.7
Table 10: Detailed BEIR results (nDCG@10). LR denotes the selected learning rate for MS MARCO fine-tuning. Best scores in each group are in bold and second best are underlined.
Model Length # Params Passkey Needle Narrative QA QMSum Summ ScreenFD 2Wiki MultihopQA Avg.
Base (≤\leq200M)
BERT 512 110M 21.5 19.2 18.8 18.1 54.2 43.8 29.3
RoBERTa 512 125M 37.3 21.0 20.5 19.9 58.6 45.6 33.8
MosaicBERT 512 137M 23.0 38.2 23.2 24.2 77.4 57.0 40.5
NomicBERT 2048 137M 54.2 30.8 22.4 25.4 77.5 59.5 45.0
GTE-en-MLM 8192 137M 60.2 50.2 43.1 33.6 91.8 71.7 58.5
ModernBERT 8192 149M 81.7 59.0 35.3 38.5 89.6 75.3 63.2
Ettin Encoder 8192 149M 80.2 50.8 38.3 33.7 90.1 67.6 60.1
UltraBERT 32768 152M 99.2 59.8 56.0 45.4 95.2 76.6 72.0
Medium (≤\leq500M)
BERT 512 334M 27.0 17.8 20.2 17.4 55.3 42.6 30.0
RoBERTa 512 355M 35.2 18.8 20.7 18.9 57.2 42.6 32.2
GTE-en-MLM 8192 434M 49.8 42.8 43.6 37.2 92.2 76.3 57.0
NeoBERT 4096 222M 16.0 14.3 19.5 23.2 70.7 50.5 32.4
LFM2.5-Encoder 8192 230M 88.8 78.2 47.3 37.5 89.2 79.0 70.0
LFM2.5-Encoder 8192 355M 92.5 72.8 52.4 41.2 93.3 78.0 71.7
ModernBERT 8192 395M 85.0 51.5 40.2 38.0 91.8 74.7 63.5
Ettin Encoder 8192 395M 85.0 56.5 42.3 37.7 91.3 72.8 64.3
UltraBERT 32768 380M 99.8 68.2 58.2 48.0 96.6 78.7 74.9
Table 11: Detailed LongEmbed results. Length denotes the maximum context length of each model. Passkey and Needle are scored by recall@1 averaged over context lengths, and the remaining tasks by nDCG@10. Best scores in each group are in bold and second best are underlined.
Concept demonstration

Open image ↗