Overview
At a glance
UltraBERT revisits bidirectional-encoder pretraining after training long enough for downstream performance to saturate. In the tested settings, low masking rates favor classification, high rates favor retrieval, and logit distillation helps primarily at low masking rates.
The resulting 150M/380M encoders mix 5% and 50% masked sequences with separate language-model heads, distilling only the 5% examples. Both support 32K context. The 380M model reaches 91.0 GLUE and 47.7 BEIR under the paper's task-specific fine-tuning protocols.
These are broad improvements across the selected benchmarks, with individual exceptions. The observed saturation is empirical for the studied architectures, data, objectives, and evaluations; it is not a universal limit on encoders.
1 Introduction
Efficient bidirectional encoders remain useful for retrieval, classification, and reward modeling [17] [18]. Recent models borrow architectural improvements from causal language models, but masking and distillation choices often inherit conclusions from much smaller training budgets.
This study trains models up to 8.8T tokens, separates convergence speed from later downstream quality, and finds opposing masking preferences for classification and retrieval. Separate prediction heads and selective distillation reconcile much of that trade-off. A shared pretrained backbone is then fine-tuned separately for downstream tasks; the results do not describe one unchanged checkpoint handling all tasks.
2 Related Work
Modernized encoders
Modern encoders add rotary positions, gated feed-forward layers, mixed local/global attention, and larger token budgets to the BERT family [69] [87] [90] [91]. This work focuses on pretraining objectives rather than another architectural redesign.
Masking ratio and pre-training objectives
Previous masking studies often use 10B–100B tokens, mixing faster convergence with final quality [97] [100] [101]. RTD improves some classification results but commonly harms retrieval [67] [68]. The paper compares objectives after longer training and across corruption rates.
Encoders for retrieval
Retrieval bi-encoders usually start with MLM and then receive contrastive fine-tuning [70] [71] [74]. Pretraining determines which representation properties this later training can exploit.
Knowledge distillation for language models
Distillation can match token logits, hidden states, or attention structure [77] [78] [79]. Cascade distillation initializes smaller models by pruning a parent, then transfers knowledge from a stronger teacher [83] [85]. Its interaction with masking is the question tested here.
3 Exploring the Pre-Training Recipe for Stronger Encoders
Controlled studies use 150M, 380M, and 1.3B encoders with shared tokenizer/data mixture, rotary positions, and alternating local/global attention. Classification evaluates fine-tuned GLUE and SQuAD 2.0; retrieval fine-tunes on MS MARCO and evaluates BEIR nDCG@10.
3.1 Training Token Scaling and Downstream Saturation
At 30% masking, 150M and 1.3B encoders improve pretraining loss through 8.8T tokens while fine-tuned classification largely plateaus around 1–2T. BEIR shows little improvement beyond the first 275B-token checkpoint.
On six multiple-choice benchmarks, base encoders and the SmolLM2 decoder both improve in five-shot evaluation. After instruction tuning, the encoder plateaus while the decoder continues improving. Thus lower MLM loss does not reliably predict these fine-tuned outcomes. Different data, objectives, architectures, or evaluations could yield different scaling behavior.
3.2 Masking Ratios and Classification Tasks
Across all three model sizes, lower masking rates yield stronger classification once slower convergence is accommodated. Training budgets are 1T tokens at 30%/50%, 2T at 15%, and 5T at 3%/6%. This reverses conclusions favoring higher rates for large models under smaller budgets.
Impact of contextual corruption
A controlled 150M comparison adjusts the RTD generator's masking rate to approximately match actual token replacement to MLM corruption. Its classification advantage peaks near 15% corruption and shrinks near 3% and 50%. CoLA is excluded because its plausibility task particularly aligns with RTD.
The proposed explanation is that plausible generator replacements preserve more context than MASK tokens. At low corruption that distinction matters less; at high corruption the generator lacks context. This interpretation is supported by the comparison, rather than a formal causal identification.
3.3 Masking Ratios and Retrieval Tasks
Retrieval favors 50% masking at every tested size. Higher masking also correlates with greater effective rank of pooled embeddings, but the study does not establish that rank causes the improvement.
Adding KoLeo regularization at 6% masking raises rank and improves BEIR, yet harms GLUE and remains below 50%-masking retrieval. Artificially spreading embeddings therefore does not eliminate the classification/retrieval trade-off.
3.4 Mixing Masking Ratios with LM Head Decoupling
Low- and high-mask models have similar input embedding geometry but different prediction-head geometry and spectra. The authors hypothesize that a shared head sends conflicting gradients and test separate heads for 5%/50% examples, mixed 4:1 because low masking converges more slowly.
For 150M, separate heads reach 89.4 GLUE/45.1 BEIR, close to the best single-rate results of 89.5/45.0. Sharing the head yields 88.9/44.7. Uniformly sampling 5%–50% behaves similarly to its average rate; decaying high to low masking does not match the best combination. These ablations support decoupling without proving a complete gradient-interference mechanism.
| Masking Ratio | GLUE | BEIR |
| Single Ratio | ||
| 6% | 89.5 | 43.5 |
| 15% | 88.9 | 43.8 |
| 30% | 88.3 | 44.2 |
| 50% | 87.5 | 45.0 |
| Decaying Ratio | ||
| 88.8 | 43.9 | |
| 88.6 | 44.1 | |
| Mixing Ratios | ||
| 88.0 | 44.2 | |
| 88.9 | 44.7 | |
| + Decoupling | 89.4 | 45.1 |
3.5 Revisiting Pruning and Distillation
The study prunes 1.3B models to 380M and distills token logits for 500B tokens. At 15% masking, GLUE rises from 90.2 to 90.8 while BEIR remains unchanged. At 50%, BEIR instead drops from 47.2 to 46.5 relative to training from scratch.
This motivates distilling only low-mask sequences. The proposed explanation of weak high-corruption teacher signals remains a hypothesis. Attention-matching distillation is excluded because quadratic sequence-length cost conflicts with the long-context training design.
4 UltraBERT: Beyond the Limit of BERT Pre-Training
UltraBERT prunes a 1.3B parent to 380M and then to 150M; both students distill from the original 1.3B teacher. Batches mix 5%/50% masking at 4:1 with separate heads. Low-mask examples use teacher–student KL loss; high-mask examples use ordinary MLM.
Each student trains for 2T tokens, extending to 32K context in the last 100B. These token counts exclude the cost of developing/training the teacher and preceding cascade stages.
4.1 Evaluation Setup
Classification covers GLUE, SQuAD 2.0, English MultiCoNER v2, supervised Few-NERD, and RewardBench. Each model/task receives a fine-tuning grid search. Retrieval uses MS MARCO fine-tuning, BEIR-based checkpoint selection, then code/math RAR-b evaluation; further English MLDR training supports long-document evaluation.
Baselines include BERT, RoBERTa, MosaicBERT, NomicBERT, GTE-en-MLM, ModernBERT, Ettin, NeoBERT, and LFM2.5-Encoder. Some classification numbers come from their papers, while remaining results use the authors' implementations.
4.2 Results
Among compared models in each size group, UltraBERT leads nearly all aggregate benchmarks. Exceptions are base-size MultiCoNER v2 and medium-size MLDR. Both sizes lead LongEmbed; code/math retrieval gains over the runner-up are 7.3/8.3 points at base size and 8.4/7.7 at medium size.
The headline 380M scores are 91.0 GLUE and 47.7 BEIR. Task-specific fine-tuning, BEIR-based selection, heterogeneous baseline score sources, and differing context limits matter when interpreting these comparisons. The claim is broad benchmark performance, not dominance on every individual task.
| Classification | Retrieval | ||||||||||
| Model | # Params | GLUE | SQuAD 2.0 | Multi- CoNER v2 | Few- NERD | Reward Bench | BEIR | RAR-b Code | RAR-b Math | MLDR | Long Embed |
| Base (200M) | |||||||||||
| BERT | 110M | 84.7 | 76.3 | 53.8 | 66.6 | 59.4 | 36.7 | 6.6 | 27.5 | 29.0 | 29.3 |
| RoBERTa | 125M | 86.4 | 83.7 | 56.5 | 67.1 | 63.2 | 39.6 | 11.7 | 33.4 | 28.7 | 33.8 |
| MosaicBERT | 137M | 85.4 | 79.4 | 57.8 | 65.7 | 62.5 | 41.4 | 12.7 | 30.9 | 29.6 | 40.5 |
| NomicBERT | 137M | 84.0 | 82.7 | 60.1 | 66.9 | 64.1 | 42.0 | 13.8 | 30.2 | 29.9 | 45.0 |
| GTE-en-MLM | 137M | 85.6 | 82.7 | 59.2 | 66.4 | 63.4 | 42.4 | 12.0 | 33.6 | 37.1 | 58.5 |
| ModernBERT | 149M | 88.4 | 86.7 | 57.4 | 66.7 | 68.0 | 42.6 | 29.6 | 46.1 | 37.7 | 63.2 |
| Ettin Encoder | 149M | 88.9 | 87.4 | 57.4 | 66.5 | 70.3 | 42.4 | 31.8 | 47.1 | 35.9 | 60.1 |
| UltraBERT | 152M | 89.5 | 89.2 | 57.9 | 67.4 | 73.0 | 45.2 | 39.1 | 55.4 | 43.5 | 72.0 |
| Medium (500M) | |||||||||||
| BERT | 334M | 85.2 | 81.8 | 57.7 | 67.8 | 62.6 | 38.6 | 12.2 | 33.0 | 29.8 | 30.0 |
| RoBERTa | 355M | 88.9 | 89.4 | 62.3 | 68.1 | 68.5 | 42.8 | 20.3 | 37.8 | 29.5 | 32.2 |
| GTE-en-MLM | 434M | 87.6 | 87.2 | 63.0 | 67.1 | 62.2 | 39.4 | 13.5 | 32.0 | 40.8 | 57.0 |
| NeoBERT | 222M | 89.0 | 86.6 | 57.4 | 66.4 | 57.6 | 42.9 | 17.3 | 33.4 | 21.7 | 32.4 |
| LFM2.5-Encoder | 230M | 88.2 | 86.1 | 59.6 | 67.6 | 67.5 | 46.7 | 30.9 | 50.2 | 43.7 | 70.0 |
| LFM2.5-Encoder | 355M | 89.2 | 87.2 | 62.7 | 67.9 | 68.5 | 47.3 | 38.6 | 50.2 | 44.5 | 71.7 |
| ModernBERT | 395M | 90.4 | 89.8 | 60.9 | 67.6 | 72.8 | 45.2 | 34.0 | 46.1 | 39.9 | 63.5 |
| Ettin Encoder | 395M | 90.8 | 90.4 | 62.3 | 67.8 | 76.7 | 45.9 | 38.5 | 54.0 | 40.9 | 64.3 |
| UltraBERT | 380M | 91.0 | 90.9 | 63.5 | 68.3 | 78.8 | 47.7 | 47.0 | 61.7 | 44.1 | 74.9 |
5 Conclusion
The experiments identify pretraining choices that jointly improve classification and retrieval: mixed masking with separate heads and low-mask-only distillation. The evidence supports this recipe at the tested sizes and budgets; it does not establish a universal token-scaling ceiling or remove the cost of teacher training.
AI use statement
The authors disclose generative-AI help formalizing individual proof steps in Appendix D, editing prose, converting logs into tables, and finding/summarizing literature. They report author-led conceptual design, experiments, implementation, and interpretation, and verification of assisted proofs, numbers, and prior-work claims.
Ethics statement
Training uses publicly released datasets under their respective licenses, without new human-subject data collection. The authors note that web-scale biases can carry into the encoders and require use-specific evaluation.
Reproducibility statement
The paper supplies training/pruning details, dataset mixture, fine-tuning grids, a geometric derivation, and full benchmark results. It announces release of evaluation code, weights, and the pretraining mixture.
Acknowledgments
The authors acknowledge Korean research and healthcare-technology funding and Google TPU Research Cloud support for most pretraining.
Appendix A Training Setup
A.1 Architecture and Optimization
All models have 24 layers with global attention every third layer and local attention otherwise. Widths are 768/1,024/2,048, with SwiGLU intermediate widths 1,024/3,072/5,632. Prediction heads are untied from input embeddings; the SmolLM2 tokenizer has 49,152 entries.
AdamW uses β=(0.9,0.95), ε=10⁻⁸, weight decay 0.1, and gradient clipping 1.0. A 10B-token warmup precedes a stable phase and final 10% decay; even intermediate evaluated checkpoints receive a branched decay to zero. Peak learning rate is 3×10⁻⁴ for controlled studies and 10⁻⁴ for distillation/final training.
The 4M-token batch initially contains 2,048 sequences of length 2,048. Final 100B-token context extension raises global RoPE base from 640K to 2.56M. Training uses 256-chip TPU slices with data, sharded, and later tensor parallelism.
A.2 Pruning and Distillation
Cascade pruning retains all 24 layers and removes width using activation-based importance from 2,000 calibration documents. Hidden-channel scores aggregate across layers; attention-head and FFN-neuron scores remain layer-specific.
When head dimension halves, query/key pruning preserves every other rotary-frequency pair and rescales queries. Shared random orthogonal projections reduce value/output dimensions while preserving their product in expectation. Both pruned models then distill directly from the original 1.3B teacher on low-mask examples.
A.3 Dataset Mixture
The mixture allocates 66.9% to web, 14.0% code, 7.1% academic, 5.8% math, 3.7% reference, and 2.5% instruction data. Table 4 retains per-dataset unique and training-token counts.
Instruction examples help the untuned encoder follow multiple-choice formats. Preliminary changing-mixture schedules barely alter fine-tuned outcomes, so final training keeps one mixture. The suggested distinction between token-distribution changes and intrinsic linguistic knowledge is an interpretation, not an established separation.
| Category | Dataset | Unique (B) | Epochs | Trained (B) | Ratio (%) |
| Web (66.9%) | DCLM (Li et al., 2024) | 756.4 | 1.17 | 881.7 | 44.1 |
| FineWeb-Edu (Penedo et al., 2024) | 195.4 | 2.33 | 455.5 | 22.8 | |
| Code (14.0%) | Stack-Edu (Ben Allal et al., 2025) | 120.3 | 2.33 | 280.5 | 14.0 |
| Academic (7.1%) | peS2o (Soldaini and Lo, 2023) | 60.7 | 2.33 | 141.6 | 7.1 |
| Math (5.8%) | MegaMath (Zhou et al., 2025) | 20.9 | 2.80 | 58.4 | 2.9 |
| FineMath (Ben Allal et al., 2025) | 10.9 | 2.80 | 30.5 | 1.5 | |
| InfiWebMath (Ben Allal et al., 2025) | 9.6 | 2.80 | 26.9 | 1.3 | |
| Reference (3.7%) | Cosmopedia v2 (Ben Allal et al., 2024) | 28.1 | 2.33 | 65.6 | 3.3 |
| Wikipedia (Soldaini et al., 2024) | 3.9 | 2.33 | 9.2 | 0.5 | |
| Instruction (2.5%) | FLAN (Longpre et al., 2023) | 19.0 | 2.33 | 44.4 | 2.2 |
| StackExchange (Weber et al., 2024) | 1.5 | 2.33 | 3.4 | 0.2 | |
| Natural Reasoning (Yuan et al., 2025) | 1.0 | 2.33 | 2.4 | 0.1 | |
| Total | 1,227.8 | 1.63 | 2,000.0 | 100.0 | |
Appendix B Evaluation Protocols
Evaluation combines published baseline scores where available with new fine-tuning runs. Protocol choices and selection criteria are part of the reported results.
B.1 Classification Tasks
Classification fine-tuning uses AdamW, 5% warmup, linear decay, mixed precision, and task/model-specific grid search. Six learning rates from 5×10⁻⁶ to 8×10⁻⁵ combine with the length, batch, and epoch grid in Table 5. Development performance selects checkpoints/configurations; labeled test scores are reported for entity-recognition tasks.
GLUE
GLUE reports eight development tasks excluding WNLI, with 128-token inputs. Metrics are Matthews correlation for CoLA, Pearson correlation for STS-B, matched MNLI accuracy, and accuracy elsewhere. RTE/MRPC/STS-B initialize from the best MNLI checkpoint. Small-task results take the median of four seeds after selecting each seed's best grid score.
SQuAD 2.0
SQuAD 2.0 uses 384-token question/context windows with stride 128. Batch sizes 16/32/64 and one to three epochs are searched; the selected development F1 is reported.
Named Entity Recognition
English MultiCoNER v2 uses 33 entity types and three seeds for development-based selection; supervised Few-NERD uses 66 types. Leading spaces align pretokenized words with byte-level BPE. Test metrics are entity-type macro-F1 and entity-level micro-F1 respectively.
RewardBench
Reward models train with Bradley–Terry loss on 135,981 single-turn preference pairs, holding out 6,800 for selection. Plain USER:/ASSISTANT: templates are truncated to 1,024 tokens. The four reported section scores use official subset weighting and are averaged.
| Task | Max Length | Batch Size | Epochs | Metric |
| MNLI, QQP, QNLI, SST-2 | 128 | 64 | {1, 2, 3} | Accuracy |
| CoLA, STS-B, MRPC, RTE | 128 | {16, 32, 64} | {2, 5, 10} | MCC / Pearson / Acc. |
| SQuAD 2.0 | 384 | {16, 32, 64} | {1, 2, 3} | F1 |
| MultiCoNER v2 | 128 | 16 | {2, 5, 10} | Macro-F1 |
| Few-NERD | 128 | 64 | {1, 2, 3} | Micro-F1 |
| RewardBench | 1024 | 64 | {1, 2, 3} | Section average |
B.2 Retrieval Tasks
Retrieval fine-tunes mean-pooled single-vector encoders using cached multiple-negatives ranking loss, mixed precision, and eight GPUs with 512 examples per device. Evaluation uses the MTEB library.
BEIR and RAR-b
MS MARCO training uses 1.25M hard-negative triplets, one epoch, and 256-token inputs. Ten learning rates are searched, and the checkpoint with the highest BEIR average is selected. Thus BEIR is a selection criterion as well as a reported benchmark.
The score averages nDCG@10 over 15 datasets, using MS MARCO development data and averaging CQADupstack's 12 forums. The same selected checkpoint transfers to RAR-b code/math retrieval with 301K/389K candidates, providing a separate domain-transfer evaluation.
MLDR and LongEmbed
The BEIR-selected model receives further training on 80K English MLDR triplets, with training context capped at 16K and evaluation at 32K. Each model remains bounded by its supported context length.
LongEmbed averages four real-task nDCG@10 scores and two synthetic recall@1 scores across lengths 256–32K. Different native context limits therefore directly affect the comparison.
B.3 Zero-Shot and Few-Shot MCQA
Six benchmarks compare encoder and decoder multiple-choice performance: ARC, BoolQ, CommonsenseQA, MMLU, OpenBookQA, and SIQA. Public test sets are used where available, validation otherwise. MMLU averages 57 subjects; ARC averages Easy/Challenge, then all six benchmark scores are averaged.
Prompting and Scoring
Prompts use Question: with options and Answer:, prepending Context: when supplied. Five-shot examples are the first five from the designated training/development split. The encoder reads option-letter logits at one appended MASK; the decoder reads next-token logits.
BoolQ sums Yes/yes/True/true versus No/no/False/false probabilities. Both models use the same prompt and left truncation to 2,048 tokens.
Instruction Tuning
Base models use five-shot evaluation; instruction-tuned models use zero-shot. Filtering 545K FLAN examples to answers of at most five tokens leaves 480K, with 1% held out. Encoders append one MASK per answer token; decoders use causal answer-token loss. Both train three epochs at learning rate 5×10⁻⁵, batch 128, and 256-token maximum length.
Appendix C Additional Analysis on Replaced Token Detection
C.1 CoLA as a Confounder
After 100B tokens of continued RTD training from a 50%-masking model, CoLA jumps 9.9 points while the other GLUE tasks move only 0.5 on average. The authors attribute the disproportionate gain to alignment between token plausibility and sentence acceptability, motivating CoLA's exclusion from the controlled objective comparison.
C.2 Comparison with RTD-Trained Encoders
RTD-trained DeBERTaV3 is omitted from the main comparison but evaluated here. Its base/large classification averages are 74.2/78.1 versus UltraBERT's 75.4/78.5; retrieval averages are 20.7/25.8 versus 45.3/48.7. These averages use the figure's benchmark grouping and differ from individual GLUE/BEIR scores.
Appendix D Why KoLeo Increases Effective Rank
The derivation connects entropy pressure, approximate isotropy, and a flatter representation spectrum. It explains a tendency of KoLeo regularization rather than proving improved retrieval or guaranteed finite-training convergence.
Setup
For normalized vectors, let Σ be the population second moment and U the uniform sphere distribution. KoLeo approximates negative entropy. If KL(P∥U) is small, bounded quadratic test functions plus Pinsker's inequality imply ||Σ−I/d||₂≤√(KL/2).
The spectral theorem identifies this operator norm with the largest eigenvalue deviation from 1/d. Thus approaching uniformity flattens population eigenvalues. When empirical second moments concentrate, singular values flatten too, raising effective rank toward its maximum.
The chain assumes that the entropy estimator's optimization meaningfully approaches the population uniform distribution and that sampling supports concentration. It does not imply every optimization step increases rank, nor that maximal rank guarantees useful retrieval. All displayed derivations are retained below.
Equations
Appendix E Detailed Evaluation Results
E.1 Reported Baseline Scores
Most baseline GLUE scores and selected SQuAD results come from prior papers; other classification and retrieval results are newly evaluated. Reproducing Ettin GLUE within 0.2 points supports the protocol's comparability, while some cross-source variation remains.
| Model | # Params | MNLI | QNLI | QQP | RTE | SST-2 | MRPC | CoLA | STS-B | Avg. |
| Ettin Encoder | 149M | 89.2 | 94.0 | 92.4 | 87.7 | 95.8 | 92.6 | 66.9 | 92.2 | 88.9 |
| (Our evaluation) | 88.9 | 93.8 | 92.1 | 88.4 | 95.4 | 92.2 | 67.0 | 92.5 | 88.8 | |
| Ettin Encoder | 395M | 91.3 | 95.2 | 93.0 | 92.8 | 96.7 | 93.6 | 71.3 | 92.7 | 90.8 |
| (Our evaluation) | 90.9 | 95.3 | 93.0 | 91.7 | 96.7 | 93.1 | 71.1 | 92.9 | 90.6 |
E.2 Full GLUE Results
Per-task GLUE results show that aggregate leadership does not mean universal task leadership. Exceptions include QQP/CoLA at base size and QQP/SST-2/MRPC/CoLA at medium size. Most deficits are within one point, except CoLA.
E.3 Full RewardBench Results
UltraBERT leads three RewardBench sections at base size and Chat Hard/Safety at medium size. Ettin remains slightly ahead on medium Chat/Reasoning. The largest margin is 4.9 points on medium Chat Hard.
E.4 Full BEIR Results
Detailed BEIR results retain all 15 datasets and selected learning rates. UltraBERT leads 10 datasets at base size and eight at medium size; LFM2.5-Encoder-350M leads several scientific and question-duplication datasets.
E.5 Full LongEmbed Results
UltraBERT leads every base-size LongEmbed task and all but Needle/2WikiMultihopQA at medium size. Figure 11 reports 99% Passkey accuracy through 32K at both sizes, while shorter-context baselines degrade beyond their supported windows. These synthetic retrieval results demonstrate long-context access under this test, not unrestricted long-context understanding.
| Model | # Params | MNLI | QNLI | QQP | RTE | SST-2 | MRPC | CoLA | STS-B | Avg. |
| Base (200M) | ||||||||||
| BERT | 110M | 85.4 | 91.6 | 91.4 | 78.2 | 93.1 | 89.5 | 59.0 | 89.4 | 84.7 |
| RoBERTa | 125M | 87.6 | 92.8 | 91.9 | 78.7 | 94.8 | 90.2 | 63.6 | 91.2 | 86.4 |
| MosaicBERT | 137M | 85.6 | 91.4 | 92.0 | 83.0 | 93.5 | 89.0 | 58.2 | 90.3 | 85.4 |
| NomicBERT | 137M | 86.0 | 92.0 | 92.0 | 82.0 | 93.0 | 88.0 | 50.0 | 90.0 | 84.0 |
| GTE-en-MLM | 137M | 86.7 | 91.9 | 88.8 | 84.8 | 93.4 | 92.1 | 57.0 | 90.2 | 85.6 |
| ModernBERT | 149M | 89.1 | 93.9 | 92.1 | 87.4 | 96.0 | 92.2 | 65.1 | 91.8 | 88.4 |
| Ettin Encoder | 149M | 89.2 | 94.0 | 92.4 | 87.7 | 95.8 | 92.6 | 66.9 | 92.2 | 88.9 |
| UltraBERT | 152M | 90.0 | 94.5 | 92.2 | 91.0 | 96.3 | 93.6 | 65.2 | 93.1 | 89.5 |
| Medium (500M) | ||||||||||
| BERT | 334M | 86.3 | 92.8 | 90.9 | 83.8 | 93.3 | 87.8 | 56.2 | 90.6 | 85.2 |
| RoBERTa | 355M | 90.2 | 94.7 | 92.2 | 86.6 | 96.4 | 90.9 | 68.0 | 92.4 | 88.9 |
| GTE-en-MLM | 434M | 89.2 | 93.9 | 89.2 | 88.1 | 95.1 | 93.5 | 60.4 | 91.4 | 87.6 |
| NeoBERT | 222M | 89.0 | 93.7 | 90.7 | 91.3 | 95.6 | 93.4 | 66.2 | 91.8 | 89.0 |
| LFM2.5-Encoder | 230M | 88.4 | 93.7 | 92.4 | 86.6 | 94.6 | 91.9 | 65.5 | 92.4 | 88.2 |
| LFM2.5-Encoder | 355M | 89.5 | 94.4 | 92.6 | 89.2 | 96.3 | 91.8 | 66.8 | 92.8 | 89.2 |
| ModernBERT | 395M | 90.8 | 95.2 | 92.7 | 92.1 | 97.1 | 91.7 | 71.4 | 92.8 | 90.4 |
| Ettin Encoder | 395M | 91.3 | 95.2 | 93.0 | 92.8 | 96.7 | 93.6 | 71.3 | 92.7 | 90.8 |
| UltraBERT | 380M | 91.3 | 95.4 | 92.6 | 93.9 | 96.4 | 93.4 | 71.2 | 93.4 | 91.0 |
| Model | # Params | Chat | Chat Hard | Safety | Reasoning | Avg. |
| Base (200M) | ||||||
| BERT | 110M | 74.0 | 52.9 | 58.6 | 51.9 | 59.4 |
| RoBERTa | 125M | 77.7 | 55.7 | 58.6 | 60.8 | 63.2 |
| MosaicBERT | 137M | 75.7 | 51.1 | 58.2 | 65.0 | 62.5 |
| NomicBERT | 137M | 74.9 | 53.9 | 66.4 | 61.4 | 64.1 |
| GTE-en-MLM | 137M | 77.7 | 55.7 | 66.5 | 53.7 | 63.4 |
| ModernBERT | 149M | 77.7 | 59.2 | 60.7 | 74.4 | 68.0 |
| Ettin Encoder | 149M | 80.7 | 61.2 | 63.1 | 76.2 | 70.3 |
| UltraBERT | 152M | 85.8 | 63.4 | 66.4 | 76.5 | 73.0 |
| Medium (500M) | ||||||
| BERT | 334M | 72.1 | 52.6 | 60.9 | 64.9 | 62.6 |
| RoBERTa | 355M | 80.4 | 62.3 | 65.3 | 66.0 | 68.5 |
| GTE-en-MLM | 434M | 78.2 | 49.8 | 61.4 | 59.6 | 62.2 |
| NeoBERT | 222M | 61.5 | 53.7 | 54.5 | 60.9 | 57.6 |
| LFM2.5-Encoder | 230M | 82.1 | 59.2 | 61.9 | 66.7 | 67.5 |
| LFM2.5-Encoder | 355M | 82.4 | 57.9 | 63.5 | 70.4 | 68.5 |
| ModernBERT | 395M | 84.6 | 64.3 | 65.1 | 77.0 | 72.8 |
| Ettin Encoder | 395M | 88.3 | 66.4 | 70.7 | 81.6 | 76.7 |
| UltraBERT | 380M | 88.0 | 71.3 | 74.9 | 81.2 | 78.8 |
| Model | # Params | LR | NF Corpus | SciFact | TREC- COVID | FiQA | ArguAna | Climate FEVER | DBPedia | FEVER | Hotpot QA | MS MARCO | NQ | Quora | SciDocs | Touche 2020 | CQA Dupstack | Avg. |
| Base (200M) | ||||||||||||||||||
| BERT | 110M | 3e-4 | 22.7 | 49.3 | 54.7 | 21.7 | 40.9 | 18.9 | 24.0 | 62.6 | 42.0 | 34.1 | 37.0 | 82.9 | 9.7 | 22.9 | 27.1 | 36.7 |
| RoBERTa | 125M | 3e-4 | 24.3 | 53.1 | 59.1 | 26.8 | 46.5 | 20.6 | 26.1 | 64.3 | 44.5 | 37.9 | 41.5 | 85.3 | 12.2 | 22.4 | 29.9 | 39.6 |
| MosaicBERT | 137M | 3e-4 | 27.5 | 53.7 | 58.9 | 28.9 | 45.0 | 21.5 | 29.6 | 67.1 | 48.9 | 40.4 | 44.1 | 86.0 | 13.6 | 23.6 | 32.3 | 41.4 |
| NomicBERT | 137M | 3e-4 | 26.7 | 55.5 | 67.8 | 26.1 | 44.6 | 22.3 | 31.7 | 67.1 | 50.4 | 39.8 | 46.2 | 85.1 | 13.6 | 21.9 | 30.6 | 42.0 |
| GTE-en-MLM | 137M | 1e-4 | 29.3 | 56.2 | 56.2 | 31.1 | 50.0 | 25.5 | 30.5 | 68.4 | 51.0 | 38.7 | 43.1 | 86.1 | 15.0 | 20.5 | 34.0 | 42.4 |
| ModernBERT | 149M | 5e-4 | 26.5 | 58.3 | 73.7 | 29.8 | 46.3 | 23.1 | 28.1 | 67.0 | 48.2 | 38.6 | 44.3 | 86.6 | 13.3 | 21.9 | 33.2 | 42.6 |
| Ettin Encoder | 149M | 5e-4 | 25.4 | 58.8 | 73.9 | 30.0 | 48.3 | 22.7 | 26.7 | 64.4 | 47.5 | 38.1 | 43.5 | 86.7 | 13.3 | 23.1 | 33.8 | 42.4 |
| UltraBERT | 152M | 5e-5 | 28.4 | 64.3 | 75.8 | 33.5 | 50.2 | 23.9 | 30.6 | 69.6 | 52.5 | 40.1 | 47.8 | 87.2 | 15.2 | 23.5 | 36.0 | 45.2 |
| Medium (500M) | ||||||||||||||||||
| BERT | 334M | 1e-4 | 24.6 | 52.5 | 55.2 | 25.1 | 46.3 | 21.4 | 23.1 | 65.7 | 44.5 | 32.8 | 39.0 | 84.3 | 13.0 | 20.7 | 30.5 | 38.6 |
| RoBERTa | 355M | 1e-4 | 27.9 | 55.4 | 60.3 | 33.0 | 51.3 | 24.9 | 28.6 | 70.0 | 49.8 | 38.6 | 46.6 | 86.9 | 14.4 | 20.1 | 34.2 | 42.8 |
| GTE-en-MLM | 434M | 1e-4 | 30.3 | 58.5 | 53.4 | 33.2 | 48.9 | 19.6 | 25.1 | 49.8 | 42.0 | 35.4 | 42.3 | 83.7 | 15.4 | 19.3 | 34.8 | 39.4 |
| NeoBERT | 222M | 3e-4 | 30.0 | 56.3 | 71.3 | 28.7 | 47.8 | 19.9 | 31.6 | 67.5 | 50.7 | 40.1 | 44.3 | 86.2 | 15.0 | 22.4 | 31.9 | 42.9 |
| LFM2.5-Encoder | 230M | 5e-4 | 30.6 | 66.1 | 78.8 | 33.0 | 52.3 | 26.1 | 32.2 | 69.2 | 55.0 | 40.8 | 50.0 | 87.3 | 16.2 | 25.9 | 37.4 | 46.7 |
| LFM2.5-Encoder | 355M | 3e-4 | 32.1 | 68.5 | 80.3 | 34.9 | 53.9 | 25.7 | 32.8 | 69.8 | 56.5 | 40.3 | 50.4 | 87.6 | 16.8 | 21.9 | 38.4 | 47.3 |
| ModernBERT | 395M | 3e-4 | 29.2 | 62.4 | 77.7 | 34.4 | 51.1 | 24.1 | 28.9 | 68.7 | 51.2 | 40.7 | 49.2 | 87.2 | 15.5 | 22.1 | 36.0 | 45.2 |
| Ettin Encoder | 395M | 5e-4 | 29.6 | 65.2 | 79.4 | 33.3 | 50.8 | 24.2 | 31.7 | 67.1 | 51.2 | 41.6 | 49.5 | 87.3 | 15.6 | 25.1 | 36.5 | 45.9 |
| UltraBERT | 380M | 3e-5 | 31.7 | 66.4 | 77.0 | 38.1 | 53.9 | 25.7 | 33.9 | 73.2 | 56.5 | 41.1 | 51.6 | 87.7 | 18.0 | 23.3 | 38.0 | 47.7 |
| Model | Length | # Params | Passkey | Needle | Narrative QA | QMSum | Summ ScreenFD | 2Wiki MultihopQA | Avg. |
| Base (200M) | |||||||||
| BERT | 512 | 110M | 21.5 | 19.2 | 18.8 | 18.1 | 54.2 | 43.8 | 29.3 |
| RoBERTa | 512 | 125M | 37.3 | 21.0 | 20.5 | 19.9 | 58.6 | 45.6 | 33.8 |
| MosaicBERT | 512 | 137M | 23.0 | 38.2 | 23.2 | 24.2 | 77.4 | 57.0 | 40.5 |
| NomicBERT | 2048 | 137M | 54.2 | 30.8 | 22.4 | 25.4 | 77.5 | 59.5 | 45.0 |
| GTE-en-MLM | 8192 | 137M | 60.2 | 50.2 | 43.1 | 33.6 | 91.8 | 71.7 | 58.5 |
| ModernBERT | 8192 | 149M | 81.7 | 59.0 | 35.3 | 38.5 | 89.6 | 75.3 | 63.2 |
| Ettin Encoder | 8192 | 149M | 80.2 | 50.8 | 38.3 | 33.7 | 90.1 | 67.6 | 60.1 |
| UltraBERT | 32768 | 152M | 99.2 | 59.8 | 56.0 | 45.4 | 95.2 | 76.6 | 72.0 |
| Medium (500M) | |||||||||
| BERT | 512 | 334M | 27.0 | 17.8 | 20.2 | 17.4 | 55.3 | 42.6 | 30.0 |
| RoBERTa | 512 | 355M | 35.2 | 18.8 | 20.7 | 18.9 | 57.2 | 42.6 | 32.2 |
| GTE-en-MLM | 8192 | 434M | 49.8 | 42.8 | 43.6 | 37.2 | 92.2 | 76.3 | 57.0 |
| NeoBERT | 4096 | 222M | 16.0 | 14.3 | 19.5 | 23.2 | 70.7 | 50.5 | 32.4 |
| LFM2.5-Encoder | 8192 | 230M | 88.8 | 78.2 | 47.3 | 37.5 | 89.2 | 79.0 | 70.0 |
| LFM2.5-Encoder | 8192 | 355M | 92.5 | 72.8 | 52.4 | 41.2 | 93.3 | 78.0 | 71.7 |
| ModernBERT | 8192 | 395M | 85.0 | 51.5 | 40.2 | 38.0 | 91.8 | 74.7 | 63.5 |
| Ettin Encoder | 8192 | 395M | 85.0 | 56.5 | 42.3 | 37.7 | 91.3 | 72.8 | 64.3 |
| UltraBERT | 32768 | 380M | 99.8 | 68.2 | 58.2 | 48.0 | 96.6 | 78.7 | 74.9 |