Literature
Display
← All papers

LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures

Hai Huang · Yann LeCun · Randall Balestriero

Preprint posted · ICLR 2026 ↗

Language models7 min text · CondensedOriginal paper ↗
Contents

Overview

Intuition: a programming request and the code that satisfies it are two descriptions of the same underlying task. LLM-JEPA trains their representations to predict one another while retaining the ordinary token-generation objective. The result is a regularized generative language model, not a replacement of language generation by a vision-only encoder. Read the reported accuracy improvements together with the paired-view requirement, extra training passes and task-specific evaluation rules.

Original figure
Original figure
Figure 1: LLM-JEPA produces strong fine-tuned models across datasets and models.

At a glance

LLM-JEPA adds an embedding-prediction loss to a language model’s usual generative loss. The study tests fine-tuning across model families and structured language tasks, plus smaller pretraining experiments. Useful paired views are supplied by the dataset: descriptions and regular expressions, questions and solutions, or paraphrases. The evidence concerns these training settings; it does not establish that arbitrary unpaired text automatically supplies equally useful JEPA views.

1 Introduction

The paper transfers a representation-learning idea from vision: related views should predict each other’s abstract features. Here the views can be English and code expressing the same functionality. The engineering choice is to keep next-token prediction so the model still emits text, then add a second incentive to organize its hidden representations. This differs from training a sentence encoder that is evaluated only by similarity. The need to construct informative views is a central constraint, not incidental preprocessing.

Refer to caption

Natural Language to Regular Expression

Refer to caption

Natural Language to SQL

Refer to caption
Figure 2: Left: JEPA applied to NLP tasks that has T​e​x​tText and C​o​d​eCode, where T​e​x​tText and C​o​d​eCode are naturally two views of the same thing. Right: (top): An illustration of the NL-RX-SYNTH dataset, where each sample consists of a description of the regular expression in natural language (T​e​x​tText) and the regular expression itself (C​o​d​eCode). (bottom): The Spider dataset, where T​e​x​tText is the database ID and description of the SQL query and C​o​d​eCode is the SQL query itself.

2 Background

A standard language model learns to predict the next token from its prefix. An embedding method instead compares vector summaries of sequences. Earlier sentence-embedding objectives improve representations but often do not preserve a usable generative model. LLM-JEPA combines the two roles: the token loss trains output generation, while a JEPA term trains cross-view abstraction. The combined objective should therefore be assessed using generated answers, training cost and representation diagnostics separately.

Equation 1
ℒLLM(Text1:L−1,TextL)=XEnt(Classifier(Enc(Text1:L−1)),TextL),\displaystyle\mathcal{L}_{\rm LLM}({\rm Text}_{1:L-1},{\rm Text}_{L})={\rm XEnt}\left({\rm Classifier}\left({\rm Enc}({\rm Text}_{1:L-1})\right),{\rm Text}_{L}\right), (1)

3 JEPA-LLM: Improving LLMs’ Reasoning and Generative Capabilities

The method has two parts: a cross-view objective and an attention mask that prevents the target representation from seeing the other view. Both matter because otherwise the apparent prediction problem can leak the information it is meant to learn.

3.1 The LLM-JEPA Objective

Each sequence is summarized by its final-token hidden state. A predictor uses the same language-model weights, with optional appended prediction tokens, to map the first view toward the second view’s embedding. Cosine similarity supplies the representation loss, weighted against ordinary token loss by a coefficient. Reusing the model avoids a separate large predictor, but still adds training computation. The number of prediction tokens controls additional processing; zero prediction tokens gives a simpler special case.

Equation 2
ℒLLM−JEPA=∑ℓ=2LℒLLM(Text1:ℓ−1,Textℓ)⏟generative capabilities (LLM)+λ×d⁡(Pred⁡(Enc⁡(Text)),Enc⁡(Code))⏟abstraction capabilities (JEPA),\displaystyle\mathcal{L}_{\rm LLM-JEPA}=\underbrace{\sum_{\ell=2}^{L}\mathcal{L}_{\rm LLM}({\rm Text}_{1:\ell-1},{\rm Text}_{\ell})}_{\text{generative capabilities (LLM)}}+\lambda\times\underbrace{d({\rm Pred}({\rm Enc}({\rm Text})),{\rm Enc}({\rm Code}))}_{\text{abstraction capabilities (JEPA)}}, (2)

3.2 Implementation with Custom Attention Mask

Packing two views into one window is insufficient: ordinary causal attention lets later tokens read earlier tokens. The custom mask separates the views into causal blocks, so each embedding is computed from its own view. This permits the JEPA computation in one additional forward pass rather than two additional passes. The complete implementation is preserved below. The paired-text examples explain the mechanism; they are not evidence that all natural-language datasets already have equally informative view pairs.

Original figure
Refer to caption
Figure 3: left: The top 100 singular values of Enc⁡(Text)−Enc⁡(Code)\operatorname{Enc}(\operatorname{Text})-\operatorname{Enc}(\operatorname{Code}). The curves of LLM-JEPA (ours) are a few magnitudes lower than that of base model and regular fine-tuning, meaning the mapping from Text\rm Text to Code\rm Code are confined within a narrow subspace, fostering the nice structure we see in Figure 4. Right: Losses in fine-tuning with ℒLLM\mathcal{L}_{\rm LLM} loss (ℒLLM\mathcal{L}_{\rm LLM}) and ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA}, our method). We measure both the cross-entropy loss for next token prediction (L​o​s​sL​L​MLoss_{LLM}, ℒLLM\mathcal{L}_{\rm LLM} in chart) and JEPA prediction loss (D⁡(⋅,⋅)D(\cdot,\cdot), pred in chart), although the latter does not contribute in the baseline case. The accuracy is 51.95%51.95\% for ℒLLM\mathcal{L}_{\rm LLM} and 71.10%71.10\% for ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA}. Since ℒLLM\mathcal{L}_{\rm LLM} and ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} share similar ℒLLM\mathcal{L}_{\rm LLM} loss, the ℒLLM\mathcal{L}_{\rm LLM} loss cannot explain the gap between the accuracy. pred stays a constant in ℒLLM\mathcal{L}_{\rm LLM}, while is minimized in ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA}, hence pred should be the main reason behind the accuracy gap.

3.3 A Good Next Token Predictor is not a Good JEPA

The controlled experiment asks whether good token prediction already minimizes the representation objective. It does not: the two monitored losses behave differently. Adding JEPA can reduce cross-view error without the corresponding deterioration in the monitored token loss. This establishes complementarity in the experiment, rather than a theorem that every generative objective lacks useful semantic structure.

4 Empirical Validation: LLM-JEPAs Outperform LLMs

The evaluation repeats experiments with five fixed training seeds and reports paired comparisons. The questions are distinct: do generated answers improve, do representations change, and which design choices are responsible? The following sections address those questions with task results, geometric diagnostics and ablations.

4.1 Fine-tuning and Pretraining Stronger Generative Models via JEPA

The study fine-tunes several roughly billion-parameter model families on regular-expression generation, mathematical answers and SQL execution. Correctness is task-specific: exact expression matching differs from matching a numerical result or a query’s execution. Gains also appear across tested sizes and LoRA ranks. The smaller pretraining experiments need separate interpretation: one uses prefix validity because generation termination is unreliable, and another pretrains on paraphrases before sentiment-label fine-tuning. These are encouraging transfer tests, not unrestricted web-scale pretraining evidence.

Table 1: Regular expressions generated by Llama-3.2-1B-Instruct after fine-tuning with ℒLLM\mathcal{L}_{\rm LLM} loss and ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (ours). Color code: wrong, extra, missing
Ground Truth ℒLLM\mathcal{L}_{\rm LLM} ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours)
lines not having the string ’dog’ followed by a number, 3 or more times
 ((dog.*[0-9].*)3,)  ((dog.*[0-9].*)3,)  ((dog.*[0-9].*){3,})
lines containing ending with a vowel, zero or more times
.*(.*)(([AEIOUaeiou])*).* (.*)(([AEIOUaeiou])*)* (.*)(.*){([AEIOUaeiou])*}
lines with a number or a character before a vowel
(([0-9])|(.)).*([AEIOUaeiou]).* (([0-9])|(.)).*([AEIOUaeiou]).*.* (([0-9])|(.)).*([AEIOUaeiou]).*
lines with words with the string ’dog’, a letter, and a number
((([0-9])&(dog))|([A-Za-z]))* ((([0-9])&(dog))|([A-Za-z]))* (([0-9])&(dog))|(([A-Za-z])*)
Table 2: Pretraining accuracy on dataset NL-RX-SYNTH by Next Token Prediction (ℒLLM\mathcal{L}_{\rm LLM}) loss vs. ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (our method). We inherit the best configuration from fine-tuning. Each case runs five times. Average accuracy and standard deviation are reported. We also report pp-value of paired, single-tailed tt-Test.
Model Method Accuracy (%) ↑\uparrow pp-value ↓\downarrow Config
Llama-3.2-1B-Instruct ℒLLM\mathcal{L}_{\rm LLM} 54.38±1.7054.38\pm 1.70 2.94​e−42.94e-4 l​r=8​e−5lr=8e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 60.59±1.0160.59\pm 1.01 λ=2,k=3\lambda=2,k=3, same l​rlr

4.2 Structured Representations Induced by LLM-JEPA

The geometry plots compare embeddings before fine-tuning, after ordinary fine-tuning and after JEPA fine-tuning. Singular-value and fitted linear-map measurements support more organized relationships between the paired views. A two-dimensional visualization can illustrate that structure but cannot establish it on its own. The map is fitted using paired examples, so its success should not be described as learning an alignment without correspondence information.

Original figure
(a) Baseline: Fine-tuned by NTP loss
Original figure
(b) LLM-JEPA (Ours) k=0k=0
Figure 4: tt-SNE plot of Text\rm Text and Code\rm Code representations in (a) Baseline that is fine-tuned with NTP loss, (b) LLM-JEPA (ours) with k=0k=0. Clearly LLM-JEPA (ours) induced nice structure on the representations while fine-tuning with NTP loss disrupted the structure in the base model. A full version of this figure is in section A.2.

4.3 Ablation Study on Design Choices

Ablations vary the vector distance, predictor-token placement and direction of view prediction. Performance depends on these choices: a seemingly similar embedding objective can work much worse. The preserved table also includes the tested contrastive-loss alternative. The practical lesson is to treat the particular objective and implementation as the method, rather than assume any added semantic-similarity term gives the same effect.

Table 3: Fine-tuning accuracy on NL-RX-SYNTH under different design choices. Reported are average accuracy and standard deviation across runs. Learning rate l​r=2​e−5lr=2\mathrm{e}{-5}, λ=1.0\lambda=1.0, and k=1k=1.
Accuracy (%) ↑\uparrow
Baseline LLM-JEPA ℓ2\ell_{2}-norm MSE Prepend Code→Text\operatorname{Code}\rightarrow\operatorname{Text} InfoNCE loss
57.29±5.3257.29\pm 5.32 71.46±1.3471.46\pm 1.34 2.22±0.072.22\pm 0.07 70.64±2.0570.64\pm 2.05 68.07±2.5768.07\pm 2.57 65.70±2.6365.70\pm 2.63 34.40±6.1034.40\pm 6.10

5 Towards Faster and More General LLM-JEPAs

The final experiments broaden the choice of views and reduce the extra compute. They examine whether the benefits survive outside description-to-code tasks and whether occasional JEPA updates can preserve much of the gain.

5.1 Beyond Code: Applying LLM-JEPA to Q&A Datasets and Reasoning Models

For open-domain questions, the two views are a question and an answer span; for multiple-choice completion, they are the context and correct continuation. These pairs are less interchangeable than English and code, yet the tested models still improve. Evaluation remains task-specific: substring matching for open-domain answers is different from ranking answer-option probabilities. The reasoning-model experiments extend the comparison to two additional models rather than establishing universal reasoning capability.

Table 4: Fine-tuning accuracy of Llama-3.2-1B-Instruct with Next Token Prediction (ℒLLM\mathcal{L}_{\rm LLM}) loss vs. ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (our method). Each case runs five times. Average accuracy and standard deviation are reported. We also report pp-value of paired, single-tailed tt-Test.
Dataset Method Accuracy (%) ↑\uparrow pp-value ↓\downarrow Config
NQ-Open ℒLLM\mathcal{L}_{\rm LLM} 20.12±0.4120.12\pm 0.41 2.44​e−32.44e-3 l​r=2​e−5lr=2e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 21.59±0.4021.59\pm 0.40 λ=1024,k=0\lambda=1024,k=0, same l​rlr
HellaSwag ℒLLM\mathcal{L}_{\rm LLM} 69.40±0.9969.40\pm 0.99 0.01360.0136 l​r=4​e−5lr=4e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 70.51±1.2070.51\pm 1.20 λ=1,k=3\lambda=1,k=3, same l​rlr
Table 5: Fine-tuning accuracy of GSM8K with Next Token Prediction (ℒLLM\mathcal{L}_{\rm LLM}) loss vs. ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (our method). Each case runs five times. Average accuracy and standard deviation are reported. We also report pp-value of paired, single-tailed tt-Test.
Model Method Accuracy (%) ↑\uparrow pp-value ↓\downarrow Config
Qwen3-1.7B ℒLLM\mathcal{L}_{\rm LLM} 44.32±0.3944.32\pm 0.39 0.01150.0115 l​r=4​e−5lr=4e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 45.00±0.4045.00\pm 0.40 λ=1,k=0\lambda=1,k=0, same l​rlr
R1-Distill-Qwen-1.5B ℒLLM\mathcal{L}_{\rm LLM} 13.87±1.0113.87\pm 1.01 0.03960.0396 l​r=8​e−5lr=8e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 15.04±0.1515.04\pm 0.15 λ=0.5,k=1\lambda=0.5,k=1, same l​rlr

5.2 Faster LLM-JEPAs via Loss Dropout

Loss dropout randomly skips the JEPA calculation for some batches. Those batches need no extra embedding pass, reducing the training overhead. The paper compares performance at a matched compute budget and studies how the loss weight should change as JEPA becomes less frequent. The reported approximate compute expression is a model of this implementation’s passes, not a promise of equal wall-clock speed on every hardware and sequence-length configuration.

Original figure
Figure 5: LLM-JEPA converges faster than regular fine-tuning at the same PFLOPs. Furthermore, random JEPA-loss dropout (LD) helps save PFLOPs and boost accuracy at the same amount of compute. L​D=0LD=0 is the regular LLM-JEPA. Learning rate l​r=2​e−5lr=2\mathrm{e}{-5} and k=1k=1. λ\lambda varies.
Table 6: Random JEPA-loss dropout (L​DLD) help save PFLOPs and at the same time, boost accuracy. L​D=0LD=0 is the regular LLM-JEPA. Reported are average accuracy and standard deviation across runs. Each row is 4.83 PFLOPs apart. Learning rate l​r=2​e−5lr=2\mathrm{e}{-5} and k=1k=1. λ\lambda varies.
Accuracy (%) ↑\uparrow
L​DLD=0, λ\lambda=1 L​DLD=0.5, λ\lambda=1 L​DLD=0.5, λ\lambda=2 L​DLD=0.75, λ\lambda=1 L​DLD=0.75, λ\lambda=2 L​DLD=0.75, λ\lambda=4
19.85±2.4419.85\pm 2.44 25.00±3.7325.00\pm 3.73 24.50±4.4024.50\pm 4.40 32.46±3.3232.46\pm 3.32 32.10±3.1132.10\pm 3.11 31.45±3.3431.45\pm 3.34
40.70±2.6740.70\pm 2.67 48.96±6.0348.96\pm 6.03 50.71±6.5250.71\pm 6.52 53.77±5.5353.77\pm 5.53 57.03±3.5157.03\pm 3.51 56.75±4.6356.75\pm 4.63
55.60±5.1655.60\pm 5.16 64.08±2.7564.08\pm 2.75 63.79±5.9663.79\pm 5.96 64.51±7.2864.51\pm 7.28 67.03±3.0867.03\pm 3.08 65.32±3.5465.32\pm 3.54
60.43±3.2160.43\pm 3.21 69.87±2.4869.87\pm 2.48 70.11±3.1570.11\pm 3.15 67.80±4.9467.80\pm 4.94 66.80±4.0666.80\pm 4.06 68.93±4.5968.93\pm 4.59
63.57±1.0163.57\pm 1.01 69.74±2.9969.74\pm 2.99 71.20±2.1771.20\pm 2.17 70.00±4.7470.00\pm 4.74 70.77±4.0870.77\pm 4.08 72.42±1.2872.42\pm 1.28
63.96±1.0763.96\pm 1.07 70.60±3.0570.60\pm 3.05 72.11±2.1872.11\pm 2.18 70.31±4.6470.31\pm 4.64 70.92±4.6270.92\pm 4.62 73.08±1.2873.08\pm 1.28

6 Conclusion and Future Work

The main contribution is a practical regularizer that preserves generation while improving the tested paired-view tasks. Its costs include additional training computation and tuning both the loss weight and predictor-token count. Pretraining results are smaller and more preliminary than the fine-tuning study. Scaling the method to broad unpaired corpora, finding useful views and reducing tuning cost remain open engineering questions.

Appendix A Appendix

The appendix retains additional generated examples and the experiments behind the headline results. The example model sometimes fails to stop generating, an important limitation when interpreting prefix-based pretraining accuracy.

Table 7: Generated samples by model pretrained by cestwc/paraphrase dataset. The pretrained model is not good at terminating sentence. prompt and generation
Ground Truth vs. Generation
Ground Truth A garden of flowers and a bench stating "City of London."
Generation A garden of flowers and a vase with a flower in it………….
Ground Truth A person that is riding on a horse in a grass field.
Generation A person that is riding in a field……………..
Ground Truth A man is riding a horse in a field.
Generation A man is riding a horse in a field…………….
Ground Truth There are two birds standing on top of a building
Generation There are two birds standing on a rock……………..
Ground Truth Two hawks sit on top of a roof spire.
Generation Two hawks sit on top of a wooden bench…………….
Ground Truth .A young woman serving herself at a cookout.
Generation .A young woman serving herself in a kitchen……………..
Ground Truth 2 bowls of fruit sit on a table.
Generation 2 bowls of fruit sit on a table……………..
Ground Truth A wooden bench written ’CITY OF LONDON’ at the park
Generation A wooden bench written ’CITY and a tree……………..

A.1 Faster LoRA Convergence

LoRA results test how many trainable low-rank parameters are needed before the model approaches full fine-tuning performance. JEPA improves the tested ranks, and the largest tested rank reaches a similar accuracy to full fine-tuning in this task. That is a task-specific comparison, not a general claim that this rank always replaces full adaptation.

Table 8: Fine-tuning accuracy on dataset NL-RX-SYNTH, LoRA vs. full fine-tuning, both by ℒLLM\mathcal{L}_{\rm LLM} loss and ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (our method). Configuration is l​r=2​e−5,λ=1,k=1lr=2e-5,\lambda=1,k=1. Each cell runs five times. Average accuracy and standard deviation are reported. At every LoRA rank, ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) has better accuracy. At LoRA rank 512 (22.59% trainable parameters), ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) achieves same accuracy as full fine-tuning, but ℒLLM\mathcal{L}_{\rm LLM} still has a significant gap from full fine-tuning.
LoRA Rank Method Accuracy (%) ↑\uparrow
3232 ℒLLM\mathcal{L}_{\rm LLM} 6.09±0.556.09\pm 0.55
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 7.45±1.877.45\pm 1.87
6464 ℒLLM\mathcal{L}_{\rm LLM} 21.09±1.9021.09\pm 1.90
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 32.46±1.2632.46\pm 1.26
128128 ℒLLM\mathcal{L}_{\rm LLM} 34.21±2.8234.21\pm 2.82
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 48.45±3.6648.45\pm 3.66
256256 ℒLLM\mathcal{L}_{\rm LLM} 45.57±4.5245.57\pm 4.52
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 60.80±2.3160.80\pm 2.31
512512 ℒLLM\mathcal{L}_{\rm LLM} 50.18±5.1550.18\pm 5.15
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 72.41±2.9472.41\pm 2.94
Full ℒLLM\mathcal{L}_{\rm LLM} 57.29±5.3257.29\pm 5.32
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 70.42±2.3670.42\pm 2.36
Table 9: Pretraining ++ fine-tuning Llama-3.2-1B-Instruct accuracy on pretraining dataset cestwc/paraphrase and fine-tuning dataset Rotten Tomatoes and Yelp by Next Token Prediction (ℒLLM\mathcal{L}_{\rm LLM}) loss vs. ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (our method). Note that ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} is applied only at pretraining. We tune l​rp​r​elr_{pre} and l​rf​tlr_{ft} by ℒLLM\mathcal{L}_{\rm LLM}, and stick to them in LLM-JEPA pretraining. We run pretraining 5 times, and for each pretrained model, we run fine-tuning 5 times. Average accuracy and standard deviation are reported. We also report pp-value of paired, single-tailed tt-Test.
FT Dataset Method Accuracy (%) ↑\uparrow pp-value ↓\downarrow Config
Rotten Tomatoes ℒLLM\mathcal{L}_{\rm LLM} 56.57±1.6656.57\pm 1.66 7.38​e−47.38e-4 l​rp​r​e=8​e−5,l​rf​t=4​e−5lr_{pre}=8e-5,lr_{ft}=4e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 57.76±1.3357.76\pm 1.33 λ=0.5,k=2\lambda=0.5,k=2, same l​rp​r​e,l​rf​tlr_{pre},lr_{ft}
Yelp ℒLLM\mathcal{L}_{\rm LLM} 26.46±0.9226.46\pm 0.92 1.00​e−31.00e-3 l​rp​r​e=8​e−5,l​rf​t=8​e−5lr_{pre}=8e-5,lr_{ft}=8e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 27.15±0.9327.15\pm 0.93 λ=0.5,k=2\lambda=0.5,k=2, same l​rp​r​e,l​rf​tlr_{pre},lr_{ft}

A.2 LLM-JEPA Induces Structured Representation

Additional embedding plots show that the qualitative paired-view structure persists for the tested predictor-token settings. They supplement the quantitative diagnostics; the visual arrangement of a projection should not be treated as a complete description of the full hidden space.

Original figure
(a) Base model: No fine-tuning
Original figure
(b) Baseline: Fine-tuned by NTP loss
Original figure
(c) LLM-JEPA (Ours) k=0k=0
Original figure
(d) LLM-JEPA (Ours) k=1k=1
Figure 6: tt-SNE plot of f​T​e​x​t{\textnormal{f}}Text and Code\rm Code representations in (a) Base mode without fine-tuning, (b) Baseline that is fine-tuned with NTP loss, (c) LLM-JEPA (ours) with k=0k=0, and (d) LLM-JEPA (ours) with k=1k=1. Clearly LLM-JEPA (ours) induced nice structure on the representations while fine-tuning with NTP loss disrupted the structure in the base model.

A.3 Ablation Study on the Role of ℒ LLM

This experiment varies the generative term separately from the JEPA term. Removing the generative contribution causes empty outputs, showing that representation alignment alone is insufficient for the tested generation task. The grid also exposes tuning sensitivity: stronger alignment is not automatically better generated-answer accuracy.

Equation 3
ℒLLM−JEPA=γ×∑ℓ=2LℒLLM(Text1:ℓ−1,Textℓ)⏟generative capabilities+λ×d⁡(Pred⁡(Enc⁡(Text)),Enc⁡(Code))⏟abstraction capabilities,\displaystyle\mathcal{L}_{\rm LLM-JEPA}=\underbrace{\gamma\times\sum_{\ell=2}^{L}\mathcal{L}_{\rm LLM}({\rm Text}_{1:\ell-1},{\rm Text}_{\ell})}_{\text{generative capabilities}}+\lambda\times\underbrace{d({\rm Pred}({\rm Enc}({\rm Text})),{\rm Enc}({\rm Code}))}_{\text{abstraction capabilities}}, (3)
Table 10: Fine-tuning accuracy on dataset NL-RX-SYNTH with ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (ours) over various γ/λ\gamma/\lambda. Configuration is l​r=2​e−5,λ=1,k=0lr=2e-5,\lambda=1,k=0. We maintain max⁡(γ,λ)=1.0\max(\gamma,\lambda)=1.0 to use a fixed l​rlr. Each cell runs five times. Average accuracy and standard deviation are reported. When γ=0.0\gamma=0.0, it generate only empty output.
γ/λ\gamma/\lambda Config Accuracy (%) ↑\uparrow
0.00.0 γ=0.0,λ=1.0\gamma=0.0,\lambda=1.0 0.00±0.000.00\pm 0.00
0.010.01 γ=0.01,λ=1.0\gamma=0.01,\lambda=1.0 1.38±0.061.38\pm 0.06
0.10.1 γ=0.1,λ=1.0\gamma=0.1,\lambda=1.0 45.80±5.0445.80\pm 5.04
1.01.0 γ=1.0,λ=1.0\gamma=1.0,\lambda=1.0 70.42±2.3670.42\pm 2.36
10.010.0 γ=1.0,λ=0.1\gamma=1.0,\lambda=0.1 67.52±1.4567.52\pm 1.45
100.0100.0 γ=1.0,λ=0.01\gamma=1.0,\lambda=0.01 66.83±3.8966.83\pm 3.89
∞\infty γ=1.0,λ=0.0\gamma=1.0,\lambda=0.0 57.29±5.3257.29\pm 5.32
Refer to caption
(a) Llama on GSM8K, l​r=2​e−5lr=2e-5
Refer to caption
(b) Llama on Spider, l​r=1​e−5lr=1e-5
Refer to caption
(c) Gemma on SYNTH, l​r=1​e−5lr=1e-5
Refer to caption
(d) OpenELM on SYNTH, l​r=8​e−5lr=8e-5
Refer to caption
(e) OLMo on SYNTH, l​r=8​e−5lr=8e-5
Refer to caption
(f) Llama on SYNTH, Pretrain, l​r=8​e−5lr=8e-5
Figure 7: In general we didn’t find any pattern on where the best accuracy could appear. It could be at either high-end or low-end of either λ\lambda or kk. Furthermore, there can be dips and spikes in random locations. Nonetheless, adjacent cells have close accuracy most of times, and sweeping (k,λ)∈{0,1,2,3,4}×{0.5,1,2,4}(k,\lambda)\in\{0,1,2,3,4\}\times\{0.5,1,2,4\} normally yield satisfiable results. Each cell is an average of five runs, e​p​o​c​h=4epoch=4.

A.4 Additional Generation Examples

The preserved table compares exact target expressions with generated alternatives and highlights errors. It helps readers see what the task metric accepts and rejects. These examples illustrate particular outcomes and do not replace the aggregate repeated-run results.

Table 11: More regular expressions generated by Llama-3.2-1B-Instruct after fine-tuning with ℒLLM\mathcal{L}_{\rm LLM} loss and ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (ours). Color code: wrong, extra, missing
Ground Truth ℒLLM\mathcal{L}_{\rm LLM} ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours)
lines ending with a vowel or starting with a character
([AEIOUaeiou].*[A-Za-z].*)+ ([AEIOUaeiou].*[A-Za-z].*)+ ([AEIOUaeiou].*[A-Za-z].*)+
ines containing either a lower-case letter, a vowel, or a letter
((.*)([AEIOUaeiou]))|((.)(.*)) (.*)(([AEIOUaeiou])|((.)(.*))) (.*)(([AEIOUaeiou])|((.)(.*)))
lines starting with the string ’dog’ before a vowel
(([A-Za-z])7,).*(dog).* (([A-Za-z])7,).*(dog).*.* (([A-Za-z])7,).*(dog).*
lines not containing a letter and the string ’dog’
((([A-Z])+)|([a-z]))(.*) ((([A-Z])+)|([a-z]))(.*)+ ((([A-Z])+)|([a-z]))(.*)
lines with a character before a vowel and the string ’dog’, zero or more times
.*(.)&([0-9])&(dog).* .*(.)&([0-9])&(dog).*.* .*(.)&([0-9])&(dog).*.*.*
lines with a vowel at least once before not a character
(([A-Za-z])+).*(~([0-9])).* (([A-Za-z])+).*(~([0-9])).*.* (([A-Za-z])+).*(~([0-9])).*

A.5 Overfitting Behavior in LoRA Fine-Tuning

Longer LoRA fine-tuning compares how test accuracy evolves after the initial improvement. JEPA is more resistant to the observed degradation in this setup, but this does not prove immunity to overfitting under different data, training lengths or adaptation methods.

Original figure
Figure 8: LLM-JEPA resists overfitting in LoRA fine-tuning. Fine-tuning with ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (our method) resists overfitting. When fine-tuning with ℒLLM\mathcal{L}_{\rm LLM} loss start to overfit, ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} kept improving. However the trend is not as stable as in full fine-tuning, possibly due to limited capacity of LoRA fine-tuning.
Table 12: Fine-tuning accuracy on dataset NL-RX-SYNTH by Next Token Prediction (ℒLLM\mathcal{L}_{\rm LLM}) loss vs. ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (our method). Each cell is the best possible accuracy over a set of configurations. Each configuration runs five times. Average accuracy and standard deviation are reported. We also report pp-value of paired, single-tailed tt-Test.
Model Method Accuracy (%) ↑\uparrow pp-value ↓\downarrow Config
Llama-3.2-1B-Instruct ℒLLM\mathcal{L}_{\rm LLM} 57.29±5.3257.29\pm 5.32 1.0​e−31.0e-3 l​r=2​e−5lr=2e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 71.46±1.3471.46\pm 1.34 λ=1,k=1\lambda=1,k=1, same l​rlr
gemma-2-2b-it ℒLLM\mathcal{L}_{\rm LLM} 33.65±3.2433.65\pm 3.24 5.5​e−35.5e-3 l​r=1​e−5lr=1e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 43.12±2.6143.12\pm 2.61 λ=2,k=4\lambda=2,k=4, same l​rlr
OpenELM-1_1B-Instruct ℒLLM\mathcal{L}_{\rm LLM} 12.07±1.8112.07\pm 1.81 5.1​e−45.1e-4 l​r=8​e−5lr=8e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 25.40±2.4025.40\pm 2.40 λ=4,k=3\lambda=4,k=3, same l​rlr
OLMo-2-0425-1B-Instruct ℒLLM\mathcal{L}_{\rm LLM} 87.09±0.3687.09\pm 0.36 2.5​e−32.5e-3 l​r=8​e−5lr=8e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 87.52±0.2987.52\pm 0.29 λ=2,k=0\lambda=2,k=0, same l​rlr
Table 13: Fine-tuning accuracy by model Llama-3.2-1B-Instruct, ℒLLM\mathcal{L}_{\rm LLM} loss vs. ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (our method). Each cell is the best possible accuracy over a set of configurations. Each configuration runs five times. Average accuracy and standard deviation are reported. We also report pp-value of paired, single-tailed tt-Test.
Dataset Method Accuracy (%) ↑\uparrow pp-value ↓\downarrow Config
NL-RX-TURK ℒLLM\mathcal{L}_{\rm LLM} 22.49±1.9122.49\pm 1.91 2.4​e−42.4e-4 l​r=2​e−5lr=2e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 30.94±1.1330.94\pm 1.13 λ=1,k=1\lambda=1,k=1, same l​rlr
GSM8K ℒLLM\mathcal{L}_{\rm LLM} 32.36±0.5832.36\pm 0.58 9.6​e−59.6e-5 l​r=2​e−5lr=2e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 36.36±0.2036.36\pm 0.20 λ=0.5,k=4\lambda=0.5,k=4, same l​rlr
Spider ℒLLM\mathcal{L}_{\rm LLM} 47.52±2.4447.52\pm 2.44 4.0​e−34.0e-3 l​r=4​e−5lr=4e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 50.55±2.0850.55\pm 2.08 λ=1,k=3\lambda=1,k=3, same l​rlr

A.6 Structured Representations Induced by LLM-JEPA

A fitted linear transformation measures how closely one view’s embeddings can predict the other’s. Smaller regression errors support the proposed cross-view organization. Since the regression is trained using paired observations, the result concerns a learned alignment and its evaluation, not an alignment discovered without paired information.

Table 14: LLM-JEPA is almost a linear transformation from Enc⁡(Text)\operatorname{Enc}(\operatorname{Text}) to Enc⁡(Code)\operatorname{Enc}(\operatorname{Code}).
minX||Enc⁡(Text)⋅X−Enc⁡(Code)||2\min_{X}{||\operatorname{Enc}(\operatorname{Text})\cdot X-\operatorname{Enc}(\operatorname{Code})||_{2}} Avg. Top 100 Singular
Base model 3953.113953.11 310.73310.73
ℒLLM\mathcal{L}_{\rm LLM} 3035.013035.01 341.80341.80
LLM-JEPA (Ours) k=1k=1 4.474.47 94.8494.84
LLM-JEPA (Ours) k=0k=0 4.044.04 16.8216.82

A.7 Performance Across Model Sizes

The size comparison extends the regular-expression experiment beyond the smallest model. The table and tuning heat map retain the actual tested configurations and uncertainty. Improvement across several sizes supports robustness within that experiment; compute cost and hyperparameter choices still change with model size.

Table 15: Fine-tuning accuracy on NL-RX-SYNTH by Next Token Prediction (ℒLLM\mathcal{L}_{\rm LLM}) loss vs. ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} loss (our method). Each case runs five times. Average accuracy and standard deviation are reported. We also report pp-value of paired, single-tailed tt-Test. Note that Llama does not have official 3.2-8B, and we have to use 3.1-8B, which has a lower accuracy. Still LLM-JEPA sees significant improvement. We also evaluated on OLMo-2-7B.
Model Method Accuracy (%) ↑\uparrow pp-value ↓\downarrow Config
Llama-3.2-1B-Instruct ℒLLM\mathcal{L}_{\rm LLM} 57.29±5.3257.29\pm 5.32 1.0​e−31.0e-3 l​r=2​e−5lr=2e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 71.46±1.3471.46\pm 1.34 λ=1,k=1\lambda=1,k=1, same l​rlr
Llama-3.2-3B-Instruct ℒLLM\mathcal{L}_{\rm LLM} 74.55±3.5874.55\pm 3.58 0.03520.0352 l​r=2​e−5lr=2e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 77.16±3.6677.16\pm 3.66 λ=2,k=0\lambda=2,k=0, same l​rlr
Llama-3.1-8B-Instruct ℒLLM\mathcal{L}_{\rm LLM} 35.77±6.6035.77\pm 6.60 0.01310.0131 l​r=2​e−5lr=2e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 63.57±16.8163.57\pm 16.81 λ=2.0,k=0\lambda=2.0,k=0, same l​rlr
OLMo-2-1124-7B-Instruct ℒLLM\mathcal{L}_{\rm LLM} 87.26±0.2787.26\pm 0.27 0.03450.0345 l​r=2​e−5lr=2e-5
ℒLLM−JEPA\mathcal{L}_{\rm LLM-JEPA} (ours) 87.75±0.3387.75\pm 0.33 λ=20,k=2\lambda=20,k=2, same l​rlr
Refer to caption
Figure 9: Fine-tuning HellaSwag with Llama-3.2-1B allows λ\lambda to be scaled up to 1024, with performance continuing to improve.
Concept demonstration

Open image ↗