Intuition: a programming request and the code that satisfies it are two descriptions of the same underlying task. LLM-JEPA trains their representations to predict one another while retaining the ordinary token-generation objective. The result is a regularized generative language model, not a replacement of language generation by a vision-only encoder. Read the reported accuracy improvements together with the paired-view requirement, extra training passes and task-specific evaluation rules.
Figure 1: LLM-JEPA produces strong fine-tuned models across datasets and models.
At a glance
LLM-JEPA adds an embedding-prediction loss to a language model’s usual generative loss. The study tests fine-tuning across model families and structured language tasks, plus smaller pretraining experiments. Useful paired views are supplied by the dataset: descriptions and regular expressions, questions and solutions, or paraphrases. The evidence concerns these training settings; it does not establish that arbitrary unpaired text automatically supplies equally useful JEPA views.
1 Introduction
The paper transfers a representation-learning idea from vision: related views should predict each other’s abstract features. Here the views can be English and code expressing the same functionality. The engineering choice is to keep next-token prediction so the model still emits text, then add a second incentive to organize its hidden representations. This differs from training a sentence encoder that is evaluated only by similarity. The need to construct informative views is a central constraint, not incidental preprocessing.
Figure 2: Left:JEPA applied to NLP tasks that has and , where and are naturally two views of the same thing. Right: (top): An illustration of the NL-RX-SYNTH dataset, where each sample consists of a description of the regular expression in natural language () and the regular expression itself (). (bottom): The Spider dataset, where is the database ID and description of the SQL query and is the SQL query itself.
2 Background
A standard language model learns to predict the next token from its prefix. An embedding method instead compares vector summaries of sequences. Earlier sentence-embedding objectives improve representations but often do not preserve a usable generative model. LLM-JEPA combines the two roles: the token loss trains output generation, while a JEPA term trains cross-view abstraction. The combined objective should therefore be assessed using generated answers, training cost and representation diagnostics separately.
Equation 1
(1)
3 JEPA-LLM: Improving LLMs’ Reasoning and Generative Capabilities
The method has two parts: a cross-view objective and an attention mask that prevents the target representation from seeing the other view. Both matter because otherwise the apparent prediction problem can leak the information it is meant to learn.
Each sequence is summarized by its final-token hidden state. A predictor uses the same language-model weights, with optional appended prediction tokens, to map the first view toward the second view’s embedding. Cosine similarity supplies the representation loss, weighted against ordinary token loss by a coefficient. Reusing the model avoids a separate large predictor, but still adds training computation. The number of prediction tokens controls additional processing; zero prediction tokens gives a simpler special case.
Equation 2
(2)
3.2 Implementation with Custom Attention Mask
Packing two views into one window is insufficient: ordinary causal attention lets later tokens read earlier tokens. The custom mask separates the views into causal blocks, so each embedding is computed from its own view. This permits the JEPA computation in one additional forward pass rather than two additional passes. The complete implementation is preserved below. The paired-text examples explain the mechanism; they are not evidence that all natural-language datasets already have equally informative view pairs.
Figure 3: left: The top 100 singular values of . The curves of LLM-JEPA (ours) are a few magnitudes lower than that of base model and regular fine-tuning, meaning the mapping from to are confined within a narrow subspace, fostering the nice structure we see in Figure 4. Right: Losses in fine-tuning with loss () and loss (, our method). We measure both the cross-entropy loss for next token prediction (, in chart) and JEPA prediction loss (, pred in chart), although the latter does not contribute in the baseline case. The accuracy is for and for . Since and share similar loss, the loss cannot explain the gap between the accuracy. pred stays a constant in , while is minimized in , hence pred should be the main reason behind the accuracy gap.
3.3 A Good Next Token Predictor is not a Good JEPA
The controlled experiment asks whether good token prediction already minimizes the representation objective. It does not: the two monitored losses behave differently. Adding JEPA can reduce cross-view error without the corresponding deterioration in the monitored token loss. This establishes complementarity in the experiment, rather than a theorem that every generative objective lacks useful semantic structure.
The evaluation repeats experiments with five fixed training seeds and reports paired comparisons. The questions are distinct: do generated answers improve, do representations change, and which design choices are responsible? The following sections address those questions with task results, geometric diagnostics and ablations.
4.1 Fine-tuning and Pretraining Stronger Generative Models via JEPA
The study fine-tunes several roughly billion-parameter model families on regular-expression generation, mathematical answers and SQL execution. Correctness is task-specific: exact expression matching differs from matching a numerical result or a query’s execution. Gains also appear across tested sizes and LoRA ranks. The smaller pretraining experiments need separate interpretation: one uses prefix validity because generation termination is unreliable, and another pretrains on paraphrases before sentiment-label fine-tuning. These are encouraging transfer tests, not unrestricted web-scale pretraining evidence.
Table 1: Regular expressions generated by Llama-3.2-1B-Instruct after fine-tuning with loss and loss (ours). Color code: wrong, extra, missing
Ground Truth
(ours)
lines not having the string ’dog’ followed by a number, 3 or more times
((dog.*[0-9].*)3,)
((dog.*[0-9].*)3,)
((dog.*[0-9].*){3,})
lines containing ending with a vowel, zero or more times
.*(.*)(([AEIOUaeiou])*).*
(.*)(([AEIOUaeiou])*)*
(.*)(.*){([AEIOUaeiou])*}
lines with a number or a character before a vowel
(([0-9])|(.)).*([AEIOUaeiou]).*
(([0-9])|(.)).*([AEIOUaeiou]).*.*
(([0-9])|(.)).*([AEIOUaeiou]).*
lines with words with the string ’dog’, a letter, and a number
((([0-9])&(dog))|([A-Za-z]))*
((([0-9])&(dog))|([A-Za-z]))*
(([0-9])&(dog))|(([A-Za-z])*)
Table 2: Pretraining accuracy on dataset NL-RX-SYNTH by Next Token Prediction () loss vs. loss (our method). We inherit the best configuration from fine-tuning. Each case runs five times. Average accuracy and standard deviation are reported. We also report -value of paired, single-tailed -Test.
Model
Method
Accuracy (%)
-value
Config
Llama-3.2-1B-Instruct
(ours)
, same
4.2 Structured Representations Induced by LLM-JEPA
The geometry plots compare embeddings before fine-tuning, after ordinary fine-tuning and after JEPA fine-tuning. Singular-value and fitted linear-map measurements support more organized relationships between the paired views. A two-dimensional visualization can illustrate that structure but cannot establish it on its own. The map is fitted using paired examples, so its success should not be described as learning an alignment without correspondence information.
Figure 4: -SNE plot of and representations in (a) Baseline that is fine-tuned with NTP loss, (b)LLM-JEPA (ours) with . Clearly LLM-JEPA (ours) induced nice structure on the representations while fine-tuning with NTP loss disrupted the structure in the base model. A full version of this figure is in sectionA.2.
4.3 Ablation Study on Design Choices
Ablations vary the vector distance, predictor-token placement and direction of view prediction. Performance depends on these choices: a seemingly similar embedding objective can work much worse. The preserved table also includes the tested contrastive-loss alternative. The practical lesson is to treat the particular objective and implementation as the method, rather than assume any added semantic-similarity term gives the same effect.
Table 3: Fine-tuning accuracy on NL-RX-SYNTH under different design choices. Reported are average accuracy and standard deviation across runs. Learning rate , , and .
The final experiments broaden the choice of views and reduce the extra compute. They examine whether the benefits survive outside description-to-code tasks and whether occasional JEPA updates can preserve much of the gain.
5.1 Beyond Code: Applying LLM-JEPA to Q&A Datasets and Reasoning Models
For open-domain questions, the two views are a question and an answer span; for multiple-choice completion, they are the context and correct continuation. These pairs are less interchangeable than English and code, yet the tested models still improve. Evaluation remains task-specific: substring matching for open-domain answers is different from ranking answer-option probabilities. The reasoning-model experiments extend the comparison to two additional models rather than establishing universal reasoning capability.
Table 4: Fine-tuning accuracy of Llama-3.2-1B-Instruct with Next Token Prediction () loss vs. loss (our method). Each case runs five times. Average accuracy and standard deviation are reported. We also report -value of paired, single-tailed -Test.
Table 5: Fine-tuning accuracy of GSM8K with Next Token Prediction () loss vs. loss (our method). Each case runs five times. Average accuracy and standard deviation are reported. We also report -value of paired, single-tailed -Test.
Loss dropout randomly skips the JEPA calculation for some batches. Those batches need no extra embedding pass, reducing the training overhead. The paper compares performance at a matched compute budget and studies how the loss weight should change as JEPA becomes less frequent. The reported approximate compute expression is a model of this implementation’s passes, not a promise of equal wall-clock speed on every hardware and sequence-length configuration.
Figure 5: LLM-JEPA converges faster than regular fine-tuning at the same PFLOPs. Furthermore, random JEPA-loss dropout (LD) helps save PFLOPs and boost accuracy at the same amount of compute. is the regular LLM-JEPA. Learning rate and . varies.
Table 6: Random JEPA-loss dropout () help save PFLOPs and at the same time, boost accuracy. is the regular LLM-JEPA. Reported are average accuracy and standard deviation across runs. Each row is 4.83 PFLOPs apart. Learning rate and . varies.
Accuracy (%)
=0, =1
=0.5, =1
=0.5, =2
=0.75, =1
=0.75, =2
=0.75, =4
6 Conclusion and Future Work
The main contribution is a practical regularizer that preserves generation while improving the tested paired-view tasks. Its costs include additional training computation and tuning both the loss weight and predictor-token count. Pretraining results are smaller and more preliminary than the fine-tuning study. Scaling the method to broad unpaired corpora, finding useful views and reducing tuning cost remain open engineering questions.
Appendix A Appendix
The appendix retains additional generated examples and the experiments behind the headline results. The example model sometimes fails to stop generating, an important limitation when interpreting prefix-based pretraining accuracy.
Table 7: Generated samples by model pretrained by cestwc/paraphrase dataset. The pretrained model is not good at terminating sentence. prompt and generation
Ground Truth vs. Generation
Ground Truth
A garden of flowers and a bench stating "City of London."
Generation
A garden of flowers anda vase with a flower in it………….
Ground Truth
A person that is riding on a horse in a grass field.
Generation
A person that is ridingin a field……………..
Ground Truth
A man is riding a horse in a field.
Generation
A man is riding ahorse in a field…………….
Ground Truth
There are two birds standing on top of a building
Generation
There are two birds standingon a rock……………..
Ground Truth
Two hawks sit on top of a roof spire.
Generation
Two hawks sit on topof a wooden bench…………….
Ground Truth
.A young woman serving herself at a cookout.
Generation
.A young woman serving herselfin a kitchen……………..
Ground Truth
2 bowls of fruit sit on a table.
Generation
2 bowls of fruit siton a table……………..
Ground Truth
A wooden bench written ’CITY OF LONDON’ at the park
LoRA results test how many trainable low-rank parameters are needed before the model approaches full fine-tuning performance. JEPA improves the tested ranks, and the largest tested rank reaches a similar accuracy to full fine-tuning in this task. That is a task-specific comparison, not a general claim that this rank always replaces full adaptation.
Table 8: Fine-tuning accuracy on dataset NL-RX-SYNTH, LoRA vs. full fine-tuning, both by loss and loss (our method). Configuration is . Each cell runs five times. Average accuracy and standard deviation are reported. At every LoRA rank, (ours) has better accuracy. At LoRA rank 512 (22.59% trainable parameters), (ours) achieves same accuracy as full fine-tuning, but still has a significant gap from full fine-tuning.
Table 9: Pretraining fine-tuning Llama-3.2-1B-Instruct accuracy on pretraining dataset cestwc/paraphrase and fine-tuning dataset Rotten Tomatoes and Yelp by Next Token Prediction () loss vs. loss (our method). Note that is applied only at pretraining. We tune and by , and stick to them in LLM-JEPA pretraining. We run pretraining 5 times, and for each pretrained model, we run fine-tuning 5 times. Average accuracy and standard deviation are reported. We also report -value of paired, single-tailed -Test.
Additional embedding plots show that the qualitative paired-view structure persists for the tested predictor-token settings. They supplement the quantitative diagnostics; the visual arrangement of a projection should not be treated as a complete description of the full hidden space.
Figure 6: -SNE plot of and representations in (a) Base mode without fine-tuning, (b) Baseline that is fine-tuned with NTP loss, (c)LLM-JEPA (ours) with , and (d)LLM-JEPA (ours) with . Clearly LLM-JEPA (ours) induced nice structure on the representations while fine-tuning with NTP loss disrupted the structure in the base model.
This experiment varies the generative term separately from the JEPA term. Removing the generative contribution causes empty outputs, showing that representation alignment alone is insufficient for the tested generation task. The grid also exposes tuning sensitivity: stronger alignment is not automatically better generated-answer accuracy.
Equation 3
(3)
Table 10: Fine-tuning accuracy on dataset NL-RX-SYNTH with loss (ours) over various . Configuration is . We maintain to use a fixed . Each cell runs five times. Average accuracy and standard deviation are reported. When , it generate only empty output.
Figure 7: In general we didn’t find any pattern on where the best accuracy could appear. It could be at either high-end or low-end of either or . Furthermore, there can be dips and spikes in random locations. Nonetheless, adjacent cells have close accuracy most of times, and sweeping normally yield satisfiable results. Each cell is an average of five runs, .
A.4 Additional Generation Examples
The preserved table compares exact target expressions with generated alternatives and highlights errors. It helps readers see what the task metric accepts and rejects. These examples illustrate particular outcomes and do not replace the aggregate repeated-run results.
Table 11: More regular expressions generated by Llama-3.2-1B-Instruct after fine-tuning with loss and loss (ours). Color code: wrong, extra, missing
Ground Truth
(ours)
lines ending with a vowel or starting with a character
([AEIOUaeiou].*[A-Za-z].*)+
([AEIOUaeiou].*[A-Za-z].*)+
([AEIOUaeiou].*[A-Za-z].*)+
ines containing either a lower-case letter, a vowel, or a letter
((.*)([AEIOUaeiou]))|((.)(.*))
(.*)(([AEIOUaeiou])|((.)(.*)))
(.*)(([AEIOUaeiou])|((.)(.*)))
lines starting with the string ’dog’ before a vowel
(([A-Za-z])7,).*(dog).*
(([A-Za-z])7,).*(dog).*.*
(([A-Za-z])7,).*(dog).*
lines not containing a letter and the string ’dog’
((([A-Z])+)|([a-z]))(.*)
((([A-Z])+)|([a-z]))(.*)+
((([A-Z])+)|([a-z]))(.*)
lines with a character before a vowel and the string ’dog’, zero or more times
.*(.)&([0-9])&(dog).*
.*(.)&([0-9])&(dog).*.*
.*(.)&([0-9])&(dog).*.*.*
lines with a vowel at least once before not a character
Longer LoRA fine-tuning compares how test accuracy evolves after the initial improvement. JEPA is more resistant to the observed degradation in this setup, but this does not prove immunity to overfitting under different data, training lengths or adaptation methods.
Figure 8: LLM-JEPA resists overfitting in LoRA fine-tuning. Fine-tuning with loss (our method) resists overfitting. When fine-tuning with loss start to overfit, kept improving. However the trend is not as stable as in full fine-tuning, possibly due to limited capacity of LoRA fine-tuning.
Table 12: Fine-tuning accuracy on dataset NL-RX-SYNTH by Next Token Prediction () loss vs. loss (our method). Each cell is the best possible accuracy over a set of configurations. Each configuration runs five times. Average accuracy and standard deviation are reported. We also report -value of paired, single-tailed -Test.
Model
Method
Accuracy (%)
-value
Config
Llama-3.2-1B-Instruct
(ours)
, same
gemma-2-2b-it
(ours)
, same
OpenELM-1_1B-Instruct
(ours)
, same
OLMo-2-0425-1B-Instruct
(ours)
, same
Table 13: Fine-tuning accuracy by model Llama-3.2-1B-Instruct, loss vs. loss (our method). Each cell is the best possible accuracy over a set of configurations. Each configuration runs five times. Average accuracy and standard deviation are reported. We also report -value of paired, single-tailed -Test.
A.6 Structured Representations Induced by LLM-JEPA
A fitted linear transformation measures how closely one view’s embeddings can predict the other’s. Smaller regression errors support the proposed cross-view organization. Since the regression is trained using paired observations, the result concerns a learned alignment and its evaluation, not an alignment discovered without paired information.
Table 14: LLM-JEPA is almost a linear transformation from to .
The size comparison extends the regular-expression experiment beyond the smallest model. The table and tuning heat map retain the actual tested configurations and uncertainty. Improvement across several sizes supports robustness within that experiment; compute cost and hyperparameter choices still change with model size.
Table 15: Fine-tuning accuracy on NL-RX-SYNTH by Next Token Prediction () loss vs. loss (our method). Each case runs five times. Average accuracy and standard deviation are reported. We also report -value of paired, single-tailed -Test. Note that Llama does not have official 3.2-8B, and we have to use 3.1-8B, which has a lower accuracy. Still LLM-JEPA sees significant improvement. We also evaluated on OLMo-2-7B.
Model
Method
Accuracy (%)
-value
Config
Llama-3.2-1B-Instruct
(ours)
, same
Llama-3.2-3B-Instruct
(ours)
, same
Llama-3.1-8B-Instruct
(ours)
, same
OLMo-2-1124-7B-Instruct
(ours)
, same
Figure 9: Fine-tuning HellaSwag with Llama-3.2-1B allows to be scaled up to 1024, with performance continuing to improve.
53 references
Amini et al. (2019)
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi.
Mathqa: Towards interpretable math word problem solving with operation-based formalisms.
arXiv preprint arXiv:1905.13319, 2019.
Assran et al. (2023)
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas.
Self-supervised learning from images with a joint-embedding predictive architecture.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619–15629, 2023.
Baevski et al. (2022)
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli.
Data2vec: A general framework for self-supervised learning in speech, vision and language.
In International conference on machine learning, pages 1298–1312. PMLR, 2022.
Balestriero and LeCun (2024)
Randall Balestriero and Yann LeCun.
Learning by reconstruction produces uninformative features for perception.
arXiv preprint arXiv:2402.11337, 2024.
Bardes et al. (2024)
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas.
Revisiting feature prediction for learning visual representations from video.
arXiv preprint arXiv:2404.08471, 2024.
Barrault et al. (2024)
Loïc Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R Costa-jussà, David Dale, et al.
Large concept models: Language modeling in a sentence representation space.
arXiv preprint arXiv:2412.08821, 2024.
Brown et al. (2020a)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
In Proceedings of NeurIPS, 2020a.
Brown et al. (2020b)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
Advances in neural information processing systems, 33:1877–1901, 2020b.
Chen et al. (2020)
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.
A simple framework for contrastive learning of visual representations.
In Proceedings of the 37th International Conference on Machine Learning, pages 1597–1607. PMLR, 2020.
URLhttps://arxiv.org/abs/2002.05709.
Chowdhery et al. (2023)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.
Palm: Scaling language modeling with pathways.
Journal of Machine Learning Research, 24(240):1–113, 2023.
Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, John Schulman, Jacob Hilton, Melanie Knight, Adrian Weller, Dario Amodei, et al.
Training verifiers to solve math word problems.
arXiv preprint arXiv:2110.14168, 2021.
Guo et al. (2019)
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang.
Towards complex text-to-sql in cross-domain database with intermediate representation.
arXiv preprint arXiv:1905.08205, 2019.
Guu et al. (2020)
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.
Retrieval augmented language model pre-training.
In Proceedings of ICML, 2020.
He et al. (2022)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.
Masked autoencoders are scalable vision learners.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022.
Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset.
arXiv preprint arXiv:2103.03874, 2021.
Hoang et al. (2020)
Thong Hoang, Hong Jin Kang, David Lo, and Julia Lawall.
Cc2vec: Distributed representations of code changes.
In Proceedings of the ACM/IEEE 42nd international conference on software engineering, pages 518–529, 2020.
Iyer et al. (2017)
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer.
Learning a neural semantic parser from user feedback.
arXiv preprint arXiv:1704.08760, 2017.
Izacard and Grave (2021)
Gautier Izacard and Edouard Grave.
Leveraging passage retrieval with generative models for open domain question answering.
arXiv preprint arXiv:2007.01282, 2021.
Jimenez et al. (2024)
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan.
Swe-bench: Can language models resolve real-world github issues?
In International Conference on Learning Representations (ICLR), 2024.
URLhttps://openreview.net/forum?id=VTF8yNQM66.
Oral presentation.
Kenneweg et al. (2025)
Tristan Kenneweg, Philip Kenneweg, and Barbara Hammer.
Jepa for rl: Investigating joint-embedding predictive architectures for reinforcement learning.
arXiv preprint arXiv:2504.16591, 2025.
LeCun (2022)
Yann LeCun.
A path towards autonomous machine intelligence (version 0.9.2).
OpenReview, 62(1):1–62, jun 2022.
URLhttps://openreview.net/forum?id=BZ5a1r-kVsf.
Version 0.9.2, released June 27, 2022.
Lee et al. (2019a)
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova.
Latent retrieval for weakly supervised open domain question answering.
In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy, July 2019a. Association for Computational Linguistics.
doi: 10.18653/v1/P19-1612.
URLhttps://aclanthology.org/P19-1612/.
Lee et al. (2019b)
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova.
Latent retrieval for weakly supervised open domain question answering.
In Proceedings of ACL, 2019b.
Li et al. (2023)
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen.
Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13067–13075, 2023.
Ling et al. (2017)
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom.
Program induction by rationale generation: Learning to solve and explain algebraic word problems.
arXiv preprint arXiv:1705.04146, 2017.
Littwin et al. (2024)
Etai Littwin, Omid Saremi, Madhu Advani, Vimal Thilak, Preetum Nakkiran, Chen Huang, and Joshua Susskind.
How jepa avoids noisy features: The implicit bias of deep linear self distillation networks.
Advances in Neural Information Processing Systems, 37:91300–91336, 2024.
Locascio et al. (2016)
Nicholas Locascio, Karthik Narasimhan, Eduardo DeLeon, Nate Kushman, and Regina Barzilay.
Neural generation of regular expressions from natural language with minimal domain knowledge.
In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1918–1923, 2016.
McCann et al. (2018)
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher.
The natural language decathlon: Multitask learning as question answering.
In Proceedings of ICLR, 2018.
Mehta et al. (2024)
Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, et al.
Openelm: An efficient language model family with open training and inference framework.
arXiv preprint arXiv:2404.14619, 2024.
Ni et al. (2021)
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang.
Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.
arXiv preprint arXiv:2108.08877, 2021.
Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision.
In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
URLhttps://arxiv.org/abs/2103.00020.
Raffel et al. (2020)
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.
Exploring the limits of transfer learning with a unified text-to-text transformer.
In Proceedings of JMLR, 2020.
Reimers and Gurevych (2019)
Nils Reimers and Iryna Gurevych.
Sentence-bert: Sentence embeddings using siamese bert-networks.
arXiv preprint arXiv:1908.10084, 2019.
Team et al. (2024)
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al.
Gemma 2: Improving open language models at a practical size.
arXiv preprint arXiv:2408.00118, 2024.
Tian et al. (2020)
Haoye Tian, Kui Liu, Abdoul Kader Kaboré, Anil Koyuncu, Li Li, Jacques Klein, and Tegawendé F Bissyandé.
Evaluating representation learning of code changes for predicting patch correctness in program repair.
In Proceedings of the 35th IEEE/ACM international conference on automated software engineering, pages 981–992, 2020.
van den Oord et al. (2018)
Aaron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding.
arXiv preprint arXiv:1807.03748, 2018.
Wang et al. (2019)
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson.
Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers.
arXiv preprint arXiv:1911.04942, 2019.
Wang and Sun (2025)
Boshi Wang and Huan Sun.
Is the reversal curse a binding problem? uncovering limitations of transformers from a basic generalization failure.
arXiv preprint arXiv:2504.01928, 2025.
Wei et al. (2022)
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.
Finetuned language models are zero-shot learners.
In Proceedings of ICLR, 2022.
Ye et al. (2020)
Xi Ye, Qiaochu Chen, Xinyu Wang, Isil Dillig, and Greg Durrett.
Sketch-driven regular expression generation from natural language and examples.
Transactions of the Association for Computational Linguistics, 8:679–694, 2020.
Yu et al. (2018)
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al.
Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.
In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
Zellers et al. (2019)
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.
Hellaswag: Can a machine really finish your sentence?
In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zhong et al. (2018)
Zexuan Zhong, Jiaqi Guo, Wei Yang, Jian Peng, Tao Xie, Jian-Guang Lou, Ting Liu, and Dongmei Zhang.
Semregex: A semantics-based approach for generating regular expressions from natural language specifications.
In Proceedings of the 2018 conference on empirical methods in natural language processing, 2018.