Literature
Display
← All papers

Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA

Hai Huang · Yann LeCun · Randall Balestriero

Preprint posted · arXiv (preprint) ↗

Language models7 min text · CondensedOriginal paper ↗
Contents

Overview

Intuition: adding a phrase to a sentence changes its meaning in context. Semantic Tube Prediction encourages hidden-state changes across related spans to point in consistent directions, while retaining next-token training. It aims to improve data efficiency without requiring the explicitly paired views used by LLM-JEPA. The geometric and stochastic descriptions are modeling hypotheses; the measured sixteen-fold data comparison is a specific regular-expression experiment with rescaled training epochs.

Refer to caption
(a) Semantic Tube
Original figure
(b) Data Efficiency
Figure 1: Semantic Tube improves data efficiency. (a) We hypothesize that error-free hidden state trajectories are geodesics, which are locally linear and approximated by the Semantic Tube. The dotted line depicts a trajectory distorted by training loss. Deviations perpendicular to the tube constitute noise, while the component along the geodesic represents the signal. (b) With our approach (ℒNTP+ℒSTP\mathcal{L}_{\rm NTP}+\mathcal{L}_{\rm STP}), accuracy shows a negligible drop when the training dataset is halved, and it matches full-dataset standard fine-tuning (ℒNTP\mathcal{L}_{\rm NTP}) accuracy using only 116\frac{1}{16} of the training data. In contrast, ℒNTP\mathcal{L}_{\rm NTP} degrades significantly when the dataset is halved.

At a glance

STP regularizes the trajectory of language-model hidden states using a local straightness assumption. The ordinary language loss remains essential for generation. The study reports better accuracy and a large reduction in distinct training examples on a synthetic regular-expression task, plus experiments on other datasets and model families. Its results should be read as changing a particular training recipe, not disproving an immutable law of all language-model scaling.

1 Introduction

Scaling curves describe measured behavior under a training setup. The paper asks whether a more informative objective can move that curve. Its proposed geometric prior encourages related hidden-state increments to align within a semantic tube. This creates three separate claims to assess: the plausibility of the representation model, the behavior of the implemented regularizer, and the observed accuracy–data tradeoff. The last does not automatically prove the first.

Equation
ℒ=ℒNTP+λ⋅ℒSTP\mathcal{L}=\mathcal{L}_{\rm NTP}+\lambda\cdot\mathcal{L}_{\rm STP}

2 Training and Inference Dynamics

The theory describes training trajectories and autoregressive inference using differential-equation analogies. These provide intuition for accumulated error, but their assumptions need to remain visible when connecting them to finite trained language models.

2.1 Training ODE

The construction rewrites token-conditioned hidden-state evolution as a continuous-time update after introducing a lifting function and related operators. This makes local direction and trajectory language available for the theory. It is a mathematical representation of the model under the stated construction, not evidence that an actual tokenizer produces a smooth semantic manifold.

Equations 1–2
xt+1\displaystyle x_{t+1} =ů∘f̊​(x≤t)\displaystyle=\mathring{u}\circ\mathring{f}(x_{\leq t}) (1)
ht\displaystyle h_{t} =f̊​(x≤t)+ϵt\displaystyle=\mathring{f}(x_{\leq t})+\epsilon_{t} (2)
x≤t+1⊖x≤t=ů∘f̊​(x≤t).x_{\leq t+1}\ominus x_{\leq t}=\mathring{u}\circ\mathring{f}(x_{\leq t}).
Proposition 2.1 (Training ODE).

The LLM training process can be modeled as a solution in the token sequence space ℝT×dmodel\mathbb{R}^{T\times d_{\rm model}} to the ODE:

Equation
d​x≤t=ů∘f̊​(x≤t)​d​t.dx_{\leq t}=\mathring{u}\circ\mathring{f}(x_{\leq t})dt.

2.2 Mode Collapse at Inference Time

At inference, generated tokens feed back into future predictions, so errors can propagate differently from teacher-forced training. The paper models that accumulation as a stochastic perturbation and uses trajectory collisions to motivate loss of output diversity. The Gaussian and infinite-width arguments are approximations; they do not establish that every finite trained model follows the stated stochastic process.

Equation 3
ht∗=ht−ϵt=f̊​(x≤t)h^{\ast}_{t}=h_{t}-\epsilon_{t}=\mathring{f}(x_{\leq t}) (3)
Refer to caption
Figure 2: Two hidden state trajectories with similar prefixes pass through the Voronoi cell of the “researcher” token at different locations, leading to different next hidden states and hence different next tokens. Since ℒNTP\mathcal{L}_{\rm NTP} cannot guarantee that hth_{t} converges to ht∗h^{\ast}_{t} (optimal hidden state), hth_{t} can be misplaced on another geodesic. This leads to mode collapse (the red dotted line mistakenly continues the generation, misattributing Hinton’s Nobel Prize to an arbitrary person, or if the error deviates in the opposite direction and precludes a winner).

3 Semantic Tube Prediction

STP turns the geometric motivation into an auxiliary training objective. The practical objective and its empirical ablations can be evaluated even when the strongest geometric interpretation remains unverified.

3.1 Semantic Tube

The objective compares contextual hidden-state differences for sampled spans and encourages compatible directions. Its tube interpretation permits a neighborhood around a proposed semantic path rather than demanding identical token sequences. The source’s lemma and theorem are retained with their assumptions. A small regularization weight matters because exact straightness need not hold for real language, and next-token prediction still supplies the generative objective.

Definition 3.1 (Local Linearity).

A time-indexed trajectory h∗h^{\ast} is defined as locally linear if ∃τ,∃ε\exists\tau,\exists\varepsilon such that for any time indices s<r<ts<r<t satisfying |t−s|≤τ|t-s|\leq\tau, we have:

Equation 4
‖(hr∗−hs∗)⟂ht∗−hs∗‖2≤ε\|(h^{\ast}_{r}-h^{\ast}_{s})_{\perp h^{\ast}_{t}-h^{\ast}_{s}}\|_{2}\leq\varepsilon (4)

where x⟂yx_{\perp y} denotes the component of vector xx that is perpendicular to vector yy.

Lemma 3.2 (Straightening Lemma).

If hs=hs∗h_{s}=h^{\ast}_{s}, ht=ht∗h_{t}=h^{\ast}_{t}, and ℒSTP≤ϵ\mathcal{L}_{\rm STP}\leq\epsilon for all rr satisfying s<r<ts<r<t, then

Equation
‖(hr−hs)⟂ht∗−hs∗‖2≤2​ϵ​‖hr−hs‖2.\|(h_{r}-h_{s})_{\perp h^{\ast}_{t}-h^{\ast}_{s}}\|_{2}\leq\sqrt{2\epsilon}\|h_{r}-h_{s}\|_{2}.
Theorem 3.3 (Semantic Tube).

If h∗h^{\ast} is locally linear and for all rr satisfying 0≤s<r<t≤τ0\leq s<r<t\leq\tau, ℒSTP→0\mathcal{L}_{\rm STP}\rightarrow 0, then

Equation
‖hr−h∗‖2≲ε\|h_{r}-h^{\ast}\|_{2}\lesssim\varepsilon
Corollary 3.4 (Random Tube).

For randomly selected s<r<ts<r<t, if ℒSTP→0\mathcal{L}_{\rm STP}\rightarrow 0, then for any ϵ\epsilon,

Equation
P⁡(‖hr−h∗‖2>ε+ϵ)→0P(\|h_{r}-h^{\ast}\|_{2}>\varepsilon+\epsilon)\rightarrow 0

3.2 Practical Considerations

The implementation chooses span boundaries and reuses hidden states already computed for ordinary training. This avoids a separate predictor or explicit pair of independently encoded views. Low extra arithmetic is a property of this implementation; it should be distinguished from the additional passes required by the earlier LLM-JEPA method.

3.3 Related Work

The related-work discussion connects manifold hypotheses, local linearity, representation learning and language objectives. These connections motivate the construction but do not make its assumptions consequences of all earlier theories. The important comparison is what information each objective requires and which aspect of a representation it constrains.

Refer to caption
Figure 3: When the sentence aligns on a geodesic, the concept direction naturally aligns.

4 Experiments

Experiments test loss behavior, accuracy, data requirements, representation diversity and design sensitivity. Those measurements provide complementary evidence. Improvements in several proxies should not be treated as direct measurements of an otherwise unobserved semantic signal-to-noise ratio.

4.1 Loss Landscape

The monitored objectives show that optimizing the generative term does not automatically optimize the geometric term. Increasing regularization smooths the measured trajectory statistic over the tested range while token loss changes less. The strongest reduction in that statistic need not coincide with the best accuracy, so smoothness is a design target rather than a complete performance metric.

Original figure
(a) Loss curve
Original figure
(b) Loss vs. λ\lambda
Figure 4: Loss landscape. (a) When ℒNTP\mathcal{L}_{\rm NTP} plateaus, ℒSTP\mathcal{L}_{\rm STP} continues to decrease. Furthermore, minimizing ℒNTP\mathcal{L}_{\rm NTP} does not automatically minimize ℒSTP\mathcal{L}_{\rm STP}. (b) Across a wide range of λ\lambda, increasing λ\lambda on a logarithmic scale reduces ℒSTP\mathcal{L}_{\rm STP} linearly, while ℒNTP\mathcal{L}_{\rm NTP} remains unchanged.

4.2 Better Accuracy

The comparisons cover regular-expression generation, question answering, mathematical problems, SQL and multiple-choice completion, with several model families and sizes. Accuracy improves across the reported experiments. Their evaluation rules differ by task, and successful adaptation of a pretrained model should not be described as evidence of broad pretraining from scratch.

Original figure
(a) Datasets
Original figure
(b) Model families
Original figure
(c) Model sizes
Figure 5: Semantic Tube (ℒNTP+ℒSTP\mathcal{L}_{\rm NTP}+\mathcal{L}_{\rm STP}, our approach) demonstrates superior performance across (a) datasets, (b) model families, and (c) model sizes compared to regular fine-tuning (ℒNTP\mathcal{L}_{\rm NTP}) and LLM-JEPA (ℒNTP+ℒJEPA\mathcal{L}_{\rm NTP}+\mathcal{L}_{\rm JEPA}).

4.3 Data Efficiency

The headline data-efficiency experiment reduces the number of distinct NL-RX-SYNTH examples while increasing training epochs proportionally. STP can match a full-data baseline using one-sixteenth of the examples in the reported setting. Because update exposure is compensated, this is not a sixteen-fold reduction in training compute. The result is a benchmark-specific shift in a training curve, not a contradiction of every empirical scaling fit.

4.4 Preserving Diversity

The diversity analysis examines two functionally equivalent regular-expression endings with different frequencies. STP retains both patterns better in this test, and normalized versus unnormalized singular spectra suggest different directional and magnitude structure. These are concrete dataset preferences and representation diagnostics; they do not certify preservation of all useful linguistic diversity.

Table 1: Accuracy on functionally equivalent regular expression suffixes “.*” and “.*.*”. Semantic Tube effectively captures nuanced preferences, whereas LLM-JEPA exhibits mode collapse by biasing towards “.*”, which is 35×\times more prevalent in the training set than “.*.*”.
Suffix Semantic Tube Regular LLM-JEPA
.* 88.5% 29.9% 68.9%
.*.* 68.0% 28.0% 32.0%
Original figure
(a) Without Normalization
Original figure
(b) With Normalization
Figure 6: SVD decomposition demonstrating Semantic Tube’s polymorphism. (a) Without normalization, the SVD profile closely resembles regular fine-tuning. (b) With normalization, the SVD aligns with LLM-JEPA. Collectively, this indicates that Semantic Tube enforces a simple structure on the directions (normalized vectors) mapping Text{\rm Text} to Code{\rm Code}, while tolerating complexity in the unnormalized vectors. Note that the relative relationships among the base model, regular fine-tuning, and LLM-JEPA remain unchanged with or without normalization.

4.5 Tuning λ

Accuracy varies non-monotonically with the auxiliary-loss weight. A useful range appears across the tested configurations, while excessive regularization can hurt. This supports tuning a modest geometric constraint alongside the token objective rather than trying to enforce perfect straightness.

Original figure
Figure 7: Impact of λ\lambda tuning on Llama-3 1B across various datasets. In most cases, peak performance is achieved within the range of 0.010.01 to 0.080.08.
Table 2: Optimal λ\lambda values yielding maximum accuracy.
SYNTH TURK GSM8K Spider NQ HS
0.02 0.04 0.005 0.04 0.16 0.02
Gemma2 Qwen3 R1 Dist OLMo OpenELM
0.005 0.02 0.04 0.01 0.04
Llama3 3B Llama3 8B
0.01 0.0025

4.6 Ablation

Ablations change span boundaries, add a predictor, use explicit paired views, mask spans and compare curvature objectives. The original formulation performs best among the tested alternatives. An auxiliary linear projector can degrade performance, illustrating that an added predictor is not automatically beneficial in a regularizer that reuses the model’s own hidden states.

Original figure
Figure 8: Ablation study. Semantic Tube (our approach) outperforms all variations. Within the Semantic Tube family, alternative configurations consistently degrade performance.

5 Conclusion

The empirical contribution is a low-overhead auxiliary loss with promising task-specific data efficiency. The broad statements about least action, universal semantic geometry and scaling require more than these experiments establish. A useful engineering reading separates the implementable loss and controlled results from the proposed explanation of why they work.

Impact Statement

The source gives a general impact statement without a task-specific risk analysis. It should not be read as evidence that changing a language-model objective removes the usual downstream reliability or misuse concerns.

Appendix A Training ODE

The appendix derives the continuous-time training representation by defining a lifted update and an operator that reproduces the token-sequence evolution. Its operators and domain assumptions are part of the construction. A representation in ODE form alone does not validate a smooth semantic interpretation of the resulting states.

Equations
x≤t=[x1,…,xt,0,…,0].x_{\leq t}=[x_{1},...,x_{t},0,...,0].
v⁡(x,t)=[0,…,0,x⏟index​t+1,0,…,0]v(x,t)=[0,...,0,\underbrace{x}_{{\rm index}\ t+1},0,...,0]
x≤t+1=v⁡(u⁡(f⁡(x≤t)),t)x_{\leq t+1}=v(u(f(x_{\leq t})),t)
x≤t+1⊖x≤t=v⁡(xt+1,t)x_{\leq t+1}\ominus x_{\leq t}=v(x_{t+1},t)
x≤t+1⊖x≤t=x≤t+1−x≤tx_{\leq t+1}\ominus x_{\leq t}=x_{\leq t+1}-x_{\leq t}

Appendix B Inference SDE

The inference model treats accumulated generation errors as Gaussian perturbations and motivates a stochastic differential equation using infinite-width results. Those idealizations should not be silently transferred to a finite, fully trained network. The exact proposition is retained so readers can check what assumptions carry the conclusion.

Proposition B.1 (Inference SDE).

The inference process of an LLM can be modeled by an SDE in the token sequence space ℝT×dmodel\mathbb{R}^{T\times d_{\rm model}},

Equation
d​x≤t=ů∘f̊​(x≤t)​d​t+σt​d​Wtdx_{\leq t}=\mathring{u}\circ\mathring{f}(x_{\leq t})dt+\sigma_{t}dW_{t}

Appendix C Context-Aware Hidden State

A hidden-state difference depends on the prefix that precedes a span. The same appended word can change meaning differently after different contexts. This explains why the implementation uses contextual increments rather than an embedding of the span in isolation.

Refer to caption
Figure 9: The same token v→F​r​a​n​c​e\vec{v}_{France} directs the geodesic along different concept directions when appended to distinct prefixes, illustrating the necessity of the context-aware state difference ht−hsh_{t}-h_{s}.

Appendix D Proof of the Straightening Lemma

The straightening argument relates local geometric quantities under small-angle assumptions. The original formal statement and proof equations are preserved. Its approximation regime matters: a local calculation does not establish that long natural-language trajectories are globally straight.

Equation
‖(hr−hs)⟂ht∗−hs∗‖2≤2​ϵ​‖hr−hs‖2.\|(h_{r}-h_{s})_{\perp h^{\ast}_{t}-h^{\ast}_{s}}\|_{2}\leq\sqrt{2\epsilon}\|h_{r}-h_{s}\|_{2}.
Refer to caption
Figure 10: Geometric illustration for the proof of Lemma 3.2
Equations
‖(hr−hs)⟂ht∗−hs∗‖2=‖hr−hs‖2⋅sin⁡θ\|(h_{r}-h_{s})_{\perp h^{\ast}_{t}-h^{\ast}_{s}}\|_{2}=\|h_{r}-h_{s}\|_{2}\cdot\sin\theta
‖(hr−hs)⟂ht∗−hs∗‖2≤‖hr−hs‖2⋅sin⁡θ′\|(h_{r}-h_{s})_{\perp h^{\ast}_{t}-h^{\ast}_{s}}\|_{2}\leq\|h_{r}-h_{s}\|_{2}\cdot\sin\theta^{\prime}
ℒSTP=1−cos⁡θ′≤ϵ\mathcal{L}_{\rm STP}=1-\cos\theta^{\prime}\leq\epsilon
θ′22≲ϵ\frac{\theta^{\prime 2}}{2}\lesssim\epsilon
θ′≲2​ϵ\theta^{\prime}\lesssim\sqrt{2\epsilon}
‖(hr−hs)⟂ht∗−hs∗‖2≤‖hr−hs‖2⋅sin⁡θ′≲2​ϵ​‖hr−hs‖2.□\|(h_{r}-h_{s})_{\perp h^{\ast}_{t}-h^{\ast}_{s}}\|_{2}\leq\|h_{r}-h_{s}\|_{2}\cdot\sin\theta^{\prime}\lesssim\sqrt{2\epsilon}\|h_{r}-h_{s}\|_{2}.\quad\quad\quad\square

Appendix E Proof of the Semantic Tube Theorem

The semantic-tube proof introduces auxiliary boundary tokens so the required endpoint conditions hold in the construction. Those boundary conditions are substantive assumptions. The theorem should be interpreted within that augmented mathematical setup rather than as an unconditional statement about arbitrary prompts.

Appendix F Inference Cone

The inference-cone argument describes how a modeled random perturbation grows along a generated trajectory. Its Gaussian-process and tangent-kernel approximations restrict its scope. In particular, decreasing a bound involving a failure probability is not by itself a direct measurement or guarantee of lower realized mode-collapse probability in every finite model.

Refer to caption
Figure 11: The inference cone defines the probabilistic range of a Brownian motion, and its radius grows ∝σt​t\propto\sigma_{t}\sqrt{t}. A larger σt\sigma_{t} leads to a wider cone, which has a high probability of colliding with a token sequence trace that is far away (blue cone and green geodesic), while a smaller σt\sigma_{t} leads to a narrower cone that may only collide with a nearby trace (yellow cone and red geodesic). The dotted red and green fine lines are the Brownian motions confined by the yellow and blue cones, respectively.
Proposition F.1 (Inference Cone).

The distortion between hth_{t} and ht∗h^{\ast}_{t} behaves as a Gaussian process, where the scale of the deviation grows as ‖ht−ht∗‖2∝σ​t\|h_{t}-h^{\ast}_{t}\|_{2}\propto\sigma\sqrt{t}

Equations
ht−ht∗=∑s≤tϵsh_{t}-h^{\ast}_{t}\ =\sum_{s\leq t}\epsilon_{s}
‖∑s≤tϵs‖2∝σ​t.\left\|\sum_{s\leq t}\epsilon_{s}\right\|_{2}\propto\sigma\sqrt{t}.
P⁡(‖h∗−g∗‖2>c⋅σ​t)≤εP(\|h^{\ast}-g^{\ast}\|_{2}>c\cdot\sigma\sqrt{t})\leq\varepsilon

Appendix G Implementation Details

Implementation details explain how to sample boundaries, account for context and skip selected prompt or answer-choice regions. Using randomly selected boundaries differs from always fixing a question–answer split. The practical advice to use a small auxiliary weight reflects the possibility that real semantic paths are curved.

Equation
ℒSTP=1−cos⁡(ht−hr′,hr−hs).\mathcal{L}_{\rm STP}=1-\cos(h_{t}-h_{r}^{\prime},h_{r}-h_{s}).

Appendix H Signal-to-Noise Ratio

This appendix relates conditional information, a Gaussian-channel approximation and data/accuracy bounds. The semantic signal and nuisance noise are not directly observed in the experiments. Consequently, higher measured accuracy or better sample efficiency does not uniquely identify an increased internal SNR; other changes can affect those outcomes. The exact assumptions and inequalities remain available for inspection.

Lemma H.1 (Data Efficiency).
Equation 5
H⁡(Y|Xm)≥H⁡(Y)−m⋅I⁡(Y,X)H(Y|X^{m})\geq H(Y)-m\cdot I(Y;X) (5)
Equations
H⁡(Y|Xm)≥H⁡(Y)−m​I​(X,Y)H(Y|X^{m})\geq H(Y)-mI(X;Y)
H⁡(Y|Xm)=H⁡(Y)−I⁡(Y,Xm)H(Y|X^{m})=H(Y)-I(Y;X^{m})
I⁡(Y,Xm)=H⁡(Xm)−H⁡(Xm|Y)I(Y;X^{m})=H(X^{m})-H(X^{m}|Y)
H⁡(Xm|Y)=∑i=1mH⁡(Xi|Y)H(X^{m}|Y)=\sum_{i=1}^{m}H(X_{i}|Y)
H⁡(Xm)≤∑i=1mH⁡(Xi)H(X^{m})\leq\sum_{i=1}^{m}H(X_{i})
I⁡(Y,Xm)≤∑i=1mH⁡(Xi)−∑i=1mH⁡(Xi|Y)I(Y;X^{m})\leq\sum_{i=1}^{m}H(X_{i})-\sum_{i=1}^{m}H(X_{i}|Y)
I⁡(Y,Xm)≤∑i=1m(H⁡(Xi)−H⁡(Xi|Y))I(Y;X^{m})\leq\sum_{i=1}^{m}\left(H(X_{i})-H(X_{i}|Y)\right)
I⁡(Y,Xm)≤∑i=1mI⁡(Y,Xi)I(Y;X^{m})\leq\sum_{i=1}^{m}I(Y;X_{i})
I⁡(Y,Xm)≤m⋅I⁡(Y,X)I(Y;X^{m})\leq m\cdot I(Y;X)
H⁡(Y|Xm)=H⁡(Y)−I⁡(Y,Xm)≥H⁡(Y)−m⋅I⁡(Y,X)H(Y|X^{m})=H(Y)-I(Y;X^{m})\geq H(Y)-m\cdot I(Y;X)
ϵ≥H⁡(Y|Xm)≥H⁡(Y)−m⋅I⁡(Y,X)\epsilon\geq H(Y|X^{m})\geq H(Y)-m\cdot I(Y;X)
SNR=𝔼⁡[‖Z‖2]𝔼⁡[‖N‖2]{\rm SNR}=\frac{\mathbb{E}[\|Z\|^{2}]}{\mathbb{E}[\|N\|^{2}]}
I⁡(X,Y)=12​log⁡(1+SNR)I(X;Y)=\frac{1}{2}\log(1+\rm{SNR})
Corollary H.2 (Signal-to-Noise Ratio).
Equation 6
m≥H⁡(Y)−ϵ12​log⁡(1+SNR)m\geq\frac{H(Y)-\epsilon}{\frac{1}{2}\log(1+{\rm SNR})} (6)
Equations 7
H⁡(Y|Xm)≤Hb​(Pe)+Pe​log⁡(|𝒱|−1)H(Y|X^{m})\leq H_{b}(P_{e})+P_{e}\log(|\mathcal{V}|-1)
H⁡(Y|Xm)≤Pe​log⁡(|𝒱|−1)H(Y|X^{m})\leq P_{e}\log(|\mathcal{V}|-1) (7)
Corollary H.3 (Accuracy).
Equation 8
Pe≳H⁡(Y)−m⋅12​log⁡(1+SNR)log⁡|𝒱|P_{e}\gtrsim\frac{H(Y)-m\cdot\frac{1}{2}\log(1+{\rm SNR})}{\log|\mathcal{V}|} (8)

Appendix I Data Efficiency

Additional data-subset curves repeat the analysis at larger tested model sizes. As in the main experiment, reducing distinct examples is paired with an altered epoch schedule. Keep the data axis, number of updates and model size separate when interpreting the comparison.

Original figure
(a) Llama3 3B
Original figure
(b) Llama3 8B
Figure 12: Semantic Tube (our approach) and regular fine-tuning with 12\frac{1}{2}, 14\frac{1}{4}, 18\frac{1}{8}, 116\frac{1}{16}, and 132\frac{1}{32} dataset on (a) Llama3 3B and (b) Llama3 8B.

Appendix J Regular Expression Samples

The examples show two equivalent regular-expression endings used to probe preservation of dataset-specific variation. The table makes the preference distinction explicit but is not an evaluation of unrestricted linguistic creativity or diversity.

Table 3: Regular expression samples from the SYNTH dataset that end with either “.*” or “.*.*”, which are functionally equivalent.
Regular Expressions
.*([a-z])||([AEIOUaeiou])||([A-Za-z]).*
.*([A-Za-z]).*([0-9]).*.*
((dog)(.*)).*([AEIOUaeiou]).*
(dog).*((truck)||([A-Z])||([0-9])).*
.*(.)&([0-9])&(dog).*
.*(dog).*((.)*).*.*
.*dog.*[a-z].*.*

Appendix K Tuning λ

The extended loss-weight sweeps cover the alternative formulations. Some settings show sharp accuracy declines and higher run-to-run variation beyond the useful range. These plots expose sensitivity that an isolated best-number comparison would hide.

Original figure
(a) Semantic Tube
Original figure
(b) Two Views
Original figure
(c) Mask
Figure 13: Tuning λ\lambda for various configurations of (a) Semantic Tube, (b) Two Views, and (c) Mask. In all cases, the accuracy vs. λ\lambda curve is concave. We also observe that when λ\lambda exceeds the optimal value, accuracy declines rapidly while the standard deviation increases sharply, indicating that λ≪1\lambda\ll 1 is preferred.

Appendix L Ablation

The complete ablations distinguish geometric span matching, explicit two-view supervision, masked-span recovery and curvature penalties. They test predictor additions, instruction handling and boundary choices. Keeping the alternatives separate helps identify what information each variant uses; the statistical comparisons apply to the tested configurations rather than every possible implementation.

Table 4: Pairwise pp-values comparing variation families. A cell is populated only if the mean accuracy of the row method exceeds that of the column method. pp-values are computed using a paired, one-tailed tt-test, restricted to the best-performing variant from each family.
Two View Mask Curvature
LLM-JEPA2 1.14e-3 1.77e-3 3.04e-5
Two View 4.76e-3 5.10e-5
Mask 1.28e-4
Table 5: Pairwise pp-values comparing options within each variation family. A cell is populated only if the mean accuracy of the row option exceeds that of the column option. pp-values are computed using a paired, one-tailed tt-test. Values exceeding 0.050.05 are struck through.
Zero Pred Inst 2View Pred Mean
LLM-JEPA2 0.0534 2.03e-3 2.34e-4 2View+Warmup 0.265 0.0426 9.16e-6
+Zero 0.0185 5.56e-4 2View 0.0689 1.19e-6
+Pred 2.97e-4 +Pred 2.78e-4
Inst,Recov Inst Mask Signed
Mask+all 0.0629 0.0159 4.02e-3
-Pred 2.34e-3 1.18e-3 Curvature 0.0368
-Recov,Pred 0.0103
Concept demonstration

Open image ↗