Literature
Display
← All papers

Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Models

Peipei Li · Dongsen Zhang · Yuchen Liu · Wenjun Xu

3 min · CondensedOriginal paper ↗
Contents

At a glance

DSEI compresses both prompts and outputs into semantic segments and predicts in latent space. Its DSAE encoder combines static and task-dependent features. On English Wanjuan data with OPT backbones, the authors report up to 48% lower perplexity than sentence-level latent inference. The experiments measure language reconstruction and throughput; they do not establish broad reasoning improvements.

1 Introduction

Token-by-token generation incurs long sequences and repeated computation. Compressing only intermediate reasoning leaves input and output costs intact. DSEI instead compresses the full sequence, balancing shorter latent sequences against the information lost when a long sentence becomes one vector.

Related references: [13], [33], [30], [1], [14].

Refer to caption
Figure 1: Comparison of token-level inference, fixed sentence-level latent inference, and the proposed dynamic segment-level latent inference. DSEI segments sentences according to token length and extracts compact semantic representations based on token-level importance, reducing both input and output sequence lengths while maintaining semantic fidelity.

2 Related Work

Prior work explores semantic compression and reasoning in continuous representations.

2.1 Semantic compression

Semantic autoencoders can replace token sequences with compact vectors, but fixed sentence-level compression struggles with long or information-rich sentences. DSAE adapts segment count to sentence length and combines a static feature with attention-weighted dynamic features.

Related references: [27], [9], [25], [26], [16], [10], [17], [19], [1], [22], [11], [40], [29], [37], [21], [43].

2.2 Latent-space inference

Latent reasoning methods often focus on internal chains of thought. DSEI extends compression across input and output and adds explicit segment-boundary and stopping predictions so generated representations can be decoded into text.

Related references: [7], [38], [12], [32], [3], [28], [4], [6], [31], [23], [24], [41], [8], [2], [5], [39], [15], [13], [33], [30], [36], [35].

3 Method

The system first trains a semantic autoencoder, then trains a latent autoregressive model with that encoder and decoder. Figure 2 shows both stages.

Refer to caption
Figure 2: Overall architecture of the proposed DSEI framework. The input text is first split into sentences, and each sentence is partitioned into one or more segments according to token length. The DSAE encoder extracts segment-level latent representations through token-level semantic weighting and gated fusion. These latent representations are then processed by the DSEI-LLM for latent-space inference, and the generated latent outputs are reconstructed into natural language by the DSAE decoder.

3.1 Semantic Autoencoder

Sentences shorter than 15 tokens use one segment; those with 15–30 tokens use two; longer sentences use three. A learned semantic probe weights token features. A sigmoid gate combines the weighted dynamic feature and static feature before normalization. The decoder reconstructs each sentence with cross-attention to its segment vectors and a focal reconstruction loss. Encoder and decoder attention weights initialize from early and late OPT layers, respectively.

Related references: [34], [18].

Equations 1–5
𝐡k={hi|⌊k⋅lK⌋≤i<⌊(k+1)⋅lK⌋},k=0,1,…,K−1,\displaystyle\mathbf{h}^{k}=\left\{h_{i}\;\middle|\;\left\lfloor\frac{k\cdot l}{K}\right\rfloor\leq i<\left\lfloor\frac{(k+1)\cdot l}{K}\right\rfloor\right\},\quad k=0,1,\dots,K-1, (1)
sk,a=LayerNorm⁡(∑𝐡k)s_{k,a}=\mathrm{LayerNorm}\left(\sum\mathbf{h}^{k}\right) (2)
sk,b=LayerNorm⁡(softmax⁡(linear⁡(𝐡k))​𝐡k⊤)s_{k,b}=\mathrm{LayerNorm}\left(\mathrm{softmax}(\mathrm{linear}(\mathbf{h}^{k}))\mathbf{h}^{k\top}\right) (3)
sk=sigmoid⁡(wg​a​t​e)⋅sk,a+(1−sigmoid⁡(wg​a​t​e))⋅sk,bs_{k}=\mathrm{sigmoid}(w_{gate})\cdot s_{k,a}+(1-\mathrm{sigmoid}(w_{gate}))\cdot s_{k,b} (4)
ℒcomp=∑i=1yFL(pi)=∑i=1y−(1−pi)2log(pi)\mathcal{L}_{\mathrm{comp}}=\sum_{i=1}^{y}\mathrm{FL}(p_{i})=\sum_{i=1}^{y}-(1-p_{i})^{2}\mathrm{log}(p_{i}) (5)

3.2 Segment-by-segment inference

The language model predicts the next segment representation rather than the next token. A boundary head groups generated segments into sentences, and a stopping head runs at sentence boundaries. Training jointly optimizes token reconstruction, segment boundaries, and termination, with auxiliary weights of 0.01. This keeps the complete input–output process in the compressed representation until decoding.

Related references: [18].

Equation 6
ℒinfer=λj​∑i=1tFL⁡(pi)⏟generation​term+∑i=1tFL⁡(bi)⏟segment​term+∑t∈ℬFL⁡(di)⏟stop​term,\displaystyle\mathcal{L}_{\mathrm{infer}}=\underbrace{\frac{\lambda}{j}\sum_{i=1}^{t}\mathrm{FL}(p_{i})}_{\mathrm{generation\;term}}+\underbrace{\sum_{i=1}^{t}\mathrm{FL}({b}_{i})}_{\mathrm{segment\;term}}+\underbrace{\sum_{t\in\mathcal{B}}\mathrm{FL}(d_{i})}_{\mathrm{stop\;term}}, (6)

4 Experiments

Experiments compare semantic reconstruction, latent inference, initialization choices, and compression components.

4.1 Experimental setup

Training uses the English portion of Wanjuan 1.0: 155 million sentences for DSAE and 6.4 million paragraphs for DSEI, with 1,000 validation examples each. Backbones are OPT-125M, OPT-350M, and OPT-1.3B. Runs use an RTX 5880 Ada, AdamW, gradient clipping, and 5,000 warmup steps. Sentence and paragraph stages use different batch sizes and learning rates; results are specific to this setup.

Related references: [14], [1], [42], [20].

4.2 Semantic extraction results

DSAE reduces perplexity by up to 34% relative to SVAE. Removing dynamic segmentation raises perplexity by about 32%; replacing the gate with a simple sum raises it by 14%. Random initialization raises it by 17%. Attention maps and decoded examples illustrate retained semantics, while Table 2 isolates the contribution of each design choice.

Refer to caption
(a) PPL for different layers with 768 hidden size.
Refer to caption
(b) PPL for different hidden sizes with 1 layer.
Figure 3: Comparison of PPL for semantic extraction between DSAE and baseline method across varying (a) layers and (b) hidden sizes. △\bigtriangleup denotes percentage decrease in PPL of DSAE compared to the baseline method.
Table 1: Test examples of sentence reconstruction between baseline method and DSAE. The darker the color, the greater the weight of the token. Text in red denotes mismatches between reconstructed sentence and input sentence, and △\triangle denotes missing output.
Weight of input sentence Output sentence
SVAE DSAE
Some also have performance venues for various
kinds of artists .
Some also have performance venues for various
kinds of artists.
Some also have performance venues for various
kinds of artists.
Intense pulsed light device emit a range
wavelength ( 515 - 1200 ) of light .
Intense pulsed light device emit a range of
puls(660-1200) △\triangle light.
Intense pulsed light device emit a range
wavelength(515-1200) of light.
Makes the perfect gift for Schylling enthusiasts
that are at least 3 years old .
Makes the perfect gift for Schreferably that
enthusiasts at △\triangle 3 years old.
Makes the perfect gift for Schylling enthusiasts
that are at least 3 years old.
His hat is accented with holly and holly berries . His hat is accented with holly holly and berries. His holly accented with holly and holly berries.
Table 2: The ablation experiment results of DSAE. △\bigtriangleup denotes percentage increase in PPL compared to the default setting. “First layer weights” indicates that the initial parameters for both the encoder and the decoder are transferred from the first layer of the base LLM, while “Last layer weights” are derived from the final layer.
Method PPL↓\downarrow △\bigtriangleup
Default 1.06 0.00%
w/o Segmentation 1.40 32.07%
w/o sas_{a} 1.14 7.55%
w/o sbs_{b} 1.16 9.43%
w/o gate 1.21 14.15%
w/o LLM weights 1.24 16.98%
First layer weights 1.13 6.60%
Last layer weights 1.17 10.38%
Refer to caption
(a) Short sentence.
Refer to caption
(b) Middle sentence.
Refer to caption
(c) Long sentence.
Figure 4: Attention heatmap comparison of decoder cross-attention weights on DSAE tokens for (a) short, (b) middle, and (c) long sentences during semantic compression. Lighter colors indicate higher attention weights, signifying that the decoder assigns greater importance to those tokens when reconstructing the original sentence.

4.3 Latent inference results

DSEI reports up to 48% lower perplexity than SLLM and, against token-level OPT, about 2.5× throughput, 90% lower memory, and 58% lower perplexity. These are the paper’s input-throughput and reconstruction measurements. Against SLLM at 125M scale, throughput improves only 0.22% and memory increases 18%; the additional components add roughly 0.6 million parameters. The advantage depends on the comparison and model scale.

Table 3: Experiment results of baseline methods and DSEI. DSEI-b-l denotes an OPT model with parameter size b, integrated with a DSAE consisting of l layers.
Model Parameters(M) PPL↓\downarrow
Mean input throughput
(k tokens/s)↑\uparrow
Mean GPU memory
(KB/token)↓\downarrow
OPT-125M 125.23 26.94 13.93 131.45
SLLM-125M-H1 214.32 21.84 35.90 10.97
DSEI-125M-H1 214.92 11.17 35.98 13.05
SLLM-125M-H2 226.14 21.20 37.49 9.94
DSEI-125M-H2 226.76 15.52 37.76 13.18
SLLM-125M-H4 249.78 20.99 24.36 10.60
DSEI-125M-H4 250.37 14.57 23.50 13.56
OPT-350M 331.19 22.60 7.71 184.72
SLLM-350M-H1 429.46 21.51 99.20 25.21
DSEI-350M-H1 432.66 14.12 98.30 27.85
OPT-1.3B 1315.75 16.02 7.9 285.24
SLLM-1.3B-H1 1605.67 19.30 60.88 50.44
DSEI-1.3B-H1 1609.88 9.86 59.53 52.07

5 Conclusion

Adaptive semantic segments improve the reconstruction–compression tradeoff in the tested models. The evidence is limited to one training corpus and relatively small OPT backbones; downstream task accuracy and larger-scale deployment remain open questions.