Overview
The opening figure compares reported agent performance and global cache size.
At a glance
DeepSeek-V4.1-Flash targets input-heavy, long-context agent workloads. The report describes a 552-billion-parameter multimodal backbone plus 196 billion Engram memory parameters, with 8 billion parameters active per token during prefill and 16 billion during decoding. CED, cross-layer sparse-attention reuse, and four-bit caching reduce global KV storage to 890 bytes per token. Persistent cache storage is reported at about one eighth of DeepSeek-V4-Flash under matched workloads. The model is trained on 45 trillion multimodal tokens and supports contexts up to one million tokens.
1 Introduction
Long-running agents repeatedly process tool outputs and move large attention caches. The design attacks prefill computation, GPU-resident global KV, and persistent cache storage separately. Its compression ratios refer to these different resources rather than total model memory or an unconditional end-to-end speedup.
Related references: [19], [21], [80], [12], [11], [70], [51], [16], [64], [48].
2 Architecture
2.1 Overview
A 20-layer causal encoder feeds a 20-layer decoder. Global cache sharing and sparse index reuse reduce attention costs, while local sliding-window attention remains layer-specific. Images are encoded and projected into the same sequence as text. Conditional memory, residual-stream fusion, and speculative decoding address additional costs.
2.1.1 Multimodal Architecture
The vision encoder produces a spatial feature grid. A 3×3 pixel-unshuffle reduces token count before an MLP projects visual features into the language-model embedding space.
DeepSeek-ViT
DeepSeek-ViT uses two-dimensional rotary positions for variable resolutions, linear patch projection, RMSNorm, and SwiGLU. Spatial downsampling reduces visual tokens ninefold and supports inputs up to roughly 1344×1344 pixels.
Multimodal Auxiliary-loss-free Load Balancing for MoEs
Separate routing-bias corrections balance expert use for image and text tokens. Combining their loads would conceal modality-specific imbalances.
Related references: [74].
2.2 Causal Encoder-Decoder (CED)
Decoder global KV is projected from the final encoder state rather than computed through every decoder layer. Most prompt tokens can therefore skip decoder computation. Decoder-local attention still needs a short replay; for long sequences, the resulting prefill cost approaches half of a full 40-layer pass.
Related references: [67], [9].
Equation 1
| (1) |
2.3 Compressed Sparse Attention 2 (CSA2)
CSA2 compresses along entry size, sequence length, and layer count. Cache sharing and sparse-selection reuse are independent choices. The compressor removes overlapping source windows and projects indexer keys from main KV, simplifying the training path.
2.3.1 Cross-Layer KV and Index Reuse
Full layers build caches and sparse selections. Reindex layers share caches but choose new entries. Reuse layers share both caches and the latest selection. Every layer still computes its own query, local KV, and attention output.
2.3.2 Hierarchical Sparse Indexer
The first decoder indexer searches the whole visible context and chooses a bounded candidate pool. Later indexers search only that pool. This bounds their cost, but the initial full-context scan remains. Post-training uses the same restriction as inference.
Related references: [82].
2.4 Efficient Architectural Extensions
2.4.1 Single-Pass m HC
Single-Pass mHC uses input-mixing coefficients produced by the preceding block. Removing the same-block dependency permits one fused residual-update, mixing, and coefficient-prediction pass, halving the original implementation’s activation-memory traffic.
Related references: [80].
Equations 2–6
| (2) |
| Residual update, contraction over | (3) | |||||
| Coefficients, contraction over | (4) | |||||
| Input mixing, contraction over | (5) |
| (6) |
2.4.2 Engram
Two Engram modules provide 196 billion sparsely accessed memory parameters using hashed token n-grams. Deterministic addresses permit prefetching; the design omits the earlier short convolution and changes the embedding optimizer.
Related references: [13].
2.4.3 DSpark
DSpark drafts five token positions in parallel with a three-block auxiliary model. A confidence-based scheduler chooses verification length for throughput under current load. It is trained after backbone pretraining and kept aligned during post-training without sending its loss gradients into the backbone.
2.4.4 FP4 Main KV Cache
Post-training introduces quantization-aware four-bit main KV storage using small per-group scales, dequantized before attention. The sensitive sliding-window cache remains FP8. Main-cache compression therefore differs from quantizing all attention state.
2.5 Optimization
Head-wise Muon addresses differences between attention heads. Large embedding and prediction matrices use momentum with Sinkhorn balancing to reduce optimizer-state memory.
Basic Configurations.
Linear matrices use Muon, selected non-matrix parameters use AdamW, and embedding-style matrices use Sinkhorn-balanced updates. The vision encoder is initially frozen and unfrozen at learning-rate decay with a smaller learning rate.
Sinkhorn-Balanced Updates for Engram / Embedding / Prediction Head.
The optimizer approximately equalizes row and column RMS of momentum updates, masking numerically negligible rows. A 0.18 scale aligns update magnitudes with the reused learning-rate schedule. The original algorithm and equations specify the procedure.
Related references: [43], [62], [94], [79], [30], [23], [86], [81].
Equation 7
| (7) |
3 General Infrastructures
3.1 Training Infrastructure
3.1.1 Multimodal Training Infrastructure
Communication-Computation Overlap in Contrastive Learning.
Contrastive vision training overlaps gathering one modality’s features with useful forward or backward work for the other, reducing communication stalls.
Equation
End-to-End Parallelism.
A separately replicated vision encoder runs before and after the language-model forward/backward phase. This allows independent image-load balancing while preserving the language model’s parallel schedule.
Long-Sequence Multimodal Training Optimization.
Images are balanced across context-parallel workers and loaded once. Reinforcement-learning rollouts transfer images incrementally and reuse decoded/preprocessed image data.
Equation
3.1.2 Attention Sharing Training for CSA2
Shared attention components crossing pipeline stages use synchronized shadow indexers, extended inter-stage payloads, and per-microbatch lifetime tracking. Ownership of optimization and checkpoints remains explicit.
3.1.3 Engram
Engram tables are row-sharded and prefetched before microbatches; gradients return to owning ranks after backbone backpropagation. FP8 transfers and fused normalization reduce memory traffic. Rollouts keep tables on GPUs to relieve host-memory pressure.
3.2 Inference System
Fused kernels reduce scheduling overhead, while vision encoding, prefill, and decoding can scale independently. The report gives 15 prefill kernels and 11 decode kernels for the common CSA2 Reuse-mode layers.
3.2.1 Persistent KV Cache Management
Persistent storage retains compressed global KV, while local sliding-window state moves to a short-lived host-memory pool. Removing persistent local state and compressing global state combine to the reported eightfold reduction. Local-cache misses are recovered through bounded replay.
3.2.2 SWA Bounded Replay
Bounded replay reconstructs local attention from only the most recent window instead of the full multilayer receptive field. It intentionally approximates the original cached state.
Encoder SWA Bounded Replay.
On an encoder local-cache miss, replay recent prefix tokens while retaining the existing global cache, then process the uncached suffix. Reconstructed states depend on the cache-hit position; the authors report little response-quality loss.
Decoder SWA Bounded Replay.
Decoder replay processes only the final attention window before decoding. The result is not mathematically equivalent to a full decoder pass, and the same approximation is simulated during post-training.
4 Pre-Training
4.1 Data Construction
Text Data Curation
Text curation emphasizes complementary information, expert quality criteria, and newer code. Low-information generated or translated content is filtered as a form of implicit duplication.
Multimodal Data Curation
Multimodal data combines image–text pairs, interleaved documents, and specialist datasets. Staged filtering and deduplication precede expensive image processing; visual grounding, document reading, image-code pairs, and computer-use trajectories fill gaps in web data.
Related references: [47].
Data Integration and Deduplication
Multimodal versions replace overlapping text-only samples. The resulting corpus uses a 7:1 text-only-to-multimodal token ratio, with coordinated sampling and packing to limit duplication and padding.
4.2 Pre-Training Setups
4.2.1 Model Setups
The backbone has 40 layers and width 5120. Encoder caches compress pairs of tokens, while decoder global caches are uncompressed along sequence length and shared across layers. Attention selects 512 entries from a decoder candidate pool of at most 16,384 positions; the local window is 128 tokens. Each MoE layer has one shared expert and activates six of 384 routed experts.
Related references: [54].
4.2.2 Training Setups
Training covers 45 trillion tokens with a fixed 100.6-million-token batch. Sequence length starts at 64K and extends to one million at 34 trillion tokens. Separate modality-routing corrections and sample-level masks are retained.
Vision Encoder Training.
The vision encoder first learns contrastively from about 47 billion image–text pairs, then trains with a temporary 4-billion-parameter language model on 236 billion tokens at higher resolutions. Only the trained vision encoder is retained for the final multimodal backbone.
Related references: [91].
4.3 Evaluations
4.3.1 Evaluation Benchmarks
Base-model evaluation covers knowledge, language understanding, reasoning, coding, mathematics, long context, and visual tasks. The original table records the benchmark-specific metrics.
Related references: [98], [76], [34], [35], [32], [26], [69], [38], [27], [89], [99], [10], [14], [33], [65], [6], [87], [50], [73], [39], [52], [46], [85].
| Benchmark (Metric) | # Shots | DeepSeek-V4-Flash | DeepSeek-V4-Pro | DeepSeek-V4.1-Flash | |
|---|---|---|---|---|---|
| Base | Base | Base | |||
| Architecture | - | MoE | MoE | MoE | |
| # Activated Params | - | 13B | 49B | 8B/16B | |
| # Backbone Params | - | 284B | 1.6T | 552B | |
| World Knowl. | AGIEval (EM) | 3-5-shot | 83.9 | 84.4 | 83.4 |
| MMLU-Pro (EM) | 5-shot | 68.3 | 73.5 | 74.1 | |
| C-Eval (EM) | 5-shot | 92.1 | 93.1 | 92.1 | |
| MultiLoKo (LLM-Judge) | 5-shot | 42.6 | 50.9 | 45.5 | |
| Simple-QA verified (EM) | 25-shot | 30.1 | 55.2 | 42.3 | |
| SuperGPQA (EM) | 5-shot | 46.5 | 53.9 | 53.1 | |
| Lang. & Reas. | BBH (EM) | 3-shot | 86.9 | 87.5 | 86.1 |
| BBEH (EM) | 1-shot | 25.4 | 29.8 | 27.2 | |
| DROP (F1) | 1-shot | 88.6 | 88.7 | 87.9 | |
| HellaSwag (EM) | 0-shot | 85.7 | 88.0 | 87.2 | |
| Code & Math | BigCodeBench (Pass@1) | 3-shot | 56.8 | 59.2 | 60.6 |
| HumanEval (Pass@1) | 0-shot | 69.5 | 76.8 | 79.4 | |
| GSM8K (EM) | 8-shot | 90.8 | 92.6 | 93.0 | |
| MATH (EM) | 4-shot | 57.4 | 64.5 | 61.1 | |
| MGSM (EM) | 8-shot | 85.7 | 84.4 | 80.2 | |
| Long Context | LongBench-V2 (EM) | 1-shot | 44.7 | 51.5 | 45.2 |
| Multimodal | MMMU-Pro (EM) | 4-shot | - | - | 56.5 |
| CVBench (EM) | 4-shot | - | - | 77.9 | |
| DocVQA (LLM-Judge) | 4-shot | - | - | 95.6 | |
| RefCOCO-avg (Acc@0.5) | 0-shot | - | - | 86.0 |
4.3.2 Evaluation Results
The authors report performance comparable to or above earlier DeepSeek base models on many tasks with a smaller active and cache footprint. An additional bits-per-byte comparison uses internal development corpora and is not an independently accessible benchmark.
5 Post-Training
5.1 Post-Training Pipeline
Post-training follows supervised fine-tuning, reinforcement learning, and on-policy distillation. The emphasized contribution is the scale and quality of synthesized tasks and environments, rather than a new optimization algorithm.
5.1.1 Large-Scale Agent Task Synthesis
Tasks combine a problem, executable environment, and verifier. General-agent tasks reconstruct consented workflow feedback; coding tasks use sessions and public repositories. Independent construction, solving, inspection, and repair stages check difficulty, correctness, leakage, and verifier weaknesses.
5.1.2 RL in Synthesized Tasks
Asynchronous learning spans diverse scaffolds. Sandboxes execute tasks while worker containers normalize trajectories for training. Checkpoint merging carries gains between successive reinforcement-learning runs; plotted discontinuities can therefore reflect reinitialization.
5.1.3 Running Agents at Massive Scale: DSec
DSec provides sharded sandbox scheduling, local admission controls, resource isolation, checkpoint resumption, and separate treatment of latency-sensitive evaluations. The report also describes environment misuse during training and the need to harden evaluators.
5.1.4 Controllable Reasoning Effort in RL
An effort scalar changes the reward penalty for reasoning length; higher effort weakens the penalty. Comparisons within training groups use the same effort level. This learns a cost–quality tradeoff rather than enforcing a hard token budget.
Equations 8–10
| (8) |
| (9) |
| (10) |
5.2 Asynchronous Post-training Infrastructure
Asynchronous generation reduces delays from long-running agent trajectories, but requires explicit handling of data age and length imbalance.
5.2.1 Overall Workflow
Sample-level dispatch maintains concurrency while rollout and training time-share devices. Training interrupts generation when enough samples are ready and replays the recorded expert routing across checkpoint boundaries.
5.2.2 Mitigating Length Bias and Off-Policy Effects
Per-dataset concurrency and early-short-sample handling reduce length bias. Dispatch limits and stale-token loss masking address off-policy samples generated by older checkpoints.
5.2.3 Performance Optimization
Token-boundary interruption and persisted cache/routing state let rollouts resume after checkpoint switches or scheduler preemption. Per-sample cleanup releases completed state.
5.2.4 Large-Scale On-Policy Distillation
The final distillation stage uses more than forty domain-specialized teachers with potentially different architectures. The infrastructure supports asynchronous teacher and data-mixture changes while samples remain in flight.
Related references: [21].
5.3 Evaluation
5.3.1 Evaluation Setup
Reasoning and agent benchmarks use task-specific contexts, sampling, and harnesses. Coding environments restrict network access and remove potentially leaking histories or caches, though the authors still observe evaluator gaming. Scores therefore depend on both model and evaluation setup.
Related references: [58], [56], [22], [51], [49], [48], [16], [84], [24], [41], [78], [77], [64], [66], [29], [8], [59].
5.3.2 Evaluation Results
Reported scores include 90.9% on GPQA Diamond, 65.6% on MathArena Apex, 74.2% on DeepSWE v1.1, and 90.6% on Terminal-Bench 2.1. These are the report’s configured evaluations, not universal performance guarantees. Visual-agent results still trail some closed-model comparisons.
| Benchmark (Metric) | Opus-5 | GPT-5.6 Sol | K3 | GLM-5.3 | DS-V4-Pro | DS-V4-Flash | DS-V4.1-Flash | |
| Max | Max | Max | Max | Max | Max | Max | ||
| Reasoning | GPQA Diamond (Pass@1) | 93.4 | 94.1 | 92.9 | 88.1 | 92.4 | 89.9 | 90.9 |
| HLE (Pass@1) | 56.3 | 44.5 | 43.5 | 42.0† | 42.7† | 37.8† | 36.8 (39.1†) | |
| Codeforces (Rating) | - | - | - | - | 3348 | 3289 | 3471 | |
| MathArena Apex (Pass@1) | - | - | 65.6 | - | 65.3 | 58.6 | 65.6 | |
| Agentic | Terminal-Bench 2.1 (Pass@1) | 89.1 | 88.8 | 88.3 | 88.2 | 87.9 | 82.7 | 90.6 |
| Terminal-Bench 3.0 (Pass@1) | 43.3 | 34.4 | 17.7 | 28.3 | 11.8 | 7.6 | 30.0 | |
| Terminal-Bench 4.0 (Pass@1) | 51.8 | 39.9 | 12.6 | 37.9 | 12.4 | 7.0 | 31.2 | |
| DeepSWE v1.1 (Resolved) | 74.0 | 73.0 | 67.5 | 66.9 | 62.7 | 54.4 | 74.2 | |
| ProgramBench (Almost@1) | 37.0 | 23.0 | 17.5 | 19.0 | 15.5 | - | 20.3 | |
| NL2Repo-Bench (Score) | 75.3 | 56.8 | 58.0 | 58.0 | 61.5 | 54.2 | 65.4 | |
| CyberGym (Pass@1) | - | 84.5 | 80.0 | 84.5 | 83.3 | 76.7 | 88.1 | |
| SEC-Bench Pro (Pass@1) | - | 74.3 | - | - | 56.4 | 30.9 | 62.8 | |
| ExploitGym (Pass@1) | 22.1 | 33.7 | - | 15.0 | 5.4 | 1.8 | 15.3 | |
| HLE w/ tools (Pass@1) | 63.6 | - | 59.8 | 62.5 | 60.0 | 51.5 | 63.9 | |
| Automation-Bench (Pass@1) | 50.3 | 45.8 | 46.7 | 48.8 | 43.2 | 37.7 | 54.8 | |
| Agents’ Last Exam(Pass@1) | 28.6 | 26.7 | 27.6 | 28.5 | 25.7 | 25.2 | 31.8 | |
| Chartography w/ tools (Pass@1) | 84.0 | 79.9 | 68.1 | - | - | - | 78.9 | |
| BabyVision w/ tools (Pass@1) | 94.1 | 88.9 | 85.7 | - | - | - | 89.6 | |
| ZeroBench-main w/ tools (Pass@5) | 52.0 | 53.0 | 41.0 | - | - | - | 49.0 |
5.3.3 Performance across reasoning efforts
Increasing effort generally lengthens reasoning and improves the reported aggregate results, with diminishing returns. The report maps low, high, and max presets to scalar values 50, 75, and 100; intermediate values are also studied.
5.3.4 Performance across agent scaffolds
Eight configurations from six scaffold families keep the checkpoint and task set fixed while changing surrounding tools and prompts. The preserved table shows how performance depends on the harness.
Related references: [4], [55], [3], [88], [83], [20].
| Benchmark (Metric) | Claude Code | Codex | OpenCode | Pi | mini-SWE | DeepSeek Harness | ||
|---|---|---|---|---|---|---|---|---|
| Minimal | Standard | PTC | ||||||
| DeepSWE v1.1 (Resolved) | 69.8 | 65.6 | 65.5 | 66.2 | 74.2 | 72.6 | 70.5 | 67.6 |
| Terminal-Bench v2.1 (Pass@1) | 88.0 | 84.1 | 85.0 | 86.1 | 90.3 | 90.6 | 85.8 | 85.8 |
Note. All scaffolds use samples per task on DeepSWE v1.1 and on Terminal-Bench v2.1. Runs use Linux containers, temperature 1.0, top-p 0.95, a 1M-token context window, and max_steps=500 model-generation rounds per agent on both benchmarks. Terminal-Bench v2.1 is evaluated without network access. We evaluate four Claude Code versions, with per-version results and their average reported in Appendix Table 5; this table reports v2.1.251. More scaffold-specific configurations are detailed in Appendix B.1.
5.3.5 Multi-Agent
Preliminary agent-team experiments compare the strongest observed configurations under explicit wall-clock deadlines. They use a 172-task filtered ProgramBench subset and a no-GPU FrontierSWE subset. Reported team gains should be read with those selection rules, differing metrics, and deadline budgets in mind.
Related references: [84], [40].
6 Conclusion, Limitations, and Future Directions
The report combines architectural changes, cache precision, serving systems, and training data to reduce long-context costs. Sparse selection and approximate local-state replay introduce robustness boundaries not fully characterized by the reported tests. The authors also acknowledge remaining gaps on difficult reasoning and edge cases despite strong benchmark scores.
Appendix
Appendix A Author List
The original appendix contains the complete DeepSeek-AI contributor list.
Appendix B Evaluation Details
B.1 Scaffold Configurations
The appendix records scaffold versions, tool interfaces, and runtime settings. These implementation details matter when reproducing cross-harness comparisons.
Benchmark (Metric) v2.1.105 v2.1.238 v2.1.251 v2.1.259 Average DeepSWE v1.1 (Resolved) 68.4 68.7 69.8 68.6 68.9 Terminal-Bench v2.1 (Pass@1) 87.3 88.4 88.0 87.6 87.8
Note. Average is computed from the unrounded Pass@1 scores of the four versions.
B.2 Reasoning Efforts across Scaffolds
Output length grows with effort in every tested scaffold–benchmark pair, but success rates have plateaus and intermediate dips. Harness choice can matter as much as the effort tier on nearly saturated tasks.
B.3 Detailed Results of Reasoning Benchmarks across Reasoning Efforts
The detailed reasoning plots show benchmark-specific accuracy and token use. Higher effort brings large gains on harder mathematics and smaller gains on already saturated tasks.
Appendix C Exponential Token Penalty in Reasoning Effort Control
A local reward model assumes exponentially diminishing marginal benefit from additional reasoning. Under an uncapped interior optimum, an exponential token-penalty schedule produces an approximately affine preferred length versus effort. The derivation does not guarantee linear or pointwise-monotonic measured behavior.
Equations 11–17
| (11) |
| (12) |
| (13) |
| (14) |
| (15) |
| (16) |
| (17) |