Overview
This edition follows the complete 30-page arXiv v3 paper: data collection, continued pretraining, instruction tuning, GRPO, ablations and mathematical appendices. It keeps the original experimental material and distinguishes corpus size, total training tokens and the different model stages. Comparisons describe the models and protocols available in this 2024 study.
At a glance
Intuition. Generate several answers to the same question, score them, then compare each answer with its peers. That group supplies a baseline for the policy update, replacing a separately trained model that predicts expected reward. Comparisons within a question help account for differences in question difficulty.
Reading the math. For outcome rewards, Ai = (ri − mean(r))/std(r): subtract the group’s average reward, then divide by its spread. Above-average answers receive positive advantages and below-average ones negative advantages. Equal scores give no comparison signal; zero spread needs numerical handling. PPO-style clipping and a reference-policy KL penalty also shape the update; the reported results reflect the full data-and-training pipeline.
DeepSeekMath combines a code-pretrained 7B model, a filtered mathematical web corpus and preference/reward-based training. The final reinforcement-trained model reaches 51.7% top-one accuracy on MATH without tools; majority voting over 64 samples reaches 60.9%. Group Relative Policy Optimization (GRPO) replaces PPO’s learned value baseline with relative rewards among responses to the same question. The evidence supports this complete training pipeline; the headline accuracy is not an isolated measure of GRPO alone.
1 Introduction
The project first mines 120B math-related tokens from Common Crawl, then continues a code model’s training, instruction-tunes it and applies reinforcement learning. The base model reaches 64.2% on GSM8K and 36.2% on MATH; instruction tuning and RL improve the latter to 46.8% and 51.7%. The paper also compares code/math data mixtures and expresses several training methods through their data sources and gradient weights. These are empirical findings for the reported models and benchmarks, not a general proof that any code corpus improves reasoning.
1.1 Contributions
The main contributions are an iterative mathematical web filter, a strong 7B mathematical model, and a critic-free group-relative policy update. Ablations study corpus choice, prior code training, online versus offline sampling, outcome versus process rewards and iterative reward-model updates. The negative arXiv-only result is scoped to these datasets, model sizes and evaluation tasks.
1.2 Summary of Evaluations and Metrics
Evaluation separates English/Chinese quantitative reasoning, Python-assisted solving, Isabelle proof automation and general language/code benchmarks. Top-one generation, majority voting, tool use and formal proof checking measure different capabilities and should not be compared as one common protocol. MMLU and BIG-Bench Hard assess broader understanding/reasoning; HumanEval and MBPP track coding after math training.
2 Math Pre-Training
Pretraining experiments examine both the quality of the mined corpus and the complete training mixture used for the 7B base model.
2.1 Data Collection and Decontamination
A fastText filter starts from 500,000 positive OpenWebMath examples and 500,000 Common Crawl negatives. URL and near-duplicate removal leave about 40B web pages. The first round keeps the highest-scoring 40B tokens. Domains with over 10% recalled pages are reviewed, and annotated mathematical URL paths supply new positive examples for subsequent rounds. Four rounds collect 35.5M pages, totaling 120B tokens.
Benchmark decontamination removes matching 10-gram segments and exact shorter benchmark strings of at least three grams. This addresses literal overlap; it does not by itself rule out paraphrased or otherwise undetected contamination. The complete collection workflow remains in Figure 2.
2.2 Validating the Quality of the DeepSeekMath Corpus
The corpus is compared with MathPile, OpenWebMath and Proof-Pile-2. Their content and size differ: MathPile is predominantly arXiv, while Proof-Pile-2 mixes arXiv, web and mathematical code. Fixed training-token budgets therefore repeat the smaller corpora more often.
2.2.1 Training Setting
Each corpus trains the same general 1.3B model for 150B tokens. AdamW uses β₁=0.9, β₂=0.95 and weight decay 0.1, with a 2,000-step warmup and scheduled learning-rate reductions. Peak learning rate is 5.3×10−4, batch size four million tokens and context length 4K. This common training budget supports the corpus comparison.
| Math Corpus | Size | English Benchmarks | Chinese Benchmarks | ||||||
| GSM8K | MATH | OCW | SAT |
MMLU STEM |
CMATH |
Gaokao MathCloze |
Gaokao MathQA | ||
| No Math Training | N/A | 2.9% | 3.0% | 2.9% | 15.6% | 19.5% | 12.3% | 0.8% | 17.9% |
| MathPile | 8.9B | 2.7% | 3.3% | 2.2% | 12.5% | 15.7% | 1.2% | 0.0% | 2.8% |
| OpenWebMath | 13.6B | 11.5% | 8.9% | 3.7% | 31.3% | 29.6% | 16.8% | 0.0% | 14.2% |
| Proof-Pile-2 | 51.9B | 14.3% | 11.2% | 3.7% | 43.8% | 29.2% | 19.9% | 5.1% | 11.7% |
| DeepSeekMath Corpus | 120.2B | 23.8% | 13.6% | 4.8% | 56.3% | 33.1% | 41.5% | 5.9% | 23.6% |
2.2.2 Evaluation Results
The mined corpus performs best across the reported eight mathematical benchmarks, including Chinese tests. Its learning curves continue improving after smaller corpora have plateaued. Better performance already around one full Proof-Pile-2 pass supports a quality advantage alongside a scale advantage, but corpus language, composition and repetition are not separately randomized factors.
2.3 Training and Evaluating DeepSeekMath-Base 7B
DeepSeekMath-Base starts from DeepSeek-Coder-Base-v1.5 7B and trains for 500B tokens: 56% mined mathematical web data, 4% AlgebraicStack, 10% arXiv, 20% GitHub code and 10% general English/Chinese web text. Thus the 120B-token corpus is reused within a larger mixed training run. The peak learning rate is 4.2×10−4 with ten million tokens per batch.
Mathematical Problem Solving with Step-by-Step Reasoning
Few-shot chain-of-thought evaluation covers quantitative and multiple-choice mathematics in English and Chinese. The base model leads the listed open base models across the eight reported benchmarks, including 36.2% on MATH versus 33.6% for Minerva 540B. These comparisons combine model-size, data and training differences; they do not isolate parameter efficiency alone.
| Model | Size | English Benchmarks | Chinese Benchmarks | ||||||
| GSM8K | MATH | OCW | SAT |
MMLU STEM |
CMATH |
Gaokao MathCloze |
Gaokao MathQA | ||
| Closed-Source Base Model | |||||||||
| Minerva | 7B | 16.2% | 14.1% | 7.7% | - | 35.6% | - | - | - |
| Minerva | 62B | 52.4% | 27.6% | 12.0% | - | 53.9% | - | - | - |
| Minerva | 540B | 58.8% | 33.6% | 17.6% | - | 63.9% | - | - | - |
| Open-Source Base Model | |||||||||
| Mistral | 7B | 40.3% | 14.3% | 9.2% | 71.9% | 51.1% | 44.9% | 5.1% | 23.4% |
| Llemma | 7B | 37.4% | 18.1% | 6.3% | 59.4% | 43.1% | 43.4% | 11.9% | 23.6% |
| Llemma | 34B | 54.0% | 25.3% | 10.3% | 71.9% | 52.9% | 56.1% | 11.9% | 26.2% |
| DeepSeekMath-Base | 7B | 64.2% | 36.2% | 15.4% | 84.4% | 56.5% | 71.7% | 20.3% | 35.3% |
Mathematical Problem Solving with Tool Use
Program-of-thought prompts ask models to write Python solutions, with libraries such as math and sympy allowed. The program’s execution result is the answer. DeepSeekMath-Base reaches 66.9% on GSM8K and 31.4% on MATH in this setup, improving over the listed Llemma baselines. These tool-assisted scores are distinct from text-only chain-of-thought results.
| Model | Size | Problem Solving w/ Tools | Informal-to-Formal Proving | ||
| GSM8K+Python | MATH+Python | miniF2F-valid | miniF2F-test | ||
| Mistral | 7B | 48.5% | 18.2% | 18.9% | 18.0% |
| CodeLlama | 7B | 27.1% | 17.2% | 16.3% | 17.6% |
| CodeLlama | 34B | 52.7% | 23.5% | 18.5% | 18.0% |
| Llemma | 7B | 41.0% | 18.6% | 20.6% | 22.1% |
| Llemma | 34B | 64.6% | 26.3% | 21.0% | 21.3% |
| DeepSeekMath-Base | 7B | 66.9% | 31.4% | 25.8% | 24.6% |
Formal Mathematics
On miniF2F, the model receives an informal statement, its formal counterpart and an informal proof, then generates an Isabelle proof sketch. Sledgehammer fills missing details. The reported validation/test results are 25.8%/24.6%; this evaluates a model-plus-automated-prover workflow, not standalone proof generation from an arbitrary natural-language problem.
| Model | Size | MMLU | BBH | HumanEval (Pass@1) | MBPP (Pass@1) |
| Mistral | 7B | 62.4% | 55.7% | 28.0% | 41.4% |
| DeepSeek-Coder-Base-v1.5 | 7B | 42.9% | 42.9% | 40.2% | 52.6% |
| DeepSeek-Coder-Base-v1.5 | 7B | 49.1% | 55.2% | 43.2% | 60.4% |
| DeepSeekMath-Base | 7B | 54.9% | 59.5% | 40.9% | 52.6% |
Natural Language Understanding, Reasoning, and Code
Against the specific pre-decay code-model checkpoint used for initialization, math training improves MMLU from 42.9% to 54.9% and BIG-Bench Hard from 42.9% to 59.5%. HumanEval remains near 40% and MBPP at 52.6%. Table 4 also includes the later code-only checkpoint, whose stronger code scores make checkpoint identity important when interpreting preservation of coding ability.
3 Supervised Fine-Tuning
Supervised fine-tuning teaches mathematical response formats before reinforcement learning.
3.1 SFT Data Curation
The 776K-example instruction dataset combines English and Chinese problems with chain-of-thought, program-of-thought and tool-integrated solutions. English material includes GSM8K, MATH, a MathInstruct subset and Lila-OOD training data. Chinese K–12 material spans 76 subtopics. The sources and solution formats matter when deciding which later evaluations are out of distribution.
3.2 Training and Evaluating DeepSeekMath-Instruct 7B
Instruction tuning concatenates examples to 4K-token contexts and trains for 500 steps, batch size 256, at learning rate 5×10−5. DeepSeekMath-Instruct reaches 82.9% GSM8K and 46.8% MATH without tools, and 57.4% MATH with tool-integrated reasoning. Table 5 retains contemporary baseline results; gray entries use 32-sample majority voting while other entries are top-one, so not every table cell has an identical inference budget.
| Model | Size | English Benchmarks | Chinese Benchmarks | ||
| GSM8K | MATH | MGSM-zh | CMATH | ||
| Chain-of-Thought Reasoning | |||||
| Closed-Source Model | |||||
| Gemini Ultra | - | 94.4% [32-vote] | 53.2% | - | - |
| GPT-4 | - | 92.0% | 52.9% | - | 86.0% |
| Inflection-2 | - | 81.4% | 34.8% | - | - |
| GPT-3.5 | - | 80.8% | 34.1% | - | 73.8% |
| Gemini Pro | - | 86.5% [32-vote] | 32.6% | - | - |
| Grok-1 | - | 62.9% | 23.9% | - | - |
| Baichuan-3 | - | 88.2% | 49.2% | - | - |
| GLM-4 | - | 87.6% | 47.9% | - | - |
| Open-Source Model | |||||
| InternLM2-Math | 20B | 82.6% | 37.7% | - | - |
| Qwen | 72B | 78.9% | 35.2% | - | - |
| Math-Shepherd-Mistral | 7B | 84.1% | 33.0% | - | - |
| WizardMath-v1.1 | 7B | 83.2% | 33.0% | - | - |
| DeepSeek-LLM-Chat | 67B | 84.1% | 32.6% | 74.0% | 80.3% |
| MetaMath | 70B | 82.3% | 26.6% | 66.4% | 70.9% |
| SeaLLM-v2 | 7B | 78.2% | 27.5% | 64.8% | - |
| ChatGLM3 | 6B | 72.3% | 25.7% | - | - |
| WizardMath-v1.0 | 70B | 81.6% | 22.7% | 64.8% | 65.4% |
| DeepSeekMath-Instruct | 7B | 82.9% | 46.8% | 73.2% | 84.6% |
| DeepSeekMath-RL | 7B | 88.2% | 51.7% | 79.6% | 88.8% |
| Tool-Integrated Reasoning | |||||
| Closed-Source Model | |||||
| GPT-4 Code Interpreter | - | 97.0% | 69.7% | - | - |
| Open-Source Model | |||||
| InternLM2-Math | 20B | 80.7% | 54.3% | - | - |
| DeepSeek-LLM-Chat | 67B | 86.7% | 51.1% | 76.4% | 85.4% |
| ToRA | 34B | 80.7% | 50.8% | 41.2% | 53.4% |
| MAmmoTH | 70B | 76.9% | 41.8% | - | - |
| DeepSeekMath-Instruct | 7B | 83.7% | 57.4% | 72.0% | 84.3% |
| DeepSeekMath-RL | 7B | 86.7% | 58.8% | 78.4% | 87.6% |
4 Reinforcement Learning
The reinforcement-learning stage updates the instruction-tuned model using sampled mathematical solutions and reward feedback.
4.1 Group Relative Policy Optimization
GRPO removes the separate learned value model while retaining a PPO-style clipped policy objective and a reference-policy penalty.
4.1.1 From PPO to GRPO
In the described PPO setup, a value model supports generalized advantage estimation, adding substantial memory when it is comparable in size to the policy. GRPO instead samples G responses to each question and derives their advantages from within-group rewards. The objective averages token-level clipped importance-ratio terms within responses and adds the reference KL penalty directly, separately from the advantages.
The displayed ratio-minus-log-ratio KL estimator is nonnegative, with equality when the compared probabilities agree. Its unbiased-KL interpretation assumes the relevant current-policy sampling distribution; stale-policy reuse does not automatically preserve that interpretation. The reported main run uses one policy update per exploration stage.
Equations 1–2
| (1) |
| (2) |
Equations 3–4
| (3) |
| (4) |
Input initial policy model ; reward models ; task prompts ; hyperparameters , ,
Output
4.1.2 Outcome Supervision RL with GRPO
Outcome supervision assigns each response one reward, subtracts the group mean and divides by the group standard deviation. Every token in that response receives the same normalized advantage. The displayed formula presumes a nonzero group standard deviation; the paper does not specify its zero-variance handling in this equation.
4.1.3 Process Supervision RL with GRPO
Process supervision scores the end of each reasoning step. Scores across the sampled group are normalized together, and a token’s advantage sums normalized rewards from subsequent steps. This supplies more detailed credit assignment than a single final score, at the cost of needing reliable step-level rewards.
4.1.4 Iterative RL with GRPO
Iterative GRPO samples new reward-training data from the evolving policy, updates the reward model with 10% historical replay, then resets the reference to the current policy for the next round. The full original algorithm specifies the rollout/update nesting. Refreshing both reward and reference changes the training regime from optimizing indefinitely against one fixed reward model.
4.2 Training and Evaluating DeepSeekMath-RL
The main RL run starts from DeepSeekMath-Instruct 7B and uses about 144K GSM8K/MATH-format questions already present in instruction tuning. The reward model starts from DeepSeekMath-Base; policy learning rate is 10−6, KL coefficient 0.04, group size 64, maximum output length 1,024 and batch size 1,024. There is one policy update after each exploration stage.
Text-only GSM8K/MATH improve to 88.2%/51.7%; CMATH improves from 84.6% to 88.8%. Other tests are “out of domain” specifically relative to the restricted RL-question set; they are not necessarily absent from all preceding pretraining or supervised data. The retained table also distinguishes tool-assisted evaluation.
5 Discussion
The discussion separates lessons from pretraining-data experiments and from reinforcement-learning comparisons.
5.1 Lessons Learnt in Pre-Training
Unless specified otherwise, these ablations use the 1.3B setup and an earlier 89B-token corpus from collection round two, rather than the final 120B-token corpus.
5.1.1 Code Training Benefits Mathematical Reasoning
The code study compares 400B code tokens followed by 150B math tokens, a matched 400B general-text precursor, math-only training, and a single mixed 400B-code/150B-math run. Matching the precursor-token budget helps distinguish code content from simply doing more training.
| Training Setting | Training Tokens | w/o Tool Use | w/ Tool Use | |||||
| General | Code | Math | GSM8K | MATH | CMATH | GSM8K+Python | MATH+Python | |
| No Continual Training | – | – | – | 2.9% | 3.0% | 12.3% | 2.7% | 2.3% |
| Two-Stage Training | ||||||||
| Stage 1: General Training | 400B | – | – | 2.9% | 3.2% | 14.8% | 3.3% | 2.3% |
| Stage 2: Math Training | – | – | 150B | 19.1% | 14.4% | 37.2% | 14.3% | 6.7% |
| Stage 1: Code Training | – | 400B | – | 5.9% | 3.6% | 19.9% | 12.4% | 10.0% |
| Stage 2: Math Training | – | – | 150B | 21.9% | 15.3% | 39.7% | 17.4% | 9.4% |
| One-Stage Training | ||||||||
| Math Training | – | – | 150B | 20.5% | 13.1% | 37.6% | 11.4% | 6.5% |
| Code & Math Mixed Training | – | 400B | 150B | 17.6% | 12.1% | 36.3% | 19.7% | 13.5% |
| Training Setting | Training Tokens | MMLU | BBH | HumanEval (Pass@1) | MBPP (Pass@1) | ||
| General | Code | Math | |||||
| No Continual Training | – | – | – | 24.5% | 28.1% | 12.2% | 13.0% |
| Two-Stage Training | |||||||
| Stage 1: General Training | 400B | – | – | 25.9% | 27.7% | 15.2% | 13.6% |
| Stage 2: Math Training | – | – | 150B | 33.1% | 32.7% | 12.8% | 13.2% |
| Stage 1: Code Training | – | 400B | – | 25.0% | 31.5% | 25.0% | 40.0% |
| Stage 2: Math Training | – | – | 150B | 36.2% | 35.3% | 12.2% | 17.0% |
| One-Stage Training | |||||||
| Math Training | – | – | 150B | 32.3% | 32.5% | 11.6% | 13.2% |
| Code & Math Mixed Training | – | 400B | 150B | 33.5% | 35.6% | 29.3% | 39.4% |
Results
Code training helps Python-assisted mathematics in both sequential and mixed settings. Mixing also limits forgetting of coding skills. For text-only reasoning, code then math gives the strongest results, whereas simultaneous mixing is less effective. The authors suggest limited 1.3B capacity as a possible explanation; this is a conjecture, not an established mechanism.
| Model | Size | ArXiv Corpus | English Benchmarks | Chinese Benchmarks | ||||||
| GSM8K | MATH | OCW | SAT |
MMLU STEM |
CMATH |
Gaokao MathCloze |
Gaokao MathQA | |||
| DeepSeek-LLM | 1.3B | No Math Training | 2.9% | 3.0% | 2.9% | 15.6% | 19.5% | 12.3% | 0.8% | 17.9% |
| MathPile | 2.7% | 3.3% | 2.2% | 12.5% | 15.7% | 1.2% | 0.0% | 2.8% | ||
| ArXiv-RedPajama | 3.3% | 3.4% | 4.0% | 9.4% | 9.0% | 7.4% | 0.8% | 2.3% | ||
| DeepSeek-Coder-Base-v1.5 | 7B | No Math Training | 29.0% | 12.5% | 6.6% | 40.6% | 38.1% | 45.9% | 5.9% | 21.1% |
| MathPile | 23.6% | 11.5% | 7.0% | 46.9% | 35.8% | 37.9% | 4.2% | 25.6% | ||
| ArXiv-RedPajama | 28.1% | 11.1% | 7.7% | 50.0% | 35.2% | 42.6% | 7.6% | 24.8% | ||
| ArXiv Corpus | miniF2F-valid | miniF2F-test |
| No Math Training | 20.1% | 21.7% |
| MathPile | 16.8% | 16.4% |
| ArXiv-RedPajama | 14.8% | 11.9% |
5.1.2 ArXiv Papers Seem Ineffective in Improving Mathematical Reasoning
ArXiv-heavy/only corpora produce little benefit or deterioration on the tested mathematical tasks after 150B tokens for the 1.3B model or 40B for the 7B code model. The authors explicitly leave open different tasks, mixtures with other data and larger models. The main 500B-token training mixture itself includes 10% arXiv, so the isolated-corpus ablation does not prove that component useless within the mixture.
5.2 Insights of Reinforcement Learning
The RL analysis compares how training examples are sampled, how rewards are obtained and how rewards become token-level gradient weights.
5.2.1 Towards to a Unified Paradigm
The common representation writes an update as a log-probability gradient multiplied by a coefficient. SFT uses selected demonstrations; rejection sampling keeps correct generated answers; DPO uses preferred/rejected pairs; PPO and GRPO use online samples and advantages. Table 10 is an analytical comparison of update components, not a proof that these methods have identical objectives, convergence guarantees or data requirements.
Equation 5
| (5) |
| Methods | Data Source | Reward Function | Gradient Coefficient |
| SFT | - | 1 | |
| RFT | , | Rule | Equation 10 |
| DPO | , | Rule | Equation 14 |
| Online RFT | , | Rule | Equation 10 |
| PPO | , | Model | Equation 18 |
| GRPO | , | Model | Equation 21 |
Observation about Data Source
Online rejection sampling draws from the changing policy; offline rejection sampling reuses outputs from the initial supervised model. The curves begin similarly but online sampling improves more later, consistent with refreshed data becoming more relevant as the policy changes.
Observation about Gradient Coefficient
Online rejection sampling positively reinforces correct answers uniformly and does not penalize incorrect ones. GRPO instead weights and can penalize responses using relative reward. In the reported comparisons it outperforms online rejection sampling; process-supervised GRPO beats outcome supervision, and two iterative reward/policy rounds improve the 7B results further. These studies vary reward granularity and sampling as well as the named algorithm.
5.2.2 Why RL Works?
At temperature 0.7, RL improves majority-vote accuracy more clearly than pass-at-K accuracy. The authors interpret this as concentrating probability on correct responses already among the model’s samples. It is evidence about these sampling curves and datasets, not a proof that RL can never add new capabilities or that all possible reasoning paths were measured.
5.2.3 How to Achieve More Effective RL?
The proposed next steps address questions and decoding, robustness of the update, and reliability of reward models. They are future directions rather than implemented results.
Data Source
The experiments reuse supervised-training questions and ordinary nucleus sampling. The authors propose new question distributions, search-based decoding and more efficient inference to broaden exploration beyond that setup.
Algorithms
All compared updates rely on reward signals that can be mistaken. The paper motivates algorithms robust to noisy feedback and weak-to-strong supervision; its cited annotation-error example is not a new audit conducted in this study.
Reward Function
Future reward models should generalize to new questions and search-generated outputs, express uncertainty and provide efficient, reliable process supervision. Otherwise training may primarily concentrate an existing output distribution around the reward model’s preferences.
6 Conclusion, Limitation, and Future Work
The full pipeline demonstrates strong 2024 mathematical benchmark performance, but geometry, theorem proving and few-shot gains remain weaker than the cited closed models. The authors report failures involving triangles and ellipses and suggest selection bias as a possible cause. Better corpus selection and reward learning remain open. The paper does not establish that benchmark accuracy alone guarantees broadly reliable mathematical reasoning.
Appendix A Appendix
The appendix preserves the mathematical update derivations underlying the common comparison of training methods.
A.1 Analysis of Reinforcement Learning
Each derivation identifies the sampling distribution, reward source and multiplicative coefficient on a token log-probability gradient. This compares the training update on sampled data rather than erasing the methods’ distinct data-collection procedures.
A.1.1 Supervised Fine-tuning
Supervised fine-tuning maximizes demonstrated-response likelihood. In the common notation, human selection determines the data and every demonstrated token has coefficient one.
Equations 6–7
| (6) |
| (7) |
A.1.2 Rejection Sampling Fine-tuning
Offline rejection sampling generates responses from the fixed supervised model and retains those passing the answer-correctness rule. The gradient reinforces only accepted responses.
Equations 8–10
| (8) |
| (9) |
| (10) |
A.1.3 Online Rejection Sampling Fine-tuning
The online variant changes the source of generated responses to the current policy. The displayed supervised-style update treats the collected responses as training data for that update.
Equation 11
| (11) |
A.1.4 Direct Preference Optimization (DPO)
DPO’s pairwise loss yields opposite-signed gradients for preferred and rejected responses, weighted by the sigmoid of their reference-adjusted implicit reward margin. The appendix frames preference labels as human feedback or answer-correctness rules in mathematical tasks.
Equations 12–14
| (12) |
| (13) |
| (14) |
A.1.5 Proximal Policy Optimization (PPO)
For a simplified one-update analysis at the rollout policy, the old and current policy coincide and the clipping/minimum are locally inactive. The resulting coefficient is the advantage from generalized advantage estimation, including the value baseline. This simplification does not describe arbitrary later epochs on stale rollout data.
Equations 15–18
| (15) |
| (16) |
| (17) |
| (18) |
A.1.6 Group Relative Policy Optimization (GRPO)
Under the same simplified old=current-policy analysis, GRPO replaces the learned-value advantage with a within-group reward advantage and adds the reference-policy regularization contribution. The original equations retain the precise coefficients and length normalization.
Equations 19–21
| (19) |
| (20) |
| (21) |