Literature
Display
← All papers

Muon is Scalable for LLM Training

Jingyuan Liu · Jianlin Su · Xingcheng Yao · Zhejun Jiang · Guokun Lai · Yulun Du · Yidao Qin · Weixin Xu · Enzhe Lu · Junjie Yan · Yanru Chen · Huabin Zheng · Yibo Liu · Shaowei Liu · Bohong Yin · Weiran He · Han Zhu · Yuzhi Wang · Jianzhou Wang · Mengnan Dong · Zheng Zhang · Yongsheng Kang · Hao Zhang · Xinran Xu · Yutao Zhang · Yuxin Wu · Xinyu Zhou · Zhilin Yang

· Technical report ↗

Contents

Muon is Scalable for LLM Training

Intuition. A matrix update can strongly favor a few directions. In a two-dimensional illustration, its singular values might be 10 and 1: one direction gets ten times the amplification. Muon approximately levels these strengths before choosing the final update scale, rather than merely rescaling individual entries.

Reading the math. Write the momentum matrix as M=UΣVᵀ. U and V describe directions; Σ contains their strengths. The ideal polar update U Vᵀ replaces nonzero strengths by one. Newton–Schulz approximates this with matrix multiplications. It changes the update, not the requirement that the weights themselves be orthogonal; the paper also adds decay and shape-dependent scaling.

This technical report studies a scalable Muon implementation and releases Moonlight models and checkpoints. Figure 1 summarizes fitted compute/loss curves and comparisons with public models; cross-model frontiers also reflect differences in data and architecture.

Complete original S0.F1 from the exact-version PDF, including every panel, label, and original color.
Figure 1: Scaling up with Muon. (a) Scaling law experiments comparing Muon and Adam. Muon is ∼2×\sim 2\times more computational efficient than Adam with compute optimal training. (b) The MMLU performance of our Moonlight model optimized with Muon and other comparable models. Moonlight advances the Pareto frontier of performance vs training FLOPs.

At a glance

Adding decoupled weight decay and shape-dependent update scaling makes Muon effective in the authors’ language-model experiments. Compute-optimal fits on dense models suggest approximately half the training FLOPs needed by their AdamW baseline at comparable loss. The resulting Moonlight mixture-of-experts model has about 3 billion active and 16 billion total parameters including embeddings, and is trained on 5.7 trillion tokens. A distributed implementation, pretraining and fine-tuning comparisons, and spectral diagnostics support the case study. The results do not establish a universal twofold saving or superiority for every fine-tuning setting.

1 Introduction

Muon replaces the singular values of gradient momentum approximately, aiming to avoid updates dominated by a few directions. Earlier results were mostly small-scale. This report addresses longer and larger runs, distributed matrix computation, and the interaction between pretraining and supervised fine-tuning. Its key interventions are decay and update-scale matching, followed by a distributed implementation and comparisons against a tuned AdamW baseline. The reported efficiency is an empirical scaling fit over the studied model/data range, while Moonlight demonstrates feasibility at larger scale.

2 Methods

2.1 Background

The Muon Optimizer

Building on Jordan and collaborators [Jordan et al., 2024], Muon accumulates gradient momentum and applies approximate matrix orthogonalization before updating matrix parameters. The implementation uses Nesterov-style momentum, as specified in the footnote. For the ideal polar factor, an SVD M=UΣVᵀ gives U Vᵀ, flattening nonzero singular values. Rectangular and rank-deficient matrices require the corresponding partial-isometry or pseudoinverse interpretation; the inverse-square-root expression alone is not an ordinary inverse in every shape. Finite Newton–Schulz iterations produce an approximation, not exact orthogonality.

Equation 1
𝐌t\displaystyle\mathbf{M}_{t} =μ​𝐌t−1+∇ℒt​(𝐖t−1)\displaystyle=\mu\mathbf{M}_{t-1}+\nabla\mathcal{L}_{t}(\mathbf{W}_{t-1})
𝐎t\displaystyle\mathbf{O}_{t} =Newton-Schulz​(𝐌t)​11 1    In practice, we follow (Jordan et al., 2024) to use a Nesterov-style momentum by putting μ​𝐌t+∇ℒt​(𝐖t−1) to the Newton-Schulz iteration instead of 𝐌t.\displaystyle=\text{Newton-Schulz}(\mathbf{M}_{t})\text{} (1)
𝐖t\displaystyle\mathbf{W}_{t} =𝐖t−1−ηt​𝐎t\displaystyle=\mathbf{W}_{t-1}-\eta_{t}\mathbf{O}_{t}

Newton-Schulz Iterations for Matrix Orthogonalization

The momentum matrix is first divided by its Frobenius norm, then updated with a quintic matrix polynomial. The coefficients are a=3.4445, b=−4.7750, and c=2.0315, inherited from the original Muon implementation. They favor rapid growth of small singular values; their practical finite-step result should not be confused with an exact polar decomposition or exact convergence of every singular value to one.

Equation 2
𝐗k\displaystyle\mathbf{X}_{k} =a​𝐗k−1+b⁡(𝐗k−1​𝐗k−1T)​𝐗k−1+c​(𝐗k−1​𝐗k−1T)2​𝐗k−1\displaystyle=a\mathbf{X}_{k-1}+b(\mathbf{X}_{k-1}\mathbf{X}_{k-1}^{\mathrm{T}})\mathbf{X}_{k-1}+c(\mathbf{X}_{k-1}\mathbf{X}_{k-1}^{\mathrm{T}})^{2}\mathbf{X}_{k-1} (2)

Steepest Descent Under Norm Constraints

Following Bernstein and Newhouse [2024], the authors interpret optimizer choice as steepest descent under different norm constraints. Exact polar-factor updates correspond to a spectral-norm constraint; their approximate iteration is discussed in relation to Schatten norms. The argument that an operator norm suits Euclidean hidden representations is a modeling motivation. It does not prove that this geometry always improves generalization over Adam’s adaptive coordinate scaling.

2.2 Scaling Up Muon

Weight Decay

Longer runs without decay show growing weight and output RMS. The proposed update adds AdamW-style decoupled shrinkage, ηλW, outside the orthogonalized momentum. An 800-million-parameter model trained on 100 billion tokens, about five times the compute-optimal token budget used here, initially favors vanilla Muon but later favors Muon with decay. Large magnitudes are discussed as a low-precision training concern; the curves establish the observed benefit in this setup rather than isolating a universal numerical-failure mechanism.

Equation 3
𝐖t=𝐖t−1−ηt​(𝐎t+λ​𝐖t−1)\displaystyle\mathbf{W}_{t}=\mathbf{W}_{t-1}-\eta_{t}(\mathbf{O}_{t}+\lambda\mathbf{W}_{t-1}) (3)
Complete original S2.F2 from the exact-version PDF, including every panel, label, and original color.
Figure 2: Validation loss curves for AdamW (green), Muon without weight decay (red), and Muon with weight decay (blue).

Consistent update RMS

For an exact full-rank polar update of shape A×B, the entrywise RMS is 1/√max(A,B). Without compensation, differently shaped matrices receive different update scales. Multiplication by √max(A,B) cancels this dependence. The result assumes the ideal factor and full rank; finite iterations and rank deficiency can change its exact value. This matters for large feed-forward matrices and small separately parameterized attention heads.

Matching update RMS of AdamW

Some parameters, including normalization scales, embeddings, and the language-model head, continue to use AdamW even when the model is described as Muon-trained. The authors empirically observe AdamW update RMS around 0.2–0.4 and scale Muon’s ideal update by 0.2√max(A,B). This permits reuse of their AdamW learning rate and decay settings. The factor is an empirical calibration, not an optimizer-independent theoretical constant.

Equation 4
𝐖t=𝐖t−1−ηt​(0.2⋅𝐎t⋅max⁡(A,B)+λ​𝐖t−1)\displaystyle\mathbf{W}_{t}=\mathbf{W}_{t-1}-\eta_{t}(0.2\cdot\mathbf{O}_{t}\cdot\sqrt{\max(A,B)}+\lambda\mathbf{W}_{t-1}) (4)

Other Hyper-parameters

Five Newton–Schulz steps are used because ten give more accurate orthogonalization without improving the tested performance. Momentum is fixed at 0.95 after no consistent benefit from tuning. These choices favor efficiency in the reported experiments.

2.3 Distributed Muon

ZeRO-1 and Megatron-LM

ZeRO-1 partitions optimizer states across data-parallel workers. Megatron-LM combines this with tensor, pipeline, and expert parallelism, so synchronization can often remain within the relevant data-parallel group.

Method

Elementwise AdamW can operate on a local shard, whereas Muon needs a complete matrix for its polynomial update. Distributed Muon reduce-scatters gradients, applies local momentum, gathers the matrix, computes the full update, keeps the local portion, and all-gathers the updated parameters. The blue lines in Algorithm 1 identify the added operations. Some full-matrix work is consequently repeated while optimizer state remains sharded.

Complete original alg1 from the exact-version PDF, including every panel, label, and original color.
Algorithm 1 Distributed Muon
Accessible original algorithm or code markup
0:  Full Gradients 𝐆\mathbf{G}, DP partitioned Momentum 𝐦\mathbf{m}, DP partitioned parameters 𝐩\mathbf{p}, momentum μ\mu.
1:  // Reduce-scatter GG on DP for correct gradients
2:  𝐠=reduce_scatter(𝐆, dp_group)\mathbf{g}=\text{reduce\_scatter($\mathbf{G}$, dp\_group)}
3:  // Apply momentum to 𝐠\mathbf{g} using local partitioned momentum 𝐦\mathbf{m}
4:  𝐠′=update_with_momentum​(𝐠,𝐦,μ)\mathbf{g}^{\prime}=\text{update\_with\_momentum}(\mathbf{g},\mathbf{m},\mu)
5:  // DP Gather: gathering 𝐠′\mathbf{g^{\prime}} across DP into a full matrix 𝐆\mathbf{G}
6:  𝐆=gather(𝐠′, dp_group)\mathbf{G}=\text{gather($\mathbf{g^{\prime}}$, dp\_group)}
7:  // Calculate Muon update
8:  𝐔=Newton-Schulz​(𝐆)\mathbf{U}=\text{Newton-Schulz}(\mathbf{G})
9:  // Discard the rest of 𝐔\mathbf{U} and only keep the local partition 𝐮{\mathbf{u}}, then apply the update rule
10:  𝐩′=apply_update​(𝐩,𝐮)\mathbf{p}^{\prime}=\text{apply\_update}(\mathbf{p},\mathbf{u})
11:  // All-gather updated 𝐩′\mathbf{p^{\prime}} into 𝐏\mathbf{P}
12:  𝐏=all_gather(𝐩′, dp_group)\mathbf{P}=\text{all\_gather($\mathbf{p^{\prime}}$, dp\_group)}
13:  // Return the update RMS for logging
14:  return 𝐮2.mean​()\sqrt{\mathbf{u}^{2}.\texttt{mean}()}

Analysis

Muon uses one momentum buffer rather than AdamW’s two, halving this component of optimizer-state memory rather than total training memory. Under the stated communication accounting, workload is between one and 1.25 times the AdamW version, using bfloat16 for the extra gather; tensor parallelism adds another gather. Optimizer work is reported to occupy roughly 1–3% of step time, with no noticeable overall overhead on the authors’ cluster. These latency and communication results depend on sharding, precision, overlap, and hardware.

3 Experiments

3.1 Consistent Update RMS

Three scaling strategies are compared: one global hidden-size factor, exact normalization of each update to RMS 0.2, and a cheaper shape-based learning-rate adjustment. Equations 5–7 retain their square-root factors and the same separate decay term.

Equations 5–7
𝐖t=𝐖t−1−ηt​(0.2⋅𝐎t⋅H+λ​𝐖t−1)\displaystyle\mathbf{W}_{t}=\mathbf{W}_{t-1}-\eta_{t}(0.2\cdot\mathbf{O}_{t}\cdot\sqrt{H}+\lambda\mathbf{W}_{t-1}) (5)
𝐖t=𝐖t−1−ηt​(0.2⋅𝐎t/RMS⁡(𝐎t)+λ​𝐖t−1)\displaystyle\mathbf{W}_{t}=\mathbf{W}_{t-1}-\eta_{t}(0.2\cdot\mathbf{O}_{t}/\mathop{\text{RMS}}(\mathbf{O}_{t})+\lambda\mathbf{W}_{t-1}) (6)
𝐖t=𝐖t−1−ηt​(0.2⋅𝐎t⋅max⁡(A,B)+λ​𝐖t−1)\displaystyle\mathbf{W}_{t}=\mathbf{W}_{t-1}-\eta_{t}(0.2\cdot\mathbf{O}_{t}\cdot\sqrt{\max(A,B)}+\lambda\mathbf{W}_{t-1}) (7)

Analysis

The diagnostic uses an 800-million-parameter model after 4 billion tokens of a 20-billion-token schedule. Replacing SwiGLU with a standard feed-forward block increases the dimension disparity to H×4H. Exact normalization and shape scaling both reduce validation loss from 2.812 to 2.789. Shape scaling approximately doubles the feed-forward weight RMS while leaving the square query matrix closer to the baseline; exact normalization affects both. The authors choose the shape rule because it is cheaper. This is an early-training ablation with a deliberately modified architecture.

Table 1: Controlling Muon’s Update RMS Across Different Model Params
Methods Training loss Validation loss query weight RMS MLP weight RMS
Baseline 2.734 2.812 3.586e-2 2.52e-2
Update Norm 2.72 2.789 4.918e-2 5.01e-2
Adjusted LR 2.721 2.789 3.496e-2 4.89e-2

3.2 Scaling Law of Muon

Dense Llama-style models range from 399 million to 1.5 billion non-embedding parameters. The AdamW baseline receives a multistage hyperparameter search, and Muon reuses the resulting settings. The fitted loss curves are 2.506 C⁻⁰·⁰⁵² and 2.608 C⁻⁰·⁰⁵⁴ in the paper’s compute units. The approximately 52% compute figure is derived from these fits over the investigated range; the slightly different exponents mean it is not a scale-independent constant, nor a direct measurement of wall-clock acceleration.

Table 2: Scaling Law Models and Hyper-Parameters
# Params. w/o Embedding Head Layer Hidden Tokens LR Batch Size*
399M 12 12 1536 8.92B 9.503e-4 96
545M 14 14 1792 14.04B 9.143e-4 128
822M 16 16 2048 20.76B 8.825e-4 160
1.1B 18 18 2304 28.54B 8.561e-4 192
1.5B 20 20 2560 38.91B 8.305e-4 256

*In terms of number of examples in 8K context length.

Complete original S3.F3 from the exact-version PDF, including every panel, label, and original color.
Figure 3: Fitted scaling law curves for Muon and AdamW optimizers.
Table 3: Fitted parameters of the scaling law curves
Muon AdamW
LM loss (seqlen=8K) 2.506×C−0.0522.506\times C^{-0.052} 2.608×C−0.0542.608\times C^{-0.054}

3.3 Pretraining with Muon

Model Architecture

Moonlight follows a modified DeepSeek-V3-Small architecture with 2.24 billion active and 15.29 billion total parameters excluding embeddings, approximately 3B/16B including them. Appendix C describes the changes, which matter in comparisons to public models.

Pretraining Data

Pretraining uses the authors’ data mixture referenced to their separate data report, with a maximum context length of 8K. The optimizer paper does not itself fully specify that proprietary mixture.

Pretraining

The first 33 billion tokens include a 2,000-step warmup to learning rate 4.2×10⁻⁴. Cosine decay then reaches 4.2×10⁻⁵ at 5.2 trillion tokens; batch size doubles from 2,048 to 4,096 examples after 200 billion tokens. A final 500-billion-token cooldown briefly raises the rate to 10⁻⁴, then decays it linearly to zero on higher-quality math, code, and reasoning data. Decay remains 0.1. The auxiliary-free routing-bias update rate is 10⁻³ in the first two stages and zero in the last.

Evaluation Benchmarks

Evaluation spans English knowledge/reasoning, code generation, mathematics, and Chinese knowledge/reasoning. The tables retain dataset names, shot counts, exact-match or pass@1 metrics, and missing entries; scores from different protocols should not be pooled as one common measure.

Performance

At 1.2 trillion tokens, the strongest optimizer comparison is Moonlight versus Moonlight-A with the same training setup and AdamW. Muon improves most reported tasks, especially code and math, but BBH is lower: 43.2 versus 45.3. These are intermediate checkpoints before cooldown. At 5.7 trillion tokens, Moonlight reaches MMLU 70.0 and HumanEval 48.1 and is competitive with similarly active-sized public models; Qwen2.5-3B remains higher on GSM8K, 79.1 versus 77.4. Public-model comparisons also change data, architecture, token count, and sometimes unknown optimizer settings, so they cannot attribute every difference to Muon. The claimed frontier is with respect to the selected models and reported compute estimates.

Table 4: Comparison of different models at around 1.2T tokens.
Benchmark (Metric) DSV3-Small Moonlight-A@1.2T Moonlight@1.2T
Activated Params† 2.24B 2.24B 2.24B
Total Params† 15.29B 15.29B 15.29B
Training Tokens 1.33T 1.2T 1.2T
Optimizer AdamW AdamW Muon
English MMLU 53.3 60.2 60.4
MMLU-pro - 26.8 28.1
BBH 41.4 45.3 43.2
TriviaQA - 57.4 58.1
Code HumanEval 26.8 29.3 37.2
MBPP 36.8 49.2 52.9
Math GSM8K 31.4 43.8 45.0
MATH 10.7 16.1 19.8
CMath - 57.8 60.2
Chinese C-Eval - 57.2 59.9
CMMLU - 58.2 58.8

† The reported parameter counts exclude the embedding parameters.

Table 5: Comparison of different models on various benchmarks.
Benchmark (Metric) Llama3.2-3B Qwen2.5-3B DSV2-Lite Moonlight
Activated Param† 2.81B 2.77B 2.24B 2.24B
Total Params† 2.81B 2.77B 15.29B 15.29B
Training Tokens 9T 18T 5.7T 5.7T
Optimizer AdamW Unknown AdamW Muon
English MMLU 54.7 65.6 58.3 70.0
MMLU-pro 25.0 34.6 25.5 42.4
BBH 46.8 56.3 44.1 65.2
TriviaQA‡ 59.6 51.1 65.1 66.3
Code HumanEval 28.0 42.1 29.9 48.1
MBPP 48.7 57.1 43.2 63.8
Math GSM8K 34.0 79.1 41.1 77.4
MATH 8.5 42.6 17.1 45.3
CMath - 80.0 58.4 81.1
Chinese C-Eval - 75.0 60.3 77.2
CMMLU - 75.0 64.3 78.2

† The reported parameter counts exclude the embedding parameters.‡ We tested all listed models with the full set of TriviaQA.

3.4 Dynamics of Singular Spectrum

Normalized entropy of squared singular values measures how evenly a weight matrix distributes its spectral energy. Muon-trained weights have higher group-average entropy across the plotted checkpoints, especially router matrices, and more than 90% of inspected matrices have higher entropy at 1.2 trillion tokens. Only a small subset of experts is analyzed. These are weight-spectrum observations; they are consistent with the proposed explanation but do not directly measure all update directions or prove that increased entropy causes better task performance.

Equation
H(σ)=−1log⁡n∑i=1nσi2∑j=1nσj2logσi2∑j=1nσj2H(\sigma)=-\frac{1}{\log n}\sum_{i=1}^{n}\frac{\sigma^{2}_{i}}{\sum_{j=1}^{n}\sigma^{2}_{j}}\log\frac{\sigma^{2}_{i}}{\sum_{j=1}^{n}\sigma^{2}_{j}}
Complete original S3.F4 from the exact-version PDF, including every panel, label, and original color.
Figure 4: SVD entropy of weight matrices across different training iterations. We categorize the weight matrices into 6 different groups: 1) AttnQO denotes the weight matrices related to the query and output projection in the attention layer; 2) AttnKV denotes the weight matrices related to the key and value projection in the attention layer; 3) Experts denotes the weight matrices in expert models; 4) SharedExperts denotes the weight matrices in shared expert models; 5) Router denotes the weight matrices in the router; 6) Dense denotes the weight matrices in the first dense layer. The SVD entropy is calculated as the macro-average of the weight matrices in each group across all layers. For weights in expert models, we only calculate 3 out of 64 experts in different layers for efficiency.

3.5 Supervised Finetuning (SFT) with Muon

The best tested combination uses Muon for both pretraining and supervised fine-tuning. Switching optimizers gives mixed results, and applying Muon only during fine-tuning does not show a general advantage.

3.5.1 Ablation Studies on the Interchangeability of Pretrain and SFT Optimizers

The 1.2-trillion-token Moonlight and Moonlight-A checkpoints are fine-tuned for two epochs on Tulu-3 SFT data at 4K context, with a linearly decaying learning rate starting at 5×10⁻⁵. The Muon/Muon combination is best on all four reported metrics. The complete four-way table is retained, so improvements from the pretrained checkpoint can be distinguished from the fine-tuning optimizer choice.

Table 6: Examining the impact of optimizer interchangeability between pretraining and SFT phases.
Benchmark (Metric) # Shots Moonlight-1.2T
Pretraining Optimizer - Muon AdamW Muon AdamW
SFT Optimzier - Muon Muon AdamW AdamW
MMLU (EM) 0-shot (CoT) 55.7 55.3 50.2 52.0
HumanEval (Pass@1) 0-shot 57.3 53.7 52.4 53.1
MBPP (Pass@1) 0-shot 55.6 55.5 55.2 55.2
GSM8K (EM) 5-shot 68.0 62.1 64.9 64.6

3.5.2 SFT with Muon on public pretrained models

Qwen2.5-7B is fine-tuned on packed 8K Tulu-3 data with cosine decay from 2×10⁻⁵ to 2×10⁻⁶. The report calls the results comparable, but Muon is numerically below Adam on all four listed metrics, including GSM8K 85.8 versus 89.8. This ablation supports caution about simply switching an existing pretrained model to Muon.

Table 7: Comparison of Adam and Muon optimizers applied to the SFT of the Qwen2.5-7B pretrained model.
Benchmark (Metric) # Shots Adam-SFT Muon-SFT
Pretrained Model - Qwen2.5-7B
MMLU (EM) 0-shot (CoT) 71.4 70.8
HumanEval (Pass@1) 0-shot 79.3 77.4
MBPP (Pass@1) 0-shot 71.9 71.6
GSM8K (EM) 5-shot 89.8 85.8

4 Discussions

The discussion identifies three unresolved extensions.

Incorporating All Parameters into the Muon Framework

The current system is hybrid: some parameters still use AdamW. A method that applies the same Muon framework to every parameter is left for future research.

Extending Muon to Schatten Norms

General Schatten-norm geometries are proposed as a possible extension beyond the spectral-norm interpretation. Their benefits are not established in this report.

Understanding and Solving the Pretraining-Finetuning Mismatch

The reason for the pretraining/fine-tuning optimizer mismatch remains unresolved. Understanding it is important before reusing the large pool of AdamW-pretrained checkpoints with a different fine-tuning optimizer.

5 Conclusions

The report demonstrates a practical large-scale Muon-based training recipe, not a theorem that it should replace AdamW everywhere. Decay, matrix-shape scaling, distributed implementation, and a hybrid treatment of parameters collectively produce strong results in the studied setup. Data and architecture differences, fitted scaling extrapolations, and the fine-tuning mismatch limit broader conclusions.

Appendix A Update RMS

Proof of Lemma 1

For a rank-r partial isometry of size n×m, orthonormality gives squared Frobenius norm r and entrywise RMS √(r/(mn)). For full rank with n≥m, this becomes 1/√n. The proof explains the shape correction and why rank deficiency changes the exact scale. The displayed sums are retained; cancellation of cross terms is understood after summing over orthogonal rows or columns.

Equations
Xi,j=∑k=1rUi,k​Vk,jX_{i,j}=\sum_{k=1}^{r}U_{i,k}V_{k,j}
RMS​(X)2\displaystyle\text{RMS}(X)^{2} =1m​n​∑i=1n∑j=1m∑k=1rUi,k2​Vk,j2\displaystyle=\frac{1}{mn}\sum_{i=1}^{n}\sum_{j=1}^{m}\sum_{k=1}^{r}U_{i,k}^{2}V_{k,j}^{2}
=1m​n​∑k=1r(∑i=1nUi,k2)​(∑j=1mVk,j2)\displaystyle=\frac{1}{mn}\sum_{k=1}^{r}\left(\sum_{i=1}^{n}U_{i,k}^{2}\right)\left(\sum_{j=1}^{m}V_{k,j}^{2}\right)
=1m​n​∑k=1r1\displaystyle=\frac{1}{mn}\sum_{k=1}^{r}1
=rm​n\displaystyle=\frac{r}{mn}

Consistent Update RMS Across Muon and AdamW

A small-model ablation compares target update RMS values 0.05, 0.1, 0.2, 0.4, and 0.8 after about 2 billion tokens. Values 0.2 and 0.4 perform similarly and better than the other tested values; 0.2 is selected. This short-run calibration supports that recipe but does not establish a universal optimum.

Table 8: Muon Update RMS Experiments
Optimizer AdamW 0.05 RMS* 0.1 RMS 0.2 RMS 0.4 RMS 0.8 RMS
LM training loss 3.512 3.355 3.239 3.198 3.199 3.386
LM validation loss 3.679 3.503 3.374 3.325 3.314 3.543
AttnQ weight RMS 1.01e-2 5.74e-3 8.44e-3 1.57e-2 2.95e-2 7.23e-2
Mlp weight RMS 1.25e-2 8.01e-3 1.27e-2 2.35e-2 4.51e-2 8.73e-2

*Except the first column, all other candidates are using Muon with controlled RMS.

Appendix B AdamW Baseline Scaling Law

On the authors’ dataset, fitted compute-optimal model and token counts scale approximately as C⁰·⁵¹¹ and C⁰·⁴⁸⁹. The appendix also reports empirical learning-rate and batch-size formulas. Their coefficients depend on the compute and parameter conventions used in these experiments.

Table 9: Empirical Relationships Between Scaling Law Parameters and Computational Budget (FLOPs)
N⁡(C)N(C) D⁡(C)D(C) η⁡(C)\eta(C) B⁡(C)B(C)
0.0483359⋅C0.51126840.0483359\cdot C^{0.5112684} 3.4480927⋅C0.48873163.4480927\cdot C^{0.4887316} 0.0127339⋅C−0.05747520.0127339\cdot C^{-0.0574752} 0.0065202⋅C0.41379150.0065202\cdot C^{0.4137915}

Hyper-Parameters Search

For each budget, model size and token count are varied under the approximation C=6ND. Learning rate and batch size are then refined with grid searches; this process is repeated two or three times with narrower ranges. The observed loss bowls motivate the fitted choices, but a finite grid does not certify a global optimum over every training recipe.

Complete original A2.F5 from the exact-version PDF, including every panel, label, and original color.
Figure 5: Optimization Landscapes for Scaling Law Hyper-parameters Across FLOPs Budgets

Appendix C Model Architecture

Moonlight starts from a DeepSeek-V3-Small-like mixture-of-experts architecture and makes the following specific changes.

Multi-token Prediction (MTP)

Multi-token-prediction layers are omitted because they did not give a significant pretraining benefit in the authors’ experiments.

Auxfree Bias Update

The expert-bias update subtracts the mean sign of expert load deviations from each signed increment. This controls the common bias drift while leaving top-k ranking unchanged for a common additive shift. The complete source update remains visible.

Gate Scaling Factor

A routing-output scale of 2.446 is selected to approximately match dense-block output RMS. Figure 6 retains the exact Python code: sample Gaussian logits, apply sigmoid, choose and renormalize top-k weights, then average the inverse square root of their squared sum. This is a calibration under that synthetic-logit model.

Complete original A3.F6 from the exact-version PDF, including every panel, label, and original color.
Figure 6: Python implementation for calculating the gate scaling factor.
Accessible original algorithm or code markup
1 import numpy as np
2
3 def sigmoid(x):
4 return 1 / (1 + np.exp(-x))
5
6 def calc_gate_scaling_factor(num_experts: int, topk: int, iter_times: int):
7 """Calculate the gate scaling factor for MoE.
8
9 Args:
10 num_experts (int): The number of experts.
11 topk (int): The number of experts to select.
12 iter_timers (int): The number of iterations.
13
14 Returns:
15 float: The gate scaling factor.
16 """
17 factors = []
18 for _ in range(iter_times):
19
20 # mock gaussian logits
21 logits = np.random.randn(num_experts)
22 # select topk logits
23 p = np.sort(sigmoid(logits))[::-1]
24 p = p[:topk]
25 # renormalize
26 p = p / p.sum()
27 # calculate the scaling factor
28 factors.append( 1/ (p**2).sum()**0.5)
29 return np.mean(factors)

Appendix D Training Stability

Figure 7 preserves all four training-dynamics panels, including attention-logit diagnostics that qualify the smooth-loss observation.

Complete original A4.F7 from the exact-version PDF, including every panel, label, and original color.
Figure 7: Training dynamics comparison between Moonlight and Moonlight-A

No Loss or Grad Norm Spike

The authors report no loss or gradient-norm spikes in the Moonlight run. This describes the observed traces rather than a stability guarantee across seeds and settings.

Max Attention Logit

Some layers’ maximum attention logits exceed 100 early in training, while AdamW controls this statistic better. The fraction above 100 remains small, around 10⁻⁴, and maximum values later decline. Stable loss and gradient norm therefore coexist with an extreme-logit concern; the separate panels should not be collapsed into an unqualified stability claim.

RMSNorm Gamma Weight Decay

Applying decay to the RMSNorm gamma parameter is reported crucial for preventing excessive layer-output RMS in this recipe.

Appendix E Comparison with More Expensive Models

Moonlight is also compared with models trained using more compute, including Llama3.1-8B, Gemma2-9B, and Qwen2.5-7B. It beats some scores and trails others; for example Qwen2.5-7B is higher on MMLU, HumanEval, MBPP, GSM8K, and MATH. Figure 8 places selected GSM8K scores against reported FLOPs. These comparisons contextualize efficiency without isolating optimizer causality.

Table 10: Comparison of different models on various benchmarks.
Benchmark (Metric) Moonlight LLAMA3.1-8B Gemma2-9B Qwen2.5-7B
Larger Training Compute Model
Activated Param† 2.24B 7.38B 8.32B 6.83B
Total Params† 15.29B 7.38B 8.32B 6.83B
Training Tokens 5.7T 15T 8T 18T
Optimizer Muon AdamW Unknown Unknown
English MMLU 70.0 66.7 71.3 74.2
MMLU-pro 42.4 37.1 44.7 45.0
BBH 65.2 57.7 68.2 70.4
TriviaQA‡ 66.3 70.3 - 60.0
Code HumanEval 48.1 37.2 37.8 57.9
MBPP 63.8 47.6 62.2 74.9
Math GSM8K 77.4 57.2 70.7 85.4
MATH 45.3 20.3 37.7 49.8

† The reported parameter counts exclude the embedding parameters.‡ We test all listed models with the full set of TriviaQA.

Complete original A5.F8 from the exact-version PDF, including every panel, label, and original color.
Figure 8: The GSM8k performance of our Moonlight model optimized with Muon and other comparable models.

Appendix F Singular Value Distributions of Weight Matrices

The appendix preserves the complete attention and feed-forward spectral grids, normalizing each spectrum by its largest singular value and marking lower-entropy Muon cases with red spines. Projection labels distinguish compressed key/value representations, RoPE components, experts, shared experts, and router matrices. Most curves are flatter with Muon. The Figure 10 caption names three selected experts, whereas the displayed grid also includes an E1 group alongside E0, E2, and E3; both are retained as printed. The sampled matrices and descriptive entropy statistic do not establish a universal performance mechanism.

Complete original A6.F9 from the exact-version PDF, including every panel, label, and original color.
Figure 9: Distribution of singular values for each weight matrix in the attention layers. We use WC to denote the weight matrices at each layer that compress the hidden states to the shared latent spaces for keys and values, WV to denote the weight matrices up-projecting the values from the latent space, WO to denote the output projection matrices, and WKR, WKC, WQR and WQC to denote the projection matrices for the part of keys and queries with and without RoPE respectively. We set the spines of each line graph red if the corresponding weight matrix optimized by Muon has a lower singular entropy than AdamW.
Complete original A6.F10 from the exact-version PDF, including every panel, label, and original color.
Figure 10: Distribution of singular values for each weight matrix in the feed-forward network (FFN) layers. We use WI, WV and WO to denote the weight matrices involved in the FFN layer with SwiGLU activation function, where WI represents the input projection to the Swish1 function, WV represents the extra input projection interacting with Swish1 activations, and WO represents the output projection. We use E0, E2, E3 to denote three arbitrarily selected expert models and SE to denote the weights in the shared expert model. We use RW to denote the weights in the router. We set the spines of each line graph red if the corresponding weight matrix optimized by Muon has a lower singular entropy than AdamW.
Concept demonstration

Open image ↗