Literature
Display
← All papers

Decoupled Weight Decay Regularization

Ilya Loshchilov · Frank Hutter

· ICLR 2019 ↗

Contents

Overview

At a glance

Intuition. Training has two jobs here: improve predictions and gently shrink weights. Adam rescales each coordinate’s gradient using its recent history. If shrinkage is included inside that gradient, it gets rescaled too, so two equally large weights can receive different shrinkage. AdamW applies shrinkage separately.

Reading the math. As a simple illustration, multiplying weights 2 and 0.2 by 0.99 gives 1.98 and 0.198: the same percentage reduction. Adam’s separate prediction-improving step still uses m, a moving average of gradients, and v, a moving average of squared gradients. These gradient statistics are not the full Fisher metric.

Weight decay and an L2 penalty on the loss are equivalent for ordinary stochastic gradient descent after rescaling the coefficient, but generally differ for adaptive optimizers such as Adam. AdamW applies parameter shrinkage separately from the gradient preconditioning. On the tested image-classification tasks, this improves generalization and makes learning-rate and decay tuning less coupled. The paper also studies normalized decay and cosine warm restarts; these empirical improvements are distinct from the algebraic equivalence result.

1 Introduction

Adaptive methods often optimize training loss quickly yet generalize less well than SGD on the cited image-classification problems. The authors argue that a common implementation detail matters: adding an L2 term to the loss lets Adam’s coordinate-wise preconditioner also scale regularization. Decoupled decay instead directly shrinks the parameters. Contributions include this distinction and the AdamW update, image-classification comparisons, a heuristic normalization of the decay parameter, and AdamWR with warm restarts. Evidence concerns the tested training settings rather than a universal ranking of optimizers. Source code is provided at AdamW-and-SGDW.

2 Decoupling the Weight Decay from the Gradient-based Update

For SGD, the update with multiplicative decay λ equals an L2-regularized update when the loss penalty coefficient is λ′=λ/α, where α is the learning rate. For a non-scalar adaptive preconditioner Mₜ, the penalty gradient is multiplied by Mₜ whereas direct decay is not; no single scalar L2 coefficient reproduces that shrinkage for every parameter vector. AdamW therefore moves the decay term outside Adam’s moment-based preconditioning. Algorithms 1 and 2 show two alternatives in one display: pink terms belong to L2 regularization, green terms to decoupled decay. They must not be applied together. The common schedule multiplier ηₜ scales the update, including decay; separate tuning parameters need not be statistically independent in every problem. Proposition 3 interprets a fixed diagonal preconditioner through a scale-adjusted quadratic penalty, but explicitly does not cover a changing preconditioner. Source-version qualification: Eq. 2 prints λ′/(2α) while also defining λ′=λ/α. Its Appendix A proof instead uses λ′/2, which yields the stated decay under that definition. Both original expressions are retained; the extra α in Eq. 2 is not silently corrected.

Equation 1
𝜽t+1=(1−λ)𝜽t−α∇ft(𝜽t),\displaystyle\bm{\theta}_{t+1}=(1-\lambda)\bm{\theta}_{t}-\alpha\nabla f_{t}(\bm{\theta}_{t}), (1)
Complete original alg1 from the exact-version PDF, including every panel, label, and original color.
Algorithm 1 SGD with L2 regularization and SGD with decoupled weight decay (SGDW), both with momentum
Accessible original algorithm markup (pink and green terms are alternatives)
1:  given initial learning rate α∈IR\alpha\in{\rm I\hskip-2.15277ptR}, momentum factor β1∈IR\beta_{1}\in{\rm I\hskip-2.15277ptR}, weight decay/L2 regularization factor λ∈IR\lambda\in{\rm I\hskip-2.15277ptR}
2:  initialize time step t←0t\leftarrow 0, parameter vector 𝜽t=0∈IRn\bm{\theta}_{t=0}\in{\rm I\hskip-2.15277ptR}^{n}, first moment vector mt=0←0\textit{{m}}_{t=0}\leftarrow\textit{{0}}, schedule multiplier ηt=0∈IR\eta_{t=0}\in{\rm I\hskip-2.15277ptR}
3:  repeat
4:   t←t+1t\leftarrow t+1
5:   ∇ft​(𝜽t−1)←SelectBatch​(𝜽t−1)\nabla f_{t}(\bm{\theta}_{t-1})\leftarrow\text{SelectBatch}(\bm{\theta}_{t-1}) ⊳\triangleright  select batch and return the corresponding gradient
6:   gt←∇ft​(𝜽t−1)\textit{{g}}_{t}\leftarrow\nabla f_{t}(\bm{\theta}_{t-1}) +λ​𝜽t−1\displaystyle+\lambda\bm{\theta}_{t-1}
7:   ηt←SetScheduleMultiplier​(t)\eta_{t}\leftarrow\text{SetScheduleMultiplier}(t) ⊳\triangleright  can be fixed, decay, be used for warm restarts
8:   mt←β1​mt−1+ηt​α​gt\textit{{m}}_{t}\leftarrow\beta_{1}\textit{{m}}_{t-1}+\eta_{t}\alpha\textit{{g}}_{t}
9:   𝜽t←𝜽t−1−mt\bm{\theta}_{t}\leftarrow\bm{\theta}_{t-1}-\textit{{m}}_{t} −ηt​λ​𝜽t−1\displaystyle-\eta_{t}\lambda\bm{\theta}_{t-1}
10:  until stopping criterion is met
11:  return optimized parameters 𝜽t\bm{\theta}_{t}
Complete original alg2 from the exact-version PDF, including every panel, label, and original color.
Algorithm 2 Adam with L2 regularization and Adam with decoupled weight decay (AdamW)
Accessible original algorithm markup (pink and green terms are alternatives)
1:  given α=0.001,β1=0.9,β2=0.999,ϵ=10−8,λ∈IR\alpha=0.001,\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8},\lambda\in{\rm I\hskip-2.15277ptR}
2:  initialize time step t←0t\leftarrow 0, parameter vector 𝜽t=0∈IRn\bm{\theta}_{t=0}\in{\rm I\hskip-2.15277ptR}^{n}, first moment vector mt=0←0\textit{{m}}_{t=0}\leftarrow\textit{{0}}, second moment vector vt=0←0\textit{{v}}_{t=0}\leftarrow\textit{{0}}, schedule multiplier ηt=0∈IR\eta_{t=0}\in{\rm I\hskip-2.15277ptR}
3:  repeat
4:   t←t+1t\leftarrow t+1
5:   ∇ft​(𝜽t−1)←SelectBatch​(𝜽t−1)\nabla f_{t}(\bm{\theta}_{t-1})\leftarrow\text{SelectBatch}(\bm{\theta}_{t-1}) ⊳\triangleright  select batch and return the corresponding gradient
6:   gt←∇ft​(𝜽t−1)\textit{{g}}_{t}\leftarrow\nabla f_{t}(\bm{\theta}_{t-1}) +λ​𝜽t−1\displaystyle+\lambda\bm{\theta}_{t-1}
7:   mt←β1​mt−1+(1−β1)​gt\textit{{m}}_{t}\leftarrow\beta_{1}\textit{{m}}_{t-1}+(1-\beta_{1})\textit{{g}}_{t} ⊳\triangleright  here and below all operations are element-wise
8:   vt←β2​vt−1+(1−β2)​gt2\textit{{v}}_{t}\leftarrow\beta_{2}\textit{{v}}_{t-1}+(1-\beta_{2})\textit{{g}}^{2}_{t}
9:   m^t←mt/(1−β1t)\hat{\textit{{m}}}_{t}\leftarrow\textit{{m}}_{t}/(1-\beta_{1}^{t}) ⊳\triangleright  β1\beta_{1} is taken to the power of tt
10:   v^t←vt/(1−β2t)\hat{\textit{{{v}}}}_{t}\leftarrow\textit{{v}}_{t}/(1-\beta_{2}^{t}) ⊳\triangleright  β2\beta_{2} is taken to the power of tt
11:   ηt←SetScheduleMultiplier​(t)\eta_{t}\leftarrow\text{SetScheduleMultiplier}(t) ⊳\triangleright  can be fixed, decay, or also be used for warm restarts
12:   𝜽t←𝜽t−1−ηt​(α​m^t/(v^t+ϵ)​+λ​𝜽t−1)\bm{\theta}_{t}\leftarrow\bm{\theta}_{t-1}-\eta_{t}\left(\alpha\hat{\textit{{m}}}_{t}/(\sqrt{\hat{\textit{{v}}}_{t}}+\epsilon)\hbox{\pagecolor{SpringGreen}$\displaystyle+\lambda\bm{\theta}_{t-1}$}\right)
13:  until stopping criterion is met
14:  return optimized parameters 𝜽t\bm{\theta}_{t}
Equation 2
ftsreg​(𝜽)=ft​(𝜽)+λ′2​α​‖𝜽⊙s‖22,f_{t}^{\text{sreg}}(\bm{\theta})=f_{t}(\bm{\theta})+\frac{\lambda^{\prime}}{2\alpha}\left\lVert\bm{\theta}\odot{}\sqrt{\textit{{s}}}\right\rVert_{2}^{2},\vskip-2.84544pt (2)

3 Justification of Decoupled Weight Decay via a View of Adaptive Gradient Methods as Bayesian Filtering

The Bayesian-filtering interpretation is attributed to Aitchison’s subsequent analysis [Aitchison, 2018]. Under a Gaussian transition model and an approximate conjugate likelihood, posterior uncertainty preconditions the mini-batch gradient. A transition mean (I−A)θ with A=λI supplies parameter shrinkage separately from that gradient update. This motivates decoupled decay within the assumed filtering model; it is not a proof that AdamW must generalize better for arbitrary data and architectures. The paper distinguishes this contribution from its earlier empirical observation.

Equations 3–4
𝝁p​o​s​t=𝝁p​r​i​o​r+𝚺p​o​s​t×𝒈,\bm{\mu}_{post}=\bm{\mu}_{prior}+\bm{\Sigma}_{post}\times\bm{g}, (3)
P⁡(𝜽t+1∣𝜽t)=𝒩⁡((𝑰−𝑨)​𝜽t,𝑸),P(\bm{\theta}_{t+1}\mid\bm{\theta}_{t})=\mathcal{N}((\bm{I}-\bm{A})\bm{\theta}_{t},\bm{Q}), (4)
Complete original S3.F1 from the exact-version PDF, including every panel, label, and original color.
Figure 1: Adam performs better with decoupled weight decay (bottom row, AdamW) than with L2L_{2} regularization (top row, Adam). We show the final test error of a 26 2x64d ResNet on CIFAR-10 after 100 epochs of training with fixed learning rate (left column), step-drop learning rate (with drops at epoch indexes 30, 60 and 80, middle column) and cosine annealing (right column). AdamW leads to a more separable hyperparameter search space, especially when a learning rate schedule, such as step-drop and cosine annealing is applied. Cosine annealing yields clearly superior results.

4 Experimental Validation

Experiments use Shake-Shake residual networks, batch size 128, and the data preprocessing and augmentation of the cited implementation. The main CIFAR-10 model is a 26 2x64d network with 11.6 million parameters; longer experiments use a 26 2x96d model with 25.6 million. ImageNet32x32 retains 1.2 million training examples and 1,000 classes at 32×32 resolution. These architecture, schedule, and budget choices matter when interpreting the optimizer comparisons.

4.1 Evaluating Decoupled Weight Decay With Different Learning Rate Schedules

On CIFAR-10, a 100-epoch learning-rate/regularization grid compares constant rates, step reductions at epochs 30, 60, and 80, and cosine annealing. AdamW improves over Adam with L2 regularization in these grids, and cosine annealing gives the best results. Later experiments consequently use cosine schedules. Figure 1 preserves all six heat maps and their separately evaluated settings.

Complete original S4.F2 from the exact-version PDF, including every panel, label, and original color.
Figure 2: The Top-1 test error of a 26 2x64d ResNet on CIFAR-10 measured after 100 epochs. The proposed SGDW and AdamW (right column) have a more separable hyperparameter space.
Complete original S4.F3 from the exact-version PDF, including every panel, label, and original color.
Figure 3: Learning curves (top row) and generalization results (bottom row) obtained by a 26 2x96d ResNet trained with Adam and AdamW on CIFAR-10. See text for details. SuppFigure 4 in the Appendix shows the same qualitative results for ImageNet32x32.

4.2 Decoupling the Weight Decay and Initial Learning Rate Parameters

Figure 2 compares SGD and Adam with either L2 regularization or decoupled decay. The favorable regions are more nearly separable in learning-rate and decay coordinates for SGDW and AdamW, simplifying tuning in these experiments. This empirical geometry is stronger evidence for practical tuning behavior than the claim that all optimal settings are independent.

4.3 Better Generalization of AdamW

With learning rate 0.001, the larger CIFAR-10 network is trained for 1,800 epochs across 12 L2 settings and seven normalized-decay settings. AdamW achieves lower training loss and lower test error, including comparisons at similar training loss. The reported relative test-error improvement is about 15% on CIFAR-10 and ImageNet32x32 at this default learning rate. The plotted range of hyperparameters and training budgets defines that comparison; the result is not a benchmark-wide guarantee.

Complete original S4.F4 from the exact-version PDF, including every panel, label, and original color.
Figure 4: Top-1 test error on CIFAR-10 (left) and Top-5 test error on ImageNet32x32 (right). For a better resolution and with training loss curves, see SuppFigure 5 and SuppFigure 6 in the supplementary material.

4.4 AdamWR with Warm Restarts for Better Anytime Performance

AdamWR combines decoupled decay with cosine warm restarts, analogous to SGDWR. Restarting the schedule keeps the learned weights and raises the learning-rate multiplier again. The first restart yields competitive checkpoints up to roughly ten times earlier in the illustrated learning curves, which is an anytime-performance statement rather than a tenfold reduction in the cost of each step. AdamWR closes much of the gap to SGDWR on CIFAR-10 and is comparable on ImageNet32x32; later checkpoints need not favor the same method.

4.5 Use of AdamW on other datasets and architectures

The authors discuss external applications rather than presenting a single controlled cross-domain benchmark. A face-analysis architecture reports faster predictions, but that system-level inference improvement cannot be attributed solely to its optimizer. In an EEG comparison supplied by collaborators, AdamW versus Adam reaches 73.68% versus 71.37% for Deep4Net and 72.04% versus 61.34% for a ResNet; the latter difference is reported significant. Language-model training and approximate natural-gradient work also use decoupled decay. These examples motivate broader study while retaining their separate architectures, data, and comparison conditions.

5 Conclusion and Future Work

The central recommendation is to distinguish direct parameter decay from an L2 loss penalty when implementing adaptive optimization. The image-classification experiments favor AdamW and show benefits from schedule design and decay normalization. Generality beyond these settings and adaptation to other optimizers remain open empirical questions. The fixed-preconditioner interpretation and the Bayesian model explain particular mechanisms without establishing universal generalization guarantees.

6 Acknowledgments

The authors acknowledge collaborators, reviewers, and compute assistance, and report support including the European Research Council and German Research Foundation. The complete original reference list is retained.

Appendix A Formal Analysis of Weight Decay vs L 2 Regularization

Appendix A proves the SGD identity by comparing coefficients of θ. For adaptive preconditioning, equality for all θ would require λI=αλ′Mₜ, impossible for the stipulated non-scalar Mₜ. For fixed diagonal scaling 1/s, a penalty λ′‖θ⊙√s‖²/2 yields a preconditioned penalty gradient λ′θ; setting λ′=λ/α reproduces direct decay. This last proof is the normalization used in the explanation above, and differs from the extra α printed in main-text Eq. 2. The derivations and their original notation remain available in full.

Equations 5–13
𝜽t+1←𝜽t−α∇ftreg(𝜽t)=𝜽t−α∇ft(𝜽t)−αλ′𝜽t.\bm{\theta}_{t+1}\leftarrow\bm{\theta}_{t}-\alpha\nabla f_{t}^{\text{reg}}(\bm{\theta}_{t})=\bm{\theta}_{t}-\alpha\nabla f_{t}(\bm{\theta}_{t})-\alpha\lambda^{\prime}\bm{\theta}_{t}. (5)
𝜽t+1←(1−λ)𝜽t−α∇ft(𝜽t).\bm{\theta}_{t+1}\leftarrow(1-\lambda)\bm{\theta}_{t}-\alpha\nabla f_{t}(\bm{\theta}_{t}). (6)
𝜽t+1\displaystyle\bm{\theta}_{t+1} ←\displaystyle\leftarrow 𝜽t−αλ′𝐌t𝜽t−α𝐌t∇ft(𝜽t).\displaystyle\bm{\theta}_{t}-\alpha\lambda^{\prime}\mathbf{M}_{t}\bm{\theta}_{t}-\alpha\mathbf{M}_{t}\nabla f_{t}(\bm{\theta}_{t}). (7)
𝜽t+1\displaystyle\bm{\theta}_{t+1} ←\displaystyle\leftarrow (1−λ)𝜽t−α𝐌t∇ft(𝜽t).\displaystyle(1-\lambda)\bm{\theta}_{t}-\alpha\mathbf{M}_{t}\nabla f_{t}(\bm{\theta}_{t}). (8)
𝜽t+1\displaystyle\bm{\theta}_{t+1} ←\displaystyle\leftarrow 𝜽t−α∇ftsreg(𝜽t)/s\displaystyle\bm{\theta}_{t}-\alpha\nabla f_{t}^{\text{sreg}}(\bm{\theta}_{t})/\textit{{s}} (9)
=\displaystyle= 𝜽t−α∇ft(𝜽t)/s−αλ′𝜽t⊙s/s\displaystyle\bm{\theta}_{t}-\alpha\nabla f_{t}(\bm{\theta}_{t})/\textit{{s}}-\alpha\lambda^{\prime}\bm{\theta}_{t}\odot\textit{{s}}/\textit{{s}} (10)
=\displaystyle= 𝜽t−α∇ft(𝜽t)/s−αλ′𝜽t,\displaystyle\bm{\theta}_{t}-\alpha\nabla f_{t}(\bm{\theta}_{t})/\textit{{s}}-\alpha\lambda^{\prime}\bm{\theta}_{t}, (11)
𝜽t+1\displaystyle\bm{\theta}_{t+1} ←\displaystyle\leftarrow (1−λ)𝜽t−α∇f(𝜽t)/s\displaystyle(1-\lambda)\bm{\theta}_{t}-\alpha\nabla f(\bm{\theta}_{t})/\textit{{s}} (12)
=\displaystyle= 𝜽t−α∇f(𝜽t)/s−λ𝜽t,\displaystyle\bm{\theta}_{t}-\alpha\nabla f(\bm{\theta}_{t})/\textit{{s}}-\lambda\bm{\theta}_{t}, (13)

Appendix B Additional Practical Improvements of Adam

The following additions address how decay and learning-rate schedules change across training budgets; they are separate from the definition of AdamW.

B.1 Normalized Weight Decay

The heuristic sets λ=λ_norm√(b/(BT)), where b is batch size, B the number of training examples, and T the number of epochs. In a restarted run, T refers to the current cycle. This reduces the amount of retuning observed when budgets change and treats λ_norm as a normalized decay hyperparameter. The authors describe the evidence as preliminary and limited; this scaling is not derived as a universally optimal regularizer.

B.2 Adam with Cosine Annealing and Warm Restarts

Cosine annealing varies η between specified maximum and minimum values as a function of the fraction of the current cycle completed. Typical bounds are one and zero. At a warm restart the schedule is reset but the parameter vector is retained. Cycle lengths can grow by a fixed factor, here commonly two. This schedule is applied to both the gradient update and decoupled shrinkage as written in the algorithms.

Equations 14–15
ηt=ηm​i​n(i)+0.5​(ηm​a​x(i)−ηm​i​n(i))​(1+cos⁡(π​Tc​u​r/Ti)),\displaystyle\eta_{t}=\eta^{(i)}_{min}+0.5(\eta^{(i)}_{max}-\eta^{(i)}_{min})(1+\cos(\pi T_{cur}/{T_{i}})), (14)
ηt=0.5+0.5​cos⁡(π​Tc​u​r/Ti).\displaystyle\eta_{t}=0.5+0.5\cos(\pi T_{cur}/{T_{i}}). (15)

Appendix C An Example Setting of the Schedule Multiplier

The example starts with 100 epochs and doubles cycle length, producing cycles of 100, 200, 400, and 800 epochs and checkpoints at 100, 300, 700, and 1,500 epochs. Supplementary Figure 1 shows the complete multiplier schedule.

Complete original A3.F1 from the exact-version PDF, including every panel, label, and original color.
SuppFigure  1: An example schedule of the learning rate multiplier as a function of epoch index. The first run is scheduled to converge at epoch Ti=0=100T_{i=0}=100, then the budget for the next run is doubled as Ti=1=Ti=0​Tm​u​l​t=200T_{i=1}=T_{i=0}T_{mult}=200, etc.

Appendix D Additional Results

The supplementary results retain all six figures. A coarse 4×4 grid of fixed-rate Adam settings after 1,800 epochs reaches performance at best comparable to the earlier 100-epoch AdamW result, but the compared network sizes differ, so this is not a clean matched-compute speedup. The best raw decay decreases as the training budget grows. Normalized values around 0.025–0.05 work across the tested budgets and datasets, while an ImageNet32x32 epoch has about 24 times as many examples as a CIFAR-10 epoch; omitting the normalization can make decay about five times too large in that comparison. Supplementary Figure 4 repeats the loss/generalization analysis for ImageNet32x32, and Figures 5–6 add full training-loss curves to the restart comparisons. These observations support the proposed normalization in this experimental range, not an unrestricted scaling law.

Complete original A4.F2 from the exact-version PDF, including every panel, label, and original color.
SuppFigure  2: Performance of “standard Adam”: Adam with L2 regularization and a fixed learning rate. We show the final test error of a 26 2x96d ResNet on CIFAR-10 after 1800 epochs of the original Adam for different settings of learning rate and weight decay used for L2 regularization.
Complete original A4.F3 from the exact-version PDF, including every panel, label, and original color.
SuppFigure  3: Effect of normalized weight decay. We show the final test Top-1 error on CIFAR-10 (first two rows for AdamW without and with normalized weight decay) and Top-5 error on ImageNet32x32 (last two rows for AdamW and SGDW, both with normalized weight decay) of a 26 2x64d ResNet after different numbers of epochs (see columns). While the optimal settings of the raw weight decay change significantly for different runtime budgets (see the first row), the values of the normalized weight decay remain very similar for different budgets (see the second row) and different datasets (here, CIFAR-10 and ImageNet32x32), and even across AdamW and SGDW.
Complete original A4.F4 from the exact-version PDF, including every panel, label, and original color.
SuppFigure  4: Learning curves (top row) and generalization results (Top-5 errors in bottom row) obtained by a 26 2x96d ResNet trained with Adam and AdamW on ImageNet32x32.
Complete original A4.F5 from the exact-version PDF, including every panel, label, and original color.
SuppFigure  5: Test error curves (top row) and training loss curves (bottom row) for CIFAR-10.
Complete original A4.F6 from the exact-version PDF, including every panel, label, and original color.
SuppFigure  6: Test error curves (top row) and training loss curves (bottom row) for ImageNet32x32.
Concept demonstration

Open image ↗