Literature
Display
← All papers

Keeping Neural Networks Simple by Minimizing the Description Length of the Weights

Geoffrey E. Hinton · Drew van Camp

· COLT 1993 ↗

Learning theory4 min text · CondensedOriginal paper ↗
Contents

At a glance

A neural network can be regularized by charging for the information in its weights. Hinton and van Camp learn independent Gaussian weight distributions and minimize expected prediction error plus a description-length penalty. Bits-back coding makes that penalty a KL divergence. Exact moment propagation is available for one nonlinear hidden layer with linear outputs. The small empirical study is promising but explicitly preliminary, and a lower-cost trivial solution exposes a weakness of the tested objective.

1 Introduction

With scarce training data, many adjustable weights can memorize outputs. Restricting connections, sharing weights or quantizing values reduces capacity, but each controls information only indirectly or complicates optimization. The paper instead learns how precisely each weight must be specified.

2 Applying the Minimum Description Length principle

A sender transmits a network’s weights and the residuals needed to reconstruct every training output. MDL trades the length of these two messages: added model complexity is worthwhile only when it reduces residual coding cost. The architecture is assumed agreed in advance [7].

3 Coding the data misfits

Assume finely quantized, Gaussian residuals for each output. Negative log probability yields a squared-error term. Optimizing residual variance gives a log mean-squared-error cost plus constants. The cost of transmitting the fitted variance is acknowledged but ignored in the implementation.

Figure 1. Gaussian probability mass for a quantized value: both density and precision determine coding cost.
Figure 1. Gaussian probability mass for a quantized value: both density and precision determine coding cost.
Equations 1–3
(1)Equation 1 from the original paper.
(2)Equation 2 from the original paper.
(3)Equation 3 from the original paper.

4 A simple method of coding the weights

A fixed zero-mean Gaussian weight code gives the familiar squared-weight penalty. An adaptive Gaussian mixture can exploit weight clusters [6], but fixed precision still wastes bits on weights whose exact values barely affect predictions. The next construction learns this precision during training.

Equation 4
(4)Equation 4 from the original paper.

5 Noisy weights

Replace one weight vector by a factorized Gaussian distribution, learning a mean and variance for every weight. More variance makes a weight cheaper to communicate but raises expected prediction error. The objective jointly balances these effects.

5.1 Expected description length of the weights

Given an agreed Gaussian coding prior P and a learned Gaussian posterior Q, the effective cost of communicating a weight is KL(Q‖P). The direction is fixed by averaging under the learned distribution Q.

Equation 5
(5)Equation 5 from the original paper.

5.2 The bits-back argument

Sample a precise weight from Q using random bits, encode it under P, then transmit the resulting output residuals. Once the receiver reconstructs the training outputs, the shared learning procedure recovers Q and the random bits used to select the sample. Subtracting those recovered bits leaves the KL cost rather than the full precision cost.

Equations 6–9
(6)Equation 6 from the original paper.
(7)Equation 7 from the original paper.
(8)Equation 8 from the original paper.
(9)Equation 9 from the original paper.

5.3 Expected description length of the data misfits

Independent Gaussian weights make each hidden preactivation Gaussian. A precomputed table maps its mean and variance through the nonlinearity. Independence across hidden units and linear output units then allow exact output moments and expected squared error. A second derivative table supports backpropagation through these moments. This exact argument applies to the stated one-hidden-layer architecture, not arbitrary deep networks.

Equations 10–14
(10)Equation 10 from the original paper.
(11)Equation 11 from the original paper.
(12)Equation 12 from the original paper.
(13)Equation 13 from the original paper.
(14)Equation 14 from the original paper.

6 Letting the data determine the prior

The coding prior’s mean and variance are learned jointly with the weight distributions. A complete Bayesian treatment could charge for transmitting these parameters using hyperpriors; the experiments ignore that cost. This makes the coding prior data-dependent and limits a literal full-description interpretation.

6.1 A more flexible prior distribution

An adaptive Gaussian mixture encodes clusters of weights more economically than one broad Gaussian. The exact KL from a Gaussian posterior to a mixture has an intractable log-sum expectation. A tractable upper bound uses componentwise Gaussian KL values inside a weighted log-sum-exp. The mixture parameters’ own coding cost is again omitted.

Equations 15–17
(15)Equation 15 from the original paper.
(16)Equation 16 from the original paper.
(17)Equation 17 from the original paper.

7 Coding with a Gaussian mixture

The four-step protocol below transmits a component choice and sampled weight, then recovers both sources of randomness. Choosing component probabilities exponentially in their KL costs produces the tractable upper bound. The steps and equations are retained in full.

Mixture coding procedure

  1. Randomly pick one of the Gaussians in the mixture with probability rᵢ given by Equation 18.
  2. Communicate the choice of Gaussian to the receiver. If we use the mixing proportions as a prior for communicating the choice, the expected code cost is Equation 19.
  3. Communicate the sample value to the receiver using the chosen Gaussian. Taking into account the random bits recovered when the receiver reconstructs the posterior, the expected sample cost is Equation 20. The combined component-choice and sample cost is Equation 21.
  4. After receiving samples from all posterior weight distributions and the training errors, the receiver runs the learning algorithm to reconstruct those posteriors and all component divergences. This recovers the random bits used to choose a Gaussian. The bits-back quantity to subtract is the component-choice entropy in Equation 22.
Equations 18–22
(18)Equation 18 from the original paper.
(19)Equation 19 from the original paper.
(20)Equation 20 from the original paper.
(21)Equation 21 from the original paper.
(22)Equation 22 from the original paper.

7.1 A statistical-mechanics lemma

At unit temperature, expected energy minus entropy is minimized by Boltzmann probabilities. Identifying each mixture component’s effective energy with its KL cost minus log mixture weight turns the optimized free energy into the negative log partition sum used in the bound.

Equations 23–25
(23)Equation 23 from the original paper.
(24)Equation 24 from the original paper.
(25)Equation 25 from the original paper.

8 Implementation

Finite-difference checks compare each predicted gradient with the actual change in the objective. A 300×300 lookup table with linear interpolation gave reasonably accurate derivatives in this implementation. The check tests derivatives; it does not establish that the objective has a useful global optimum.

9 Preliminary results

One peptide-effectiveness task uses 128 inputs, four hidden units and 521 weights, trained on 105 examples and tested on 420. Five Gaussian prior components and all weight distributions are optimized jointly. Starting the weight penalty at full strength collapses predictions to the mean, so the authors gradually raise its coefficient from 0.05 to 1. Across five starts, the selected useful solution has relative test squared error 0.286, versus 0.967 without information regularization and 0.317 for validation-tuned weight decay. Test-tuned 0.291 comparisons are explicitly acknowledged as invalid model selection.

The evidence is a single small task, not a broad benchmark. More seriously, the constant predictor still attains a lower total description length than the useful solution and has relative error near one. The paper calls this a failure of the MDL principle or of the chosen coding scheme; it must not be hidden by the favorable result.

Figure 2. Final incoming and outgoing weights of the four hidden units. Sign is shown by black/white, magnitude by rectangle area.
Figure 2. Final incoming and outgoing weights of the four hidden units. Sign is shown by black/white, magnitude by rectangle area.
Figure 3. Learned coding distribution, represented by an adaptive mixture of five Gaussians.
Figure 3. Learned coding distribution, represented by an adaptive mixture of five Gaussians.
Equations 26–27
(26)Equation 26 from the original paper.
(27)Equation 27 from the original paper.

10 Discussion

Exact Bayesian prediction averages over the full posterior but is generally intractable. Monte Carlo [5] preserves complex posterior shape; Laplace methods [4] approximate a local optimum. This method instead optimizes a diagonal Gaussian throughout learning. Ignoring weight correlations may matter, and suppression of those correlations is only a proposed mechanism. Weight noise also smooths activation probabilities, potentially permitting threshold hidden units.

11 Acknowledgments

The work acknowledges NSERC support and discussions with David MacKay, Radford Neal, Chris Williams and Richard Zemel. Hinton was a fellow of the Canadian Institute for Advanced Research.

Concept demonstration

Open image ↗