Overview
A Neural Turing Machine combines a neural controller with an addressable memory matrix. Differentiable reads, writes, and address shifts allow training from examples with gradient descent. Synthetic tasks show learning and some extrapolation of simple memory procedures; the system is finite and approximate, rather than an unrestricted reliable computer.
1 Introduction
An RNN may represent complex procedures in principle, yet learning them from data can be difficult. External memory provides an explicit place to store temporary variables and revisit them. The paper draws an analogy to working memory and digital computers while designing a trainable architecture rather than a biological model.
2 Foundational Research
2.1 Psychology and Neuroscience
Working-memory research motivates separating a controller from selective storage. The proposed system emphasizes flexible addressing, which is absent from many earlier neural accounts. It does not seek to reproduce human capacity limits or establish a neural implementation in the brain.
2.2 Cognitive Science and Linguistics
Variable binding and variable-length structures have long challenged connectionist models of cognition. Addressable memory offers one mechanism for assigning transient information to reusable locations. The discussion situates this design within earlier distributed representations and debates about recursive computation.
2.3 Recurrent Neural Networks
LSTM gates protect additive memory updates against vanishing or exploding sensitivity. Recurrent networks already support variable-length inputs; the new contribution is an explicitly addressable memory with learned access. Differentiable attention and earlier neural program-learning work are important precursors.
3 Neural Turing Machines
The controller receives external inputs and previous read vectors, produces outputs, and emits parameters for read and write heads. A head assigns normalized weights to memory rows rather than selecting an inherently discrete address. Sharply concentrated weights approximate ordinary memory access while retaining differentiability.
3.1 Reading
A read vector is the weighted average of memory rows. Its weights are nonnegative and sum to one, making the operation differentiable with respect to both memory contents and the chosen address distribution.
Equations 1–2
| (1) |
| (2) |
3.2 Writing
Writing first erases selected components and then adds a vector, with both operations weighted by address. Separate erase and add vectors permit fine-grained updates within a row. Multiple heads’ erase operations commute, as do their additions.
Equations 3–4
| (3) |
| (4) |
3.3 Addressing Mechanisms
Content addressing locates a row by its stored value; location addressing moves relative to a prior position. Their combination supports both associative jumps and sequential traversal. Figure 2 shows content lookup, interpolation, shifting, and sharpening in their actual order.
3.3.1 Focusing by Content
Each head compares a learned key with all rows using cosine similarity. A positive key-strength parameter controls concentration before softmax normalization. A partial content match can therefore retrieve an associated stored vector.
Equations 5–6
| (5) |
| (6) |
3.3.2 Focusing by Location
An interpolation gate blends the previous address distribution with a fresh content lookup. Circular convolution shifts that distribution, and an exponent then sharpens it to counter diffusion. The head can retain its position, jump by content, move relative to a match, or iterate through locations; finite circular memory can eventually wrap and overwrite data.
Equations 7–9
| (7) |
| (8) |
| (9) |
3.4 Controller Network
A recurrent controller has internal state in addition to external memory, analogous to temporary registers. A feed-forward controller must use external memory for temporal information and may expose a more readable access pattern. Its number of simultaneous read heads limits how many stored vectors it can combine in one step.
4 Experiments
Five supervised synthetic tasks compare memory systems with feed-forward or LSTM controllers against a standalone LSTM. Dynamic state resets between episodes. Binary cross-entropy is reported as bits per sequence, and testing beyond training lengths probes whether a reusable procedure was learned.
4.1 Copy
Training copies random eight-bit vectors in sequences of length 1–20. The memory models learn faster and extrapolate better than the standalone LSTM. A 120-step example still contains local mistakes and an insertion that shifts later outputs. The authors infer the preserved copy pseudocode from access traces: write sequentially, return to the start, and read sequentially. Memory capacity ultimately limits this extrapolation.
4.2 Repeat Copy
The repeat task adds a requested repetition count and an end marker. The memory model generalizes to longer input sequences, but beyond trained repetition counts it fails to stop correctly. This separates learning a reusable copying loop from learning an extrapolating numerical counter.
4.3 Associative Recall
After observing several items, the network receives one as a query and must return its successor. A feed-forward controller remains nearly perfect at twice the largest training length. Traces suggest it writes a compressed key at each delimiter, retrieves that key by content, then shifts to the next item. This is the authors’ interpretation of the learned access behavior.
4.4 Dynamic N-Grams
Each episode samples a new binary six-gram distribution. Predicting the next bit requires tracking the five-bit context and updating counts within the episode. The paper derives an optimal Bayesian predictor; memory models outperform the tested LSTM but do not reach it exactly. Access errors in the preserved examples expose limitations of the inferred counting mechanism.
Equation 10
| (10) |
4.5 Priority Sort
The sorting task presents twenty vectors with priorities and requests the sixteen highest-priority vectors in order. Memory access suggests writing near locations determined by priority and then reading sequentially. Both controller types outperform the LSTM baseline; the best feed-forward version uses eight read/write heads. This is a specific finite sorting task, not a proof of general exact sorting.
4.6 Experimental Details
All models use RMSProp with momentum and elementwise gradient clipping. Tables 1–3 retain controller sizes, memory dimensions, learning rates, heads, and parameter counts. Adding memory rows need not add learned parameters, unlike increasing LSTM hidden width, though memory operations still cost computation.
| Task | #Heads | Controller Size | Memory Size | Learning Rate | #Parameters |
|---|---|---|---|---|---|
| Copy | 1 | 100 | 128 20 | ||
| Repeat Copy | 1 | 100 | 128 20 | ||
| Associative | 4 | 256 | 128 20 | ||
| N-Grams | 1 | 100 | 128 20 | ||
| Priority Sort | 8 | 512 | 128 20 |
| Task | #Heads | Controller Size | Memory Size | Learning Rate | #Parameters |
|---|---|---|---|---|---|
| Copy | 1 | 100 | 128 20 | ||
| Repeat Copy | 1 | 100 | 128 20 | ||
| Associative | 1 | 100 | 128 20 | ||
| N-Grams | 1 | 100 | 128 20 | ||
| Priority Sort | 5 | 128 20 |
| Task | Network Size | Learning Rate | #Parameters |
|---|---|---|---|
| Copy | |||
| Repeat Copy | |||
| Associative | |||
| N-Grams | |||
| Priority Sort |
5 Conclusion
Differentiable external memory can make simple algorithmic tasks easier to learn and support extrapolation beyond training lengths. The demonstrations also reveal stopping, addressing, and capacity failures. The results are preliminary evidence on synthetic tasks rather than general program synthesis.
6 Acknowledgments
The authors thank colleagues from DeepMind, Google, and cognitive science for discussions and feedback.