At a glance
The Relational Memory Core combines a fixed recurrent memory with self-attention between memory slots. It improves synthetic sequential reasoning, program evaluation, partially observed games, and language modeling. The evidence demonstrates better task performance; the authors explicitly avoid claiming that attention maps establish a unique internal reasoning mechanism.
1 Introduction
Memory models commonly emphasize storage and retrieval. This work adds a bias toward interactions among remembered items, so a model can compare information received at different times without retaining an ever-growing history of hidden states.
2 Relational reasoning
Relational reasoning compares entities rather than processing their properties independently. Graph networks and relation modules make this bias explicit in spatial settings. A recurrent memory can represent entities across time, but a single undifferentiated state or a passive slot store does not directly encourage interactions between them.
3 Model
The model keeps a matrix of memory slots, applies attention among those slots and new input, and integrates proposed updates through recurrent gates. Unlike attention over a complete history, attention operates on a fixed-size state at the current timestep.
3.1 Allowing memories to interact using multi-head dot product attention
Each slot produces queries, keys, and values. Scaled dot-product attention returns a weighted combination of slot values; multiple heads can exchange different information in parallel. The concatenated head outputs propose an update with the same dimensions as the original memory.
Equation 1
| (1) |
3.2 Encoding new memories
Appending the current observation to the keys and values, while querying only from existing memory slots, lets the same operation encode new information and combine old memories. The number of slots stays fixed. Even a single slot can use input-dependent attention, though it lacks multiple separate memory rows.
Equation 2
| (2) |
3.3 Introducing recurrence and embedding into an LSTM
LSTM-style gates blend existing memory with the attention-derived update. The implementation uses a row-wise multilayer perceptron, layer normalization, and no output gate. Gating can operate per unit or per slot. Shared slot parameters allow the number of memories to change independently of parameter count, while slot width, head count, and repeated attention blocks control capacity and cost.
Equations 3–9
| (3) | ||||
| (4) | ||||
| (5) | ||||
| (6) | ||||
| (7) | ||||
| (8) | ||||
| (9) |
4 Experiments
Evaluation combines deliberately relational synthetic tasks with program execution, partially observed reinforcement learning, and natural-language prediction.
4.1 Illustrative supervised tasks
N t h Farthest
The farthest-vector task presents vectors sequentially and asks which is the specified rank in distance from a reference vector. A model must retain values, compare distances, and implicitly rank relationships; copying stored values is insufficient.
Program Evaluation
Learning to Execute maps character-level program snippets to their execution outputs. Addition, control flow, and full programs require manipulation of remembered variables. Separate copying/permutation tasks test conventional sequence memory.
4.2 Reinforcement learning
Mini Pacman with viewport
Mini Pacman restricts perception to a 5 × 5 viewport. Success requires remembering food locations and predicting unseen ghosts. The appendix also tests BoxWorld, where remembered key-lock relations determine a valid route to a gem.
4.3 Language Modeling
Word-level experiments use WikiText-103, Project Gutenberg, and GigaWord, covering articles, books, and news at different scales. Each retains contiguous sentences and uses a vocabulary of roughly 250,000 words. Validation-selected settings are transferred from WikiText-103 to the other corpora.
5 Results
5.1 N t h Farthest
For 16-dimensional input vectors, the relational model reaches about 91% accuracy while the compared LSTM and differentiable-memory models remain below 30% best-batch accuracy. Some configurations also succeed with 32-dimensional vectors, but less robustly. Attention patterns change after the reference vector arrives; because memory rapidly becomes distributed, these patterns do not identify exact stored meanings.
5.2 Program Evaluation
The relational core matches or improves most program-evaluation and memorization baselines. Training without ground-truth output feedback works best in these experiments. Reported scores use the strongest tuned configuration for each model; the largest LSTM baselines contain more parameters than the relational models.
| Model | Add | Control | Program | Copy | Reverse | Double |
|---|---|---|---|---|---|---|
| LSTM [3, 37] | 99.8 | 97.4 | 66.1 | 99.8 | 99.7 | 99.7 |
| EntNet [38] | 98.4 | 98.0 | 73.4 | 91.8 | 100.0 | 62.3 |
| DNC [5] | 99.4 | 83.8 | 69.5 | 100.0 | 100.0 | 100.0 |
| Relational Memory Core | 99.9 | 99.6 | 79.0 | 100.0 | 100.0 | 99.8 |
| WikiText-103 | Gutenberg | GigaWord | |||
| Valid. | Test | Valid | Test | Test | |
| LSTM [40] | - | 48.7 | - | - | - |
| Temporal CNN [41] | - | 45.2 | - | - | - |
| Gated CNN [42] | - | 37.2 | - | - | - |
| LSTM [32] | 34.1 | 34.3 | 41.8 | 45.5 | 43.7 |
| Quasi-RNN [43] | 32 | 33 | - | - | - |
| Relational Memory Core | 30.8 | 31.6 | 39.2 | 42.0 | 38.3 |
5.3 Mini-Pacman
In viewport Mini Pacman, the relational agent averages 677 points versus 550 for the LSTM. With full observations, scores are 1,159 versus 598. These are task-specific comparisons rather than evidence that recurrent attention universally improves reinforcement learning.
5.4 Language Modeling
Perplexity improves by 1.4–5.4 points over the reported published comparisons across the three corpora. Improvements concentrate on frequent words, and the model needs less evaluation context than the tested LSTM. The preserved tables distinguish validation/test scores and vocabulary-frequency effects.
6 Discussion
Fixed recurrent state bounds memory cost across arbitrarily long streams. Slots and heads offer different ways to divide computation, with task-dependent tradeoffs. The language model uses one large slot because of parameter needs and computation limits, so the study does not establish that one slot is intrinsically best. Behavioral gains support the architecture, while causal claims about its internal reasoning remain unresolved.
Acknowledgements
The authors acknowledge colleagues at DeepMind for feedback, discussion, and support.
Appendix A Further task details, analyses, and model configurations
The supplement defines memory capacity as slot count times slot width, and distinguishes attention heads, repeated attention blocks, and unit-level versus slot-level gates. Complete task examples, sweeps, and diagnostic figures follow.
A.1 N t h Farthest
Sequences contain eight randomly labeled 16-dimensional vectors, with reference identity and requested rank supplied at every step. Models share a four-layer output perceptron and train with Adam. The relational sweep varies 1/8/16 slots and heads at fixed total memory size; baseline hidden/memory sizes are also swept. Runtime-normalized plots expose the cost differences between narrow and wide slots.
A.2 Program Evaluation
Program samples are generated online with mixed literal lengths and nesting depths; evaluation uses the maximum difficulty. Models train for 200,000 iterations with batch size 128. An encoder state initializes a recurrent decoder, and the strongest results avoid ground-truth output feedback. Baselines include an LSTM, differentiable neural computer, and entity memory. All original program and memorization examples are retained.
A.3 Viewport BoxWorld
Viewport BoxWorld hides most of a 14 × 14 board. Keys are consumed when opening locks, and distractor branches can make the goal unreachable. A convolutional encoder with coordinate features feeds four slots and four attention heads. The distributed actor-critic setup uses 100 actors, trajectories of length 40, and rewards for progress and collecting the gem.
A.3.1 Results
After one billion environment steps, the relational agent solves 98% of generated levels compared with 73% for the convolutional LSTM baseline. Training covers solution paths of one to five boxes and up to five distractor branches.
A.4 Language Modeling
Language models use Adam at learning rate 0.001, gradient norm clipping at 0.1, truncated backpropagation over 100 tokens, and total batch size 384 across six accelerators. Input/output word embeddings are tied. WikiText-103 selects 2,500 memory units, one slot, four heads, a five-layer post-attention perceptron, and one attention block; the other corpora reuse those settings.