Literature
Display
← All papers

Relational recurrent neural networks

Adam Santoro · Ryan Faulkner · David Raposo · Jack Rae · Mike Chrzanowski · Théophane Weber · Daan Wierstra · Oriol Vinyals · Razvan Pascanu · Timothy Lillicrap

Published · NeurIPS 2018 ↗

Reasoning5 min text · CondensedOriginal paper ↗
Contents

At a glance

The Relational Memory Core combines a fixed recurrent memory with self-attention between memory slots. It improves synthetic sequential reasoning, program evaluation, partially observed games, and language modeling. The evidence demonstrates better task performance; the authors explicitly avoid claiming that attention maps establish a unique internal reasoning mechanism.

1 Introduction

Memory models commonly emphasize storage and retrieval. This work adds a bias toward interactions among remembered items, so a model can compare information received at different times without retaining an ever-growing history of hidden states.

2 Relational reasoning

Relational reasoning compares entities rather than processing their properties independently. Graph networks and relation modules make this bias explicit in spatial settings. A recurrent memory can represent entities across time, but a single undifferentiated state or a passive slot store does not directly encourage interactions between them.

3 Model

The model keeps a matrix of memory slots, applies attention among those slots and new input, and integrates proposed updates through recurrent gates. Unlike attention over a complete history, attention operates on a fixed-size state at the current timestep.

Original research figure
Figure 1: Relational Memory Core. (a) The RMC receives a previous memory matrix and input vector as inputs, which are passed to the MHDPA module labeled with an “A”. (b). Linear projections are computed for each memory slot, and input vector, using row-wise shared weights WqW^{q} for the queries, WkW^{k} for the keys, and WvW^{v} for the values. (c) The queries, keys, and values are then compiled into matrices and softmax​(Q​KT)​V\text{softmax}(QK^{T})V is computed. The output of this computation is a new memory where information is blended across memories based on their attention weights. An MLP is applied row-wise to the output of the MHDPA module (a), and the resultant memory matrix is gated, and passed on as the core output or next memory state.

3.1 Allowing memories to interact using multi-head dot product attention

Each slot produces queries, keys, and values. Scaled dot-product attention returns a weighted combination of slot values; multiple heads can exchange different information in parallel. The concatenated head outputs propose an update with the same dimensions as the original memory.

Equation 1
Aθ​(M)=softmax​(M​Wq​(M​Wk)Tdk)​M​Wv, where ​θ=(Wq,Wk,Wv)\displaystyle A_{\theta}(M)=\text{softmax}\left(\frac{MW^{q}(MW^{k})^{T}}{\sqrt{d_{k}}}\right)MW^{v},\text{ where }\theta=(W^{q},W^{k},W^{v}) (1)

3.2 Encoding new memories

Appending the current observation to the keys and values, while querying only from existing memory slots, lets the same operation encode new information and combine old memories. The number of slots stays fixed. Even a single slot can use input-dependent attention, though it lacks multiple separate memory rows.

Equation 2
M~=softmax​(M​Wq​([M;x]​Wk)Tdk)​[M;x]​Wv,\displaystyle\widetilde{M}=\text{softmax}\left(\frac{MW^{q}([M;x]W^{k})^{T}}{\sqrt{d^{k}}}\right)[M;x]W^{v}, (2)

3.3 Introducing recurrence and embedding into an LSTM

LSTM-style gates blend existing memory with the attention-derived update. The implementation uses a row-wise multilayer perceptron, layer normalization, and no output gate. Gating can operate per unit or per slot. Shared slot parameters allow the number of memories to change independently of parameter count, while slot width, head count, and repeated attention blocks control capacity and cost.

Equations 3–9
si,t\displaystyle s_{i,t} =(hi,t−1,mi,t−1)\displaystyle=(h_{i,t-1},m_{i,t-1}) (3)
fi,t\displaystyle f_{i,t} =Wf​xt+Uf​hi,t−1+bf\displaystyle=W^{f}x_{t}+U^{f}h_{i,t-1}+b^{f} (4)
ii,t\displaystyle i_{i,t} =Wi​xt+Ui​hi,t−1+bi\displaystyle=W^{i}x_{t}+U^{i}h_{i,t-1}+b^{i} (5)
oi,t\displaystyle o_{i,t} =Wo​xt+Uo​hi,t−1+bo\displaystyle=W^{o}x_{t}+U^{o}h_{i,t-1}+b^{o} (6)
mi,t\displaystyle m_{i,t} =σ⁡(fi,t+b~f)∘mi,t−1+σ⁡(ii,t)∘gψ​(m~i,t)⏟\displaystyle=\sigma(f_{i,t}+\tilde{b}^{f})\circ m_{i,t-1}+\sigma(i_{i,t})\circ\underbrace{{g_{\psi}(\widetilde{m}_{i,t})}} (7)
hi,t\displaystyle h_{i,t} =σ⁡(oi,t)∘tanh​(mi,t)\displaystyle=\sigma(o_{i,t})\circ\text{tanh}(m_{i,t}) (8)
si,t+1\displaystyle s_{i,t+1} =(mi,t,hi,t)\displaystyle=(m_{i,t},h_{i,t}) (9)

4 Experiments

Evaluation combines deliberately relational synthetic tasks with program execution, partially observed reinforcement learning, and natural-language prediction.

Refer to caption
Figure 2: Tasks. We tested the RMC on a suite of supervised and reinforcement learning tasks. Notable are the Nt​hN^{th} Farthest toy task and language modeling. In the former, the solution requires explicit relational reasoning since the model must sort distance relations between vectors, and not the vectors themselves. The latter tests the model on a large quantity of natural data and allows us to compare performance to well-tuned models.

4.1 Illustrative supervised tasks

N t ​ h Farthest

The farthest-vector task presents vectors sequentially and asks which is the specified rank in distance from a reference vector. A model must retain values, compare distances, and implicitly rank relationships; copying stored values is insufficient.

Program Evaluation

Learning to Execute maps character-level program snippets to their execution outputs. Addition, control flow, and full programs require manipulation of remembered variables. Separate copying/permutation tasks test conventional sequence memory.

4.2 Reinforcement learning

Mini Pacman with viewport

Mini Pacman restricts perception to a 5 × 5 viewport. Success requires remembering food locations and predicting unseen ghosts. The appendix also tests BoxWorld, where remembered key-lock relations determine a valid route to a gem.

4.3 Language Modeling

Word-level experiments use WikiText-103, Project Gutenberg, and GigaWord, covering articles, books, and news at different scales. Each retains contiguous sentences and uses a vocabulary of roughly 250,000 words. Validation-selected settings are transferred from WikiText-103 to the other corpora.

5 Results

5.1 N t ​ h Farthest

For 16-dimensional input vectors, the relational model reaches about 91% accuracy while the compared LSTM and differentiable-memory models remain below 30% best-batch accuracy. Some configurations also succeed with 32-dimensional vectors, but less robustly. Attention patterns change after the reference vector arrives; because memory rapidly becomes distributed, these patterns do not identify exact stored meanings.

Refer to caption
Figure 3: Model analysis. Each row depicts the attention matrix at each timestep of a particular sequence. The text beneath spells out the particular task for the sequence, which was encoded and provided to the model as an input. We mark in red the vector that is referenced in the task: e.g., if the model is to choose the 2n​d2^{nd} farthest vector from vector 77, then the time point at which vector 77 was input to the model is depicted in red. A single attention matrix shows the attention weights from one particular memory slot (y-axis) to another memory slot (columns), or the input (offset column), with the numbers denoting the memory slot and “input” denoting the input embedding.

5.2 Program Evaluation

The relational core matches or improves most program-evaluation and memorization baselines. Training without ground-truth output feedback works best in these experiments. Reported scores use the strongest tuned configuration for each model; the largest LSTM baselines contain more parameters than the relational models.

Table 1: Test per character Accuracy on Program Evaluation and Memorization tasks.
Model Add Control Program Copy Reverse Double
LSTM [3, 37] 99.8 97.4 66.1 99.8 99.7 99.7
EntNet [38] 98.4 98.0 73.4 91.8 100.0 62.3
DNC [5] 99.4 83.8 69.5 100.0 100.0 100.0
Relational Memory Core 99.9 99.6 79.0 100.0 100.0 99.8
Table 2: Validation and test perplexities on WikiText-103, Project Gutenberg, and GigaWord v5.
WikiText-103 Gutenberg GigaWord
Valid. Test Valid Test Test
LSTM [40] - 48.7 - - -
Temporal CNN [41] - 45.2 - - -
Gated CNN [42] - 37.2 - - -
LSTM [32] 34.1 34.3 41.8 45.5 43.7
Quasi-RNN [43] 32 33 - - -
Relational Memory Core 30.8 31.6 39.2 42.0 38.3

5.3 Mini-Pacman

In viewport Mini Pacman, the relational agent averages 677 points versus 550 for the LSTM. With full observations, scores are 1,159 versus 598. These are task-specific comparisons rather than evidence that recurrent attention universally improves reinforcement learning.

5.4 Language Modeling

Perplexity improves by 1.4–5.4 points over the reported published comparisons across the three corpora. Improvements concentrate on frequent words, and the model needs less evaluation context than the tested LSTM. The preserved tables distinguish validation/test scores and vocabulary-frequency effects.

6 Discussion

Fixed recurrent state bounds memory cost across arbitrarily long streams. Slots and heads offer different ways to divide computation, with task-dependent tradeoffs. The language model uses one large slot because of parameter needs and computation limits, so the study does not establish that one slot is intrinsically best. Behavioral gains support the architecture, while causal claims about its internal reasoning remain unresolved.

Acknowledgements

The authors acknowledge colleagues at DeepMind for feedback, discussion, and support.

Appendix A Further task details, analyses, and model configurations

The supplement defines memory capacity as slot count times slot width, and distinguishes attention heads, repeated attention blocks, and unit-level versus slot-level gates. Complete task examples, sweeps, and diagnostic figures follow.

A.1 N t ​ h Farthest

Sequences contain eight randomly labeled 16-dimensional vectors, with reference identity and requested rank supplied at every step. Models share a four-layer output perceptron and train with Adam. The relational sweep varies 1/8/16 slots and heads at fixed total memory size; baseline hidden/memory sizes are also swept. Runtime-normalized plots expose the cost differences between narrow and wide slots.

Refer to caption
Figure 4: Nt​hN^{th} Farthest hyperparameter analysis. Timestamp refers to hours of training. There is a clear effect with the number of memories, with 88 or 1616 memories being better than 11. Interestingly, when the model had 11 memory we observed an effect with the number of heads, with more heads (88 or 1616) being better than one, possibly indicating that the RMC can learn to compartmentalise and relate information across heads in addition to across memories.
Original research figure
Figure 5: LSTM and DNC training curves for the Nt​hN^{th} Farthest task.

A.2 Program Evaluation

Program samples are generated online with mixed literal lengths and nesting depths; evaluation uses the maximum difficulty. Models train for 200,000 iterations with batch size 128. An encoder state initializes a recurrent decoder, and the strongest results avoid ground-truth output feedback. Baselines include an LSTM, differentiable neural computer, and entity memory. All original program and memorization examples are retained.

Refer to caption
Figure 6: Samples of programmatic tasks. Note that training samples will sample literal length up to including the maximum length.
Refer to caption
Figure 7: Memorization tasks. Each sub-task takes the form of a list permutation.
Refer to caption
Figure 8: Programmatic results. From left to right: full program, addition, control. The top row depicts per character accuracy scores from tasks with nesting = 2 and literal length = 5 while the bottom row shows scores from more difficult tasks with nesting = 3 and literal length = 6.

A.3 Viewport BoxWorld

Viewport BoxWorld hides most of a 14 × 14 board. Keys are consumed when opening locks, and distractor branches can make the goal unreachable. A convolutional encoder with coordinate features feeds four slots and four attention heads. The distributed actor-critic setup uses 100 actors, trajectories of length 40, and rewards for progress and collecting the gem.

Refer to caption
Figure 9: Example BoxWorld level. The left panel shows the full-view frame of a BoxWorld level. The agent, the dark grey pixel, only has access to a 5×55\times 5 view surrounding it (light gray area). The right panel shows the underlying graph that was sampled to generate the level. In this example the solution path has length 5 and there are 4 distractor branches.

A.3.1 Results

After one billion environment steps, the relational agent solves 98% of generated levels compared with 73% for the convolutional LSTM baseline. Training covers solution paths of one to five boxes and up to five distractor branches.

Refer to caption
Figure 10: Mini Pacman Results.

A.4 Language Modeling

Language models use Adam at learning rate 0.001, gradient norm clipping at 0.1, truncated backpropagation over 100 tokens, and total batch size 384 across six accelerators. Input/output word embeddings are tied. WikiText-103 selects 2,500 memory units, one slot, four heads, a five-layer post-attention perceptron, and one attention block; the other corpora reuse those settings.

Original research figure
Figure 11: Validation perplexity on WikiText-103. LSTM comparison from [32]. Visual display of data may not match numbers from table 2 because of curve smoothing.
Original research figure
Figure 12: Perplexity as a function of test unroll length. Increase in perplexity when models are unrolled for shorter sequence lengths at test time without state transfer between unrolls. Perplexities are compared against the ‘best’ perplexity where the model is unrolled continuously over the full test set. We see that both models incorporate little information beyond 500500 words. Furthermore, the RMC has a smaller gain in perplexity (drop in performance) when unrolled over shorter time steps in comparison to the LSTM, e.g. a regression of 11 perplexity for the RMC vs 55 for the LSTM at 100100 time steps. This suggests it is focusing on more recent words in the text.
Table 3: Test perplexity split by word frequency on GigaWord v5. Words are bucketed by the number of times they occur in training set, >10>10K contains the most frequent words.
>10>10K 1010K−1\mathchar 45\relax 1K <1<1K All
LSTM [32] 39.4 6.5e3 3.7e4 53.5
LSTM + Hebbian Softmax [32] 33.2 3.2e3 1.6e4 43.7
RMC 28.3 3.1e3 6.9e4 38.3
Concept demonstration

Open image ↗