Literature
Display
← All papers

A simple neural network module for relational reasoning

Adam Santoro · David Raposo · David G. T. Barrett · Mateusz Malinowski · Razvan Pascanu · Peter Battaglia · Timothy Lillicrap

Published · NeurIPS 2017 ↗

Reasoning6 min text · CondensedOriginal paper ↗
Contents

At a glance

A Relation Network applies one learned function to every object pair, sums the results, and predicts an answer. Coupled to image or language encoders, it reaches 95.5% CLEVR test accuracy, solves 18 of 20 joint bAbI tasks, and infers physical relationships. Its advantage is an explicit bias toward comparing entities; the all-pairs computation grows quadratically.

1 Introduction

Recognizing individual entities is different from comparing their relationships. Symbolic systems encode relations explicitly but need grounded symbols; ordinary neural encoders can learn from raw inputs but may struggle with relational structure. Relation Networks combine a dedicated pairwise computation with jointly learned upstream representations.

Refer to caption
Figure 1: An illustrative example from the CLEVR dataset of relational reasoning. An image containing four objects is shown alongside non-relational and relational questions. The relational question requires explicit reasoning about the relations between the four objects in the image, whereas the non-relational question requires reasoning about the attributes of a particular object.

2 Relation Networks

The same relation function processes every ordered pair of object vectors. Its outputs are summed and passed to an answer function; both functions are multilayer perceptrons. “Object” means an input element, not necessarily a detected physical object. The model learns which pairwise features matter for the supervised task.

Equation 1
RN​(O)=fϕ​(∑i,jgθ​(oi,oj)),\displaystyle\text{RN}(O)=f_{\phi}\left(\sum_{i,j}g_{\theta}(o_{i},o_{j})\right), (1)

RNs learn to infer relations

The complete directed graph of pairs supplies candidate relations, without specifying which relations actually exist. When a known graph is available, computation could be restricted to its edges; the experiments mainly use all pairs.

RNs are data efficient

Sharing one relation function across pairs avoids learning separate comparison rules for each input position. This improves the inductive bias but still requires a forward computation for every pair.

RNs operate on a set of objects

Summation makes the final representation invariant to permutations of the input set. Spatial or temporal information must therefore be carried in object features when the task needs it.

3 Tasks

Experiments test visual questions, text-based questions, and hidden physical connections to separate the general relational module from its input encoder.

3.1 CLEVR

CLEVR uses rendered objects and questions about attributes, counts, and relationships, reducing language and world-knowledge biases found in natural-image question answering. Separate experiments provide either pixels or explicit object-state descriptions; the model does not receive both together.

3.2 Sort-of-CLEVR

Sort-of-CLEVR isolates the relational challenge with six colored shapes and binary-coded questions. Single-object questions ask shape or position; relational questions ask nearest/farthest shape or matching-shape counts. This removes natural-language parsing as the main explanation of performance differences.

3.3 bAbI

The bAbI benchmark has 20 synthetic text-reasoning tasks. Models train jointly on 10,000 examples per task; exceeding 95% accuracy counts as solving a task. Comparisons distinguish this joint setting from separately training one model per task.

3.4 Dynamic physical systems

Simulated scenes contain ten balls, some linked by invisible springs or rigid constraints. Sixteen timesteps of position and color observations support either explicit pairwise connection prediction or counting connected components. Counting requires the model to discover useful relations without edge-level supervision.

4 Models

The relation module can operate on distributed features as well as explicit entities. Joint training lets upstream encoders develop useful object-like descriptions without object segmentation labels.

Dealing with pixels

A four-layer convolutional encoder produces a spatial feature grid. Each cell, augmented with its coordinates, becomes an object vector; it may represent background, texture, a part, or several physical objects.

Refer to caption
Figure 2: Visual QA architecture. Questions are processed with an LSTM to produce a question embedding, and images are processed with a CNN to produce a set of objects for the RN. Objects (three examples illustrated here in yellow, red, and blue) are constructed using feature-map vectors from the convolved image. The RN considers relations across all pairs of objects, conditioned on the question embedding, and integrates all these relations to answer the question.

Conditioning RNs with question embeddings

An LSTM encodes the question. Its final state is appended to every object pair so that the relation function can select question-relevant comparisons.

Dealing with state descriptions

When explicit object states are available, they replace the image encoder’s spatial features while question conditioning remains the same.

Dealing with natural language

For bAbI, an LSTM separately encodes each of the 20 most recent support sentences, with tags for their order. Sentence states form the object set; another LSTM encodes the question. This uses sentence boundaries as a modest structural prior.

Model configuration details

The pixel model uses four 24-channel convolutional layers, a 128-unit question LSTM, a four-layer 256-unit relation network, and an answer multilayer perceptron. Training uses Adam and dropout. Task-specific sizes and complete training details appear below.

5 Results

5.1 CLEVR from pixels

The pixel model reaches 95.5% CLEVR test accuracy, above the reported 92.6% human baseline and contemporary general-purpose comparison models. This is a result on the controlled benchmark, not a claim of general human-level reasoning.

Results using privileged training information

A contemporaneous system reports 96.9% accuracy but trains with ground-truth question-generation programs. The Relation Network uses no such program supervision, making that comparison different from the general-purpose baselines. The retained table and figure show scores by question type and identify the authors’ tuned stacked-attention baseline.

Model Overall Count Exist Compare Numbers Query Attribute Compare Attribute
Human 92.6 86.7 96.6 86.5 95.0 96.0
Q-type baseline 41.8 34.6 50.2 51.0 36.0 51.3
LSTM 46.8 41.7 61.1 69.8 36.8 51.8
CNN++LSTM 52.3 43.7 65.2 67.1 49.3 53.0
CNN++LSTM++SA 68.5 52.2 71.1 73.5 85.3 52.3
CNN++LSTM++SA* 76.6 64.4 82.7 77.4 82.6 75.4
CNN+LSTM+RN 95.5 90.1 97.8 93.6 97.9 97.1
  • •

    * Our implementation, with optimized hyperparameters and trained fully end-to-end.

Table 1: Results on CLEVR from pixels. Performances of our model (RN) and previously reported models [16], measured as accuracy on the test set and broken down by question category.
Original research figure
Figure 3: Results on CLEVR from pixels. The RN augmented model outperformed all other models and exhibited super-human performance overall. In particular, it solved “compare attribute” questions, which trouble all other models because they heavily depend on relational reasoning.

5.2 CLEVR from state descriptions

Providing explicit object states yields 96.4% accuracy. The same relation architecture can therefore exploit a different representation, although this is a separately trained model.

5.3 Sort-of-CLEVR from pixels

On Sort-of-CLEVR, the relation model exceeds 94% accuracy for both question types. A comparable convolution-plus-perceptron baseline succeeds on nonrelational questions but plateaus near 63% on relational questions. The controlled contrast supports the inductive-bias argument without proving that conventional networks can never learn relations.

5.4 bAbI

The jointly trained model solves 18 of 20 bAbI tasks, including basic induction. It misses the threshold on two- and three-supporting-fact tasks. The reported evaluation uses one selected seed; performance variability across repeated runs was not established.

5.5 Dynamic physical systems

The model predicts every connection correctly in 93% of held-out scenes and the correct component count in 95%. Parameter-comparable perceptrons perform near chance. Qualitative transfer to human motion capture illustrates learned motion relationships but is narrower evidence than a broad transfer benchmark.

6 Discussion and Conclusions

An explicit pairwise module can make simple perception/language encoders effective on structured reasoning tasks. Internal objects and relations remain learned rather than supplied symbolically. Main limitations include quadratic pair counts, occlusion, and fine spatial distinctions; selecting only relevant pairs is a proposed efficiency improvement.

Acknowledgments

The authors acknowledge colleagues at DeepMind for discussion and feedback.

Supplementary Material

The supplement specifies the tasks, training settings, failure examples, and relationship to earlier approaches.

A Related Work

Related work spans symbolic reasoning, graph networks, spatial grounding, and question answering.

Relational reasoning

Graph models also make relations explicit. This work emphasizes using a simple shared relation function on object-like features learned from less structured inputs.

Grounding spatial relations

Earlier spatial-language systems often relied on manually designed spatial features or rules; the relation module learns comparisons jointly with input representations.

Visual question answering

The visual pipeline resembles other image/question encoders, but adds the relation module instead of relying mainly on attention or hand-designed reasoning programs. Comparisons distinguish systems with access to ground-truth programs.

Text-based question answering

Memory-based text models provide another route to structured question answering. The relation module reaches competitive synthetic-task results through pairwise sentence comparisons.

B CLEVR from pixels

CLEVR training uses 70,000 scenes and 699,989 questions, small crop/rotation augmentation, ten synchronous workers, batch size 64 per worker, learning rate 0.00025, and 50% dropout in the penultimate answer layer. Training plateaus near 1.4 million iterations; validation accuracy reaches 96.8%. Smaller encoders perform better in the tested search.

Failure cases

The twelve retained failure panels illustrate heavy occlusion, delicate positional distinctions, and ambiguous scenes. Some are also difficult for humans; the examples do not constitute a systematic error decomposition.

[Uncaptioned image] [Uncaptioned image] [Uncaptioned image]
What shape is the small object What number of things are either tiny What number of objects are blocks
that is in front of the yellow matte green rubber objects or shiny things that are in front of the large
thing and behind the gray sphere? that are behind the big metal block? red cube or green balls?
RN: cylinder 1 2
GT: cube 2 3
[Uncaptioned image] [Uncaptioned image] [Uncaptioned image]
Is the shape of the small red object How many gray objects are in front What number of objects are big
the same as the large matte object of the tiny green shiny ball and right red matte cubes or things on the right
that is right of the small rubber ball? of the big blue matte thing? side of the large red matte block?
RN: no 0 5
GT: yes 1 6
[Uncaptioned image] [Uncaptioned image] [Uncaptioned image]
There is a brown ball; How many objects are big purple How many things are rubber
what number of things are left of it? rubber blocks or red blocks in front cylinders in front of the tiny yellow
of the tiny yellow rubber thing? block or blocks that are to the right
of the small brown rubber thing?
RN: 3 3 2
GT: 4 2 3
[Uncaptioned image] [Uncaptioned image] [Uncaptioned image]
What number of objects are either Are there the same number of small What number of other things
big things that are left blue objects that are to the right of are there of the same
of the cylinder or cylinders? the blue cube and blue metal cubes? material as the green cube?
RN: 2 no 6
GT: 3 yes 5
Table 2: Failures on CLEVR; RN – predicted answers, GT – ground-truth answer.

C CLEVR from state descriptions

The state-description version omits convolution, uses a 256-unit question LSTM and 512-unit relation layers, and trains with ten synchronous workers at learning rate 0.0001.

D Sort-of-CLEVR

The generated dataset contains 10,000 images, with 200 held out and 20 questions per image. Eleven-bit question encodings specify object color and query type. A large relation module and matched-depth perceptron baseline share the same image-processing setup; the original examples and curves are preserved.

Refer to caption
Figure 4: “Sort-of-CLEVR” task: examples and results. The Sort-of-CLEVR example here consists of an image of six objects and two questions – a relational question, and a non-relational question – along with the corresponding answers. The fraction of correctly answered relational questions (inset bar plot) for our model (CNN+RN) is much larger than the comparable MLP based model (CNN+MLP), whereas both models have similar performance levels for non-relational questions.

E bAbI model for language understanding

Each support sentence and question is encoded with a 32-unit LSTM. Four 256-unit relation layers feed a three-layer answer network over 159 tokens. Adam uses learning rate 0.0002.

F Dynamic physical system reasoning

Physical reasoning uses object histories as rows of a state matrix. Connection prediction outputs 100 binary decisions; component counting outputs ten class logits. The tasks share the same relation architecture but different final outputs. The original supplementary demonstrations are available on the authors’ video channel.

Concept demonstration

Open image ↗