At a glance
A Relation Network applies one learned function to every object pair, sums the results, and predicts an answer. Coupled to image or language encoders, it reaches 95.5% CLEVR test accuracy, solves 18 of 20 joint bAbI tasks, and infers physical relationships. Its advantage is an explicit bias toward comparing entities; the all-pairs computation grows quadratically.
1 Introduction
Recognizing individual entities is different from comparing their relationships. Symbolic systems encode relations explicitly but need grounded symbols; ordinary neural encoders can learn from raw inputs but may struggle with relational structure. Relation Networks combine a dedicated pairwise computation with jointly learned upstream representations.
2 Relation Networks
The same relation function processes every ordered pair of object vectors. Its outputs are summed and passed to an answer function; both functions are multilayer perceptrons. “Object” means an input element, not necessarily a detected physical object. The model learns which pairwise features matter for the supervised task.
Equation 1
| (1) |
RNs learn to infer relations
The complete directed graph of pairs supplies candidate relations, without specifying which relations actually exist. When a known graph is available, computation could be restricted to its edges; the experiments mainly use all pairs.
RNs are data efficient
Sharing one relation function across pairs avoids learning separate comparison rules for each input position. This improves the inductive bias but still requires a forward computation for every pair.
RNs operate on a set of objects
Summation makes the final representation invariant to permutations of the input set. Spatial or temporal information must therefore be carried in object features when the task needs it.
3 Tasks
Experiments test visual questions, text-based questions, and hidden physical connections to separate the general relational module from its input encoder.
3.1 CLEVR
CLEVR uses rendered objects and questions about attributes, counts, and relationships, reducing language and world-knowledge biases found in natural-image question answering. Separate experiments provide either pixels or explicit object-state descriptions; the model does not receive both together.
3.2 Sort-of-CLEVR
Sort-of-CLEVR isolates the relational challenge with six colored shapes and binary-coded questions. Single-object questions ask shape or position; relational questions ask nearest/farthest shape or matching-shape counts. This removes natural-language parsing as the main explanation of performance differences.
3.3 bAbI
The bAbI benchmark has 20 synthetic text-reasoning tasks. Models train jointly on 10,000 examples per task; exceeding 95% accuracy counts as solving a task. Comparisons distinguish this joint setting from separately training one model per task.
3.4 Dynamic physical systems
Simulated scenes contain ten balls, some linked by invisible springs or rigid constraints. Sixteen timesteps of position and color observations support either explicit pairwise connection prediction or counting connected components. Counting requires the model to discover useful relations without edge-level supervision.
4 Models
The relation module can operate on distributed features as well as explicit entities. Joint training lets upstream encoders develop useful object-like descriptions without object segmentation labels.
Dealing with pixels
A four-layer convolutional encoder produces a spatial feature grid. Each cell, augmented with its coordinates, becomes an object vector; it may represent background, texture, a part, or several physical objects.
Conditioning RNs with question embeddings
An LSTM encodes the question. Its final state is appended to every object pair so that the relation function can select question-relevant comparisons.
Dealing with state descriptions
When explicit object states are available, they replace the image encoder’s spatial features while question conditioning remains the same.
Dealing with natural language
For bAbI, an LSTM separately encodes each of the 20 most recent support sentences, with tags for their order. Sentence states form the object set; another LSTM encodes the question. This uses sentence boundaries as a modest structural prior.
Model configuration details
The pixel model uses four 24-channel convolutional layers, a 128-unit question LSTM, a four-layer 256-unit relation network, and an answer multilayer perceptron. Training uses Adam and dropout. Task-specific sizes and complete training details appear below.
5 Results
5.1 CLEVR from pixels
The pixel model reaches 95.5% CLEVR test accuracy, above the reported 92.6% human baseline and contemporary general-purpose comparison models. This is a result on the controlled benchmark, not a claim of general human-level reasoning.
Results using privileged training information
A contemporaneous system reports 96.9% accuracy but trains with ground-truth question-generation programs. The Relation Network uses no such program supervision, making that comparison different from the general-purpose baselines. The retained table and figure show scores by question type and identify the authors’ tuned stacked-attention baseline.
| Model | Overall | Count | Exist | Compare Numbers | Query Attribute | Compare Attribute |
|---|---|---|---|---|---|---|
| Human | 92.6 | 86.7 | 96.6 | 86.5 | 95.0 | 96.0 |
| Q-type baseline | 41.8 | 34.6 | 50.2 | 51.0 | 36.0 | 51.3 |
| LSTM | 46.8 | 41.7 | 61.1 | 69.8 | 36.8 | 51.8 |
| CNNLSTM | 52.3 | 43.7 | 65.2 | 67.1 | 49.3 | 53.0 |
| CNNLSTMSA | 68.5 | 52.2 | 71.1 | 73.5 | 85.3 | 52.3 |
| CNNLSTMSA* | 76.6 | 64.4 | 82.7 | 77.4 | 82.6 | 75.4 |
| CNN+LSTM+RN | 95.5 | 90.1 | 97.8 | 93.6 | 97.9 | 97.1 |
- •
* Our implementation, with optimized hyperparameters and trained fully end-to-end.
5.2 CLEVR from state descriptions
Providing explicit object states yields 96.4% accuracy. The same relation architecture can therefore exploit a different representation, although this is a separately trained model.
5.3 Sort-of-CLEVR from pixels
On Sort-of-CLEVR, the relation model exceeds 94% accuracy for both question types. A comparable convolution-plus-perceptron baseline succeeds on nonrelational questions but plateaus near 63% on relational questions. The controlled contrast supports the inductive-bias argument without proving that conventional networks can never learn relations.
5.4 bAbI
The jointly trained model solves 18 of 20 bAbI tasks, including basic induction. It misses the threshold on two- and three-supporting-fact tasks. The reported evaluation uses one selected seed; performance variability across repeated runs was not established.
5.5 Dynamic physical systems
The model predicts every connection correctly in 93% of held-out scenes and the correct component count in 95%. Parameter-comparable perceptrons perform near chance. Qualitative transfer to human motion capture illustrates learned motion relationships but is narrower evidence than a broad transfer benchmark.
6 Discussion and Conclusions
An explicit pairwise module can make simple perception/language encoders effective on structured reasoning tasks. Internal objects and relations remain learned rather than supplied symbolically. Main limitations include quadratic pair counts, occlusion, and fine spatial distinctions; selecting only relevant pairs is a proposed efficiency improvement.
Acknowledgments
The authors acknowledge colleagues at DeepMind for discussion and feedback.
Supplementary Material
The supplement specifies the tasks, training settings, failure examples, and relationship to earlier approaches.
A Related Work
Related work spans symbolic reasoning, graph networks, spatial grounding, and question answering.
Relational reasoning
Graph models also make relations explicit. This work emphasizes using a simple shared relation function on object-like features learned from less structured inputs.
Grounding spatial relations
Earlier spatial-language systems often relied on manually designed spatial features or rules; the relation module learns comparisons jointly with input representations.
Visual question answering
The visual pipeline resembles other image/question encoders, but adds the relation module instead of relying mainly on attention or hand-designed reasoning programs. Comparisons distinguish systems with access to ground-truth programs.
Text-based question answering
Memory-based text models provide another route to structured question answering. The relation module reaches competitive synthetic-task results through pairwise sentence comparisons.
B CLEVR from pixels
CLEVR training uses 70,000 scenes and 699,989 questions, small crop/rotation augmentation, ten synchronous workers, batch size 64 per worker, learning rate 0.00025, and 50% dropout in the penultimate answer layer. Training plateaus near 1.4 million iterations; validation accuracy reaches 96.8%. Smaller encoders perform better in the tested search.
Failure cases
The twelve retained failure panels illustrate heavy occlusion, delicate positional distinctions, and ambiguous scenes. Some are also difficult for humans; the examples do not constitute a systematic error decomposition.
![]() |
![]() |
![]() | |
| What shape is the small object | What number of things are either tiny | What number of objects are blocks | |
| that is in front of the yellow matte | green rubber objects or shiny things | that are in front of the large | |
| thing and behind the gray sphere? | that are behind the big metal block? | red cube or green balls? | |
| RN: | cylinder | 1 | 2 |
| GT: | cube | 2 | 3 |
![]() |
![]() |
![]() | |
| Is the shape of the small red object | How many gray objects are in front | What number of objects are big | |
| the same as the large matte object | of the tiny green shiny ball and right | red matte cubes or things on the right | |
| that is right of the small rubber ball? | of the big blue matte thing? | side of the large red matte block? | |
| RN: | no | 0 | 5 |
| GT: | yes | 1 | 6 |
![]() |
![]() |
![]() | |
| There is a brown ball; | How many objects are big purple | How many things are rubber | |
| what number of things are left of it? | rubber blocks or red blocks in front | cylinders in front of the tiny yellow | |
| of the tiny yellow rubber thing? | block or blocks that are to the right | ||
| of the small brown rubber thing? | |||
| RN: | 3 | 3 | 2 |
| GT: | 4 | 2 | 3 |
![]() |
![]() |
![]() | |
| What number of objects are either | Are there the same number of small | What number of other things | |
| big things that are left | blue objects that are to the right of | are there of the same | |
| of the cylinder or cylinders? | the blue cube and blue metal cubes? | material as the green cube? | |
| RN: | 2 | no | 6 |
| GT: | 3 | yes | 5 |
C CLEVR from state descriptions
The state-description version omits convolution, uses a 256-unit question LSTM and 512-unit relation layers, and trains with ten synchronous workers at learning rate 0.0001.
D Sort-of-CLEVR
The generated dataset contains 10,000 images, with 200 held out and 20 questions per image. Eleven-bit question encodings specify object color and query type. A large relation module and matched-depth perceptron baseline share the same image-processing setup; the original examples and curves are preserved.
E bAbI model for language understanding
Each support sentence and question is encoded with a 32-unit LSTM. Four 256-unit relation layers feed a three-layer answer network over 159 tokens. Adam uses learning rate 0.0002.
F Dynamic physical system reasoning
Physical reasoning uses object histories as rows of a state matrix. Connection prediction outputs 100 binary decisions; component counting outputs ten class logits. The tasks share the same relation architecture but different final outputs. The original supplementary demonstrations are available on the authors’ video channel.
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/c6/c686a97607d463a82b3453ecd82153e8b47ca2ae3285cd3558d54973898e8116.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/b2/b2b777e02779d0a7abe532f661716e27c912b858e45c7493fa6aae305703bfa4.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/b9/b9c05a7ee4802ca13ed7a2cc540126c3bc1007d5fe792c70fb25c8f912aad28e.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/5e/5eecb6de0c549818661a8dc3f545f6bb0cc4d261ba20b8962b06308e8ef6c9f4.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/b2/b24054732eab0700da1d658a5b4ee931047ad29185405dfabeeae030e61720e3.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/29/29aa82f573538567eb72fed7901f445b3ccd3b6e0bc2cc611d524f239765c340.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/94/94c3d5c0d947b463dbb41282d6f1bc81cee3aa555712d3e77574a574a7d37fe8.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/72/7273de5ed5437cc9c5f6d0fd237e9fa65a6d9d86475d67414986c7d7088e7020.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/33/33bdbc26bfea605fc7099f7034713682ac86b9382bfae5a6959bb63c8271f947.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/cd/cd1ef124ab2fa5a991fd751ff286bfa07a734fdb6f63e2e8c1b74ab75d384002.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/cd/cd230113c69eaba1886b4db33178781b2e0b2c08bb2f550fb38d8ead07d41c8d.jpg)
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/87/87d98632f956171811511950dee2d500403269c066d8143eee235d93bab57b8f.jpg)