Literature
Display
← All papers

Language-Mediated, Object-Centric Representation Learning

Ruocheng Wang · Jiayuan Mao · Samuel J. Gershman · Jiajun Wu

· ACL 2021 (Findings) ↗

Contents

Overview

At a glance

LORL couples object discovery with grounded language learning. A reconstruction model proposes object slots; a program executor uses their representations to answer questions or judge descriptions. Errors in these language tasks help refine the segmentation and concept embeddings. Integrating LORL improves MONet and Slot Attention in the tested synthetic scenes and supports retrieval, reasoning, and referring-expression tasks. The method avoids direct segmentation labels during object learning, but uses language supervision and a separately supervised semantic parser.

1 Introduction

Reconstruction alone may split one object into several pieces or merge conceptually distinct parts. Language can penalize those errors: identifying a pan requires its handle to belong to the pan, while counting chair legs requires separating them. LORL learns object masks, representations, and concept embeddings through reconstruction and language-task losses. The experiments test whether those representations improve segmentation and transfer to other concept-based tasks.

Refer to caption
Figure 1: Two illustrative cases of Language-mediated, Object-centric Representation Learning. Different colors in the segmentation masks indicate individual objects recognized by the model. LORL can learn from visual and language inputs to associate various concepts: black, pan, leg, with the visual appearance of individual objects. Furthermore, language provides cues about how an input scene should be segmented into individual objects: (a) segmenting the frying pan and its handle into two parts (Segmentation II) yields an incorrect answer to the question ; (b) an incorrect parsing of the chair image makes the counting result wrong.

2 Related Work

Unsupervised object representation learning.

Object-centric generative models explain an image through a collection of component images and masks, usually trained with reconstruction losses. Other approaches use motion or multiple views. LORL adds questions or descriptions to static images as a source of object-individuation supervision.

Visual concept learning.

Earlier concept-grounding systems often start from pretrained detectors and object proposals. LORL couples concept learning to the discovery of object slots themselves, so language can influence segmentation rather than only label fixed proposals.

Refer to caption
Figure 2: Our LORL contains four modules. (a) An image encoder encodes the input image into a factorized, object-centric representation. (b) An image decoder learns to reconstruct the image from the learned representations. It also decodes an objectness score based on the representation. (c) A pre-trained semantic parser translates the input sentence into an executable program and associates concepts with learnable vector embeddings. (d) A neuro-symbolic program executor takes the object-centric representation, the objectness scores, the parsed program, and concept embeddings as input to predict the answer (if the input is a question) or TRUE/FALSE (if the input is a descriptive sentence).

3 Preliminaries

The method combines a reconstruction-based object-slot model with a differentiable program executor that grounds words in those slots.

3.1 Unsupervised Object-Centric Representation Learning

Each slot contains a latent feature, reconstructed component image, and mask; unused slots may represent no object. MONet uses recurrent spatial attention and a variational autoencoder, with reconstruction and latent-prior losses. Slot Attention iteratively extracts slot features and decodes their images and masks, trained by reconstruction. Both supply the perception module used here.

Equations 1–2
ℒ=‖(∑k=1Kmk​xk)−I‖22+β⋅∑k=1KKL⁡(zk),\mathcal{L}=\left\|\left(\sum_{k=1}^{K}m_{k}x_{k}\right)-I\right\|_{2}^{2}+\beta\cdot\sum_{k=1}^{K}\mathrm{KL}(z_{k}), (1)
ℒ=‖(∑k=1Kmk​xk)−I‖22.\mathcal{L}=\left\|\left(\sum_{k=1}^{K}m_{k}x_{k}\right)-I\right\|_{2}^{2}. (2)

3.2 Neuro-Symbolic Concept Learning

NS-CL represents concepts as learned vectors and executes a parsed question over object features. Filtering selects objects by concept similarity; subsequent operations query attributes or count selected objects. LORL adapts this machinery to uncertain, learned slots instead of assuming a pretrained detector supplies valid objects.

Refer to caption
Figure 3: In Shop-VRB-Simple, questions involve seven different attributes of objects in the scene: name, size, weight, material, color, shape, mobility. In PartNet-Chairs, images are paired with sentences describing the names and colors of the parts in the scene, and their relationships.

4 Language-mediated, Object-centric Representation Learning

The architecture has four components, but the parser is pretrained separately. The staged LORL optimization updates the perception modules and grounded concept representations, using the fixed parsed programs to route language supervision.

Image encoder.

The image encoder maps an image to a set of latent slot vectors, intended to separate individual objects or parts.

Image decoder.

The decoder predicts a component image, mask, and scalar objectness score for each slot. Objectness weights the reconstructed component and later limits its selection by the executor. A linear head on the penultimate decoder layer predicts the score for both MONet and Slot Attention.

Semantic parser.

A semantic parser trained on question-program pairs translates language into executable primitives such as filter and query. Programs use a CLEVR-like grammar, with separate learnable embeddings for concepts such as white. This supervised parsing stage is a material part of the setup.

Neuro-symbolic program executor.

The executor combines concept similarity with objectness: a filter output is capped by the slot’s objectness score. A slot must therefore match the requested concept and plausibly represent an object. The resulting soft selections feed the remaining program operations.

Training paradigm.

Training first fits perception through image reconstruction, then freezes perception while learning concepts from answer supervision, and finally jointly refines the trainable components. The total loss combines perception loss with answer cross-entropy. The appendix records dataset-specific schedules; the second stage is skipped on PartNet-Chairs because initial segmentation is too weak for useful fixed-feature grounding.

Equation
ℒ=α⋅ℒperception+β⋅ℒreasoning.\mathcal{L}=\alpha\cdot\mathcal{L}_{\text{perception}}+\beta\cdot\mathcal{L}_{\text{reasoning}}.

5 Experiments

Evaluation distinguishes segmentation quality from semantic structure in the embeddings, question answering, and transfer to referring expressions.

5.1 Image Segmentation

Refer to caption
Figure 4: Visualization on Shop-VRB-Simple. Pixels with the same color represent a mask produced by the models. The Slot Attention model often fails to segment blenders, coffee makers, and toasters. LORL helps to greatly improve its results.

Data.

Shop-VRB-Simple deliberately removes tiny or highly transparent objects and complex backgrounds that the base discovery systems cannot handle. It contains 10,000 training images with 90,000 questions and 960 test images with about 8,600 questions. PartNet-Chairs uses six chair shapes with randomly colored parts: 5,000 training images paired with 40,000 template-generated descriptions and 960 test images. These controlled datasets test complex shapes and part segmentation, but do not establish performance on unrestricted real photographs.

Baselines.

Slot Attention is evaluated on both datasets. MONet performs poorly on the varied colors and reflective objects in Shop-VRB-Simple, so its reported comparison is restricted to PartNet-Chairs. The experiments therefore do not show that every base architecture benefits on every dataset.

(a) ARI↑\uparrow GT Split↓\downarrow Pred Split↓\downarrow
Slot Attention 83.51±2.383.51_{\pm 2.3} 15.68±1.915.68_{\pm 1.9} 13.19±1.513.19_{\pm 1.5}
LORL + SA 89.23±1.6\bf 89.23_{\pm 1.6} 9.95±1.6\bf 9.95_{\pm 1.6} 10.18±1.3\bf 10.18_{\pm 1.3}
(b) Slot Attention LORL + SA
Coffee maker 39.4±7.039.4_{\pm 7.0} 21.9±8.3\bf 21.9_{\pm 8.3}
Blender 38.9±11.538.9_{\pm 11.5} 17.7±3.4\bf 17.7_{\pm 3.4}
Toaster 33.4±3.933.4_{\pm 3.9} 16.4±7.2\bf 16.4_{\pm 7.2}
Table 1: (a) Results on Shop-VRB-Simple. LORL helps to improve Slot Attention (Locatello et al., 2020, SA; ) in all metrics. (b) The Ground Truth split ratio for the three object categories where SA most commonly fail. LORL helps SA to reduce the ratio by 50%.
Refer to caption
Figure 5: Visualization on PartNet-Chairs. Pixels with the same color represent a mask produced by the models. LORL successfully recognizes different parts in various situations.

Metrics.

Adjusted Rand Index compares pixel clustering and can underweight errors on small objects. The paper supplements it with object-level split ratios: ground-truth objects split among several masks, and predicted masks covering several objects. Both ratios should be low. The main text describes a 20% object-pixel coverage threshold, while the appendix calls the threshold IoU; the retained formulas and threshold-sensitivity table preserve that source distinction.

Equation
GTSplit\displaystyle\mathrm{GTSplit} =#​of objects that are covered by > 1 masks#​of objects that are covered by > 0 masks,\displaystyle=\frac{\#~\text{of objects that are covered by $>$ 1 masks}}{\#~\text{of objects that are covered by $>$ 0 masks}},
PredSplit\displaystyle\mathrm{PredSplit} =#​of masks that cover > 1 objects#​of masks that cover > 0 objects.\displaystyle=\frac{\#~\text{of masks that cover $>$ 1 objects}}{\#~\text{of masks that cover $>$ 0 objects}}.
ARI↑\uparrow GT Split↓\downarrow Pred Split↓\downarrow
MONet 91.41±3.791.41_{\pm{3.7}} 10.3±5.1{10.3_{\pm{5.1}}} 14.09±5.2{14.09_{\pm{5.2}}}
LORL + MONet 94.91±2.1{94.91_{\pm{2.1}}} 4.95±0.7{4.95_{\pm{0.7}}} 4.02±2.5{4.02_{\pm{2.5}}}
Slot Attention 87.32±3.6{87.32_{\pm{3.6}}} 12.54±6.6{12.54_{\pm{6.6}}} 22.99±5.0{22.99_{\pm{5.0}}}
LORL + SA 95.81±1.0{\bf 95.81_{\pm{1.0}}} 3.39±1.1{\bf 3.39_{\pm{1.1}}} 2.92±1.0{\bf 2.92_{\pm{1.0}}}
Table 2: Quantitative results on PartNet-Chairs. All numbers are in percentage. LORL consistently improves MONet’s and Slot Attention’s performance on segmentation.

Results.

On Shop-VRB-Simple, language raises Slot Attention’s ARI from 83.51 to 89.23 and lowers both split ratios. Difficult coffee-maker, blender, and toaster categories show roughly halved ground-truth split rates. On PartNet-Chairs, both backbones improve, with larger gains for Slot Attention. Results report standard errors across three runs for Shop-VRB-Simple and four for PartNet-Chairs. The separately trained parser exceeds 99.9% test accuracy in this generated-language setting.

k=1k=1 k=3k=3 k=5k=5
Slot Attention 54.07±2.654.07_{\pm 2.6} 43.70±1.743.70_{\pm 1.7} 37.02±1.737.02_{\pm 1.7}
LORL + SA 94.03±0.6\bf 94.03_{\pm 0.6} 91.03±1.3\bf 91.03_{\pm 1.3} 87.71±1.7\bf 87.71_{\pm 1.7}
Table 3: The percentage (%) of retrieved objects that belong to the same category as the query object. With LORL, objects within the same category are more likely to be close to each other in the feature space. The results are averaged over 3 runs and standard errors are given.
QA Accuracy
LORL + SA (No FT) 62.79±1.662.79_{\pm 1.6}
LORL + SA 92.72±1.0\bf 92.72_{\pm 1.0}
Table 4: Our three-stage training paradigm improves visual concept learning. Without fine-tuning (i.e., the third training stage), the question answering accuracy drops by 30%. The results are averaged over 3 runs and standard errors are given.

5.2 Instance Retrieval

Retrieval evaluates 1,000 object features per model, retaining only masks with at least 0.75 IoU against an annotated object. Neighbors in the learned feature space are more often in the query’s category after LORL training; top-one category agreement increases from 54.07% to 94.03%. This measures semantic quality among well-segmented objects, not end-to-end retrieval over every predicted slot.

5.3 Visual Reasoning

Removing the final joint-training stage drops question-answering accuracy from 92.72% to 62.79%. Learning concepts on frozen visual features is therefore insufficient in this setup: language needs to reshape the visual representation as well.

5.4 Referring Expression Comprehension

After learning concepts from questions, LORL can execute referring expressions without new task-specific visual training. At mask IoU 0.5, its recall is 84.4%, compared with 90.1% for IEP-Ref trained on 17,000 referring-expression examples with segmentation labels. Both assume a pretrained parser and use the same concept inventory. This is transfer within the constructed domain, with a remaining accuracy gap to the directly supervised baseline.

6 Conclusion

Grounded language supervision improves object-centric representations in the evaluated synthetic scenes and supports several concept-based tasks. The main limits are controlled scene complexity, a restricted concept/program inventory, supervised parsing, and dependence on the underlying discovery architecture. The appendix’s failure cases and ablations clarify which improvements come from language, objectness, and joint training.

Acknowledgments

The authors thank Ruidong Wu and acknowledge the listed institutional and research-award support.

Appendix A Domain-Specific Language (DSL)

The program grammar extends CLEVR with Equal(X, y), allowing descriptive sentences to supervise the probability that a queried attribute, existence result, or count equals a specified value. Descriptions thus become executable truth judgments rather than separate free-text objectives.

Appendix B Hyperparameters

Table 5 retains loss weights, batch sizes, and the three stage lengths. Training stops when question-answering performance converges. PartNet-Chairs skips the concept-only second stage; keeping it slows final convergence by about 15 epochs without changing final performance.

Shop-VRB PartNet-Chairs
Slot Attention MONet Slot Attention
α\alpha 1 0.01 1
β\beta 0.1 1 0.1
Batch Size 64 64 64
N1N_{1} 90 100 50
N2N_{2} 20 0 0
N3N_{3} 80 200 200
Table 5: Hyperparameters of LORL

Learning rate scheduling.

Slot Attention initially warms Adam’s learning rate to 0.0004 over 10,000 iterations, then halves it every 100,000 iterations. PartNet-Chairs continues that schedule. Shop-VRB-Simple uses 0.001 for the 20-epoch concept stage, then 0.0002, followed by 0.00002 after another 65 epochs. MONet uses RMSProp at 0.01 initially and 0.001 thereafter. A curriculum begins with three-object scenes, then increases object counts and question complexity.

Appendix C Implementation Details

The appendix details the object-level segmentation metrics and the generation and evaluation of referring expressions.

GT and Pred split ratios.

Figures 6 and 7 retain examples where pixel-weighted ARI and object-level split ratios rank errors differently. Small chair parts can be merged with little impact on ARI, while the split measures expose those failures. The default reported threshold is 0.2; Table 6 repeats the comparison at 0.1 and 0.3 and still favors LORL.

Refer to caption
(a) There are 4 objects in the scene; one of them is split into 3 masks. Thus, GT Split =1/4=0.25=1/4=0.25.
Refer to caption
(b) There are 5 masks in the scene; one of them covers two objects. Thus, Pred Split =1/5=0.2=1/5=0.2.
Figure 6: Two examples on how GT/Pred Split Ratios are computed.
Refer to caption
Figure 7: Comparison of ARI and GT/Pred Split Ratios on example images. Pixels with the same color represent a mask produced by the models. We find that ARI is very sensitive to the size of objects, while split ratios capture object-level failures where an object is split or multiple objects are merged.
Thres =0.1=0.1 Thres =0.3=0.3
GT Split↓\downarrow Pred Split↓\downarrow GT Split↓\downarrow Pred Split↓\downarrow
Slot Attention 23.71±1.523.71_{\pm 1.5} 23.45±4.123.45_{\pm 4.1} 8.91±0.78.91_{\pm 0.7} 7.47±0.97.47_{\pm 0.9}
LORL + SA 17.74±1.4\bf 17.74_{\pm 1.4} 18.15±1.2\bf 18.15_{\pm 1.2} 5.86±1.3\bf 5.86_{\pm 1.3} 5.67±0.7\bf 5.67_{\pm 0.7}
Table 6: GT/Pred split ratios on Shop-VRB-Simple using different IoU thresholds. The results are averaged over 3 runs and standard errors are given.

Referring expression comprehension.

Expressions either name object properties or refer through another object, such as an object in front of a mug. The baseline chains learned neural modules and requires target segmentation masks during training. LORL instead multiplies the executor’s soft object selections by its predicted masks. Both approaches use pretrained parsing and the existing scene concepts.

Appendix D Additional Results

Additional experiments separate the effects of objectness, question types, language-data quantity, joint training, and base-model choice.

ARI↑\uparrow GT Split↓\downarrow Pred Split↓\downarrow
Slot Attention 83.51±2.3\bf 83.51_{\pm 2.3} 15.68±1.9\bf 15.68_{\pm 1.9} 13.19±1.513.19_{\pm 1.5}
SA + Obj score 83.4±1.883.4_{\pm 1.8} 16.57±1.516.57_{\pm 1.5} 12.71±0.8\bf 12.71_{\pm 0.8}
Table 7: Ablation study of the objectness score module on Shop-VRB-Simple. The results are averaged over 3 runs and standard errors are given.

Objectness score.

Adding objectness to Slot Attention without language does not improve segmentation. The segmentation gains therefore cannot be attributed to this extra head alone.

ARI↑\uparrow GT Split↓\downarrow Pred Split↓\downarrow
SA (image-only) 83.51±2.383.51_{\pm 2.3} 15.68±1.915.68_{\pm 1.9} 13.19±1.513.19_{\pm 1.5}
Count only 85.52±1.785.52_{\pm 1.7} 13.86±2.013.86_{\pm 2.0} 12.56±1.412.56_{\pm 1.4}
Exist only 86.29±2.286.29_{\pm 2.2} 15.34±2.915.34_{\pm 2.9} 10.33±1.110.33_{\pm 1.1}
Query only 88.79±1.188.79_{\pm 1.1} 10.64±1.710.64_{\pm 1.7} 9.03±0.59.03_{\pm 0.5}
All types 89.23±1.6\bf 89.23_{\pm 1.6} 9.95±1.69.95_{\pm 1.6} 10.18±1.3\bf 10.18_{\pm 1.3}
Table 8: Ablation study of using different types of questions to train LORL + SA on Shop-VRB-Simple. The standard errors are computed based on 3 runs.

Question type.

Counting, existence, and attribute-query questions each improve segmentation at equal total question counts. Attribute queries provide the largest individual benefit; the original mixture, with counting:existence:query ratio 1:1:7, performs best overall.

ARI↑\uparrow GT Split↓\downarrow Pred Split↓\downarrow
25% (22.5K) 81.0181.01 14.3114.31 15.3415.34
50% (45K) 84.3984.39 14.7914.79 9.379.37
75% (67.5K) 86.5386.53 11.6711.67 11.5211.52
100% (90K) 89.2389.23 9.959.95 10.1810.18
Table 9: Ablation study of using different number of questions to train LORL + SA on Shop-VRB-Simple.
Precision Recall
@0.5 @0.75 @1 @0.5 @0.75 @1
LORL + SA (NO) 59.52±0.759.52_{\pm 0.7} 54.13±2.254.13_{\pm 2.2} 36.36±3.636.36_{\pm 3.6} 94.96±1.2{\bf 94.96_{\pm 1.2}} 86.4±3.6{\bf 86.4_{\pm 3.6}} 58.07±5.7{\bf 58.07_{\pm 5.7}}
LORL + SA 89.47±0.7{\bf 89.47_{\pm 0.7}} 79.37±1.2{\bf 79.37_{\pm 1.2}} 52.48±1.6{\bf 52.48_{\pm 1.6}} 94.72±0.194.72_{\pm 0.1} 84.02±0.684.02_{\pm 0.6} 55.55±1.255.55_{\pm 1.2}
Table 10: Concept quantification evaluation. The number after @ indicates the IoU threshold. The results suggest that objectness score improves the precision of concept quantification.The results are averaged over 3 runs.
ARI↑\uparrow GT Split↓\downarrow Pred Split↓\downarrow
SPACE (Lin et al., 2020) 72.3472.34 29.229.2 12.3812.38
LORL + SPACE 97.82\bf 97.82 2.04\bf 2.04 2.17\bf 2.17
Table 11: Segmentation performance of SPACE and LORL +SPACE on CLEVR. The integration of LORL improves the result.

Data efficiency.

Using 25%, 50%, 75%, and 100% of the language pairs shows progressively better segmentation as supervision grows. This is a within-dataset data-efficiency study, with the visual setting and backbone held fixed.

QA Accuracy
LORL + SA (No FT) 62.79±1.662.79_{\pm 1.6}
IEP (Johnson et al., 2017b) 78.3±0.178.3_{\pm 0.1}
NSCL (Mao et al., 2019) 97.9±0.0\bf 97.9_{\pm 0.0}
LORL + SA 92.72±1.092.72_{\pm 1.0}
Table 12: Question answering accuracy on the Shop-VRB-Simple dataset. The results are averaged over 3 runs and standard errors are given.

Visual reasoning baselines.

With the same question-answer pairs, LORL outperforms the IEP reasoning baseline. NS-CL scores higher but receives a pretrained object detector, so it has additional supervision. The complete table preserves these different information budgets.

Integration with SPACE.

LORL also improves SPACE on CLEVR, raising ARI from 72.34 to 97.82. SPACE performs poorly on Shop-VRB-Simple and PartNet-Chairs with or without LORL, frequently fragmenting complex objects. Language supervision does not overcome every limitation of a base discovery model.

Baseline using language supervision.

A simpler baseline encodes questions and answers with a GRU and concatenates them to image features. It does not improve over Slot Attention on Shop-VRB-Simple, suggesting that merely adding language features is insufficient; the structured grounding objective matters in this comparison.

Concept quantification.

For concept quantification, each slot receives concepts with positive cosine similarity; slots with no concepts or objectness below 0.5 are discarded. Predicted and reference objects are matched by the Hungarian algorithm using concept-set IoU. Objectness substantially improves precision, with some recall tradeoffs, even when question-answering accuracy is similar. The results depend on these specified thresholds and matching rules.

Equation
I​o​Ui​j=|Ci∩Cj||Ci∪Cj|.IoU_{ij}=\frac{|C_{i}\cap C_{j}|}{|C_{i}\cup C_{j}|}.
Concept demonstration

Open image ↗