Overview
At a glance
LORL couples object discovery with grounded language learning. A reconstruction model proposes object slots; a program executor uses their representations to answer questions or judge descriptions. Errors in these language tasks help refine the segmentation and concept embeddings. Integrating LORL improves MONet and Slot Attention in the tested synthetic scenes and supports retrieval, reasoning, and referring-expression tasks. The method avoids direct segmentation labels during object learning, but uses language supervision and a separately supervised semantic parser.
1 Introduction
Reconstruction alone may split one object into several pieces or merge conceptually distinct parts. Language can penalize those errors: identifying a pan requires its handle to belong to the pan, while counting chair legs requires separating them. LORL learns object masks, representations, and concept embeddings through reconstruction and language-task losses. The experiments test whether those representations improve segmentation and transfer to other concept-based tasks.
2 Related Work
Unsupervised object representation learning.
Object-centric generative models explain an image through a collection of component images and masks, usually trained with reconstruction losses. Other approaches use motion or multiple views. LORL adds questions or descriptions to static images as a source of object-individuation supervision.
Visual concept learning.
Earlier concept-grounding systems often start from pretrained detectors and object proposals. LORL couples concept learning to the discovery of object slots themselves, so language can influence segmentation rather than only label fixed proposals.
3 Preliminaries
The method combines a reconstruction-based object-slot model with a differentiable program executor that grounds words in those slots.
3.1 Unsupervised Object-Centric Representation Learning
Each slot contains a latent feature, reconstructed component image, and mask; unused slots may represent no object. MONet uses recurrent spatial attention and a variational autoencoder, with reconstruction and latent-prior losses. Slot Attention iteratively extracts slot features and decodes their images and masks, trained by reconstruction. Both supply the perception module used here.
Equations 1–2
| (1) |
| (2) |
3.2 Neuro-Symbolic Concept Learning
NS-CL represents concepts as learned vectors and executes a parsed question over object features. Filtering selects objects by concept similarity; subsequent operations query attributes or count selected objects. LORL adapts this machinery to uncertain, learned slots instead of assuming a pretrained detector supplies valid objects.
4 Language-mediated, Object-centric Representation Learning
The architecture has four components, but the parser is pretrained separately. The staged LORL optimization updates the perception modules and grounded concept representations, using the fixed parsed programs to route language supervision.
Image encoder.
The image encoder maps an image to a set of latent slot vectors, intended to separate individual objects or parts.
Image decoder.
The decoder predicts a component image, mask, and scalar objectness score for each slot. Objectness weights the reconstructed component and later limits its selection by the executor. A linear head on the penultimate decoder layer predicts the score for both MONet and Slot Attention.
Semantic parser.
A semantic parser trained on question-program pairs translates language into executable primitives such as filter and query. Programs use a CLEVR-like grammar, with separate learnable embeddings for concepts such as white. This supervised parsing stage is a material part of the setup.
Neuro-symbolic program executor.
The executor combines concept similarity with objectness: a filter output is capped by the slot’s objectness score. A slot must therefore match the requested concept and plausibly represent an object. The resulting soft selections feed the remaining program operations.
Training paradigm.
Training first fits perception through image reconstruction, then freezes perception while learning concepts from answer supervision, and finally jointly refines the trainable components. The total loss combines perception loss with answer cross-entropy. The appendix records dataset-specific schedules; the second stage is skipped on PartNet-Chairs because initial segmentation is too weak for useful fixed-feature grounding.
Equation
5 Experiments
Evaluation distinguishes segmentation quality from semantic structure in the embeddings, question answering, and transfer to referring expressions.
5.1 Image Segmentation
Data.
Shop-VRB-Simple deliberately removes tiny or highly transparent objects and complex backgrounds that the base discovery systems cannot handle. It contains 10,000 training images with 90,000 questions and 960 test images with about 8,600 questions. PartNet-Chairs uses six chair shapes with randomly colored parts: 5,000 training images paired with 40,000 template-generated descriptions and 960 test images. These controlled datasets test complex shapes and part segmentation, but do not establish performance on unrestricted real photographs.
Baselines.
Slot Attention is evaluated on both datasets. MONet performs poorly on the varied colors and reflective objects in Shop-VRB-Simple, so its reported comparison is restricted to PartNet-Chairs. The experiments therefore do not show that every base architecture benefits on every dataset.
Metrics.
Adjusted Rand Index compares pixel clustering and can underweight errors on small objects. The paper supplements it with object-level split ratios: ground-truth objects split among several masks, and predicted masks covering several objects. Both ratios should be low. The main text describes a 20% object-pixel coverage threshold, while the appendix calls the threshold IoU; the retained formulas and threshold-sensitivity table preserve that source distinction.
Equation
Results.
On Shop-VRB-Simple, language raises Slot Attention’s ARI from 83.51 to 89.23 and lowers both split ratios. Difficult coffee-maker, blender, and toaster categories show roughly halved ground-truth split rates. On PartNet-Chairs, both backbones improve, with larger gains for Slot Attention. Results report standard errors across three runs for Shop-VRB-Simple and four for PartNet-Chairs. The separately trained parser exceeds 99.9% test accuracy in this generated-language setting.
| Slot Attention | |||
| LORL + SA |
5.2 Instance Retrieval
Retrieval evaluates 1,000 object features per model, retaining only masks with at least 0.75 IoU against an annotated object. Neighbors in the learned feature space are more often in the query’s category after LORL training; top-one category agreement increases from 54.07% to 94.03%. This measures semantic quality among well-segmented objects, not end-to-end retrieval over every predicted slot.
5.3 Visual Reasoning
Removing the final joint-training stage drops question-answering accuracy from 92.72% to 62.79%. Learning concepts on frozen visual features is therefore insufficient in this setup: language needs to reshape the visual representation as well.
5.4 Referring Expression Comprehension
After learning concepts from questions, LORL can execute referring expressions without new task-specific visual training. At mask IoU 0.5, its recall is 84.4%, compared with 90.1% for IEP-Ref trained on 17,000 referring-expression examples with segmentation labels. Both assume a pretrained parser and use the same concept inventory. This is transfer within the constructed domain, with a remaining accuracy gap to the directly supervised baseline.
6 Conclusion
Grounded language supervision improves object-centric representations in the evaluated synthetic scenes and supports several concept-based tasks. The main limits are controlled scene complexity, a restricted concept/program inventory, supervised parsing, and dependence on the underlying discovery architecture. The appendix’s failure cases and ablations clarify which improvements come from language, objectness, and joint training.
Acknowledgments
The authors thank Ruidong Wu and acknowledge the listed institutional and research-award support.
Appendix A Domain-Specific Language (DSL)
The program grammar extends CLEVR with Equal(X, y), allowing descriptive sentences to supervise the probability that a queried attribute, existence result, or count equals a specified value. Descriptions thus become executable truth judgments rather than separate free-text objectives.
Appendix B Hyperparameters
Table 5 retains loss weights, batch sizes, and the three stage lengths. Training stops when question-answering performance converges. PartNet-Chairs skips the concept-only second stage; keeping it slows final convergence by about 15 epochs without changing final performance.
Learning rate scheduling.
Slot Attention initially warms Adam’s learning rate to 0.0004 over 10,000 iterations, then halves it every 100,000 iterations. PartNet-Chairs continues that schedule. Shop-VRB-Simple uses 0.001 for the 20-epoch concept stage, then 0.0002, followed by 0.00002 after another 65 epochs. MONet uses RMSProp at 0.01 initially and 0.001 thereafter. A curriculum begins with three-object scenes, then increases object counts and question complexity.
Appendix C Implementation Details
The appendix details the object-level segmentation metrics and the generation and evaluation of referring expressions.
GT and Pred split ratios.
Figures 6 and 7 retain examples where pixel-weighted ARI and object-level split ratios rank errors differently. Small chair parts can be merged with little impact on ARI, while the split measures expose those failures. The default reported threshold is 0.2; Table 6 repeats the comparison at 0.1 and 0.3 and still favors LORL.
Referring expression comprehension.
Expressions either name object properties or refer through another object, such as an object in front of a mug. The baseline chains learned neural modules and requires target segmentation masks during training. LORL instead multiplies the executor’s soft object selections by its predicted masks. Both approaches use pretrained parsing and the existing scene concepts.
Appendix D Additional Results
Additional experiments separate the effects of objectness, question types, language-data quantity, joint training, and base-model choice.
Objectness score.
Adding objectness to Slot Attention without language does not improve segmentation. The segmentation gains therefore cannot be attributed to this extra head alone.
Question type.
Counting, existence, and attribute-query questions each improve segmentation at equal total question counts. Attribute queries provide the largest individual benefit; the original mixture, with counting:existence:query ratio 1:1:7, performs best overall.
| ARI | GT Split | Pred Split | |
| 25% (22.5K) | |||
| 50% (45K) | |||
| 75% (67.5K) | |||
| 100% (90K) |
Data efficiency.
Using 25%, 50%, 75%, and 100% of the language pairs shows progressively better segmentation as supervision grows. This is a within-dataset data-efficiency study, with the visual setting and backbone held fixed.
Visual reasoning baselines.
With the same question-answer pairs, LORL outperforms the IEP reasoning baseline. NS-CL scores higher but receives a pretrained object detector, so it has additional supervision. The complete table preserves these different information budgets.
Integration with SPACE.
LORL also improves SPACE on CLEVR, raising ARI from 72.34 to 97.82. SPACE performs poorly on Shop-VRB-Simple and PartNet-Chairs with or without LORL, frequently fragmenting complex objects. Language supervision does not overcome every limitation of a base discovery model.
Baseline using language supervision.
A simpler baseline encodes questions and answers with a GRU and concatenates them to image features. It does not improve over Slot Attention on Shop-VRB-Simple, suggesting that merely adding language features is insufficient; the structured grounding objective matters in this comparison.
Concept quantification.
For concept quantification, each slot receives concepts with positive cosine similarity; slots with no concepts or objectness below 0.5 are discarded. Predicted and reference objects are matched by the Hungarian algorithm using concept-set IoU. Objectness substantially improves precision, with some recall tradeoffs, even when question-answering accuracy is similar. The results depend on these specified thresholds and matching rules.