At a glance
The Neuro-Symbolic Concept Learner (NS-CL) learns visual concepts and translates questions into executable programs from images and question-answer pairs. An object-based visual representation, learned concept embeddings, and a fixed differentiable executor connect language with perception. Explicit execution traces expose intermediate reasoning and some sources of error.
On CLEVR, the model reaches 98.9% question-answer accuracy using roughly 10% of the training images and 99.2% with the full set. This efficiency depends on substantial structure: a predefined domain-specific language, a known concept lexicon, curriculum design, pretrained visual features, and an object detector trained with bounding-box labels. Generalization experiments often freeze or pretrain individual components; natural-image results are more limited.
1 Introduction
Learning from scenes and ordinary descriptions should connect words to objects, properties, and relations without annotating every intermediate reasoning step. NS-CL separates visual grounding from symbolic composition: perception produces object features, concepts classify those features, and programs combine the resulting predictions.
Figure 1 illustrates this interface. The contribution is joint learning of concepts and question interpretation from answers, combined with explicit execution. It does not remove all supervision or hand-designed knowledge: object proposals and the reasoning language are supplied. The study tests data efficiency, controlled forms of generalization, new operations, and preliminary transfer to natural images.
2 Related Work
Earlier visual reasoning systems use neural attention, learned neural modules, or symbolic programs over supervised scene descriptions. NS-CL combines learned object-centric concept representations with predefined symbolic operations and learns the parser without ground-truth program annotations. Table 1 compares the required supervision and model components.
Related concept-learning approaches use human attributes, language, or structured scene descriptions. Here, question-answer loss trains concept grounding while the program language constrains how concepts can be combined. The explicit representation favors compositional reuse, at the cost of choosing an ontology and executor in advance.
| Models | Visual Features | Semantics | Extra Labels | Inference | |
| # Prog. | Attr. | ||||
| FiLM (Perez et al., 2018) | Convolutional | Implicit | 0 | No | Feature Manipulation |
| IEP (Johnson et al., 2017b) | Convolutional | Explicit | 700K | No | Feature Manipulation |
| MAC (Hudson & Manning, 2018) | Attentional | Implicit | 0 | No | Feature Manipulation |
| Stack-NMN (Hu et al., 2018) | Attentional | Implicit | 0 | No | Attention Manipulation |
| TbD (Mascharka et al., 2018) | Attentional | Explicit | 700K | No | Attention Manipulation |
| NS-VQA (Yi et al., 2018) | Object-Based | Explicit | 0.2K | Yes | Symbolic Execution |
| NS-CL | Object-Based | Explicit | 0 | No | Symbolic Execution |
3 Neuro-Symbolic concept learner
Figure 2 separates visual perception, concept quantization, semantic parsing, and quasi-symbolic execution. Training connects the visual and language components through the probability of the correct answer.
3.1 Model details
Visual perception.
Mask R-CNN proposes objects. A ResNet-34 backbone and RoIAlign extract each object’s features, concatenated with full-scene context so relative size and spatial relations remain accessible. Concept learning uses these continuous object representations rather than supervised discrete scene attributes.
The proposal model is trained on 4,000 CLEVR images with bounding-box annotations, and the backbone is initialized from ImageNet. Consequently, the absence of attribute and program labels should not be read as absence of visual supervision.
Concept quantization.
Each attribute has a learned mapping into a visual-semantic space; each concept has a learned embedding and an attribute association. A sigmoid of scaled cosine similarity gives concept-membership probabilities. Pairwise features similarly support relations. Figure 3 illustrates how visual and conceptual representations align; Appendix C retains the precise formulas.
DSL and semantic parsing.
A bidirectional GRU encodes the question, and a recursive decoder predicts a tree of operations and their concept arguments. The operations and types come from a predefined domain-specific language (DSL). Hand-coded rules identify concept words, while abstract concept-type tokens help the parser reuse structures with different words. The complete type system, algorithm, and parsing example remain in Appendices A and B.
Quasi-symbolic program execution.
Programs operate on probabilistic object sets rather than prematurely discretized scene labels. A set is a vector of memberships; a single-object representation sums to one. Fixed operators implement filtering, relations, intersections, counting, and queries, passing gradients to visual representations and concept parameters. Figure 4 shows an example.
The executor itself is designed rather than learned. Explicit intermediate sets aid inspection, but erroneous perception or an answer-consistent wrong program can still yield a misleading trace.
3.2 Training paradigm
Optimization objective.
The objective maximizes the probability of the correct answer, averaging over programs proposed by the semantic parser. Visual parameters receive gradients through execution; discrete program choices use REINFORCE or an off-policy search over answer-consistent programs. Equation 1 retains the objective. Appendix D explains the search and its treatment of spurious programs.
Equation 1
| (1) |
Curriculum visual concept learning.
The curriculum first learns simple object concepts in small scenes, then relations, then complex programs with perception frozen; final joint fine-tuning unfreezes the components. Appendix E describes these as three lessons followed by joint tuning. Scene size and question complexity increase gradually, using program-depth information to select training questions.
In the reported ablations, removing the scene-complexity curriculum makes concept recognition nearly random, and removing the question-complexity curriculum prevents successful joint training. The learning procedure therefore relies materially on this structured progression.
4 Experiments
The main CLEVR setting uses 5,000 training images, under 10% of the full 70,000, with question selection following the curriculum. Experiments separately examine grounded concepts, question answering, controlled generalization, and extensions beyond the original task.
4.1 Visual concept learning
Classification-based concept evaluation.
The authors evaluate concept classification on the CLEVR validation split; attribute labels do not supervise the learned concept classifier. Object-attribute classification is near 99%; learned relations are less accurate because their supervision is indirect. These tests inspect grounding beyond aggregate question-answer accuracy.
Count-based concept evaluation.
Count questions probe whether a system consistently identifies every object matching a concept. Table 2 reports a mean of 98.7 for NS-CL, versus 90.6 for IEP, 95.9 for MAC, and 96.5 for TbD under this diagnostic. Counts can reveal grounding errors hidden by simpler answer categories, though they still combine perception and execution.
| Visual | Mean | Color | Mat. | Shape | Size | |
|---|---|---|---|---|---|---|
| IEP | Conv. | 90.6 | 91.0 | 90.0 | 89.9 | 90.6 |
| MAC | Attn. | 95.9 | 98.0 | 91.4 | 94.4 | 94.2 |
| TbD (hres.) | Attn. | 96.5 | 96.6 | 92.2 | 95.4 | 92.6 |
| NS-CL | Obj. | 98.7 | 99.0 | 98.7 | 98.1 | 99.1 |
| Model | Visual | Accuracy ( Data) | Accuracy ( Data) |
|---|---|---|---|
| TbD | Attn. | 99.1 | 54.2 |
| TbD-Object | Obj. | 84.1 | 52.6 |
| TbD-Mask | Attn. | 99.0 | 55.0 |
| MAC | Attn. | 98.9 | 67.3 |
| MAC-Object | Obj. | 79.5 | 51.2 |
| MAC-Mask | Attn. | 98.7 | 68.4 |
| NS-CL | Obj. | 99.2 | 98.9 |
4.2 Data-efficient and interpretable visual reasoning
With the full CLEVR training set, NS-CL reaches 99.2% accuracy; with about 10%, it reaches 98.9%. In the same data-reduction comparison, MAC falls from 98.9 to 67.3 and TbD from 99.1 to 54.2. Table 3 also tests object-based and mask-guided baseline variants; object features alone do not reproduce the proposed system’s data efficiency.
Table 4 provides question-type results and supervision requirements. NS-CL’s 98.9 overall ties MAC in that comparison and is slightly below TbD’s 99.1, which uses program annotations. The main advantage is strong performance with fewer examples and inspectable execution, rather than a uniform accuracy win. Appendix F gives the baseline construction and ablations.
| Model | Prog. Anno. | Overall | Count | Cmp. Num. | Exist | Query Attr. | Cmp. Attr. |
|---|---|---|---|---|---|---|---|
| Human | N/A | 92.6 | 86.7 | 86.4 | 96.6 | 95.0 | 96.0 |
| NMN | 700K | 72.1 | 52.5 | 72.7 | 79.3 | 79.0 | 78.0 |
| N2NMN | 700K | 88.8 | 68.5 | 84.9 | 85.7 | 90.0 | 88.8 |
| IEP | 700K | 96.9 | 92.7 | 98.7 | 97.1 | 98.1 | 98.9 |
| DDRprog | 700K | 98.3 | 96.5 | 98.4 | 98.8 | 99.1 | 99.0 |
| TbD | 700K | 99.1 | 97.6 | 99.4 | 99.2 | 99.5 | 99.6 |
| RN | 0 | 95.5 | 90.1 | 93.6 | 97.8 | 97.1 | 97.9 |
| FiLM | 0 | 97.6 | 94.5 | 93.8 | 99.2 | 99.2 | 99.0 |
| MAC | 0 | 98.9 | 97.2 | 99.4 | 99.5 | 99.3 | 99.5 |
| NS-CL | 0 | 98.9 | 98.2 | 99.0 | 98.8 | 99.3 | 99.1 |
| Model | Test | |||
| Split A | Split B | Split C | Split D | |
| MAC | 97.3 | N/A | 92.9 | N/A |
| IEP | 96.1 | 92.1 | 91.5 | 90.9 |
| TbD | 98.8 | 94.5 | 94.3 | 91.9 |
| NS-CL | 98.9 | 98.9 | 98.7 | 98.8 |
4.3 Generalization to new attributes and compositions
Generalizing to new visual compositions.
CLEVR-CoGenT changes which colors co-occur with particular shapes. Ordinary joint training can entangle shape and color. The successful experiment freezes pretrained attribute operators and learns new concept embeddings and the parser on condition A, obtaining 98.8% on A and 98.9% on B.
This supports reuse of a suitably disentangled representation. It does not show that unrestricted joint learning automatically discovers composition-independent attributes from the biased condition-A data.
Generalizing to new visual concepts.
To add the previously withheld color purple, the model is fine-tuned with 100 images containing purple objects. It reaches 93.9% question-answer accuracy, exceeding the compared IEP and TbD settings by 4.6 and 6.1 percentage points. The parser was pretrained on the full CLEVR question set, so the experiment tests adaptation of visual concepts with an established linguistic structure.
4.4 Combinatorial generalization to new scenes and questions
Training scenes contain fewer than six objects and questions have program depth below five. Tests increase scene size, program depth, or both; Figure 5 reports 98.9, 98.9, 98.7, and 98.8% across the four splits. Figure 6 shows an example.
The semantic parser is pretrained on the full dataset and fixed, and counting questions with answers above six are excluded. These controls isolate generalization of concept grounding and execution; they do not establish unrestricted extrapolation of the complete learned system.
4.5 Extending to other program domain
The model is adapted to a restricted image-caption retrieval task using a new program language for relational statements. Visual components are fixed while a new semantic parser is learned. Table 5 retains the retrieval examples and comparison results.
On the comparable subset, NS-CL scores 96.9%, versus 97.0 for TbD and 95.5 for IEP. On the full constructed task, it scores 97.0 versus 68.9 for CNN-LSTM. This demonstrates reuse of grounded concepts with another executor/parser, under templated caption and task assumptions.
4.6 Extending to natural images and language
VQS supplies natural images and questions grounded in annotated regions. This extension uses pretrained object proposals and a pretrained dependency parser to construct programs, together with word embeddings. Each question has 18 candidate answers. Figures 7 and 8 show examples and learned concepts.
NS-CL achieves 44.3% accuracy, compared with 43.9 for the MLP baseline and 46.2 for MAC. The result demonstrates an inspectable pipeline in a more difficult domain, but remains below the strongest compared baseline. Appendix G details the additional pretrained components and restricted evaluation.
5 Discussion and Future Work
Explicit object representations and programs can make grounded concept learning efficient and reusable within a chosen language of operations. The controlled experiments support this combination, while also exposing dependence on proposals, curriculum, predefined concepts, and component pretraining.
Broader natural-language understanding, automatic discovery of concept words and operations, dynamic scenes, actions, and richer three-dimensional representations remain open directions. Transparent traces help diagnose mistakes; they do not guarantee correct grounding or reasoning.
Acknowledgements.
The authors acknowledge collaborators, institutional support, and research funding listed in the original paper.
Appendix A CLEVR domain-specific language and implementations
Tables 6 and 7 retain every CLEVR operation, signature, semantic description, and type. Operations requiring one object implicitly cast a set through the Unique operation. The intended reference must be nonempty and unambiguous.
The original table describes Exist as asking whether a set is empty, while the implementation in Table 9 returns its maximum membership, corresponding to existence. Both source tables are retained without silently rewriting this wording.
| Operation | Signature | Semantics |
| Scene | () ObjectSet | Return all objects in the scene. |
| Filter | (ObjectSet, ObjConcept) ObjectSet | Filter out a set of objects having the object-level concept (e.g., red) from the input object set. |
| Relate | (Object, RelConcept) ObjectSet | Filter out a set of objects that have the relational concept (e.g., left) with the input object. |
| AERelate | (Object, Attribute) ObjectSet | (Attribute-Equality Relate) Filter out a set of objects that have the same attribute value (e.g., same color) as the input object. |
| Intersection | (ObjectSet, ObjectSet) ObjectSet | Return the intersection of two object sets. |
| Union | (ObjectSet, ObjectSet) ObjectSet | Return the union of two object sets. |
| Query | (Object, Attribute) ObjConcept | Query the attribute (e.g., color) of the input object. |
| AEQuery | (Object, Object, Attribute) Bool | (Attribute-Equality Query) Query if two input objects have the same attribute value (e.g., same color). |
| Exist | (ObjectSet) Bool | Query if the set is empty. |
| Count | (ObjectSet) Integer | Query the number of objects in the input set. |
| CLessThan | (ObjectSet, ObjectSet) Bool | (Counting LessThan) Query if the number of objects in the first input set is less than the one of the second set. |
| CGreaterThan | (ObjectSet, ObjectSet) Bool | (Counting GreaterThan) Query if the number of objects in the first input set is greater than the one of the second set. |
| CEqual | (ObjectSet, ObjectSet) Bool | (Counting Equal) Query if the number of objects in the first input set is the same as the one of the second set. |
| Type | Example | Semantics |
| ObjConcept | Red, Cube, etc. | Object-level concepts. |
| Attribute | Color, Shape, etc. | Object-level attributes. |
| RelConcept | Left, Front, etc. | Relational concepts. |
| Object | A single object in the scene. | |
| ObjectSet | A set of objects in the scene. | |
| Integer | A single integer. | |
| Bool | True, False | A single boolean value. |
Appendix B Semantic parsing
The string-to-tree parser combines a bidirectional GRU encoder, an operation decoder, a concept decoder, and output encoders that recursively generate child states. Algorithm 1 retains the complete procedure.
The concept inventory is known for CLEVR, and hand-coded rules extract concept words. Replacing specific words with indexed concept-type tokens lets the network predict reusable structures and then restore the arguments. Concept embeddings used by the parser are separate from the visual-semantic embeddings used to recognize objects. Natural-language extensions instead rely on linguistic tagging or parsing tools.
A Running Example
The running question asks about an object identified by attributes and a spatial relation. Table 8 follows the recursive generation of Query, Filter, Relate, Filter, and Scene, showing the latent state and selected concept at each step. This makes the interface between question structure and executable program explicit.
| Step | Inputs | Outputs | Recursive Invocation |
| 1 |
;
; |
||
| 2 |
;
; |
||
| 3 |
;
; |
||
| 4 |
;
; |
||
| 5 | ; | (End of branch.) |
Appendix C Program Execution
The executor treats object memberships as continuous values, allowing question-answer loss to train perception and concepts through fixed operations.
Object-typed and ObjectSet-typed variables.
An ObjectSet contains one membership probability per detected object. An Object is a distribution over those objects. Casting a set into an object applies a softmax to the logits of its memberships; the intended interpretation is a unique referent. Empty or ambiguous references can be detected during symbolic execution.
Concept quantization.
Object concepts have embedding vectors and normalized associations with attributes. Attribute-specific visual mappings and scaled cosine similarity yield concept probabilities. Relations use analogous pairwise classification, and equality of an attribute compares two mapped object representations. The original equations retain the scaling constants and aggregation over attribute associations.
Equations
Quasi-symbolic program execution.
Table 9 preserves the full implementations of filtering, relations, set operations, queries, existence, counting, and count comparisons. Probabilities are stored in log space for stability. At inference, the operations can be quantized as in prior symbolic execution. These fixed formulas define the reasoning semantics rather than learning each operation from scratch.
| Signature | Implementation | |||
| Scene() : ObjectSet | ||||
|
Filter(: ObjectSet, : ObjConcept)
: ObjectSet |
||||
|
Relate(: Object, : RelConcept)
: ObjectSet |
||||
|
AERelate(: Object, : Attribute)
: ObjectSet |
||||
|
Intersection(: ObjectSet,
: ObjectSet) : ObjectSet |
||||
|
Union(: ObjectSet, : ObjectSet)
: ObjectSet |
||||
|
| |||
|
AEQuery(: Object, : Object,
: Attribute) : Bool |
||||
| Exist(: ObjectSet) : Bool | ||||
| Count(: ObjectSet) : Integer | ||||
|
CLessThan(: ObjectSet,
: ObjectSet) : Bool |
||||
|
CGreaterThan(: ObjectSet,
: ObjectSet) : Bool |
||||
|
CEqual(: ObjectSet,
: ObjectSet) : Bool |
Appendix D Optimization of the Semantic Parser
REINFORCE estimates the parser gradient by sampling programs. An alternative freezes perception, enumerates candidate programs, and records those producing the correct answer. This off-policy set supports an exact sum over answer-consistent programs in the stated language, avoiding repeated Monte Carlo sampling.
The resulting programs are inferred supervision, not annotated ground truth. Search tractability depends on the restricted operation vocabulary, program constraints, and curriculum stage.
Equation
Spurious program suppression.
Different programs may be semantically equivalent, or may agree only accidentally on the current scene. Training all answer-consistent programs uniformly can reinforce those accidental explanations. The proposed loss weights each program’s negative log probability by its current probability, with the weight detached from gradient computation.
The retained equations give the loss and gradient, and the original program examples illustrate equivalence and spurious agreement. Diverse scenes help suppress accidental programs, but correct answers alone cannot certify a correct interpretation.
Equations
| P1: | |||
| P2: |
| P1: | |||
| P2: |
Appendix E Curriculum Learning Setup
The curriculum starts with single-object attributes in small scenes, adds relations and compositional questions, then learns more complex programs with perception fixed. Scenes grow from three to ten objects. Question depth and permitted templates control each lesson; an additional count-after-relation template helps train relational concepts. The final stage jointly fine-tunes the full model.
These choices supply task structure beyond raw question-answer examples. The reported failures when removing scene-size or question-complexity progression show why the curriculum is part of the method rather than a minor optimization detail.
Appendix F Ablation study
F.1 Semantic parsing accuracy.
The reported parser accuracy exceeds 99.9% when its programs execute over ground-truth scenes. This is answer-equivalence evaluation, not exact matching of predicted and annotated program strings, so alternative correct or accidentally equivalent programs can count as successful.
F.2 Impacts of the ImageNet pre-training.
Removing ImageNet pretraining reduces overall question-answer accuracy from 98.9 to 98.2% and shape recognition from 98.7 to 97.5%. These explicit appendix values imply a 0.7-point answer-accuracy drop; the main-text description gives 0.5. The qualitative conclusion is a modest but measurable benefit in this synthetic setting.
F.3 Data Efficiency and Object-based Representations
This ablation asks whether object representations alone explain the gains under limited training data.
Baselines.
Object-based MAC and TbD variants attend over object features, while mask-guided variants retain spatial maps but incorporate proposal masks. Their architectures adapt the original attention mechanisms to use the same object information. Table 3 retains the full-data and reduced-data results.
Results.
Object-based variants do not match NS-CL’s data efficiency. The authors argue that permutation-invariant object attention is poorly suited to quantities such as counts, while mask guidance accelerates convergence without substantially improving final accuracy. NS-CL instead supplies counting and other reasoning operations explicitly.
For the full-data experiment, 3,500 training images form a hyperparameter-selection holdout, and the official validation split is used as the reported test set. The resulting NS-CL accuracy is 99.2%.
Appendix G Extending to Other Scene and Language Domains
G.1 Minecraft Dataset
Minecraft tests different visual appearances and additional reasoning operations. Figure 9 gives a scene, question, answer, and recovered program.
Setup.
The generated dataset contains 10,000 scenes, each with three to six objects drawn from twelve categories and four facing directions. Two operations extend the language: FilterMost selects an extremal object, such as the closest, and BelongTo tests membership in a set.
Results.
Table 10 reports 93.3% overall accuracy for NS-CL and 87.7% for the supervised NS-VQA comparison. The authors attribute part of the baseline deficit to noisy scene-label supervision caused by occlusion and proposal-to-box matching. This is a result under that rendering and training setup; it does not establish that weaker supervision is inherently superior. Figure 12 retains the execution traces and a missed-object failure.
G.2 VQS Dataset
VQS is a subset of VQA 1.0 with region annotations identifying evidence relevant to each question. Figure 10 shows the proposal and program interface.
Setup.
Training uses 63,509 images, with another 5,000 training-split images held out for model selection, and evaluation on the designated test split. Answers are selected from 18 candidates per question. A pretrained dependency parser extracts latent programs and concepts; bag-of-words or GloVe representations initialize concepts. These supplied language tools differ from the question-answer-trained parser used in CLEVR.
Baselines.
The MLP baseline scores image-question-answer triples using an ImageNet-pretrained ResNet-34 and word features. MAC receives each candidate answer concatenated with the question and predicts its score, using the same visual backbone. This compares candidate ranking in the specified multiple-choice task.
Results.
The 44.3% NS-CL result is close to the 43.9% MLP baseline and below MAC’s 46.2%. Its distinctive benefit here is the visible execution trace. Figure 13 also reveals limits: a correct answer can be grounded in the wrong object, and visual misclassification can propagate through otherwise understandable steps.
Appendix H Visualization of execution traces and visual concepts
Figures 11–15 preserve all qualitative traces and concept visualizations. CLEVR examples show successful composition, material misclassification, and ambiguous references. Minecraft examples show extremal-object reasoning and a missed pig behind a tree. Natural-image examples include both successful answers and incorrect grounding or action classification.
These cases illustrate what the intermediate representation makes inspectable. In particular, the correct “cake” answer grounded on a dish shows that answer accuracy and faithful localization are distinct. The final figures display learned single concepts and compositions in CLEVR and Minecraft.