Literature
Display
← All papers

The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

Jiayuan Mao · Chuang Gan · Pushmeet Kohli · Joshua B. Tenenbaum · Jiajun Wu

· ICLR 2019 ↗

Reasoning12 min text · CondensedOriginal paper ↗
Contents

At a glance

The Neuro-Symbolic Concept Learner (NS-CL) learns visual concepts and translates questions into executable programs from images and question-answer pairs. An object-based visual representation, learned concept embeddings, and a fixed differentiable executor connect language with perception. Explicit execution traces expose intermediate reasoning and some sources of error.

On CLEVR, the model reaches 98.9% question-answer accuracy using roughly 10% of the training images and 99.2% with the full set. This efficiency depends on substantial structure: a predefined domain-specific language, a known concept lexicon, curriculum design, pretrained visual features, and an object detector trained with bounding-box labels. Generalization experiments often freeze or pretrain individual components; natural-image results are more limited.

1 Introduction

Learning from scenes and ordinary descriptions should connect words to objects, properties, and relations without annotating every intermediate reasoning step. NS-CL separates visual grounding from symbolic composition: perception produces object features, concepts classify those features, and programs combine the resulting predictions.

Figure 1 illustrates this interface. The contribution is joint learning of concepts and question interpretation from answers, combined with explicit execution. It does not remove all supervision or hand-designed knowledge: object proposals and the reasoning language are supplied. The study tests data efficiency, controlled forms of generalization, new operations, and preliminary transfer to natural images.

Complete original figure with every panel, label and legend.
Figure 1: Humans learn visual concepts, words, and semantic parsing jointly and incrementally. I. Learning visual concepts (red vs. green) starts from looking at simple scenes, reading simple questions, and reasoning over contrastive examples (Fazly et al., 2010). II. Afterwards, we can interpret referential expressions based on the learned object-based concepts, and learn relational concepts (e.g., on the right of, the same material as). III Finally, we can interpret complex questions from visual cues by exploiting the compositional structure.

2 Related Work

Earlier visual reasoning systems use neural attention, learned neural modules, or symbolic programs over supervised scene descriptions. NS-CL combines learned object-centric concept representations with predefined symbolic operations and learns the parser without ground-truth program annotations. Table 1 compares the required supervision and model components.

Related concept-learning approaches use human attributes, language, or structured scene descriptions. Here, question-answer loss trains concept grounding while the program language constrains how concepts can be combined. The explicit representation favors compositional reuse, at the cost of choosing an ontology and executor in advance.

Models Visual Features Semantics Extra Labels Inference
# Prog. Attr.
FiLM (Perez et al., 2018) Convolutional Implicit 0 No Feature Manipulation
IEP (Johnson et al., 2017b) Convolutional Explicit 700K No Feature Manipulation
MAC (Hudson & Manning, 2018) Attentional Implicit 0 No Feature Manipulation
Stack-NMN (Hu et al., 2018) Attentional Implicit 0 No Attention Manipulation
TbD (Mascharka et al., 2018) Attentional Explicit 700K No Attention Manipulation
NS-VQA (Yi et al., 2018) Object-Based Explicit 0.2K Yes Symbolic Execution
NS-CL Object-Based Explicit 0 No Symbolic Execution
Table 1: Comparison with other frameworks on the CLEVR VQA dataset, w.r.t. visual features, implicit or explicit semantics and supervisions.

3 Neuro-Symbolic concept learner

Figure 2 separates visual perception, concept quantization, semantic parsing, and quasi-symbolic execution. Training connects the visual and language components through the probability of the correct answer.

Refer to caption
Figure 2: We propose to use neural symbolic reasoning as a bridge to jointly learn visual concepts, words, and semantic parsing of sentences.
Refer to caption
Figure 3: We treat attributes such as Shape and Color as neural operators. The operators map object representations into a visual-semantic space. We use similarity-based metric to classify objects.

3.1 Model details

Visual perception.

Mask R-CNN proposes objects. A ResNet-34 backbone and RoIAlign extract each object’s features, concatenated with full-scene context so relative size and spatial relations remain accessible. Concept learning uses these continuous object representations rather than supervised discrete scene attributes.

The proposal model is trained on 4,000 CLEVR images with bounding-box annotations, and the backbone is initialized from ImageNet. Consequently, the absence of attribute and program labels should not be read as absence of visual supervision.

Concept quantization.

Each attribute has a learned mapping into a visual-semantic space; each concept has a learned embedding and an attribute association. A sigmoid of scaled cosine similarity gives concept-membership probabilities. Pairwise features similarly support relations. Figure 3 illustrates how visual and conceptual representations align; Appendix C retains the precise formulas.

DSL and semantic parsing.

A bidirectional GRU encodes the question, and a recursive decoder predicts a tree of operations and their concept arguments. The operations and types come from a predefined domain-specific language (DSL). Hand-coded rules identify concept words, while abstract concept-type tokens help the parser reuse structures with different words. The complete type system, algorithm, and parsing example remain in Appendices A and B.

Quasi-symbolic program execution.

Programs operate on probabilistic object sets rather than prematurely discretized scene labels. A set is a vector of memberships; a single-object representation sums to one. Fixed operators implement filtering, relations, intersections, counting, and queries, passing gradients to visual representations and concept parameters. Figure 4 shows an example.

The executor itself is designed rather than learned. Explicit intermediate sets aid inspection, but erroneous perception or an answer-consistent wrong program can still yield a misleading trace.

Refer to caption
Figure 4: A. Demonstration of the curriculum learning of visual concepts, words, and semantic parsing of sentences by watching images and reading paired questions and answers. Scenes and questions of different complexities are illustrated to the learner in an incremental manner. B. Illustration of our neuro-symbolic inference model for VQA. The perception module begins with parsing visual scenes into object-based deep representations, while the semantic parser parse sentences into executable programs. A symbolic execution process bridges two modules.

3.2 Training paradigm

Optimization objective.

The objective maximizes the probability of the correct answer, averaging over programs proposed by the semantic parser. Visual parameters receive gradients through execution; discrete program choices use REINFORCE or an off-policy search over answer-consistent programs. Equation 1 retains the objective. Appendix D explains the search and its treatment of spurious programs.

Equation 1
Θv,Θs←argmaxΘv,Θs𝔼P[Pr[A=Executor(Perception(S;Θv),P)]],\Theta_{v},\Theta_{s}\leftarrow\arg\max_{\Theta_{v},\Theta_{s}}\mathbb{E}_{P}[\Pr[A=\mathrm{Executor}(\mathrm{Perception}(S;{\Theta_{v}}),P)]], (1)

Curriculum visual concept learning.

The curriculum first learns simple object concepts in small scenes, then relations, then complex programs with perception frozen; final joint fine-tuning unfreezes the components. Appendix E describes these as three lessons followed by joint tuning. Scene size and question complexity increase gradually, using program-depth information to select training questions.

In the reported ablations, removing the scene-complexity curriculum makes concept recognition nearly random, and removing the question-complexity curriculum prevents successful joint training. The learning procedure therefore relies materially on this structured progression.

4 Experiments

The main CLEVR setting uses 5,000 training images, under 10% of the full 70,000, with question selection following the curriculum. Experiments separately examine grounded concepts, question answering, controlled generalization, and extensions beyond the original task.

4.1 Visual concept learning

Classification-based concept evaluation.

The authors evaluate concept classification on the CLEVR validation split; attribute labels do not supervise the learned concept classifier. Object-attribute classification is near 99%; learned relations are less accurate because their supervision is indirect. These tests inspect grounding beyond aggregate question-answer accuracy.

Count-based concept evaluation.

Count questions probe whether a system consistently identifies every object matching a concept. Table 2 reports a mean of 98.7 for NS-CL, versus 90.6 for IEP, 95.9 for MAC, and 96.5 for TbD under this diagnostic. Counts can reveal grounding errors hidden by simpler answer categories, though they still combine perception and execution.

Visual Mean Color Mat. Shape Size
IEP Conv. 90.6 91.0 90.0 89.9 90.6
MAC Attn. 95.9 98.0 91.4 94.4 94.2
TbD (hres.) Attn. 96.5 96.6 92.2 95.4 92.6
NS-CL Obj. 98.7 99.0 98.7 98.1 99.1

Table 2: We also evaluate the learned visual concepts using a diagnostic question set containing simple questions such as “How many red objects are there?”. NS-CL outperforms both convolutional and attentional baselines. The suggested object-based visual representation and symbolic reasoning approach perceives better interpretation of visual concepts.
Model Visual Accuracy (100%100\% Data) Accuracy (10%10\% Data)
TbD Attn. 99.1 54.2
TbD-Object Obj. 84.1 52.6
TbD-Mask Attn. 99.0 55.0
MAC Attn. 98.9 67.3
MAC-Object Obj. 79.5 51.2
MAC-Mask Attn. 98.7 68.4
NS-CL Obj. 99.2 98.9

Table 3: We compare different variants of baselines for a systematic study on visual features and data efficiency. Using only 10% of the training images, our model is able to achieve a comparable results with the baselines trained on the full dataset. See the text for details.

4.2 Data-efficient and interpretable visual reasoning

With the full CLEVR training set, NS-CL reaches 99.2% accuracy; with about 10%, it reaches 98.9%. In the same data-reduction comparison, MAC falls from 98.9 to 67.3 and TbD from 99.1 to 54.2. Table 3 also tests object-based and mask-guided baseline variants; object features alone do not reproduce the proposed system’s data efficiency.

Table 4 provides question-type results and supervision requirements. NS-CL’s 98.9 overall ties MAC in that comparison and is slightly below TbD’s 99.1, which uses program annotations. The main advantage is strong performance with fewer examples and inspectable execution, rather than a uniform accuracy win. Appendix F gives the baseline construction and ablations.

Model Prog. Anno. Overall Count Cmp. Num. Exist Query Attr. Cmp. Attr.
Human N/A 92.6 86.7 86.4 96.6 95.0 96.0
NMN 700K 72.1 52.5 72.7 79.3 79.0 78.0
N2NMN 700K 88.8 68.5 84.9 85.7 90.0 88.8
IEP 700K 96.9 92.7 98.7 97.1 98.1 98.9
DDRprog 700K 98.3 96.5 98.4 98.8 99.1 99.0
TbD 700K 99.1 97.6 99.4 99.2 99.5 99.6
RN 0 95.5 90.1 93.6 97.8 97.1 97.9
FiLM 0 97.6 94.5 93.8 99.2 99.2 99.0
MAC 0 98.9 97.2 99.4 99.5 99.3 99.5
NS-CL 0 98.9 98.2 99.0 98.8 99.3 99.1

Table 4: Our model outperforms all baselines using no program annotations. It achieves comparable results with models trained by full program annotations such as TbD.
Model Test
Split A Split B Split C Split D
MAC 97.3 N/A 92.9 N/A
IEP 96.1 92.1 91.5 90.9
TbD 98.8 94.5 94.3 91.9
NS-CL 98.9 98.9 98.7 98.8
Figure 5: We test the combinatorial generalization w.r.t. the number of objects in scenes and the complexity of questions (i.e. the depth of the program trees). We makes four split of the data containing various complexities of scenes and questions. Our object-based visual representation and explicit program semantics enjoys the best (and almost-perfect) combinatorial generalization compared with strong baselines.

4.3 Generalization to new attributes and compositions

Generalizing to new visual compositions.

CLEVR-CoGenT changes which colors co-occur with particular shapes. Ordinary joint training can entangle shape and color. The successful experiment freezes pretrained attribute operators and learns new concept embeddings and the parser on condition A, obtaining 98.8% on A and 98.9% on B.

This supports reuse of a suitably disentangled representation. It does not show that unrestricted joint learning automatically discovers composition-independent attributes from the biased condition-A data.

Generalizing to new visual concepts.

To add the previously withheld color purple, the model is fine-tuned with 100 images containing purple objects. It reaches 93.9% question-answer accuracy, exceeding the compared IEP and TbD settings by 4.6 and 6.1 percentage points. The parser was pretrained on the full CLEVR question set, so the experiment tests adaptation of visual concepts with an established linguistic structure.

Complete original figure with every panel, label and legend.
Figure 6: Samples collected from four splits in Section 4.3 for illustration. Models are trained on split A but evaluated on all splits for testing the combinatorial generalization.

4.4 Combinatorial generalization to new scenes and questions

Training scenes contain fewer than six objects and questions have program depth below five. Tests increase scene size, program depth, or both; Figure 5 reports 98.9, 98.9, 98.7, and 98.8% across the four splits. Figure 6 shows an example.

The semantic parser is pretrained on the full dataset and fixed, and counting questions with answers above six are excluded. These controls isolate generalization of concept grounding and execution; they do not establish unrestricted extrapolation of the complete learned system.

4.5 Extending to other program domain

The model is adapted to a restricted image-caption retrieval task using a new program language for relational statements. Visual components are fixed while a new semantic parser is learned. Table 5 retains the retrieval examples and comparison results.

On the comparable subset, NS-CL scores 96.9%, versus 97.0 for TbD and 95.5 for IEP. On the full constructed task, it scores 97.0 versus 68.9 for CNN-LSTM. This demonstrates reuse of grounded concepts with another executor/parser, under templated caption and task assumptions.

Refer to caption
(a) An illustrative pair of image and caption in our synthetic dataset.
Model Retrieval Accuracy
IEP 95.5
TbD 97.0
NS-CL 96.9
(b) Image-caption retrieval accuracy on a subset of data. Our model archives comparable results with VQA baselines.
Model Retrieval Accuracy
CNN-LSTM 68.9
NS-CL 97.0
(c) Image-caption retrieval accuracy on the full dataset. Our model outperforms baselines and requires no extra training or fine-tuning of the visual perception module.
Table 5: We introduce a new simple DSL for image-caption retrieval to evaluate how well the learned visual concepts transfer. Due to the difference between VQA and caption retrieval, VQA baselines are only able to infer the result on a partial set of data. The learned object-based visual concepts can be directly transferred into the new domain for free.

4.6 Extending to natural images and language

VQS supplies natural images and questions grounded in annotated regions. This extension uses pretrained object proposals and a pretrained dependency parser to construct programs, together with word embeddings. Each question has 18 candidate answers. Figures 7 and 8 show examples and learned concepts.

NS-CL achieves 44.3% accuracy, compared with 43.9 for the MLP baseline and 46.2 for MAC. The result demonstrates an inspectable pipeline in a more difficult domain, but remains below the strongest compared baseline. Appendix G details the additional pretrained components and restricted evaluation.

Refer to caption
Model Accuracy
MLP 43.9
MAC 46.2
NS-CL 44.3
Figure 7: Left: An example image-question pair from the VQS dataset and the corresponding execution trace of NS-CL. Right: Results on the VQS test set. Our model achieves a comparable results with the baselines.
Refer to caption
Figure 8: Concepts learned from VQS, including object categories, attributes, and relations.

5 Discussion and Future Work

Explicit object representations and programs can make grounded concept learning efficient and reusable within a chosen language of operations. The controlled experiments support this combination, while also exposing dependence on proposals, curriculum, predefined concepts, and component pretraining.

Broader natural-language understanding, automatic discovery of concept words and operations, dynamic scenes, actions, and richer three-dimensional representations remain open directions. Transparent traces help diagnose mistakes; they do not guarantee correct grounding or reasoning.

Acknowledgements.

The authors acknowledge collaborators, institutional support, and research funding listed in the original paper.

Appendix A CLEVR domain-specific language and implementations

Tables 6 and 7 retain every CLEVR operation, signature, semantic description, and type. Operations requiring one object implicitly cast a set through the Unique operation. The intended reference must be nonempty and unambiguous.

The original table describes Exist as asking whether a set is empty, while the implementation in Table 9 returns its maximum membership, corresponding to existence. Both source tables are retained without silently rewriting this wording.

Operation Signature Semantics
Scene () ⟶\longrightarrow ObjectSet Return all objects in the scene.
Filter (ObjectSet, ObjConcept) ⟶\longrightarrow ObjectSet Filter out a set of objects having the object-level concept (e.g., red) from the input object set.
Relate (Object, RelConcept) ⟶\longrightarrow ObjectSet Filter out a set of objects that have the relational concept (e.g., left) with the input object.
AERelate (Object, Attribute) ⟶\longrightarrow ObjectSet (Attribute-Equality Relate) Filter out a set of objects that have the same attribute value (e.g., same color) as the input object.
Intersection (ObjectSet, ObjectSet) ⟶\longrightarrow ObjectSet Return the intersection of two object sets.
Union (ObjectSet, ObjectSet) ⟶\longrightarrow ObjectSet Return the union of two object sets.
Query (Object, Attribute) ⟶\longrightarrow ObjConcept Query the attribute (e.g., color) of the input object.
AEQuery (Object, Object, Attribute) ⟶\longrightarrow Bool (Attribute-Equality Query) Query if two input objects have the same attribute value (e.g., same color).
Exist (ObjectSet) ⟶\longrightarrow Bool Query if the set is empty.
Count (ObjectSet) ⟶\longrightarrow Integer Query the number of objects in the input set.
CLessThan (ObjectSet, ObjectSet) ⟶\longrightarrow Bool (Counting LessThan) Query if the number of objects in the first input set is less than the one of the second set.
CGreaterThan (ObjectSet, ObjectSet) ⟶\longrightarrow Bool (Counting GreaterThan) Query if the number of objects in the first input set is greater than the one of the second set.
CEqual (ObjectSet, ObjectSet) ⟶\longrightarrow Bool (Counting Equal) Query if the number of objects in the first input set is the same as the one of the second set.
Table 6: All operations in the domain-specific language for CLEVR VQA.
Type Example Semantics
ObjConcept Red, Cube, etc. Object-level concepts.
Attribute Color, Shape, etc. Object-level attributes.
RelConcept Left, Front, etc. Relational concepts.
Object A single object in the scene.
ObjectSet {      ,      }\{\hbox to4.71pt{\vbox to4.71pt{\pgfpicture\makeatletter\hbox{\hskip 2.35277pt\lower-2.35277pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {}{{}}{}{{{}}{\lx@inpgf@ignorespaces}{}{\lx@inpgf@ignorespaces}{}{}{}{}{}}\lxSVG@begingroup@{_scopebegin=1} \color[rgb]{1,0,0}\lxSVG@begingroup@{fill=#FF0000} {}\lxSVG@fillstroke\lxSVG@drawpath@unclipped{M 0 0 M 2.98 0 C 2.98 1.65 1.65 2.98 0 2.98 C -1.65 2.98 -2.98 1.65 -2.98 0 C -2.98 -1.65 -1.65 -2.98 0 -2.98 C 1.65 -2.98 2.98 -1.65 2.98 0 Z M 0 0}{} \lx@inpgf@ignorespaces \lxSVG@closescope \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}},\hbox to4.71pt{\vbox to4.71pt{\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {}{{}}{} {\lx@inpgf@ignorespaces}{{}}{}{}{}{}{{}}{}\lxSVG@begingroup@{_scopebegin=1} \color[rgb]{0,0,1}\lxSVG@begingroup@{fill=#0000FF} {}\lxSVG@fillstroke\lxSVG@drawpath@unclipped{M 0 0 M 0 0 L 0 5.96 L 5.96 5.96 L 5.96 0 Z M 5.96 5.96}{} \lx@inpgf@ignorespaces \lxSVG@closescope \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}\} A set of objects in the scene.
Integer 0,1,2,⋯0,1,2,\cdots A single integer.
Bool True, False A single boolean value.
Table 7: The type system of the domain-specific language for CLEVR VQA.

Appendix B Semantic parsing

The string-to-tree parser combines a bidirectional GRU encoder, an operation decoder, a concept decoder, and output encoders that recursively generate child states. Algorithm 1 retains the complete procedure.

The concept inventory is known for CLEVR, and hand-coded rules extract concept words. Replacing specific words with indexed concept-type tokens lets the network predict reusable structures and then restore the arguments. Concept embeddings used by the parser are separate from the visual-semantic embeddings used to recognize objects. Natural-language extensions instead rely on linguistic tagging or parsing tools.

Algorithm 1 The String-to-Tree Semantic Parser.
Function parse(ff, {ci}\{c_{i}\}):
   p​r​o​g​r​a​m←EmptyProgram​()program\leftarrow\texttt{EmptyProgram}();
   p​r​o​g​r​a​m.o​p←OpDecoder​(f)program.op\leftarrow\texttt{OpDecoder}(f);
   if p​r​o​g​r​a​m.o​pprogram.op requires a concept parameter then
      p​r​o​g​r​a​m.c​o​n​c​e​p​t←ConceptDecoder​(f,{ci})program.concept\leftarrow\texttt{ConceptDecoder}(f,\{c_{i}\});
   for i=0,1,⋯i=0,1,\cdotsnumber of non-concept inputs of p​r​o​g​r​a​m.o​pprogram.op do
      p​r​o​g​r​a​m.i​n​p​u​t​[i]←program.input[i]\leftarrow parse ( OEncoderi(ff, p​r​o​g​r​a​m.o​pprogram.op) ,{ci}\{c_{i}\} );
   return p​r​o​g​r​a​mprogram

A Running Example

The running question asks about an object identified by attributes and a spatial relation. Table 8 follows the recursive generation of Query, Filter, Relate, Filter, and Scene, showing the latent state and selected concept at each step. This makes the interface between question structure and executable program explicit.

Step Inputs Outputs Recursive Invocation
1 f0f_{0} OpDecoder​(f0)→Query\texttt{OpDecoder}(f_{0})\rightarrow\texttt{Query};
ConceptDecoder​(f0)→<Attribute 1>\texttt{ConceptDecoder}(f_{0})\rightarrow<\textnormal{Attribute 1}>;
OEncoder0​(f0,Query)→f1\texttt{OEncoder}_{0}(f_{0},\texttt{Query})\rightarrow f_{1}
p​a​r​s​e​(f1)parse(f_{1})
2 f1f_{1} OpDecoder​(f1)→Filter\texttt{OpDecoder}(f_{1})\rightarrow\texttt{Filter};
ConceptDecoder​(f1)→<ObjConcept 1>\texttt{ConceptDecoder}(f_{1})\rightarrow<\textnormal{ObjConcept 1}>;
OEncoder0​(f1,Filter)→f2\texttt{OEncoder}_{0}(f_{1},\texttt{Filter})\rightarrow f_{2}
p​a​r​s​e​(f2)parse(f_{2})
3 f2f_{2} OpDecoder​(f2)→Relate\texttt{OpDecoder}(f_{2})\rightarrow\texttt{Relate};
ConceptDecoder​(f2)→<RelConcept 1>\texttt{ConceptDecoder}(f_{2})\rightarrow<\textnormal{RelConcept 1}>;
OEncoder0​(f2,Relate)→f3\texttt{OEncoder}_{0}(f_{2},\texttt{Relate})\rightarrow f_{3}
p​a​r​s​e​(f3)parse(f_{3})
4 f3f_{3} OpDecoder​(f3)→Filter\texttt{OpDecoder}(f_{3})\rightarrow\texttt{Filter};
ConceptDecoder​(f3)→<ObjConcept 2>\texttt{ConceptDecoder}(f_{3})\rightarrow<\textnormal{ObjConcept 2}>;
OEncoder0​(f3,Filter)→f4\texttt{OEncoder}_{0}(f_{3},\texttt{Filter})\rightarrow f_{4}
p​a​r​s​e​(f4)parse(f_{4})
5 f4f_{4} OpDecoder​(f3)→Scene\texttt{OpDecoder}(f_{3})\rightarrow\texttt{Scene}; (End of branch.)
Table 8: A step-by-step running example of the recursive parsing procedure. The parameter {ci}\{c_{i}\} is omitted for better visualization.

Appendix C Program Execution

The executor treats object memberships as continuous values, allowing question-answer loss to train perception and concepts through fixed operations.

Object-typed and ObjectSet-typed variables.

An ObjectSet contains one membership probability per detected object. An Object is a distribution over those objects. Casting a set into an object applies a softmax to the logits of its memberships; the intended interpretation is a unique referent. Empty or ambiguous references can be detected during symbolic execution.

Concept quantization.

Object concepts have embedding vectors and normalized associations with attributes. Attribute-specific visual mappings and scaled cosine similarity yield concept probabilities. Relations use analogous pairwise classification, and equality of an attribute compares two mapped object representations. The original equations retain the scaling constants and aggregation over attribute associations.

Equations
Pr⁡[object i is Red]=σ⁡(∑a∈A(baRed⋅⟨ua​(oi),vRed⟩−γτ)),\Pr[\text{object $i$ is {Red}}]=\sigma\left(~\sum_{a\in A}\left(b^{\texttt{Red}}_{a}\cdot\frac{{~\langle u^{a}(o_{i}),v_{\texttt{Red}}\rangle-\gamma}~}{\tau}\right)\right),
Pr⁡[object i has the same Color as object j]=σ⁡(⟨uColor​(oi),uColor​(oj)⟩−γτ),\Pr[\text{object $i$ has the same {Color} as object $j$}]=\sigma\left(\frac{{~\langle u^{\texttt{Color}}(o_{i}),u^{\texttt{Color}}(o_{j})\rangle-\gamma}~}{\tau}\right),

Quasi-symbolic program execution.

Table 9 preserves the full implementations of filtering, relations, set operations, queries, existence, counting, and count comparisons. Probabilities are stored in log space for stability. At inference, the operations can be quantized as in prior symbolic execution. These fixed formulas define the reasoning semantics rather than learning each operation from scratch.

Signature Implementation
Scene() →\rightarrow o​u​tout: ObjectSet o​u​ti:=1out_{i}:=1
Filter(i​nin: ObjectSet, o​coc: ObjConcept) →\rightarrow
   o​u​tout: ObjectSet
o​u​ti:=min⁡(i​ni,ObjClassify​(o​c)i)out_{i}:=\min(in_{i},\mathrm{ObjClassify}(oc)_{i})
Relate(i​nin: Object, r​crc: RelConcept) →\rightarrow
   o​u​tout: ObjectSet
o​u​ti:=∑j(i​nj⋅RelClassify​(r​c)j,i))out_{i}:=\sum_{j}(in_{j}\cdot\mathrm{RelClassify}(rc)_{j,i}))
AERelate(i​nin: Object, aa: Attribute) →\rightarrow
   o​u​tout: ObjectSet
o​u​ti:=∑j(i​nj⋅AEClassify​(a)j,i))out_{i}:=\sum_{j}(in_{j}\cdot\mathrm{AEClassify}(a)_{j,i}))
Intersection(i​n(1)in^{(1)}: ObjectSet,
   i​n(2)in^{(2)}: ObjectSet) →\rightarrow o​u​tout: ObjectSet
o​u​ti:=min⁡(i​ni(1),i​ni(2))out_{i}:=\min(in^{(1)}_{i},in^{(2)}_{i})
Union(i​n(1)in^{(1)}: ObjectSet, i​n(2)in^{(2)}: ObjectSet) →\rightarrow
   o​u​tout: ObjectSet
o​u​ti:=max⁡(i​ni(1),i​ni(2))out_{i}:=\max(in^{(1)}_{i},in^{(2)}_{i})
Query(i​nin: Object, aa: Attribute) →\rightarrow
     o​u​tout: ObjConcept
Pr[out=oc]:=∑iini⋅ObjClassify​(o​c)i⋅bao​c∑o​c′ObjClassify​(o​c′)i⋅bao​c′\Pr[out=oc]:=\sum_{i}in_{i}\cdot\displaystyle\frac{\mathrm{ObjClassify}(oc)_{i}\cdot b^{oc}_{a}}{\sum_{oc^{\prime}}\mathrm{ObjClassify}(oc^{\prime})_{i}\cdot b^{oc^{\prime}}_{a}}
AEQuery(i​n(1)in^{(1)}: Object, i​n(2)in^{(2)}: Object,
    aa: Attribute) →\rightarrow bb: Bool
b:=∑i∑j(i​ni(1)⋅i​nj(2)⋅AEClassify​(a)j,i))b:=\sum_{i}\sum_{j}(in^{(1)}_{i}\cdot in^{(2)}_{j}\cdot\mathrm{AEClassify}(a)_{j,i}))
Exist(i​nin: ObjectSet) →\rightarrow bb: Bool b:=maxi⁡i​nib:=\max_{i}in_{i}
Count(i​nin: ObjectSet) →\rightarrow ii: Integer i:=∑ii​nii:=\sum_{i}in_{i}
CLessThan(i​n(1)in^{(1)}: ObjectSet,
    i​n(2)in^{(2)}: ObjectSet) →\rightarrow bb: Bool
b:=σ⁡((∑ii​ni(2)−∑ii​ni(1)−1+γc)/τc)b:=\sigma\big((\sum_{i}in^{(2)}_{i}-\sum_{i}in^{(1)}_{i}-1+\gamma_{c})/\tau_{c}\big)
CGreaterThan(i​n(1)in^{(1)}: ObjectSet,
    i​n(2)in^{(2)}: ObjectSet) →\rightarrow bb: Bool
b:=σ⁡((∑ii​ni(1)−∑ii​ni(2)−1+γc)/τc)b:=\sigma\big((\sum_{i}in^{(1)}_{i}-\sum_{i}in^{(2)}_{i}-1+\gamma_{c})/\tau_{c}\big)
CEqual(i​n(1)in^{(1)}: ObjectSet,
    i​n(2)in^{(2)}: ObjectSet) →\rightarrow bb: Bool
b:=σ⁡((−|∑ii​ni(1)−∑ii​ni(2)|+γc)/(γc⋅τc))b:=\sigma\big((-|\sum_{i}in^{(1)}_{i}-\sum_{i}in^{(2)}_{i}|+\gamma_{c})/(\gamma_{c}\cdot\tau_{c})\big)
Table 9: All operations in the domain-specific language for CLEVR VQA. γc=0.5\gamma_{c}=0.5 and τc=0.25\tau_{c}=0.25 are constants for scaling and shift the probability. During inference, one can quantify all operations as Yi et al. (2018).

Appendix D Optimization of the Semantic Parser

REINFORCE estimates the parser gradient by sampling programs. An alternative freezes perception, enumerates candidate programs, and records those producing the correct answer. This off-policy set supports an exact sum over answer-consistent programs in the stated language, avoiding repeated Monte Carlo sampling.

The resulting programs are inferred supervision, not annotated ground truth. Search tractability depends on the restricted operation vocabulary, program constraints, and curriculum stage.

Equation
∇Θs=∇Θs𝔼P​[r⋅log⁡Pr⁡[P]],\nabla_{\Theta_{s}}=\nabla_{\Theta_{s}}\mathbb{E}_{P}[r\cdot\log\Pr[P]],

Spurious program suppression.

Different programs may be semantically equivalent, or may agree only accidentally on the current scene. Training all answer-consistent programs uniformly can reinforce those accidental explanations. The proposed loss weights each program’s negative log probability by its current probability, with the weight detached from gradient computation.

The retained equations give the loss and gradient, and the original program examples illustrate equivalence and spurious agreement. Diverse scenes help suppress accidental programs, but correct answers alone cannot certify a correct interpretation.

Equations
P1: AEQuery​(Color,Filter​(Cube),Filter​(Sphere))​ and\displaystyle\texttt{AEQuery}(\texttt{Color},\texttt{Filter}(\texttt{Cube}),\texttt{Filter}(\texttt{Sphere}))\text{~and}
P2: Exist​(Filter​(Sphere,AERelate​(Color,Filter​(Cube))))\displaystyle\texttt{Exist}(\texttt{Filter}(\texttt{Sphere},\texttt{AERelate}(\texttt{Color},\texttt{Filter}(\texttt{Cube}))))
P1: Filter​(Red,Relate​(Left,Filter​(Sphere)))​ and\displaystyle\texttt{Filter}(\texttt{Red},\texttt{Relate}(\texttt{Left},\texttt{Filter}(\texttt{Sphere})))\text{~and}
P2: Filter​(Red,Relate​(Left,Filter​(Cube)))\displaystyle\texttt{Filter}(\texttt{Red},\texttt{Relate}(\texttt{Left},\texttt{Filter}(\texttt{Cube})))
ℓ=∑p∈ℚstop_gradient​(Pr⁡[p])⋅(−log⁡Pr⁡[p]).\ell=\sum_{p\in\mathbb{Q}}\text{stop\_gradient}(\Pr[p])\cdot(-\log\Pr[p]).
∇Θs=∑p∈ℚPr⁡[p]⋅∇Θs(r⋅log⁡Pr⁡[P])=∇Θs(∑p∈ℚr⋅Pr⁡[p]).\nabla_{\Theta_{s}}=\sum_{p\in\mathbb{Q}}\Pr[p]\cdot\nabla_{\Theta_{s}}\left(r\cdot\log\Pr[P]\right)=\nabla_{\Theta_{s}}\left(\sum_{p\in\mathbb{Q}}r\cdot\Pr[p]\right).

Appendix E Curriculum Learning Setup

The curriculum starts with single-object attributes in small scenes, adds relations and compositional questions, then learns more complex programs with perception fixed. Scenes grow from three to ten objects. Question depth and permitted templates control each lesson; an additional count-after-relation template helps train relational concepts. The final stage jointly fine-tunes the full model.

These choices supply task structure beyond raw question-answer examples. The reported failures when removing scene-size or question-complexity progression show why the curriculum is part of the method rather than a minor optimization detail.

Appendix F Ablation study

F.1 Semantic parsing accuracy.

The reported parser accuracy exceeds 99.9% when its programs execute over ground-truth scenes. This is answer-equivalence evaluation, not exact matching of predicted and annotated program strings, so alternative correct or accidentally equivalent programs can count as successful.

F.2 Impacts of the ImageNet pre-training.

Removing ImageNet pretraining reduces overall question-answer accuracy from 98.9 to 98.2% and shape recognition from 98.7 to 97.5%. These explicit appendix values imply a 0.7-point answer-accuracy drop; the main-text description gives 0.5. The qualitative conclusion is a modest but measurable benefit in this synthetic setting.

F.3 Data Efficiency and Object-based Representations

This ablation asks whether object representations alone explain the gains under limited training data.

Baselines.

Object-based MAC and TbD variants attend over object features, while mask-guided variants retain spatial maps but incorporate proposal masks. Their architectures adapt the original attention mechanisms to use the same object information. Table 3 retains the full-data and reduced-data results.

Results.

Object-based variants do not match NS-CL’s data efficiency. The authors argue that permutation-invariant object attention is poorly suited to quantities such as counts, while mask guidance accelerates convergence without substantially improving final accuracy. NS-CL instead supplies counting and other reasoning operations explicitly.

For the full-data experiment, 3,500 training images form a hyperparameter-selection holdout, and the official validation split is used as the reported test set. The resulting NS-CL accuracy is 99.2%.

Appendix G Extending to Other Scene and Language Domains

G.1 Minecraft Dataset

Minecraft tests different visual appearances and additional reasoning operations. Figure 9 gives a scene, question, answer, and recovered program.

Refer to caption
Figure 9: An example image and a related question-answering pair from the Minecraft dataset.

Setup.

The generated dataset contains 10,000 scenes, each with three to six objects drawn from twelve categories and four facing directions. Two operations extend the language: FilterMost selects an extremal object, such as the closest, and BelongTo tests membership in a set.

Results.

Table 10 reports 93.3% overall accuracy for NS-CL and 87.7% for the supervised NS-VQA comparison. The authors attribute part of the baseline deficit to noisy scene-label supervision caused by occlusion and proposal-to-box matching. This is a result under that rendering and training setup; it does not establish that weaker supervision is inherently superior. Figure 12 retains the execution traces and a missed-object failure.

Model Overall Count Exist Belong Query
NS-VQA 87.7 83.3 91.5 91.1 86.4
NS-CL 93.3 91.3 95.6 93.9 94.3
Table 10: Our model achieves comparable results on the Minecraft dataset with baselines trained by full program annotations.

G.2 VQS Dataset

VQS is a subset of VQA 1.0 with region annotations identifying evidence relevant to each question. Figure 10 shows the proposal and program interface.

Refer to caption
Figure 10: An example image from the VQS dataset. The orange bounding boxes are object proposals. On the right, we show the original question and answer in natural language, as well as the latent program recovered by our parser. To answer this question, models are expected to attend to the man and his pen in the pocket.

Setup.

Training uses 63,509 images, with another 5,000 training-split images held out for model selection, and evaluation on the designated test split. Answers are selected from 18 candidates per question. A pretrained dependency parser extracts latent programs and concepts; bag-of-words or GloVe representations initialize concepts. These supplied language tools differ from the question-answer-trained parser used in CLEVR.

Baselines.

The MLP baseline scores image-question-answer triples using an ImageNet-pretrained ResNet-34 and word features. MAC receives each candidate answer concatenated with the question and predicts its score, using the same visual backbone. This compares candidate ranking in the specified multiple-choice task.

Results.

The 44.3% NS-CL result is close to the 43.9% MLP baseline and below MAC’s 46.2%. Its distinctive benefit here is the visible execution trace. Figure 13 also reveals limits: a correct answer can be grounded in the wrong object, and visual misclassification can propagate through otherwise understandable steps.

Appendix H Visualization of execution traces and visual concepts

Figures 11–15 preserve all qualitative traces and concept visualizations. CLEVR examples show successful composition, material misclassification, and ambiguous references. Minecraft examples show extremal-object reasoning and a missed pig behind a tree. Natural-image examples include both successful answers and incorrect grounding or action classification.

These cases illustrate what the intermediate representation makes inspectable. In particular, the correct “cake” answer grounded on a dish shows that answer accuracy and faithful localization are distinct. The final figures display learned single concepts and compositions in CLEVR and Minecraft.

Refer to caption
Figure 11: Visualization of the execution trace generated by our Neuro-Symbolic Concept Learner on the CLEVR dataset. Example A and B are successful executions that generate correct answers. In example C, the execution aborts at the first operator. To inspect the reason why the execution engine fails to find the corresponding object, we can read out the visual representation of the object, and locate the error source as the misclassification of the object material. Example D shows how our symbolic execution engine can detect invalid or ambiguous programs during the execution by performing sanity checks.
Complete original figure with every panel, label and legend.
Figure 12: Exemplar execution trace generated by our Neuro-Symbolic Concept Learner on the Minecraft reasoning dataset. Example A, B and C are successful execution. Example C demonstrates the semantics of the FilterMost operation. Example D shows a failure case: the detection model fails to detect a pig hiding behind the big tree.
Refer to caption
Figure 13: Illustrative execution trace generated by our Neuro-Symbolic Concept Learner on the VQS dataset. Execution traces A and B shown in the figure leads to the correct answer to the question. Our model effectively learns visual concepts from data. The symbolic reasoning process brings transparent execution trace and can easily handle quantities (e.g., object counting in Example A). In Example C, although NS-CL answers the question correctly, it locates the wrong object during reasoning: a dish instead of the cake. In Example D, our model misclassifies the sport as frisbee.
Refer to caption
Figure 14: Concepts learned on the CLEVR dataset.
Refer to caption
Figure 15: Concepts learned on the Minecraft dataset.
Concept demonstration

Open image ↗