Literature
Display
← All papers

FALCON: Fast Visual Concept Learning by Integrating Images, Linguistic Descriptions, and Conceptual Relations

Lingjie Mei · Jiayuan Mao · Ziqi Wang · Chuang Gan · Joshua B. Tenenbaum

· ICLR 2022 ↗

Reasoning12 min text · CondensedOriginal paper ↗
Contents

At a glance

FALCON learns a new visual concept from one or a few image examples, a sentence identifying the relevant object, and optional sentences relating the concept to known concepts. A pretrained language interface resolves the reference; a meta-learned module then predicts an axis-aligned box in a shared embedding space. The new concept can be reused in downstream visual questions.

The strongest results are on birds and synthetic scenes, where supplied conceptual relations reduce ambiguity. Improvements on natural-image GQA are modest. The system assumes pretrained perception and semantic parsing, generated language templates, and structured concept relations; it does not learn these components or discover the ontology from a single example.

1 Introduction

A single purple cylinder cannot reveal whether a new word names its color or shape. A description can identify the object, while a supplemental relation such as “is a color” constrains the intended property. Figure 1 illustrates this combination across CLEVR, birds, and natural images.

FALCON stores explicit concept embeddings and learns how to infer a new embedding from examples and related concepts. The main contribution is a sample-update-select inference procedure trained over many base-concept tasks. Its learned concepts support subsequent questions and can supply context for learning later concepts.

Refer to caption
Figure 1: Three illustrative examples of our fast concept learning task. Our model learns from three naturally occurring data streams: (a) looking at images, reading sentences that describe the objects in the scene, and (b) interpreting supplemental sentences that relate the novel concept with other concepts. (c) The acquired novel concepts (e.g., red and white-eyed vireo transfer to downstream tasks, such as visual reasoning. We present a meta-learning framework to solve this task.

2 Related Works

The approach extends neuro-symbolic concept learning and concept-relation learning to a meta-learning setting. It also draws on few-shot classifier generation and geometric embeddings. Boxes represent category extent and containment, while object features occupy very small regions in the same space. Figure 2 gives the complete model pipeline.

Refer to caption
Figure 2: Overview of FALCON-G. FALCON-G starts from extracting object-centric representations and parse input sentences into semantic programs. It executes the program to locate the objects being referred to. The example objects, together with the reconstructed concept graphs are fed into two GNNs sequentially to predict the concept embedding for the “white-eyed vireo”. The derived concept embedding can be used in downstream tasks such as visual reasoning.

3 FALCON

Each task supplies a novel concept, paired example images and descriptions, optional supplemental relations, and downstream image-question queries. Training concepts, validation concepts, and test concepts are separated. The system learns an inference procedure on base concepts, then applies it to previously unseen concept identities.

3.1 Visual Representations and Embedding Spaces

Mask R-CNN detects objects, and ResNet-34 extracts object features from the image and masks. Object-pair features support relational concepts in a separate embedding space. GQA uses the dataset’s supplied pretrained detector features.

A concept box is specified by its center and offsets; an object is represented by a tiny box. Box volumes and intersection volumes define conditional membership scores. Containment naturally expresses a narrow category belonging to a broader one. The original definitions below preserve the precise notation, and softplus smoothing makes the geometric scoring easier to optimize.

These geometric relations are an inductive bias. Predicted boxes are not guaranteed to satisfy every supplied relation, because the system learns to infer embeddings rather than enforcing all constraints exactly.

Equationsec=(Cen⁡(ec),Off⁡(ec))e_{c}=(\operatorname{Cen}(e_{c}),\operatorname{Off}(e_{c}))Min⁡(e)=Cen⁡(e)−Off⁡(e)\operatorname{Min}(e)=\operatorname{Cen}(e)-\operatorname{Off}(e)Max⁡(e)=Cen⁡(e)+Off⁡(e)\operatorname{Max}(e)=\operatorname{Cen}(e)+\operatorname{Off}(e)Pr⁡[e]=∏i(Max⁡(c)i−Min⁡(c)i),\Pr[e]=\prod_{i}(\operatorname{Max}(c)_{i}-\operatorname{Min}(c)_{i}),Pr⁡[e1∩e2]=∏imax⁡((M​(e1,e2)i−m​(e1,e2)i),0)\Pr[e_{1}\cap e_{2}]=\prod_{i}\operatorname{max}\left((M(e_{1},e_{2})_{i}-m(e_{1},e_{2})_{i}),0\right)M⁡(e1,e2)=Max⁡(e1)∨Max⁡(e2),m⁡(e1,e2)=Min⁡(e1)∧Min⁡(e2))M(e_{1},e_{2})=\operatorname{Max}(e_{1})\vee\operatorname{Max}(e_{2}),m(e_{1},e_{2})=\operatorname{Min}(e_{1})\wedge\operatorname{Min}(e_{2}))Pr⁡[e1|e2]=Pr⁡[e1∩e2]/Pr⁡[e2]\Pr[e_{1}|e_{2}]=\Pr[e_{1}\cap e_{2}]/\Pr[e_{2}]softplus⁡(x)=τ​log⁡(1+exp⁡(x/τ))\operatorname{softplus}(x)=\tau\log(1+\exp(x/\tau))

3.2 Neuro-Symbolic Reasoning

Pretrained semantic parsers turn descriptions and questions into typed programs and supplemental sentences into concept-relation tuples. A fixed differentiable executor uses known concepts to locate the example object, then uses the inferred new concept to answer later questions. Intermediate object sets are soft membership vectors.

This modular interface exposes grounding errors, but an incorrect reference can contaminate the new concept. Parser supervision and the restricted operation vocabulary are part of the setup.

3.3 Learning to Learn Concepts

FALCON samples candidate concept embeddings from a learned prior, updates them using related concepts and visual examples, then selects the candidate assigning the highest likelihood to the observed example objects. The retained equations give the updates and the training objective.

FALCON-G applies two graph networks in sequence: one over concept relations, one over example objects. FALCON-R processes relation/example edges recurrently in randomized order; Appendix A describes its unified implementation. Both receive downstream question-answer loss plus a prior-density term. Their learned updates replace a hand-written geometric constraint solver.

Equations
eci,0\displaystyle e_{c}^{i,0} ∼p⁡(θ),\displaystyle\sim p(\theta), eci,1=GNN1​(eci,0,Gc​o​n​c​e​p​t),\displaystyle e_{c}^{i,1}=\mathrm{GNN}_{1}(e_{c}^{i,0},G_{concept}),
eci,2\displaystyle e_{c}^{i,2} =GNN2​(eci,1,Ge​x​a​m​p​l​e),\displaystyle=\mathrm{GNN}_{2}(e_{c}^{i,1},G_{example}), ec=eck,2;k=arg⁡max⁡∏j=0Mi⁡Pr⁡[eci,2|oi(c)].\displaystyle e_{c}=e_{c}^{k,2};k=\arg\max_{i}\prod_{j=0}^{M}\Pr[e_{c}^{i,2}|o_{i}^{(c)}].
ℒm​e​t​a=ℒQ​A−λprior​log⁡pθ​(ec)\mathcal{L}_{meta}=\mathcal{L}_{QA}-\lambda_{\text{prior}}\log p_{\theta}(e_{c})

3.4 Training and Testing Paradigm

Pretraining learns visual representations and base-concept embeddings. Meta-training fixes base embeddings and learns the new-concept inference modules from tasks built around base concepts, selecting models on validation concepts. Meta-testing supplies examples and relations for held-out concepts and measures downstream answers. Appendix B retains the training schedule and the adaptations needed for comparison models.

4 Experiments

Experiments use procedurally generated descriptions and questions to control which concepts and relations are available.

4.1 Dataset

CUB’s 200 bird species are split into 100 base, fifty validation, and fifty test species. Adding higher taxonomic categories yields 211, 83, and 72 concepts. Relations include hypernyms and concepts sharing a hypernym.

CLEVR uses fifteen attribute concepts split into ten base, two validation, and three test concepts, with results averaged over four splits. GQA uses 105 frequent concepts split 89/6/10. Images may be natural, but the descriptions, supplemental sentences, and evaluation questions are generated from annotations and templates rather than unrestricted user language.

4.2 Baseline

CNN+LSTM and MAC process examples, descriptions, and test queries together. Concept-centric comparisons instead infer a reusable concept embedding and share the same parser/executor. NSCL+LSTM combines recurrent encodings; NSCL+GNN initializes from averaged examples and propagates through the relation graph. FALCON adds sampled initialization, learned example updates, and candidate selection. Figure 3 contrasts the designs.

The study compares box, hyperplane, and hypercone representations. Their original scoring formulas are retained. Appendix A clarifies that the one-example box baseline uses an average training-concept box size, despite a main-text sentence suggesting that this baseline cannot operate in that setting.

Original figure
Figure 3: Different models for predicting concept embedding ece_{c}. They differ in how they process DcD_{c}: supplemental sentences, Gc​o​n​c​e​p​tG_{concept}: concept graphs, and oi(c)o_{i}^{(c)}: objects being referred to.
EquationsPr⁡[eo|ec]=σ⁡(eoT​ec/τ−γ)\Pr[e_{o}|e_{c}]=\sigma(e_{o}^{T}e_{c}/\tau-\gamma)Pr⁡[eo|ec]=σ⁡((⟨eo,ec⟩−γ)/τ)\Pr[e_{o}|e_{c}]=\sigma((\langle e_{o},e_{c}\rangle-\gamma)/\tau)

4.3 Results on CUB

CUB queries ask whether the bird belongs to the learned category, emphasizing classification rather than complex reasoning. With one example plus relations, Table 1 reports 81.33% for FALCON-G and 79.83 for FALCON-R, compared with 79.53 for NSCL+LSTM and 73.88 for MAC. The main text’s statement that FALCON-R is highest conflicts with this table; this edition follows the reported values.

Without supplemental relations, Table 2 gives 76.37 for FALCON-G and 75.84 for FALCON-R. Figure 4 preserves successes and mistakes. Later ablations examine continuing to learn through previously acquired concepts, reducing the base vocabulary, and withholding related-concept information.

Model(Type) QA Accuracy
CNN+LSTM (E2E) 51.37
MAC (E2E) 73.88
Box Hyperplane Hypercone
NSCL+LSTM (C.C.) 79.53 71.71 72.48
NSCL+GNN (C.C.) 78.50 67.60 72.82
FALCON-G (C.C.) 81.33 74.11 68.44
FALCON-R (C.C.) 79.83 72.82 69.74
Table 1: Fast concept learning performance on the CUB dataset. We use “E2E” for end-to-end methods and “C.C.” for concept-centric methods. Our model FALCON-G and FALCON-R outperforms all baselines.
Model(Type) QA Accuracy
CNN+LSTM (E2E) 51.50
MAC (E2E) 73.55
Box Hyperplane Hypercone
NSCL+LSTM (C.C.) 72.23 63.18 68.48
NSCL+GNN (C.C.) 73.38 62.00 72.91
FALCON-G (C.C.) 76.37 64.60 70.42
FALCON-R (C.C.) 75.84 63.32 69.69
Table 2: Fast concept learning with only visual examples, evaluated on the CUB dataset. The task is also referred to as one-shot learning. In this setting, NSCL+GNN will fallback to a Prototypical Network (Snell et al., 2017).
Complete original figure with every panel, label and legend.
Figure 4: Qualitative results of FALCON-G on the CUB and CLEVR dataset. The bottom left corner of each images shows the model prediction and the confidence score. ∗* marks a wrong prediction.

4.4 Results on CLEVR

On CLEVR, one object often leaves a new attribute ambiguous. Table 3 shows FALCON-G improving from 68.47% with examples alone to 88.40 with supplemental sentences; NSCL+GNN rises from 65.01 to 84.11. The result supports using supplied concept relations to resolve ambiguity.

A separate biased-data test forces the novel concept to co-occur with another novel concept. Appendix Table 9 reports 87.29% for FALCON-G with supplemental sentences, whereas the main narrative states 88.29. The appendix also gives different baseline values; the condensed discussion uses its complete table.

Model Example Only Example+Supp. Δ\Delta
CNN+LSTM 41.92 42.69 0.77
MAC 61.81 62.05 0.24
NSCL+LSTM 56.82 61.35 4.53
NSCL+GNN 65.01 84.11 19.11
FALCON-G 68.47 88.40 19.93
FALCON-R 68.57 87.36 18.78

Table 3: Fast concept learning performance on the CLEVR dataset. We compare both Example only and the Example +Supp. settings. The results are evaluated and averaged on the four splits of the dataset. See the main text for analysis.

4.5 Results on GQA

GQA asks whether a referred object has the new property, using a restricted frequent-concept vocabulary. FALCON-G with boxes scores 55.96%, compared with 55.41 for NSCL+LSTM and 54.99 for MAC. These are small gains, and without supplemental sentences FALCON-R’s 54.60 is close to MAC’s 54.45. Natural-scene complexity remains a substantial limitation.

5 Conclusion

Explicit reusable concepts and learned inference from visual examples plus relational guidance can improve data efficiency in controlled tasks. Their usefulness depends on reliable grounding, supplied language structure, the base vocabulary, and the geometry of the new category. The paper demonstrates downstream question answering, with broader task transfer remaining to be established. The authors acknowledge ONR, CBMM, MIT Quest for Intelligence, and the MIT–IBM AI Lab.

Appendix A Model Details

Appendix A retains the detailed representations, update equations, code, parser operations, and comparison-model implementations.

A.1 Remarks on the overall model structure

The concept-centric design separates learning a reusable object classifier from answering queries. Images and descriptions are compressed into a concept embedding, which avoids reprocessing every learning example for each future question.

A.2 Concept Embedding

Box embeddings use 100 dimensions for centers and another 100 for offsets; hyperplane and hypercone vectors use 512 dimensions. Box smoothing uses temperature 0.2. Thus the representation comparisons also differ in dimensionality and parameterization.

A.3 Prior Distribution

A shared Dirichlet prior samples the position and extent of each box dimension, while vector alternatives use isotropic Gaussian priors. The original sampling formulas are preserved below. The prior parameters are learned through the meta-learning objective, with the prior-density loss coefficient set to one.

EquationsCen⁡(c)p=12​(x1​p−x0​p)\operatorname{Cen}(c)_{p}=\frac{1}{2}(x_{1p}-x_{0p})Off⁡(c)p=12​x2​p\operatorname{Off}(c)_{p}=\frac{1}{2}x_{2p}ec∼𝒩⁡(0,θ0​Id)e_{c}\sim\mathcal{N}(\textbf{0},\theta_{0}\textbf{I}_{d})ℒmeta=ℒQ​A−λprior ​log⁡pθ​(ec)\mathcal{L}_{\text{meta}}=\mathcal{L}_{QA}-\lambda_{\text{prior }}\log p_{\theta}\left(e_{c}\right)

A.4 FALCON-G

Each graph network has two message-passing layers. Relation-specific networks produce messages, elementwise maximum aggregates neighbors, and gated residual updates modify the target embedding. Box updates share small networks across dimensions; vector embeddings use larger fully connected networks. The equations retain the exact center/offset and vector updates.

Equations
mj​i=MPk⁡(hj(ℓ),hi(ℓ)),m_{ji}=\operatorname{MP}_{k}(h_{j}^{(\ell)},h_{i}^{(\ell)}),
hi(ℓ+1)=UPDATE⁡(hi(ℓ),h𝒩G​(i)(ℓ)).h_{i}^{(\ell+1)}=\operatorname{UPDATE}(h_{i}^{(\ell)},h_{\mathcal{N}_{G}(i)}^{(\ell)}).
Cen⁡(hi​p(ℓ+1))\displaystyle\operatorname{Cen}(h_{ip}^{(\ell+1)}) =\displaystyle= Cen⁡(hi​p(ℓ))+σ⁡(oi​p(ℓ))​δi​p(ℓ);\displaystyle\operatorname{Cen}(h_{ip}^{(\ell)})+\sigma(o_{ip}^{(\ell)})\delta_{ip}^{(\ell)};
Off⁡(hi​p(ℓ+1))\displaystyle\operatorname{Off}(h_{ip}^{(\ell+1)}) =\displaystyle= Off⁡(hi​p(ℓ))+σ⁡(oi​p′(ℓ))​δi​p′(ℓ),\displaystyle\operatorname{Off}(h_{ip}^{(\ell)})+\sigma(o_{ip}^{\prime(\ell)})\delta_{ip}^{\prime(\ell)},
hi(ℓ+1)=hi(ℓ)+σ⁡(oi)⊙δi,h_{i}^{(\ell+1)}=h_{i}^{(\ell)}+\sigma(o_{i})\odot\delta_{i},

A.5 FALCON-R

The appendix combines concept and example edges into one graph, randomly orders the relational tuples, and processes each through a recurrent update implemented as message passing on a two-node graph. Snippet 1 retains the complete published PyTorch-style propagation code. It is source pseudocode, not a newly validated executable implementation.

for j, i, k in G.edges:
    split = torch.stack([
        *h[j].chunk(2, -1),
        *h[i].chunk(2, -1)
    ], dim=-1)
    m[(j, i)] = mlp_1[k](split)
for i in G.nodes:
    h_N[i] = torch.stack([
        m[(jj, ii)] for (jj, ii) in m if i == ii
    ]).max(dim=0).values
for i in G.nodes:
    o, delta, o_prime, delta_prime = mlp_2(torch.stack([
        *h[i].chunk(2, dim=-1),
        *h_N[i].unbind(dim=-1)
    ])).unbind(dim=-1)
    h[i] = torch.cat([
        center + torch.sigmoid(o) * delta,
        offset + torch.sigmoid(o_prime) * delta_prime
    ], dim=-1)
Snippet 1: A snippet of one-step concept graph propagation based on the box embedding space. The code is written in PyTorch Paszke et al. (2019). The code snippet updates the node embedding h[i] by first computing the messages passing from node j to node i in the first for-loop, aggregating the messages to node i by taking the element-wise max, and finally updating the node embedding of i.

A.6 Semantic parser

The sequence-to-sequence parser predicts flattened operation/concept pairs, then reconstructs hierarchical programs. It is trained on 10% of the generated data. Table 4 preserves all CLEVR operations; CUB uses only Scene, Filter, and Exist because each image has one bird. Table 5 retains description and query examples across all datasets.

Operation Semantics
Scene Return all objects in the scene.
Filter
Filter out the set of objects from input objects that have the specific
   object-level concept.
Relate
Filter out the set of objects that have the specific relational concept
   with as input object.
AERelate Filter out the set of objects that have the same attribute value
   as the input object.
Intersection Return the intersection of two object sets.
Union Return the union of two object sets.
Query Query the attribute of the input object.
AEQuery Query if two objects have the same attribute value (e.g., color).
Exist Query if an object set is empty.
Count Return the size of an object set.
CountLessThan
Query if the size of the first object set is smaller than the size of
   the second object set.
CountGreaterThan
Query if the size of the first object set is greater than the size of
   the second object set.
CountEqual
Query if the size of the first set is the same as the size of
   the second object set.
Table 4: All program operations supported in CLEVR.
Dataset Input Output
CLEVR
(Description Sentences)
The red objects that are both
in front of the cyan object and
behind the cube is a sphere.
Filter(Intersection(
Relate(Filter(Scene(),
  cyan), front),
Relate(Filter(Scene(),
cube), behind)
), red)
CLEVR
(Questions)
There is a rubber object to
the left of the metal object;
What is its color?
Query(Filter(
Relate(Filter(Scene(),
  metal), left),
rubber), color)
CUB
(Description Sentences)
There is a frigatebird. Scene()
CUB
(Questions)
Is there a frigatebird?
Exist(Filter(Scene(),
frigatebird))
GQA
(Description Sentences)
The clouds are a white object. Filter(Scene(),clouds)
GQA
(Questions)
Is the standing man
a red object?
Exist(Filter(Filter(
Filter(Scene(), standing),
man), red))
Table 5: Example questions and the programs recovered by the semantic parser. For description sentences, the semantic parser parse the referential expression in the sentence into a program. On CUB, since there is only one object in the image, the operation Scene() returns the only object in the image.

A.7 Neuro-Symbolic Progrm Execution

Execution follows NS-CL with two changes: concept membership uses the selected embedding geometry, and same-attribute tests are represented as learned relational concepts such as has_same_color. Soft object memberships allow downstream answer loss to train the underlying representations. Table 4’s original Exist wording says “empty,” whereas the intended operation tests existence; the table is preserved as published.

A.8 Baseline Implementation

The following implementations distinguish reusable-concept models from comparisons that process the full task context for each query.

CNN+LSTM

CNN+LSTM extracts global ResNet-34 features for the example and query images, encodes concatenated descriptions and questions with a bidirectional LSTM, and predicts the answer through an MLP. It does not explicitly construct a reusable novel-concept classifier.

MAC

MAC concatenates example/query image feature maps and description/query text, then uses its attention-based reasoning network. The image backbone is ResNet-101, and other settings follow the original architecture.

NSCL+LSTM

NSCL+LSTM shares the parser and executor, then separately encodes grounded visual examples and related-concept information with bidirectional LSTMs. Their final states predict the new concept embedding through a linear layer. This contrasts with FALCON’s sequential or graph-based updates to sampled candidates.

NSCL+GNN

NSCL+GNN initializes a concept from normalized mean vectors or the smallest box enclosing its example embeddings, then applies a relation graph network. Without supplemental relations, the single-example box version initializes size from the average training-concept box. This appendix detail explains the numerical box baseline in Table 2.

Appendix B Training Details

Figure 5 preserves the three-stage learning protocol. CLEVR pretraining uses a curriculum over scene size and question complexity. Pretraining runs 50,000 iterations with batch size ten and Adam learning rate 0.0001; meta-training runs 10,000 task iterations with batch size one and learning rate 0.001. Validation occurs every 1,000 iterations, with a tenfold learning-rate reduction after five evaluations without improvement.

For CNN+LSTM and MAC, the novel concept name is replaced by an unknown token during meta-training and meta-testing because those models assume a fixed vocabulary. This is an implementation choice affecting the comparison, not a universal limit of all end-to-end architectures.

Refer to caption
Figure 5: An overview of the overall training paradigm.

Appendix C Experiment Details

Appendix C preserves the data templates, additional result tables, biased-concept examples, graph alternatives, and entailment evaluation.

C.1 The CUB Dataset

Table 6 gives the bird description, taxonomy, and query templates. The generated data include roughly 1,800 descriptions, 54,000 downstream questions, and 11,800 pretraining image-question pairs. Training supplemental sentences randomly omit a related concept with probability 0.2. Standard image resizing, cropping, and horizontal flips augment the training data.

(a) The descriptive sentence template in CUB.
Template There is a/an <CONCEPT>.
Example There is a white-eyed vireo.
Description <CONCEPT> is the word representing concept cc.
(b) The supplemental sentence template in CUB.
Template <CONCEPT-1>, <CONCEPT-2>⋯\cdots <CONCEPT-N> are a kind of
   <HYPERNYM>.
Example Black-capped vireo, philadelphia vireo, warbling vireo, white-eyed vireo are
   a kind of vireos.
Description <CONCEPT-i> are words representing cc and c′c^{\prime} that share a cohypernym relation
   with cc. <HYPERNYM> denotes cc’s hypernym.
(c) The test question template in CUB.
Template Is there a/an <CONCEPT>.
Example Is there a white-eyed vireo.
Description <CONCEPT> is the word representing concept cc.
Table 6: Templates for generating descriptive sentences, supplemental sentences and test questions in CUB.

Extension: continual learning.

Continual learning introduces test concepts in increasing taxonomy distance from known concepts, freezing newly learned concepts before using them to support later ones. Table 7 reports 79.67% for FALCON-G with relations versus 75.21 for NSCL+LSTM. This tests ordered expansion of the concept repository; it is not a general benchmark of forgetting under repeated updates to all previous classifiers.

Model Performance
Box Hyperplane Hypercone
NSCL+LSTM 75.21 67.78 71.06
NSCL+GNN 71.04 64.74 72.37
FALCON-G 79.67 73.14 64.37
FALCON-R 78.76 65.38 68.11
(a) The continual concept learning performance on the CUB dataset.
Model Performance
Box Hyperplane Hypercone
NSCL+LSTM 71.31 62.45 67.50
NSCL+GNN 72.53 61.56 71.25
FALCON-G 75.28 64.16 68.73
FALCON-R 75.01 62.78 67.96
(b) The continual concept learning performance with only visual examples on the CUB dataset. This task is also referred as few-shot learning, where NSCL+GNN will fall back to a Prototypical Network Snell et al. (2017).
Table 7: The continual concept learning performance on the CUB dataset. Only concept-centric methods can be applied to this setting. FALCON-G and FALCON-R outperform all baselines. The table b serves as comparison with table a. FALCON-G and FALCON-R make the best use supplemental sentences to help concept learning.

C.2 The CLEVR Dataset.

Each CLEVR split holds out two colors for validation and two further colors plus one shape for testing. Descriptions identify an object using only base concepts; test questions contain the novel concept once. Table 8 retains the templates. The protocol generates about 7,500 descriptions, 225,000 downstream image-question pairs, and 402,000 pretraining questions.

(a) The descriptive sentence template in CLEVR.
Template There is a <REFEXP>; it is <CONCEPT>.
Example There is a small object to the right of the yellow object; it is a cyan object.
Description <REFEXP> is a referential expression generated by CLEVR-Ref+
   Liu et al. (2019). <CONCEPT> is the word for concept cc.
(b) The supplemental sentence template in CLEVR.
Template <CONCEPT-1>,<CONCEPT-2>⋯\cdots<CONCEPT-N> are <ATTRIBUTE>.
Example Cube, sphere and cylinder are shapes.
Description <CONCEPT-i> are words representing cc and its related concepts c′c^{\prime}.
   <ATTRIBUTE> is the word for an attribute of a object, one of
   shape, color, material and size.
(c) The test question generation template in CLEVR.
Template <QUEST>
Example Does the small blue thing have the same material as the gray object?
Description <QUEST> are question generated from the original CLEVR paper
   Johnson et al. (2017a).
Table 8: Templates for generating descriptive sentences, supplemental sentences and test questions in CLEVR.

Learning from biased data.

Figure 6 uses the same purple-cylinder appearance for two different novel concepts. Supplemental sentences distinguish the shape interpretation from the color interpretation. Table 9 reports FALCON-G improving from 69.71 to 87.29%; NSCL+GNN improves from 64.75 to 84.83. The table’s FALCON improvement is smaller than NSCL+GNN’s absolute gain, despite the higher final score.

Refer to caption
Figure 6: A visualization of fast concept learning from biased data with supplemental sentences on CLEVR. Notice that in both tasks, the visual examples are purple cylinders. The association between the new concept and the visual appearance (shape or color) is ambiguous, without supplemental sentences.
Model Example Only Example+Supp. Δ\Delta
CNN+LSTM 41.61 42.38 0.77
MAC 62.38 62.40 0.02
NSCL+LSTM 57.76 61.79 4.03
NSCL+GNN 64.75 84.83 20.08
FALCON-G 69.71 87.29 17.57
FALCON-R 69.70 86.21 16.50
Table 9: Ablated study of FALCON tasks w/. and w/o. supplemental sentences (+Supp.) on the biased CLEVR dataset. All concept-centric methods use the box embedding space.

Handling non-prototypical concepts.

Attribute categories such as “color” do not have a single visual prototype. Figure 7 compares connecting a new concept to sibling concepts against introducing the abstract attribute itself as a graph node. The former reaches 88.40%, the latter 71.34; the experiments therefore use sibling links. Table 10 also preserves the natural-image language templates.

Refer to caption
Figure 7: Two implementations of graph Gc​o​n​c​e​p​tG_{concept}.
(a) The descriptive sentence template in GQA.
Template The <CONCEPT-1> (<CONCEPT-2>) (object) is a/an <CONCEPT> (object).
Example The small man is a sitting object.
Description <CONCEPT> is the word for concept cc. <CONCEPT-i> are other concepts.
(b) The supplemental sentence template in GQA.
Template <CONCEPT-1>,<CONCEPT-2>⋯\cdots<CONCEPT-N> describes the same
property of an object.
Example Sitting, standing describes the same property of an object.
Description <CONCEPT-i> are words representing cc and its related concepts c′c^{\prime}.
(c) The test question generation template in GQA.
Template Is the <CONCEPT-1> (<CONCEPT-1>) (object) a/an <CONCEPT> (object)?
Example Is the white woman a sitting object?
Description <CONCEPT> is the word for concept cc. <CONCEPT-i> are other concepts.
Table 10: Templates for generating descriptive sentences, supplemental sentences and test questions in GQA.

C.3 The GQA dataset

GQA uses supplied Faster R-CNN object features and generated descriptions and questions. Tables 11(a) and 11(b) retain results with and without relations across all embedding geometries. The source’s stated total and split counts for generated questions do not sum consistently, so they should not be treated as a verified dataset-size audit. The reported accuracies show only modest advantages over the strongest comparisons.

Model QA Accuracy
CNN+LSTM 54.47
MAC 54.99
Box Hyperplane Hypercone
NSCL+LSTM 55.41 54.19 54.81
NSCL+GNN 55.32 50.07 54.75
FALCON-G 55.96 54.22 54.56
FALCON-R 55.42 55.73 55.79
(a) Fast concept learning performance on the GQA dataset. Our model FALCON-G and FALCON-R outperforms all baselines.
Model QA Accuracy
CNN+LSTM 48.73
MAC 54.45
Box Hyperplane Hypercone
NSCL+LSTM 53.54 53.16 52.83
NSCL+GNN 50.15 50.74 52.03
FALCON-G 53.89 53.24 52.22
FALCON-R 54.60 52.86 52.37
(b) Fast concept learning with only visual examples, evaluated on the GQA dataset. In this setting, NSCL+GNN will fallback to a Prototypical Network (Snell et al., 2017).

C.4 The connections between the box embedding and our formulation

The geometric motivation and the empirical entailment test are distinct: containment can express category inclusion, but learned updates need not satisfy every relation exactly.

Modeling concept entailments (intuition).

A narrower concept’s box should lie inside its broader hypernym’s box. Same-kind relations, such as two colors, do not have the same simple containment interpretation. The source appendix reverses the containment direction in one explanatory sentence; the interpretation here follows the main text and its conditional-probability definition.

Comparison with other box embedding spaces (empirical support).

The authors score every concept pair for entailment and vary a threshold to compute ROC AUC. The original classifier and scoring formulas are retained. Table 12 reports 0.855 for boxes, 0.638 for hyperplanes, and 0.585 for hypercones. The alternatives use symmetric similarity measures that are poorly suited to directional entailment, so the comparison favors the box representation’s intended geometry.

Equationsentailst(c,c′)=𝟏[Pr[c′|c]>t]entails_{t}(c,c^{\prime})=\mathbf{1}[\Pr[c^{\prime}|c]>t]Pr⁡[c′|c]=Pr⁡[ec′∩ec]/Pr⁡[ec]\Pr[c^{\prime}|c]=\Pr[e_{c^{\prime}}\cap e_{c}]/\Pr[e_{c}]Pr⁡[c′|c]=σ⁡(ecT​ec′/τ−γ)\Pr[c^{\prime}|c]=\sigma(e_{c}^{T}e_{c^{\prime}}/\tau-\gamma)Pr⁡[c′|c]=σ⁡((⟨ec′,ec⟩−γ)/τ)\Pr[c^{\prime}|c]=\sigma((\langle e_{c^{\prime}},e_{c}\rangle-\gamma)/\tau)
Methods FALCON (Box) FALCON (Hyperplane) FALCON (Hypercone)
AUC-ROC Score 0.855 0.638 0.585
Table 12: AUC-ROC score of classifying entailment relationship based on concept embeddings from different embedding spaces.

Appendix D Ablation Details

Two ablations vary the base-concept vocabulary and the amount of relational context.

D.1 The Effect of the Number of Base Concepts

Reducing the training species from 100 to fifty changes the base vocabulary from 211 to 130 concepts and also changes validation/test composition. FALCON-G accuracy falls from 81.33 to 76.32%; Table 13 retains the comparison models. This supports the value of prior concepts, while the changed evaluation split limits an interpretation as a pure vocabulary-size intervention.

# of base concepts FALCON-G NSCL+GNN MAC
130 76.32 75.21 65.16
211 81.33 78.50 73.88
Table 13: Fast concept learning (with supplemental sentence) performance under different number of base concepts.

D.2 The Effect of The Number of Related Concepts in Supplemental Sentences

Table 14 gives FALCON-G accuracies of 76.37, 80.20, 80.78, 81.20, and 81.33% with 0, 25, 50, 75, and 100% of related-concept information. Most of the gain comes from the first quarter of relations.

Figure 8 is retained unchanged, but its MAC and FALCON-G legend labels conflict with the corresponding series in Table 14: the highest curve matches FALCON-G’s tabulated values. The numerical discussion follows the table.

% of all related concepts FALCON-G NSCL+GNN MAC
0 76.37 73.38 73.55
25 80.20 77.16 73.94
50 80.78 76.93 74.13
75 81.20 77.77 74.23
100 81.33 78.50 73.88
Table 14: Fast concept learning performance under different percentage of related concepts in the supplemental sentence.
Refer to caption
Figure 8: Fast concept learning performance under different percentage of related concepts in the supplemental sentence.

Appendix E Failure Mode Details

The failure analysis separates object detection, language parsing, and concept-grounding errors. A correct final answer can coexist with an incorrect intermediate representation.

E.1 Failure mode in detection module

Table 15 preserves missed-object and false-positive examples. Missing a small or occluded object can redirect the description to the wrong instance; a spurious mask may have no effect if execution ignores it. Errors in the chosen training example can alter the learned concept and affect later questions.

E.2 Failure mode in semantic parser

Table 16 retains syntactically invalid programs repaired by heuristics and syntactically valid but semantically wrong programs. A wrong program can still produce the expected answer when scene properties are redundant. The implementation has no general semantic-error recovery; visual-feedback parser refinement is proposed as future work.

E.3 Failure mode in concept learning module

Figure 9 preserves two concept-grounding failures: an overly broad spatial relation chooses the wrong example object, while an overly narrow newly learned shape excludes the correct test object. Such errors can arise in base, relational, or newly learned concepts and propagate through otherwise valid programs.

Descriptive sentence:The blue matte thing to the right of the big cyan object is a cube. Expected example object: The blue cube around the upper-right corner of the cyan cube. Predicted example object: The purple metal cylinder. ✗ Detection error: The blue cube is not detected by the detection module.
[Uncaptioned image]
Descriptive sentence: The white small object is a plate. Expected example object: The white plate. Predicted example object: The lower side of the plate. ✗ Detection error: The entirety of the white plate is not detected as one object
[Uncaptioned image]
(a) Error case when the detection module misses an object. The blue boxes indicates the expected object that the detection module misses; the red boxes indicates the wrongly predicted objects.
Descriptive sentence:The metallic thing that is right of the tiny sphere is a cube. Expected example object: The green cube. Predicted example object: The green cube. ✓ Detection error: The union of the blue cube and the gray cube is detected as one object by the detection module.
[Uncaptioned image]
Descriptive sentence:The large long object is a silver object. Expected example object: The long knife. Predicted example object: The long knife. ✓ Detection error: Shadows of the knife and the apple are detected as objects by the detection module.
[Uncaptioned image]
(b) Error case when the detection module recognize a false positive as an object. The green boxes indicates the expected and predicted object; the red boxes indicates false positives.
Table 15: A visualization of three failure cases of the detection module. The first two descriptive sentences are evaluated to find the wrong example object, and the last two are evaluated to find the right example object.
[Uncaptioned image]
Text
Are there an equal number of small balls in front of the small ball
and big cyan things?
Expected Program CountEqual(Filter(Filter(Relate(
      Filter(Filter(Scene(), small), ball),
          front), small), ball),
      Filter(Filter(Scene(), big), cyan))
Predicted Program CountEqual(Filter(Filter(Relate(
      Filter(Filter(Scene(), small), ball),
          front), small), Filter),
      Filter(Filter(Scene(), big), cyan))
Fixed Program CountEqual(Filter(Relate(
      Filter(Filter(Scene(), small), ball),
          front), small),
      Filter(Filter(Scene(), big), cyan))
(a) A syntactically infeasible program. The program is fixed with heuristics.
Text
Is there a metal cyan object to the left of the big brown object?
Expected Program Exists(Filter(Filter(Relate(
      Filter(Filter(Scene(), big), brown),
left), metal), cyan))
Predicted Program Exists(Filter(Filter(Relate(
      Filter(Filter(Scene(), big), purple),
left), metal), cyan))
Expected answer Yes. Predicted answer No.
(b) A syntactically feasible but semantically wrong program, which yields a wrong answer.
Text
Is there a big cyan object to the right of the big brown object?
Expected Program Exists(Filter(Filter(Relate(
      Filter(Filter(Scene(), big), brown),
right), big), cyan))
Predicted Program Exists(Filter(Filter(Relate(
      Filter(Filter(Scene(), big), purple),
right), big), cyan))
Expected answer Yes. Predicted answer Yes.
(c) A syntactically feasible but semantically wrong program, but the answer is correct.
Table 16: A visualization of three failure cases of the semantic parser. All examples are from the question answering part of tasks on learning the novel concept cyan based on the image above.
Refer to caption

Descriptive sentence: The shiny block that are behind small rubber thing is a red object.
Expected example object: The metallic red cube.
Predicted example object: The metallic grey cube.
Explanation: The relational concept behind is not accurate enough; it is so extensive that even the grey cube is behind the yellow cube.

(a) Error case when the imperfect visual grounding leads to a wrong object example in the training image. This is part of the task that learns the novel concept red.
Refer to caption

Question: How many red cylinders are to the right of the blue sphere?
Expected answer: 1. (The red cylinder)
Predicted answer: 0.
Explanation: The meta-learned concept cylinder is not accurate enough; it is too exclusive that even the red cylinder is not considered a cylinder.

(b) Error case when the imperfect visual grounding leads to a wrong answer in downstream question-answering. This is part of the task that learns the novel concept cylinder.
Figure 9: A visualization of two failure cases of the concept learning module.
Concept demonstration

Open image ↗