Literature
Display
← All papers

Visual Concept-Metaconcept Learning

Chi Han · Jiayuan Mao · Chuang Gan · Joshua B. Tenenbaum · Jiajun Wu

· NeurIPS 2019 ↗

Reasoning9 min text · CondensedOriginal paper ↗
Contents

At a glance

The Visual Concept-Metaconcept Learner (VCML) connects visual categories with relations between categories, such as synonymy, shared attribute type, hypernymy, and part-whole relations. Images and visual questions ground concept embeddings; text-only questions train operators over those embeddings. Information can therefore flow from relational knowledge to visual recognition and back.

Controlled experiments show improved learning from scarce or biased examples and prediction of relations between held-out concepts. The method assumes pretrained proposals and a semantic parser, and obtains metaconcept supervision from external ontologies. Results are strongest on synthetic scenes; natural-image zero-shot recognition remains near chance.

1 Introduction

A few red cubes do not establish whether “red” names their color or shape. Knowing that red and green describe the same property can reduce this ambiguity. Conversely, grounded representations may suggest that cube and sphere share a property even when that relation was not explicitly taught. Figure 1 illustrates these two directions of transfer.

VCML shares concept embeddings between visual and text-only reasoning. The study tests new combinations of attributes, concepts with few visual examples, and relations between previously unseen pairs. The latter still have visual grounding; they are not wholly unknown concepts.

Refer to caption
Figure 1: Our model learns concepts and metaconcepts from images and two types of questions. The learned knowledge helps visual concept learning (generalizing to unseen visual concept compositions, or to concepts with limited visual data) and metaconcept generalization (generalizing to relations between unseen pairs of concepts.)

2 Related Work

Visual concept learning has used labels, language, structured embeddings, and external knowledge. Neuro-symbolic reasoning separates perception from executable operations. VCML extends this setting with trainable relations between concept representations.

The work also connects compositional visual recognition with knowledge-base completion. Its particular claim is bidirectional transfer through visual grounding, rather than discovering a concept ontology or an unrestricted reasoning language automatically.

3 Visual Concept-Metaconcept Learning

The architecture in Figure 2 combines object perception, question-to-program parsing, and differentiable execution. Concepts are vectors in a shared space; metaconcepts are small neural classifiers over geometric relationships between vectors. Both visual answers and text-only relation answers train this representation.

Refer to caption
Figure 2: The Visual Concept-Metaconcept Learner. The model comprises three modules: (a) a perception module for extracting object-based visual representations, (b) a semantic parsing module for recovering latent programs from natural language, and (c) a neuro-symbolic reasoning module that executes the program to answer the question.

3.1 Motivating Examples

Visual reasoning.

For “Is there any red object?”, perception extracts candidate objects and the parser produces Exist(Filter(red)). Concept-object similarity supplies soft membership scores; taking their maximum answers the existence question.

Metaconcept questions.

A text-only question about whether red and green describe the same property becomes MetaVerify(red, green, same_kind). A metaconcept classifier reads the two concept embeddings without requiring an image for that particular question.

3.2 Model Details

Table 1 retains the MetaVerify signature, semantics, and example. This operation extends the existing visual-reasoning domain-specific language (DSL).

Table 1: Our extension of the visual-reasoning DSL (Johnson et al., 2017; Hudson and Manning, 2019), including one new primitive function for metaconcepts.
MetaVerify
Signature Concept, Concept, MetaConcept ⟶\longrightarrow Bool
Semantics Returns whether two input concepts have the specified metaconcept relation.
Example MetaVerify(Sphere, Ball, Synonym) ⟶\longrightarrow True

Perception module.

Mask R-CNN generates object proposals, and ResNet-34 extracts a feature vector for each region. The GQA experiment instead uses its supplied ground-truth bounding boxes.

Semantic parsing module.

The semantic parser converts questions into hierarchical programs containing fixed visual operations and the new metaconcept operation. The experiments assume access to a pretrained parser, so joint learning here concerns representations and metaconcepts rather than discovery of the language interface.

Concept embeddings.

Each object or concept vector defines a half-space in a shared high-dimensional space. A standard Gaussian measure assigns probabilities to these regions and their intersections. Conditional region probability gives an asymmetric entailment score, used to decide whether an object belongs to a concept. Figure 3 and the retained equations specify the geometry.

This construction supplies structured scores for concept relations while allowing gradients from both kinds of questions. The formulas are preserved as printed, including their Gaussian-normalization notation.

EquationsV⁡(𝐱)={𝐲∈ℝN∣(𝐲−𝐱)T​𝐱>0}V({\mathbf{x}})=\{{\mathbf{y}}\in\mathbb{R}^{N}\mid({\mathbf{y}}-{\mathbf{x}})^{T}{\mathbf{x}}>0\}
Pr⁡(a)=Vol𝒩⁡(𝟎,𝐈)​(Va)=∫𝐳∈Va12​π​e−12​‖𝐳‖22​𝑑𝐳=12​[1−erf​(‖𝐱‖22)]\Pr(a)=\textnormal{Vol}_{\mathcal{N}(\mathbf{0},\mathbf{I})}(V_{a})=\int_{{\mathbf{z}}\in V_{a}}{\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}{\left\|{\mathbf{z}}\right\|}_{2}^{2}}d{\mathbf{z}}}=\frac{1}{2}[1-\text{erf}(\frac{{\left\|{\mathbf{x}}\right\|}_{2}}{\sqrt{2}})]
Pr⁡(a,b)=Vol𝒩⁡(𝟎,𝐈)​(Va∩Vb)=∫𝐳∈Va∩Vb12​π​e−12​‖𝐳‖22​𝑑𝐳\Pr(a,b)=\textnormal{Vol}_{\mathcal{N}(\mathbf{0},\mathbf{I})}(V_{a}\cap V_{b})=\int_{{\mathbf{z}}\in V_{a}\cap V_{b}}{\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}{\left\|{\mathbf{z}}\right\|}_{2}^{2}}d{\mathbf{z}}}
Pr⁡(b∣a)=Pr⁡(a,b)Pr⁡(a)=Vol𝒩⁡(𝟎,𝐈)​(Vb∩Va)Vol𝒩⁡(𝟎,𝐈)​(Va)\Pr(b\mid a)=\frac{\Pr(a,b)}{\Pr(a)}=\frac{\textnormal{Vol}_{\mathcal{N}(\mathbf{0},\mathbf{I})}(V_{b}\cap V_{a})}{\textnormal{Vol}_{\mathcal{N}(\mathbf{0},\mathbf{I})}(V_{a})}
Refer to caption
Figure 3: A graphical illustration of our concept-metaconcept embedding space. Each concept or object is embedded as a high-dimensional vector, which is associated with a half-space supported by this vector. Each metaconcept is associated with a multi-layer perceptron as a classifier. The programs are executed based on the scene representation, concept embeddings and metaconcept operators.

Metaconcept operators

Each binary metaconcept has a multilayer perceptron. Its inputs are the logits of both directional conditional probabilities and the log ratio of joint probability to independent marginals. The retained formulas show how these three geometric features predict, for example, synonymy. Only pairwise metaconcept relations are modeled.

Equations
g1​(a,b)=logit​(Pr⁡(a∣b)),g2​(a,b)=ln⁡Pr⁡(a,b)Pr⁡(a)​Pr⁡(b),g_{1}(a,b)=\textnormal{logit}(\Pr(a\mid b)),\quad g_{2}(a,b)=\ln{\frac{\Pr(a,b)}{\Pr(a)\Pr(b)}}~,
MetaVerify​(𝑟𝑒𝑑,c​u​b​e,s​y​n​o​n​y​m)=fs​y​n​o​n​y​m​(g1​(r​e​d,c​u​b​e),g1​(c​u​b​e,r​e​d),g2​(r​e​d,c​u​b​e)).\text{MetaVerify}(\mathit{red},cube,synonym)=f_{synonym}(g_{1}(red,cube),g_{1}(cube,red),g_{2}(red,cube))~.

Neuro-symbolic reasoning module.

The executor relaxes Boolean values to probabilities and applies fixed operations such as filtering and maximum-based existence. These visual reasoning operations require no learned weights. Gradients through their soft outputs train object features, concept embeddings, and the separate metaconcept classifiers.

3.3 Training

The object proposal generator and semantic parser are pretrained. ResNet-34 starts from ImageNet weights and is fine-tuned. Concept embeddings and metaconcept operators are optimized from answer supervision with Adam, learning rate 0.001 and batch size 10. This supplied structure limits any interpretation as learning all perception and reasoning solely from raw questions.

4 Experiments

The experiments test whether metaconcept learning improves visual-concept learning and compositional generalization, and whether grounding helps predict relations between unseen concept pairs. The paper reports averages over four runs, with ± denoting standard deviation. This run-level variation should be read separately from the supplementary figures whose shaded regions are explicitly half a standard deviation.

4.1 Dataset

Questions are generated from annotated concepts to control the experimental splits. CLEVR supplies 70,000 training and 15,000 test images and 22 concepts including synonyms. GQA uses 75,000 training and 10,000 test images, its ground-truth object boxes, and the 584 most frequent concepts from a larger ontology. Attribute labels are not used directly to train the models.

CUB supplies roughly 8,000 training and 1,000 test bird images. Its 200 species are extended with 166 higher taxonomic categories. Metaconcept questions come from the CLEVR ontology, WordNet for GQA, and eBird taxonomy plus CUB attributes for bird relations. Thus the relational knowledge is explicit extra supervision.

4.2 Baseline

Comparisons include general visual question-answering networks, an adapted neuro-symbolic concept learner, and language-only systems. Their access to metaconcept questions should be distinguished from the architectural manner in which they use them.

GRU-CNN.

GRU-CNN combines a ResNet-34 image encoding and a GRU question encoding, followed by an answer classifier. Text-only questions use the question features alone.

MAC.

MAC uses its dual-attention reasoning architecture for visual questions and receives a blank image for text-only metaconcept questions.

NS-CL.

The NS-CL baseline is extended with two-layer networks that predict metaconcepts from concept embeddings. It therefore has access to relational supervision, unlike an unmodified concept-only baseline.

GRU (language only).

The language-only GRU uses fixed pretrained GloVe word embeddings, a learned recurrent question encoder, and an answer classifier. It tests what can be predicted without visual grounding.

BERT.

One BERT variant encodes the entire question before classification. The other uses a pretrained parser and classifies concatenated concept-word embeddings. Both freeze BERT’s pretrained parameters; these results do not compare against unrestricted fine-tuning of the language model.

4.3 Metaconcepts Help Concept Learning

Three experiments use synonymy, shared attribute type, or hypernymy to aid visual learning. A further referring-expression task measures whether the learned embeddings help select the described object. Tables 2–5 retain all reported values and uncertainty estimates.

4.3.1 synonym Supports Zero-Shot Learning of Novel Concepts

Synonym relations can transfer visual knowledge to a new name. Held-out concept names appear in text-only synonym questions during training but never in visual questions. Test questions require recognizing them in images, so “zero-shot” refers specifically to visual supervision for those names.


GRU-CNN MAC NS-CL VCML
CLEVR 50.0±0.050.0_{\pm 0.0} 68.7±3.868.7_{\pm 3.8} 80.2±3.180.2_{\pm 3.1} 94.1±4.6\mathbf{94.1_{\pm 4.6}}
GQA 50.0±0.050.0_{\pm 0.0} 49.5±0.249.5_{\pm 0.2} 49.3±0.649.3_{\pm 0.6} 50.5±0.1\mathbf{50.5_{\pm 0.1}}
Table 2: The metaconcept synonym provides abstract-level supervision for concepts. This enables zero-shot learning of novel concepts.

GRU-CNN MAC NS-CL VCML
CLEVR-200 50.0±0.050.0_{\pm 0.0} 94.2±3.394.2_{\pm 3.3} 98.5±0.398.5_{\pm 0.3} 98.9±0.2\mathbf{98.9_{\pm 0.2}}
CLEVR-20 50.0±0.050.0_{\pm 0.0} 79.7±2.679.7_{\pm 2.6} 95.7±0.0\mathbf{95.7_{\pm 0.0}} 95.1±1.695.1_{\pm 1.6}
Table 3: The metaconcept same_kind helps the model learn from biased data and generalize to novel combinations of visual attributes.
GRU-CNN MAC NS-CL VCML
CUB 50.0±0.050.0_{\pm 0.0} 70.8±3.470.8_{\pm 3.4} 80.0±2.380.0_{\pm 2.3} 80.2±1.7\mathbf{80.2_{\pm 1.7}}
Table 4: The metaconcept hypernym enables few-shot learning of new concepts.
#Train w/. w/o.
Ref. Expr. 10K 74.9±0.1\mathbf{74.9_{\pm 0.1}} 73.8±1.773.8_{\pm 1.7}
1K 59.7±0.2\mathbf{59.7_{\pm 0.2}} 51.6±2.651.6_{\pm 2.6}
Table 5: Application of VCML on Referential Expression task on CLEVR dataset

Dataset.

The experiment holds out three of CLEVR’s 22 concepts and thirty GQA concepts from visual-question training while retaining their synonym relations.

Results.

VCML reaches 94.1% on CLEVR, compared with 80.2 for adapted NS-CL and 68.7 for MAC. On GQA it reaches only 50.5%, close to the 50% baseline. The reported GQA training/validation accuracies of 83.1/52.3 show a substantial visual-generalization problem; the natural-image result should not be presented as strong zero-shot recognition.

4.3.2 same_kind Supports Learning from Biased Data

The shared-property relation is intended to prevent a shape concept from absorbing a color bias, or vice versa. The test changes the color-shape combinations seen during training.

Dataset.

The CoGenT-style split uses all training images from condition A plus either twenty or two hundred images from condition B. Metaconcept questions identify shared attribute types, and held-out condition-B images test the changed combinations. This includes limited target-condition visual data rather than testing transfer with none.

Results.

With two hundred condition-B images, VCML scores 98.9% and adapted NS-CL 98.5%. With twenty, VCML scores 95.1 and NS-CL 95.7; the proposed model does not win every setting. Both exceed MAC in these comparisons. The supplement varies the number of condition-B examples and removes metaconcept supervision to clarify its contribution.

4.3.3 hypernym Supports Few-Shot Learning Concepts

Hypernym relations constrain a rare concept through a better-known broader category. For example, linking Arctic Tern to Sterna supplies category-level information while learning from a few bird images.

Dataset.

Ninety-one of 366 taxonomic concepts receive five training images per concept; the others receive about forty. Class labels generate visual questions, and held-out images evaluate the rare concepts.

Results.

VCML reaches 80.2%, essentially comparable to adapted NS-CL’s 80.0, and above MAC’s 70.8. Without metaconcept questions, VCML reaches 78.5, NS-CL 79.6, and MAC 60.4. Relational knowledge helps, but the improvement over the strongest adapted baseline is small relative to the reported variation.

4.3.4 Application of concept embeddings to downstream task

The referring-expression task selects the object matching a description. Recall@1 compares VCML trained with and without metaconcept questions under limited visually grounded question data.

Results.

With 1,000 visual questions, metaconcepts increase Recall@1 from 51.6 to 59.7%; with 10,000, from 73.8 to 74.9. The larger gain at lower data volume supports their usefulness as additional constraints when direct visual supervision is scarce.

4.4 Concepts Help Metaconcept Generalization

The reverse direction asks whether visual grounding helps predict unseen concept relations. Table 6 reports synonym, shared-property, hypernym, and meronym results; Figure 4 illustrates the split.

Table 6: Metaconcept generalization evaluation on the CLEVR, GQA and CUB dataset. (Two variants of BERT are shown here; see Section 4.2 for details.)
Q.Type GRU (Lang. Only) GRU-CNN BERT (Variant I ; Variant II) NS-CL VCML
CLEVR Synonym 50.0 66.3±1.466.3_{\pm 1.4} 60.9±10.660.9_{\pm 10.6} 76.2±10.276.2_{\pm 10.2} ; 80.2±16.180.2_{\pm 16.1} 100.0±0.0\mathbf{100.0_{\pm 0.0}} 100.0±0.0\mathbf{100.0_{\pm 0.0}}
Same-kind 50.0 64.7±5.164.7_{\pm 5.1} 61.5±6.661.5_{\pm 6.6} 75.4±5.475.4_{\pm 5.4} ; 80.1±10.080.1_{\pm 10.0} 92.3±4.992.3_{\pm 4.9} 99.3±1.0\mathbf{99.3_{\pm 1.0}}
GQA Synonym 50.0 80.8±1.080.8_{\pm 1.0} 76.2±0.876.2_{\pm 0.8} 76.2±2.476.2_{\pm 2.4} ; 83.1±1.583.1_{\pm 1.5} 81.2±2.881.2_{\pm 2.8} 91.1±1.7\mathbf{91.1_{\pm 1.7}}
Same-kind 50.0 56.3±2.356.3_{\pm 2.3} 57.3±5.357.3_{\pm 5.3} 59.5±2.759.5_{\pm 2.7} ; 68.2±4.068.2_{\pm 4.0} 66.8±4.166.8_{\pm 4.1} 69.1±1.7\mathbf{69.1_{\pm 1.7}}
CUB Hypernym 50.0 74.3±5.274.3_{\pm 5.2} 76.7±8.876.7_{\pm 8.8} 75.6±1.275.6_{\pm 1.2} ; 61.7±10.361.7_{\pm 10.3} 80.1±7.380.1_{\pm 7.3} 94.8±1.3\mathbf{94.8_{\pm 1.3}}
Meronym 50.0 80.1±5.980.1_{\pm 5.9} 78.1±4.878.1_{\pm 4.8} 63.1±3.263.1_{\pm 3.2} ; 72.9±9.972.9_{\pm 9.9} 97.7±1.1\mathbf{97.7_{\pm 1.1}} 92.5±1.092.5_{\pm 1.0}
Refer to caption
Figure 4: Data split used for metaconcept generalization tests. The models are required to leverage the visual analogy between concepts to predict metaconcepts about unseen pairs of concepts (shown in blue). More details for other metaconcepts can be found in the supplementary material.

Dataset.

For each relation, concepts are partitioned into training and test groups. Training relation questions involve two training concepts; test relation questions involve two test concepts. Visual questions still ground the test concepts, allowing the relational classifier to transfer to their embeddings.

Results.

VCML scores 100.0% for CLEVR synonyms and 99.3 for shared property; GQA results are 91.1 and 69.1. CUB hypernym accuracy is 94.8, but meronym accuracy is 92.5, below adapted NS-CL’s 97.7. These results support visually grounded relation transfer with important variation across relation types and domains.

5 Conclusion

Sharing an embedding space lets explicit relational knowledge constrain visual concepts and lets visual grounding support relation prediction. The controlled tests support both directions, particularly with synthetic scenes or scarce supervision. Limits include pretrained detection and parsing, supplied ontologies, generated questions, a restricted pairwise relation vocabulary, and weak natural-image zero-shot recognition.

Acknowledgement.

The authors thank Jon Gauthier and acknowledge support from CBMM, ONR, the MIT–IBM Watson AI Lab, and Facebook.

Supplementary Material for: Visual Concept-Metaconcept Learning

The supplement retains the detailed split visualizations and an additional bias ablation. Its figure numbering restarts at one.

1 Dataset Visualization

Supplementary Figure 1 displays all three concept-learning settings: a new synonym name, changed attribute combinations, and rare bird categories. The blue test examples distinguish held-out visual information from metaconcept information supplied during training.

Supplementary Figure 2 shows all four relation-generalization settings, including bird hypernyms and part-whole relations. Test concept pairs lack relational training labels but retain visual grounding. The full image panels and question-answer examples are preserved.

Complete original figure with every panel, label and legend.
Figure 1: Data split used for tests on "Metaconcepts help concept learning". In training, the visual data are insufficient or even biased for some concepts. The models are then required to leverage the information provided in metaconcept questions in order to correctly reason on visual questions about these concepts (shown in blue).
Complete original figure with every panel, label and legend.
Figure 2: Data split used for metaconcept generalization tests. The models are required to leverage the visual analogy between concepts to predict metaconcepts about unseen pairs of concepts (shown in blue). See the main text for details.

2 Ablation: same_kind Supports Learning from Biased Data

Supplementary Figure 3 varies condition-B training images from zero to one thousand and compares VCML, adapted NS-CL, MAC, and VCML without metaconcept questions. Metaconcepts are most helpful in some small-data regimes; NS-CL closes the gap as more examples arrive, and VCML without relational supervision also approaches the full model with enough visual data.

The curves are not a uniform dominance result at every sample count. Shaded bands represent one-half of a standard deviation, as specified in the original caption, rather than confidence intervals.

Refer to caption
Figure 3: Models results under different levels of visual compositional bias in the experiment "same_kind Supports Learning from Biased Data". The y-axis is the test accuracy in percent, and the x-axis is the number of split B images in training. Plots are results for different models/settings. The transparent band denotes ±12\pm\frac{1}{2} standard deviation.
Concept demonstration

Open image ↗