Literature
Display
← All papers

Neuro-Symbolic Concepts

Jiayuan Mao · Joshua B. Tenenbaum · Jiajun Wu

· Communications of the ACM ↗

AI agents7 min text · CondensedOriginal paper ↗
Contents

At a glance

This perspective organizes a family of neuro-symbolic learning methods around a common concept representation. Its examples span images, videos, 3D scenes, human motion, and robot manipulation; their results come from separate systems and evaluations.

This edition follows the supplied arXiv manuscript. The journal article was published as Building Intelligent Agents with Neuro-Symbolic Concepts in Communications of the ACM.

Concept library connecting image understanding, 3D scene reasoning, video prediction, and robot actions.
Figure 1. Our framework for building intelligent agents by internalizing a vocabulary of “concepts,” which are represented as compositional programs and neural network embeddings. These concepts can be grounded in various domains: 2D images, videos, 3D scenes, and robotic actions, and be recombined based on different types of user queries: visually grounded questions, physical and causal reasoning questions, referring expressions, and manipulation instructions.

Key Insights

The central proposal is to give an agent reusable concepts for objects, relations, and actions. Neural representations connect those concepts to perception and control; typed programs specify how to combine them. The article uses earlier studies to illustrate data efficiency, compositional generalization, incremental concept learning, and transfer. A unified system sharing knowledge across all these domains remains a research goal.

1. Overview

Concepts are reusable units of knowledge: an object property such as orange, a relation such as left of, or an action such as placing one object beside another. Drawing on cognitive science and traditions of logic-based and probabilistic learning, the authors propose learning their sensory grounding separately from the rules for composing them. [24] [26] [27] [13] [20]

For example, moving an orange cylinder beside a bottle requires recognizing color and shape, locating the target relation, and controlling motion. A concept library can learn these parts from different data and recombine them for new instructions. The hoped-for benefits are lower supervision costs, novel combinations, gradual learning, and reuse across tasks. The following studies provide evidence for specific instances of these goals.

2. Neuro-Symbolic Concepts

A concept consists of parameters, a program, and neural representations. For an object property, a learned embedding and a filtering operation score whether an object belongs to the concept; a relation scores object pairs. The examples use cosine similarity between concept embeddings and object or pair representations.

Action concepts add a controller plus preconditions and postconditions. Placing an object to the left of another, for example, requires holding the first object, produces the target spatial relation, and invokes a learned movement controller. Other concepts can express those conditions.

Typed inputs and outputs—objects, events, actions, Booleans, and integers—constrain valid combinations. Neural modules provide perceptual judgments, while specified operations implement composition and operations such as counting. The four original formulas below define the general representation and three examples.

Object, relation, and action concepts combine neural embeddings with symbolic programs.
Figure 2. In a neuro-symbolic concept-centric framework, different concepts such as object categories, properties, relations, and actions are represented as a combination of programmatic and neural representations. In (a), the neural representation connects the concept with sensory and actuation representations. In (b), different concepts can be combined to form new compound concepts.
Equations
c=⟨parameter,program,neural-nets⟩.c=\langle\textit{parameter},\textit{program},\textit{neural-nets}\rangle.
orange=⟨{x},filter​(x,ORANGE),{ORANGE}⟩,\textit{orange}=\langle\{x\},\textit{filter}(x,\textbf{ORANGE}),\{\textbf{ORANGE}\}\rangle,
left-of=⟨{x,y},relate​(x,y,LEFT-OF),{LEFT-OF}⟩.\textit{left-of}=\langle\{x,y\},\textit{relate}(x,y,\textbf{LEFT-OF}),\{\textbf{LEFT-OF}\}\rangle.
put-left-of=⟨{x,y},{pre=holding(x),post=left-of(x,y),controller=PUT-LEFT-OF},{PUT-LEFT-OF}⟩.
Examples of compositional generalization, continual learning, and transfer between tasks.
Figure 3. Three challenges in (a): Compositional generalization. (b): Continual learning of concepts for reasoning. (c): Transfer learned concepts across domains.

Data efficiency.

Modularity supplies a learning bias: colors and shapes can be learned from images, while movement requires interaction data. Reusing these components can reduce the data needed for a composite task, provided the chosen decomposition fits that task.

Compositional generalization.

Programs can recombine familiar concepts into unfamiliar instructions and apply the same operations to scenes with more objects. This generalization depends on learning appropriate concept groundings and on the available programming primitives.

Continual learning.

Separate concept representations make it possible to add knowledge from different streams, including visual examples, language, and user preferences. Later experiments demonstrate adding new categories; broader revision of existing knowledge remains unresolved.

Zero-shot transfer.

The same object concept can be reused in caption understanding, VQA, or robot manipulation. Here, zero-shot reuse describes transferring an existing grounding: a new task can still require a newly trained parser or action policy.

3. Learning Neuro-Symbolic Visual Concepts

NS-CL learns object concepts and sentence interpretation from images paired with questions and answers. Its curriculum starts with simple questions and scenes containing at most three objects, then introduces relations and more complex compositions. Concept labels and ground-truth semantic programs are not supplied as direct supervision; the approach still uses object-centered perception, a designed VQA DSL, and a curriculum. [1] [12] [21]

Three components cooperate. A perception module extracts object representations; a semantic parser translates a question into a program; an executor evaluates that program against the scene. Concept classifiers estimate object attributes and relations, while deterministic functions combine their outputs.

The executor represents an intermediate object set as a soft mask, with each entry between zero and one. Filtering and relation operations transform these masks until a final operation returns the answer. This makes execution differentiable with respect to visual representations. Figure 5 distinguishes backpropagation through perception and concept learning from REINFORCE training of the discrete parser; Figure 6 shows an execution trace.

Curriculum progressing from object attributes to relations and compositional visual questions.
Figure 4. Humans learn visual concepts, words, and semantic parsing jointly and incrementally. I. Learning visual concepts (red vs. green) starts from looking at simple scenes, reading simple questions, and reasoning over contrastive examples (Fazly et al., 2010). II. Afterwards, we can interpret referential expressions based on the learned object-based concepts, and learn relational concepts (e.g., on the right of, the same material as). III Finally, we can interpret complex questions from visual cues by exploiting the compositional structure.
NS-CL architecture: object perception, a semantic parser, and an executor learn from questions and answers.
Figure 5. NS-CL uses neural symbolic reasoning to bridge the learning of visual concepts, words, and semantic parsing.
Execution trace: soft object masks pass through filtering, relation, and attribute-query operations.
Figure 6. The neuro-symbolic execution procedure of a program based on the visual representation and concept embeddings.
CLEVR accuracy bars for data efficiency and compositional generalization, with all numerical results labeled.
Figure 7. Data efficiency test and compositional generalization test for the Neuro-Symbolic Concept Learner (NS-CL).

Data efficiency.

In the reported 2019 CLEVR experiment, NS-CL reaches 98.9% test accuracy with 10% of the training data. Figure 7(a) shows 85.5% for the strongest displayed baseline: a 13.4 percentage-point difference. These are the earlier study’s benchmark results, not a new comparison against current systems. [17] [21]

Compositional generalization.

Figure 7(b) compares training on simpler scenes and questions with testing on more complex ones. NS-CL changes from 99.4% to 99.3% accuracy; the three displayed baselines fall from 96.1–98.8% to 90.9–92.9%. The experiment supports compositional generalization within this visual reasoning setting.

Continual learning.

FALCON meta-learns to add a visual concept from one or a few image–sentence examples plus statements relating it to known concepts. It first identifies the referred objects, then infers an embedding that fits both those examples and relationships such as category membership or shared properties. Evaluations include synthetic and real images. This is class-incremental learning: it adds categories rather than solving every form of continual adaptation. [25] [14] [35]

Transfer learning.

In a synthetic image-retrieval experiment, learned visual concepts are reused with fixed visual scenes. A new semantic parser learns a different DSL for captions describing object–relation–object triples. Such existential descriptions can involve multiple matching objects, whereas the original CLEVR question programs impose different assumptions. Transfer therefore reuses visual grounding but still trains the new language-to-program mapping.

4. Applications

The following applications extend the representation principles in separate published systems. Their capabilities should not be read as a single agent’s combined performance.

Accurate and robust image captioning.

The cited application performs image–caption retrieval. Pretrained parsers turn captions into structured representations of objects, properties, and relations. This factorization improves matching and resistance to misleading captions that differ from correct descriptions by only a few words. [38]

Video and counterfactual reasoning.

Object-centered representations can be coupled with learned physics models to predict future interactions or counterfactual outcomes, such as removing an object. One system jointly grounds objects and events from videos and language; a later system also infers physical properties online. [5] [9]

3D concept grounding.

Learning in 3D must handle scarce labels and language whose meaning depends on viewpoint or multiple objects. Disentangled object properties support learning concepts from few examples. NS3D adds operators for relations involving more than two objects, enabling viewpoint-sensitive reference and demonstrating transfer to 3D question answering in its evaluated setting. [29] [16]

Human motion.

Motion reasoning requires identifying relevant time intervals as well as recognizing actions. The cited model jointly learns temporal localization, motion concepts, attribute operators, and temporal relations, because motion sequences lack a universal segmentation into predefined actions. [11]

Robotic manipulation.

Robot manipulation systems use instruction structure to compose object recognition with action policies. One transfers learned visual concepts and trains additional movement policies through RL. Another uses pretrained VL models for recognition and reports better data efficiency and generalization to unseen object/task combinations. Transferred perception does not eliminate the need to learn control. [18] [37]

5. General Discussion

Neuro-symbolic concepts sit within a broader family of methods linking neural representations to rules, logical inference, probabilistic models, and algorithms. Related approaches inject rules into embeddings, learn interpretable concepts or logical programs, and combine neural predictions with satisfiability or answer-set solvers. Programs expose intermediate reasoning and allow external constraints; these properties can help inspection and safety, but do not by themselves guarantee correct or safe behavior. [6] [30] [4] [2] [32] [33] [36] [34] [40] [28] [39]

Compared with Neural Module Networks, which learn transformations for reasoning modules, the concept-centric approach separates learned grounding from explicitly specified composition operations. Its cost is a reliance on suitable DSL primitives and concept vocabularies. More expressive languages with loops and recursion expand what programs can represent, but complicate search and execution; differentiable relaxations can become computationally intractable or suffer unstable gradients. [3]

The article discusses three routes to expanding a vocabulary. Grammar-based learning uses a word’s syntactic role to constrain its meaning. Experience-driven induction constructs new concepts or reusable program fragments, including predicate invention from contrasting examples. Language models can propose programs and new symbols; those symbols still need representations grounded and learned from data. [23] [1] [10] [31] [8]

Open problems include concepts involving variable numbers of objects or whole layouts, automatically adapting curricula, revising or retracting old concepts, learning without explicit supervision, and reasoning rigorously under perceptual uncertainty. Existing class-incremental demonstrations address only part of continual learning. [15] [19] [35]

Cross-domain integration remains unfinished. Most image, video, motion, and robotics systems discussed here were developed separately and share little learned knowledge. The proposed next step is a scalable concept library that reuses abstractions across modalities—for example, a common notion of distance with different sensory groundings—and supports reasoning and planning. This is a research direction rather than a demonstrated general-purpose agent. [21] [16] [11] [5] [37] [22]

Acknowledgements.

The authors acknowledge support from AFOSR and ONR, including YIP and MURI awards; the NSF RI program; and an AI2050 Senior Fellowship. The original manuscript lists the grant identifiers.

Author Information

The manuscript identifies Jiayuan Mao and Joshua B. Tenenbaum with MIT and Jiajun Wu with Stanford University. These affiliations describe the supplied manuscript.

Concept demonstration

Open image ↗