Literature
Display
← All papers

What's Left? Concept Grounding with Logic-Enhanced Foundation Models

Joy Hsu · Jiayuan Mao · Joshua B. Tenenbaum · Jiajun Wu

· NeurIPS 2023 ↗

Reasoning7 min text · CondensedOriginal paper ↗
Contents

Overview

The complete article and supplementary material are condensed below, preserving all four figures, eight numbered tables, execution formulas, and references.

Refer to caption
Figure 1: LEFT is a unified concept learning and reasoning framework that grounds modular concepts across domains, and flexibly reasons with concepts across tasks with a foundation model.

At a glance

LEFT combines a language model that produces first-order logic programs, a differentiable logic executor, and trainable concept-grounding networks. The same interpreter and execution vocabulary can be reused for images, point clouds, motion sequences, and manipulation, while each domain supplies its own representations and learned groundings. Experiments show strong concept learning and transfer to new reasoning tasks built from learned concepts. Reusing the framework across domains should be distinguished from transferring one domain’s trained perception weights without adaptation.

1 Introduction

Concept words such as “left” express a reusable abstraction but require different perceptual or action groundings. Inference-only program-generation systems depend on available pretrained modules and cannot readily learn missing groundings. LEFT makes execution differentiable so task supervision can train modular concept networks. A general logical vocabulary replaces a separate hand-written reasoning language for each domain. This reduces custom program implementations while retaining domain-specific encoders, training data, and task interfaces.

2 Related Work

The paper connects language-model decomposition, general vision-language systems, and neuro-symbolic learning. Earlier program-based methods can be data efficient but often require a fixed domain language and manually implemented primitives. LEFT instead lets the language model propose lexical predicates and their argument counts, then instantiates generic neural grounding modules. Its comparison is with the tested systems and datasets, not all possible multimodal or program-generation approaches.

3 Logic-Enhanced Foundation Model (LEFT)

Three components separate language interpretation, compositional reasoning, and perceptual grounding. The language model emits hierarchical logic programs; a shared executor evaluates tensor-valued expressions; modality-specific encoders and concept networks provide their inputs. Gradients flow through execution to grounding networks, while the pretrained language interpreter is not fine-tuned.

Refer to caption
Figure 2: At the core of LEFT is a foundation model-powered language interpreter that generates first-order logic (FOL) programs from language queries, a domain-independent, differentiable FOL executor that operates on all types of entities, and domain-specific grounding modules that connect symbols in the first-order logic language with domain-specific features.

3.1 Domain-independent LLM interpreter

GPT-3.5 is prompted with syntax rules and simple generic examples, such as cats and apples, rather than examples tailored to each evaluated domain. It first simplifies or reasons about the query in language, then emits a logic program. Interpretation includes reference resolution and pattern inference. The generated predicate names and arities define which grounding modules are needed; the language model does not itself supply reliable visual recognition for those predicates.

3.2 Domain-independent first-order logic executor

A unary predicate yields one score per entity; binary relations yield an entity-by-entity matrix, and higher arities add tensor axes. The executor recursively combines them: exists uses maximum, forall uses minimum, conjunction/disjunction use elementwise minimum/maximum, and negation flips logits. Iota applies softmax to select an entity; count sums sigmoid-transformed scores. Equality uses a smooth comparison with scale 8 and tolerance 0.25. Most internal values remain logits for stability. Built-ins also describe an entity, change a reference view, or execute a grounded action. These soft operations enable gradient training; they are not exact Boolean inference over perfectly known predicates. Table 1 groups the two quantifiers under an existential description, while Appendix Table 7 specifies their distinct max/min semantics.

Name Syntax Description
variables x,y,z,⋯x,y,z,\cdots Variables that refer to entities in the domain.
∃,∀\exists,\forall exists​(var,expr)\textit{exists}(\textit{var},\textit{expr}) expr is an expression that contains the variable var; return true if there is at least one assignment of var that satisfies expr.
iota (ι\iota) iota​(var,expr)\textit{iota}(\textit{var},\textit{expr}) expr is an expression that contains the variable var; return an assignment to var’s that satisfies expr.
not not expr Compute the negation of an expression.
and, or expr1​ and expr2\textit{expr}_{1}\textit{~and~}\textit{expr}_{2} Compute the conjunction/disjunction of an expression.
count count​(var,expr)\textit{count}(\textit{var},\textit{expr}) Count the number of assignments to var that will make expr true.
=,<,>=,<,> eq​(expr1,expr2)\textit{eq}(\textit{expr}_{1},\textit{expr}_{2}) Built-in number comparison functions.
view view​(expr)\textit{view}(\textit{expr}) expr is an object, e.g., view(ιx.blue(x))\textit{view}(\iota x.\textit{blue}(x)); it changes the view direction to face the object.
describe describe​(expr)\textit{describe}(\textit{expr}) expr is a text description, e.g., describe(ιc.color(c,ιx.sphere(x)))\textit{describe}(\iota c.\textit{color}(c,\iota x.\textit{sphere}(x))); it returns the description (the color of the sphere).
do do​(expr)\textit{do}(\textit{expr}) expr is an action, e.g., do(ιa.pick-up(a,ιx.sphere(x)))\textit{do}(\iota a.\textit{pick-up}(a,\iota x.\textit{sphere}(x))); it executes the action (pick up the sphere).
concepts left ​(x,y)\textit{left }(x,y) A function that takes objects, actions, and texts as inputs and returns Boolean values.
Table 1: LEFT’s first-order logic language, a general reasoning language across domains.

3.3 Domain-specific grounding modules

Domain encoders extract entity and relation features: Faster R-CNN and DenseCLIP for image-related inputs, PointNet++ for point clouds, and a two-stream adaptive graph convolutional network for human motion. A separate multilayer perceptron is initialized for each concept and arity produced by the interpreter. Applying it to unary, binary, or higher-order features produces grounding scores. Supporting a new predicate name means creating a trainable module; accurate grounding still requires informative domain data or an appropriate pretrained module.

4 Experiments

Experiments evaluate four domains and six named datasets, followed by three unseen reasoning tasks in the CLEVR image domain. Metrics differ across tasks and should not be combined into one accuracy score.

Refer to caption
Figure 3: LEFT is evaluated on four broad categories of tasks, ranging from concept learning in 2D images, 3D point clouds, temporal motion sequences, and robotic manipulation.

4.1 Concept learning across domains

The same reasoning framework is trained with the appropriate grounding representation for each domain.

2D visual concept learning.

CLEVR and CLEVR-Humans evaluate exact-answer visual question answering over rendered objects. The former uses generated language; the latter uses more diverse human-written questions.

3D visual concept learning.

ReferIt3D evaluates selecting a target object from indoor-scene point clouds. SR3D contains synthetic spatial descriptions; NR3D contains natural descriptions. The experiment uses the NS3D subset of NR3D restricted to the SR3D concept set. View-dependent relations are represented with higher-arity grounding functions rather than a separately implemented logical operation.

Motion capture concept learning.

HumanMotionQA asks about joint-parameterized motion sequences, including actions, body parts, directions, and temporal relationships. Evaluation uses exact-match answers.

Robotic manipulation.

The CLIPort experiment uses sequential packing of Google Scanned Objects. Given a tabletop image and a language instruction, LEFT generates actions and is evaluated by average total task reward. This is a particular manipulation benchmark rather than unrestricted robot planning.

4.1.1 Accuracy

On the synthetic-language settings, LEFT scores 0.996 on CLEVR, 0.620 on SR3D, 0.582 on HumanMotionQA, and 0.793 task reward on CLIPort, broadly matching prior neuro-symbolic systems. Ground-truth programs are available for these comparisons, so they should not be interpreted as measuring language-model parsing alone. BUTD’s higher SR3D score uses a restricted 485-class setting instead of all 607 classes. For natural language, LEFT reports executable-only/all-query scores of 0.788/0.698 on CLEVR-Humans and 0.496/0.485 on NR3D. Counting invalid programs matters materially, especially in the former. MDETR reaches 0.817 on CLEVR-Humans and NS3D 0.526 on NR3D; LEFT does not lead every comparison.

Pre-def. CLEVR SR3D HumanMotionQA Cliport
Prior NS Yes 0.9920.992 (NSCL) 0.6270.627 (NS3D) 0.5780.578 (NSPose) 0.7970.797 (ProgramPort)
End-to-End No 0.9890.989 (MAC) 0.670\mathbf{0.670} (BUTD ** * BUTD operates on a constrained setting, assuming 485 classes instead of the full 607 classes in ReferIt3D.) 0.4300.430 (MotionCLIP) 0.7080.708 (Cliport)
LEFT No 0.996\mathbf{0.996} 0.620{0.620} 0.582\mathbf{0.582} 0.793\mathbf{0.793}
Table 2: Results comparing LEFT to prior top-performing neuro-symbolic and end-to-end works. Our unified LEFT framework yields strong performance on all settings without any pre-defined programs.
Pre-def. CLEVR-Humans NR3D
Prior NS Yes 0.6780.678 (NSVQA) 0.5260.526 (NS3D)
End-to-End No 0.817\mathbf{0.817} (MDETR) 0.4300.430 (MVT)
LEFT No 0.788{0.788} / 0.698{0.698} 0.496\mathbf{0.496} / 0.485\mathbf{0.485}
Table 3: Comparisons on natural language tasks.

4.1.2 Data efficiency

At 0.5% of SR3D data, LEFT reaches 0.410 versus NS3D’s 0.426 and much lower scores for the listed end-to-end models. With 10% of CLEVR, LEFT reaches 0.941 versus MAC’s 0.673, but NSCL reaches 0.989. NSCL’s additional attribute-group and mutual-exclusivity assumptions help in that setting. The results support substantial data efficiency from modular reasoning, while showing that useful domain priors can still improve performance.

Pre-def. 0.5% 1.5% 2.5% 5% 10% 100%
NS3D Yes 0.426{0.426} 0.5200.520 0.5560.556 0.591{0.591} 0.600{0.600} 0.6270.627
BUTD-DETR No 0.0830.083 0.1580.158 0.2590.259 0.3950.395 0.5280.528 0.670\mathbf{0.670}
MVT No 0.1610.161 0.2750.275 0.3220.322 0.3800.380 0.4910.491 0.6450.645
SAT No 0.1720.172 0.2600.260 0.2980.298 0.3300.330 0.3620.362 0.5790.579
LEFT No 0.410\mathbf{0.410} 0.521\mathbf{0.521} 0.565\mathbf{0.565} 0.579\mathbf{0.579} 0.591\mathbf{0.591} 0.6200.620
Table 4: Results on SR3D in data efficient settings. LEFT performs comparably to prior neuro-symbolic works, and outperforms end-to-end works significantly.
Pre-def. 10% 25% 50% 100%
NSCL Yes 0.989{0.989} 0.9920.992 0.9940.994 0.9920.992
TbD-Net Yes 0.5410.541 0.5600.560 0.9120.912 0.9910.991
MAC No 0.6730.673 0.9400.940 0.9740.974 0.9890.989
LEFT No 0.941\mathbf{0.941} 0.991\mathbf{0.991} 0.992\mathbf{0.992} 0.996\mathbf{0.996}
Table 5: Results on CLEVR in data efficient settings.

4.2 Reasoning generalization across tasks

Transfer reuses CLEVR-trained concept groundings for new task structures within the same visual domain.

CLEVR reasoning tasks.

Each transfer task has 100 evaluation examples. CLEVR-Ref selects an object from a description; CLEVR-Puzzles solves a joint assignment of four objects under properties and relations; CLEVR-RPM reasons about a text-described 3×3 attribute grid and checks whether an appropriate missing object exists in the image.

Refer to caption
Figure 4: The three zero-shot generalization tasks. As seen in the CLEVR-Ref example, LEFT’s domain-independent modules resolve the language into first-order logic and generate the answer.

4.2.1 Accuracy

With language-model programs, LEFT scores 0.94, 0.75, and 0.87 on Ref, Puzzles, and RPM. Ground-truth programs give 1.00, 0.92, and 1.00, exposing a meaningful interpretation gap, especially on puzzles. The tested VisProg, ViperGPT, and OpenFlamingo variants perform worse; some fail to emit executable programs or usable output formats. ViperGPT’s RPM result is 0.04 overall, combining 40% executable queries with 10% accuracy among them. These small, constructed tasks demonstrate recombination of existing concepts, not grounding of unseen predicates or broad domain transfer.

CLEVR-Ref CLEVR-Puzzles CLEVR-RPM
LEFT + GT programs 1.001.00 0.920.92 1.001.00
LEFT + LLM programs 0.94\mathbf{0.94} 0.75\mathbf{0.75} 0.87\mathbf{0.87}
VisProg (Gupta and Kembhavi, 2023) 0.350.35 0.270.27 0.510.51
ViperGPT (Surís et al., 2023) 0.080.08 0.340.34 0.040.04
OpenFlamingo 4-shot (Awadalla et al., 2023) N/A 0.540.54 0.520.52
OpenFlamingo 8-shot N/A 0.570.57 0.520.52
Table 6: Zero-shot generalization of LEFT to three unseen tasks; LEFT recomposes domain-independent programs and learned concepts with the LLM interpreter for flexible transfer.

5 Discussion

LEFT remains limited by semantic mistakes in language-model programs, insufficient grounding data, and separate concept representations that may fragment synonyms. Syntax failures can be detected and resampled, but semantically wrong executable programs are not repaired. Pretrained modules can be inserted when available; domains lacking such modules still require learning. The current architecture does not feed perception or execution evidence back into language interpretation. Pragmatic reasoning, speaker intent, and feedback-driven program correction are proposed extensions.

Acknowledgments.

The authors acknowledge feedback, institutional and industry support, and the listed fellowships and grants.

Appendix A LEFT

A.1 Function definitions

Table 7 gives every logical primitive’s signature, tensor implementation, and intended semantics. Concrete selected entities are combined with predicate scores by dot products; text-valued arguments use a generative form. Higher-arity predicates extend the same mechanism. All original formulas and definitions are preserved in the selectable table.

Signature & Implementation Semantics
exists(var, expr) ⟶\longrightarrow bb: boolean
Implementation: b=max​(ℰ​(expr))b=\textit{max}({\mathcal{E}}(\textit{expr}))
Return true if there is at least one assignment of var that satisfies expr.
forall(var, expr) ⟶\longrightarrow bb: boolean
Implementation: b=min​(ℰ​(expr))b=\textit{min}({\mathcal{E}}(\textit{expr}))
Return true if there is all assignments to var satisfy expr.
iota(var, expr) ⟶\longrightarrow ee: entity
Implementation: e=softmax​(ℰ​(expr))e=\textit{softmax}({\mathcal{E}}({\textit{expr}}))
Return an assignment to var’s that satisfies expr.
not(expr) ⟶\longrightarrow ee: boolean
Implementation: ei=−ℰ​(expr)ie_{i}=-{\mathcal{E}}(\textit{expr})_{i}, for all i∈{1,2,⋯,N}i\in\{1,2,\cdots,N\}
Compute the negation of an expression.
and(expr1\textit{{expr}}_{1}, expr2\textit{{expr}}_{2}) ⟶\longrightarrow ee: boolean
Implementation: ei=min​(ℰ​(expr1)i,ℰ​(expr2)i)e_{i}=\textit{min}({{\mathcal{E}}(\textit{{expr}}_{1})}_{i},{{\mathcal{E}}(\textit{{expr}}_{2})}_{i}), for all i∈{1,2,⋯,N}i\in\{1,2,\cdots,N\}
Compute the conjunction of two expressions.
or(expr1\textit{{expr}}_{1}, expr2\textit{{expr}}_{2}) ⟶\longrightarrow ee: boolean
Implementation: ei=max​(ℰ​(expr1)i,ℰ​(expr2)i)e_{i}=\textit{max}({{\mathcal{E}}(\textit{{expr}}_{1})}_{i},{{\mathcal{E}}(\textit{{expr}}_{2})}_{i}), for all i∈{1,2,⋯,N}i\in\{1,2,\cdots,N\}
Compute the disjunction of two expressions.
count(var, expr) ⟶\longrightarrow cc: integer
Implementation: c=∑i[σ⁡(ℰ​(expr)i)]c=\sum_{i}\left[\sigma({\mathcal{E}}(\textit{expr})_{i})\right]
Return the count of assignments to var’s that satisfies expr.
greater_than(expr1\textit{expr}_{1}, expr2\textit{expr}_{2}) ⟶\longrightarrow bb: boolean
Implementation: b=σ⁡(τ⋅(ℰ⁡(expr1)−ℰ⁡(expr2)−γ))b=\sigma(\tau\cdot({\mathcal{E}}(\textit{expr}_{1})-{\mathcal{E}}(\textit{expr}_{2})-\gamma))
Return a score indicating whether ℰ⁡(expr1){\mathcal{E}}(\textit{expr}_{1}) is greater than ℰ⁡(expr2){\mathcal{E}}(\textit{expr}_{2}).
view(expr)
Implementation: ℰ⁡(expr){\mathcal{E}}(\textit{expr})
Set the object returned by ℰ⁡(expr){\mathcal{E}}(\textit{expr}) as the object that the agent is looking at.
do(expr)
Implementation: ℰ⁡(expr){\mathcal{E}}(\textit{expr})
Perform action computed by ℰ⁡(expr){\mathcal{E}}(\textit{expr}).
describe(var, expr) ⟶\longrightarrow ee: text
Implementation: e=softmax​(ℰ​(expr))e=\textit{softmax}({\mathcal{E}}({\textit{expr}}))
Return the best assignment to var (e.g., colors, shapes, etc.) that describes the object returned by ℰ⁡(expr){\mathcal{E}}(\textit{expr}).
concept(varx\textit{var}_{x}) ⟶\longrightarrow ee: boolean
Implementation: ei=MLPconcept​(fiunary)e_{i}=\text{MLP}^{\text{concept}}\left(f^{\text{unary}}_{i}\right)
A unary function that takes objects, actions, and texts as inputs and returns Boolean values.
concept(varx\textit{var}_{x}, vary\textit{var}_{y}) ⟶\longrightarrow ee: boolean
Implementation: ei,j=MLPconcept​(fi,jbinary)e_{i,j}=\text{MLP}^{\text{concept}}(f^{\text{binary}}_{i,j})
A binary function that takes objects, actions, and texts as inputs and returns Boolean values.
Table 7: Function definitions for first-order logic programs in LEFT.

A.2 Broader impact

Because the language interpreter is a pretrained model outside the authors’ direct control, it can introduce harmful biases into generated programs. The authors state that this limitation is documented in their code release.

Appendix B Experiments

B.1 Error bars

Three-seed results report means and standard errors: 0.942±0.008 for CLEVR at 10% data and 0.996±0.001 for full CLEVR. CLEVR-Humans gives 0.780±0.003 on executable queries and 0.694±0.001 overall. These repeated-run summaries cover the listed CLEVR settings, not every domain or transfer task.

CLEVR-10% CLEVR-100% CLEVR-Humans
LEFT 0.942±0.008\text{0.942}_{\pm\text{0.008}} 0.996±0.001\text{0.996}_{\pm\text{0.001}} 0.780±0.003{0.780}_{\pm\text{0.003}} / 0.694±0.001{0.694}_{\pm\text{0.001}}
Table 8: Results on the CLEVR dataset with standard errors. Average performance and standard errors are computed based on three random seeds.

B.2 Transfer task construction

Transfer examples are generated from existing CLEVR attributes and relations.

CLEVR-Ref

CLEVR-Ref uses four template families: direct property descriptions, one relation, two relations, and a nested relation with a referring pronoun. It selects a subset of CLEVR-Ref+ templates and excludes ordinal concepts. Object descriptions and relations are filled from the dataset vocabulary.

CLEVR-Puzzles

Each puzzle specifies four object descriptions and three relations. No individual object description uniquely identifies its referent, but the combined constraints have one global assignment. This tests joint relational reasoning rather than independent property lookup.

CLEVR-RPM

RPM examples describe eight positions in a nine-object grid and ask for a scene object that completes the last position. Row and column progression dimensions are sampled from color, shape, material, and size. Pattern inference is mainly expressed in language before visual grounding checks the candidate.

Appendix C Model

C.1 Implementation details

The implementation separates GPT-3.5 interpretation, concept initialization from parsed arity, modality-specific features, and differentiable execution. This architecture permits reuse of the interpreter and executor while groundings vary by domain.

C.2 Training details

Training uses the official dataset splits and 128-dimensional concept embeddings. Learning rates follow the prior neuro-symbolic codebases used as baselines; detailed per-domain settings are delegated to the released code.

C.3 Compute

Each experiment uses one NVIDIA Titan RTX from an internal cluster; the source does not give total training hours.

C.4 Code

The authors provide public code and additional details through the project website.

Concept demonstration

Open image ↗