Overview
The complete article and supplementary material are condensed below, preserving all four figures, eight numbered tables, execution formulas, and references.
At a glance
LEFT combines a language model that produces first-order logic programs, a differentiable logic executor, and trainable concept-grounding networks. The same interpreter and execution vocabulary can be reused for images, point clouds, motion sequences, and manipulation, while each domain supplies its own representations and learned groundings. Experiments show strong concept learning and transfer to new reasoning tasks built from learned concepts. Reusing the framework across domains should be distinguished from transferring one domain’s trained perception weights without adaptation.
1 Introduction
Concept words such as “left” express a reusable abstraction but require different perceptual or action groundings. Inference-only program-generation systems depend on available pretrained modules and cannot readily learn missing groundings. LEFT makes execution differentiable so task supervision can train modular concept networks. A general logical vocabulary replaces a separate hand-written reasoning language for each domain. This reduces custom program implementations while retaining domain-specific encoders, training data, and task interfaces.
2 Related Work
The paper connects language-model decomposition, general vision-language systems, and neuro-symbolic learning. Earlier program-based methods can be data efficient but often require a fixed domain language and manually implemented primitives. LEFT instead lets the language model propose lexical predicates and their argument counts, then instantiates generic neural grounding modules. Its comparison is with the tested systems and datasets, not all possible multimodal or program-generation approaches.
3 Logic-Enhanced Foundation Model (LEFT)
Three components separate language interpretation, compositional reasoning, and perceptual grounding. The language model emits hierarchical logic programs; a shared executor evaluates tensor-valued expressions; modality-specific encoders and concept networks provide their inputs. Gradients flow through execution to grounding networks, while the pretrained language interpreter is not fine-tuned.
3.1 Domain-independent LLM interpreter
GPT-3.5 is prompted with syntax rules and simple generic examples, such as cats and apples, rather than examples tailored to each evaluated domain. It first simplifies or reasons about the query in language, then emits a logic program. Interpretation includes reference resolution and pattern inference. The generated predicate names and arities define which grounding modules are needed; the language model does not itself supply reliable visual recognition for those predicates.
3.2 Domain-independent first-order logic executor
A unary predicate yields one score per entity; binary relations yield an entity-by-entity matrix, and higher arities add tensor axes. The executor recursively combines them: exists uses maximum, forall uses minimum, conjunction/disjunction use elementwise minimum/maximum, and negation flips logits. Iota applies softmax to select an entity; count sums sigmoid-transformed scores. Equality uses a smooth comparison with scale 8 and tolerance 0.25. Most internal values remain logits for stability. Built-ins also describe an entity, change a reference view, or execute a grounded action. These soft operations enable gradient training; they are not exact Boolean inference over perfectly known predicates. Table 1 groups the two quantifiers under an existential description, while Appendix Table 7 specifies their distinct max/min semantics.
| Name | Syntax | Description |
| variables | Variables that refer to entities in the domain. | |
| expr is an expression that contains the variable var; return true if there is at least one assignment of var that satisfies expr. | ||
| iota () | expr is an expression that contains the variable var; return an assignment to var’s that satisfies expr. | |
| not | not expr | Compute the negation of an expression. |
| and, or | Compute the conjunction/disjunction of an expression. | |
| count | Count the number of assignments to var that will make expr true. | |
| Built-in number comparison functions. | ||
| view | expr is an object, e.g., ; it changes the view direction to face the object. | |
| describe | expr is a text description, e.g., ; it returns the description (the color of the sphere). | |
| do | expr is an action, e.g., ; it executes the action (pick up the sphere). | |
| concepts | A function that takes objects, actions, and texts as inputs and returns Boolean values. |
3.3 Domain-specific grounding modules
Domain encoders extract entity and relation features: Faster R-CNN and DenseCLIP for image-related inputs, PointNet++ for point clouds, and a two-stream adaptive graph convolutional network for human motion. A separate multilayer perceptron is initialized for each concept and arity produced by the interpreter. Applying it to unary, binary, or higher-order features produces grounding scores. Supporting a new predicate name means creating a trainable module; accurate grounding still requires informative domain data or an appropriate pretrained module.
4 Experiments
Experiments evaluate four domains and six named datasets, followed by three unseen reasoning tasks in the CLEVR image domain. Metrics differ across tasks and should not be combined into one accuracy score.
4.1 Concept learning across domains
The same reasoning framework is trained with the appropriate grounding representation for each domain.
2D visual concept learning.
CLEVR and CLEVR-Humans evaluate exact-answer visual question answering over rendered objects. The former uses generated language; the latter uses more diverse human-written questions.
3D visual concept learning.
ReferIt3D evaluates selecting a target object from indoor-scene point clouds. SR3D contains synthetic spatial descriptions; NR3D contains natural descriptions. The experiment uses the NS3D subset of NR3D restricted to the SR3D concept set. View-dependent relations are represented with higher-arity grounding functions rather than a separately implemented logical operation.
Motion capture concept learning.
HumanMotionQA asks about joint-parameterized motion sequences, including actions, body parts, directions, and temporal relationships. Evaluation uses exact-match answers.
Robotic manipulation.
The CLIPort experiment uses sequential packing of Google Scanned Objects. Given a tabletop image and a language instruction, LEFT generates actions and is evaluated by average total task reward. This is a particular manipulation benchmark rather than unrestricted robot planning.
4.1.1 Accuracy
On the synthetic-language settings, LEFT scores 0.996 on CLEVR, 0.620 on SR3D, 0.582 on HumanMotionQA, and 0.793 task reward on CLIPort, broadly matching prior neuro-symbolic systems. Ground-truth programs are available for these comparisons, so they should not be interpreted as measuring language-model parsing alone. BUTD’s higher SR3D score uses a restricted 485-class setting instead of all 607 classes. For natural language, LEFT reports executable-only/all-query scores of 0.788/0.698 on CLEVR-Humans and 0.496/0.485 on NR3D. Counting invalid programs matters materially, especially in the former. MDETR reaches 0.817 on CLEVR-Humans and NS3D 0.526 on NR3D; LEFT does not lead every comparison.
| Pre-def. | CLEVR | SR3D | HumanMotionQA | Cliport | |
| Prior NS | Yes | (NSCL) | (NS3D) | (NSPose) | (ProgramPort) |
| End-to-End | No | (MAC) | (BUTD ** * BUTD operates on a constrained setting, assuming 485 classes instead of the full 607 classes in ReferIt3D.) | (MotionCLIP) | (Cliport) |
| LEFT | No |
4.1.2 Data efficiency
At 0.5% of SR3D data, LEFT reaches 0.410 versus NS3D’s 0.426 and much lower scores for the listed end-to-end models. With 10% of CLEVR, LEFT reaches 0.941 versus MAC’s 0.673, but NSCL reaches 0.989. NSCL’s additional attribute-group and mutual-exclusivity assumptions help in that setting. The results support substantial data efficiency from modular reasoning, while showing that useful domain priors can still improve performance.
4.2 Reasoning generalization across tasks
Transfer reuses CLEVR-trained concept groundings for new task structures within the same visual domain.
CLEVR reasoning tasks.
Each transfer task has 100 evaluation examples. CLEVR-Ref selects an object from a description; CLEVR-Puzzles solves a joint assignment of four objects under properties and relations; CLEVR-RPM reasons about a text-described 3×3 attribute grid and checks whether an appropriate missing object exists in the image.
4.2.1 Accuracy
With language-model programs, LEFT scores 0.94, 0.75, and 0.87 on Ref, Puzzles, and RPM. Ground-truth programs give 1.00, 0.92, and 1.00, exposing a meaningful interpretation gap, especially on puzzles. The tested VisProg, ViperGPT, and OpenFlamingo variants perform worse; some fail to emit executable programs or usable output formats. ViperGPT’s RPM result is 0.04 overall, combining 40% executable queries with 10% accuracy among them. These small, constructed tasks demonstrate recombination of existing concepts, not grounding of unseen predicates or broad domain transfer.
| CLEVR-Ref | CLEVR-Puzzles | CLEVR-RPM | |
| LEFT + GT programs | |||
| LEFT + LLM programs | |||
| VisProg (Gupta and Kembhavi, 2023) | |||
| ViperGPT (Surís et al., 2023) | |||
| OpenFlamingo 4-shot (Awadalla et al., 2023) | N/A | ||
| OpenFlamingo 8-shot | N/A |
5 Discussion
LEFT remains limited by semantic mistakes in language-model programs, insufficient grounding data, and separate concept representations that may fragment synonyms. Syntax failures can be detected and resampled, but semantically wrong executable programs are not repaired. Pretrained modules can be inserted when available; domains lacking such modules still require learning. The current architecture does not feed perception or execution evidence back into language interpretation. Pragmatic reasoning, speaker intent, and feedback-driven program correction are proposed extensions.
Acknowledgments.
The authors acknowledge feedback, institutional and industry support, and the listed fellowships and grants.
Appendix A LEFT
A.1 Function definitions
Table 7 gives every logical primitive’s signature, tensor implementation, and intended semantics. Concrete selected entities are combined with predicate scores by dot products; text-valued arguments use a generative form. Higher-arity predicates extend the same mechanism. All original formulas and definitions are preserved in the selectable table.
| Signature & Implementation | Semantics | ||
|
Return true if there is at least one assignment of var that satisfies expr. | ||
|
Return true if there is all assignments to var satisfy expr. | ||
|
Return an assignment to var’s that satisfies expr. | ||
|
Compute the negation of an expression. | ||
|
Compute the conjunction of two expressions. | ||
|
Compute the disjunction of two expressions. | ||
|
Return the count of assignments to var’s that satisfies expr. | ||
|
Return a score indicating whether is greater than . | ||
|
Set the object returned by as the object that the agent is looking at. | ||
|
Perform action computed by . | ||
|
Return the best assignment to var (e.g., colors, shapes, etc.) that describes the object returned by . | ||
|
A unary function that takes objects, actions, and texts as inputs and returns Boolean values. | ||
|
A binary function that takes objects, actions, and texts as inputs and returns Boolean values. |
A.2 Broader impact
Because the language interpreter is a pretrained model outside the authors’ direct control, it can introduce harmful biases into generated programs. The authors state that this limitation is documented in their code release.
Appendix B Experiments
B.1 Error bars
Three-seed results report means and standard errors: 0.942±0.008 for CLEVR at 10% data and 0.996±0.001 for full CLEVR. CLEVR-Humans gives 0.780±0.003 on executable queries and 0.694±0.001 overall. These repeated-run summaries cover the listed CLEVR settings, not every domain or transfer task.
B.2 Transfer task construction
Transfer examples are generated from existing CLEVR attributes and relations.
CLEVR-Ref
CLEVR-Ref uses four template families: direct property descriptions, one relation, two relations, and a nested relation with a referring pronoun. It selects a subset of CLEVR-Ref+ templates and excludes ordinal concepts. Object descriptions and relations are filled from the dataset vocabulary.
CLEVR-Puzzles
Each puzzle specifies four object descriptions and three relations. No individual object description uniquely identifies its referent, but the combined constraints have one global assignment. This tests joint relational reasoning rather than independent property lookup.
CLEVR-RPM
RPM examples describe eight positions in a nine-object grid and ask for a scene object that completes the last position. Row and column progression dimensions are sampled from color, shape, material, and size. Pattern inference is mainly expressed in language before visual grounding checks the candidate.
Appendix C Model
C.1 Implementation details
The implementation separates GPT-3.5 interpretation, concept initialization from parsed arity, modality-specific features, and differentiable execution. This architecture permits reuse of the interpreter and executor while groundings vary by domain.
C.2 Training details
Training uses the official dataset splits and 128-dimensional concept embeddings. Learning rates follow the prior neuro-symbolic codebases used as baselines; detailed per-domain settings are delegated to the released code.
C.3 Compute
Each experiment uses one NVIDIA Titan RTX from an internal cluster; the source does not give total training hours.
C.4 Code
The authors provide public code and additional details through the project website.