Overview
The complete edition includes the main method, benchmark experiments, the supplementary language specification, additional data splits, and all qualitative success and failure examples. The central design separates language structure from learned 3D object and relation classifiers.
At a glance
NS3D converts a referring expression into a symbolic program and executes it with neural modules over segmented object point clouds. Explicit three-object relations handle viewpoint anchoring and spatial relationships such as between. On the structured SR3D benchmark, the method is strong for view-dependent references, small training sets, and selected distribution shifts. Reusing the trained grounding modules also answers a small new question-answering set without further neural-network training, after supplying task-specific parser examples and calibrating a threshold.
1 Introduction
Grounding a sentence in a 3D scene often requires identifying reference objects before applying a spatial relation. NS3D represents this order explicitly, using programs whose intermediate outputs are soft selections of scene objects. Its key extension is a learned ternary relation module that can consider a target, a spatial reference, and a viewpoint anchor together.
The approach is evaluated on object sets supplied with the scene, rather than discovering all objects from raw scans. Its modular structure supports diagnosing object classification separately from relational reasoning and reusing those components for new output types.
2 Related Work
Prior 3D grounding systems either operate on segmented object point clouds or model whole scenes and generate object proposals. NS3D belongs to the first group. It builds on neuro-symbolic concept learning [24], extending compositional neural execution to 3D relations of higher arity.
The comparison distinguishes object-centric methods from systems using full-scene modeling, pretrained detection, or additional image supervision. The evidence supports the chosen explicit relational representation; it does not establish that transformer architectures generally cannot represent multi-object reasoning.
3 NS3D
The input is a language expression and a set of colored object point clouds; the output is the uniquely referred object. A semantic parser creates the program, separate encoders extract object and spatial features, and a differentiable executor recursively applies learned predicates to select the target.
3.1 Semantic parser
A domain-specific language provides scene, filter, relate, ternary-relation, and viewpoint-anchor operations. Each operation consumes and returns object selections, allowing nested references to be expressed compositionally. Parsing identifies the reasoning structure, while learned visual modules determine which objects actually satisfy it.
Semantic parsing with Codex.
The paper uses the historical Codex language-to-code model [11] with Synchromesh [27] to constrain generated programs to valid syntax. A small set of sentence–program examples supplies the desired grammar and structures, and the pretrained model can recognize category and relation words absent from those examples.
The authors report perfect parsing in the tested structured setup. That claim does not imply universal semantic correctness for arbitrary language, and syntax constraints alone cannot guarantee the intended meaning. The parser’s extensive pretraining is separate from the small task-specific annotation counts.
3.2 3D object-centric encoder
An object PointNet++ encoder uses 1,024 XYZRGB points to capture category information. A separate shallower PointNet++ encoder uses XYZ coordinates for spatial relations, preventing the object-classification and relation tasks from sharing all features.
Two-layer perceptrons combine two or three spatial embeddings into binary or ternary relation features. The binary and ternary branches share the underlying spatial encoder. Explicit triples improve representation of viewpoint-dependent relations, although enumerating higher-arity combinations can increase computation as object counts grow.
Equations
3.3 Neural program executor
Execution represents an object selection by a score vector. The scene operator selects every object; filter intersects the input selection with a learned category score. Binary relate computes learned pairwise relation scores and softly weights them by the reference-object distribution.
The ternary operator similarly combines two reference selections with learned three-object relation scores. A viewpoint anchor is implemented through this operation, sometimes using the same object as both spatial reference and anchor. Recursively composing these modules yields the final target scores. The full original equations and execution diagrams preserve the exact soft-selection and intersection rules.
Equations
3.4 Training
Training uses object-category cross-entropy plus referred-target cross-entropy, with equal weights after initial object-classifier pretraining. Thus the experiments use object category labels as well as expression targets, despite the architecture also allowing target-only supervision in principle. Intermediate selections support diagnostic ablations. Training examples contain ten objects, while evaluation can include up to 88 objects.
4 Experiments
The principal benchmark is SR3D, the spatially structured variant of ReferIt3D [2]. The experiments separately study full-data referring expressions, reduced annotations, new combinations and room types, and a small question-answering transfer task.
4.1 3D referring expression comprehension
NS3D achieves 62.7% overall and 62.0% view-dependent accuracy. It leads the listed object-centric methods, while full-scene MVT and BUTD-DETR have higher overall scores, 64.5% and 67.0%. Its view-dependent score exceeds MVT’s 58.4% by 3.6 percentage points. BUTD-DETR also uses a more restricted 485-class setting, compared with the benchmark’s 607 classes.
Replacing learned object classification with ground-truth categories yields 96.9% overall but 82.3% view-dependent accuracy: categorization is a major bottleneck, while view-dependent reasoning remains imperfect. Sharing object and relation encoders reduces overall accuracy to 58.1%; removing the second reference argument reduces view-dependent accuracy to 56.2%. These ablations support separate features and explicit ternary structure.
| Overall | View-dep. | |
| - Without 3D scene modeling | ||
| NS3D (ours) | ||
| SAT [37] | ||
| TransRefer [16] | ||
| LanguageRefer [30] | ||
| 3DRefTransformer [1] | ||
| ReferIt3D [2] | ||
| + With 3D scene modeling | ||
| BUTD-DETR [20] | †† † We note that BUTD-DETR operates on a more constrained setting, assuming 485 classes instead of the full 607 classes in ReferIt3D. | |
| MVT [18] | ||
| 3DVG-Transformer [40] | ||
| InstanceRefer [39] | ||
| Text-Guided-GNNs [17] | ||
| All | V-dep. | All | V-dep. | All | V-dep. | All | V-dep. | All | V-dep. | |||||
| NS3D + full (Ours) | ||||||||||||||
| NS3D (Ours) | ||||||||||||||
| BUTD-DETR [20] | ||||||||||||||
| MVT [18] | ||||||||||||||
| SAT [37] | ||||||||||||||
| TransRefer [16] | ||||||||||||||
4.2 Data efficiency
At 1.5% of the expression training set, or 987 examples, NS3D reaches 52.0% accuracy versus the best listed baseline’s 27.5%, a 24.5-point gap. At 5%, it reaches 59.1%, compared with 62.7% at full data. A separately labeled NS3D + full variant pretrains object classification on the full training set and reaches 57.6% at 1.5% expression data. The two variants must remain distinct when interpreting annotation efficiency; baselines also differ in object pretraining.
4.3 Generalization
PAIRS trains on the most frequent 5% of object–relation–object combinations and tests the remaining combinations. NS3D reaches 59.9% overall, or 61.2% with full-set category pretraining, compared with 44.0% for the strongest listed baseline. This tests new combinations of known concepts rather than unrestricted new visual categories.
SCENE holds out living-room examples and yields 54.4% overall, or 56.3% with full-set category pretraining, versus 51.5% for the strongest listed baseline. Only 36 test examples in this split are view-dependent, making the corresponding percentages unstable. The separate category-pretraining variant carries additional supervision and should not be conflated with the restricted-training result.
4.4 Zero-shot transfer
A manually constructed fifty-question set tests existence, counting, object labels, and relation labels, including anchored and ternary variants. Ten sentence–program examples adapt the parser. The learned visual modules are reused unchanged, with new deterministic query operators and an existence/count threshold selected on a separate training-scene question set.
Exact-match accuracy is 68% with Codex, 30% with T5 finetuned on the same parser examples, and 16% with randomly initialized visual modules. The result demonstrates reuse without further neural training, not transfer without any task-specific examples or calibration. The small, closed-vocabulary evaluation does not establish broad open-ended 3D question answering.
| All | Exist | Count | Obj | Rel | |
| NS3D + Codex | |||||
| NS3D + T5 | |||||
| Random |
5 Conclusion
Explicit programs and specialized neural predicates improve data efficiency and selected generalization tests for 3D grounding. The main limitations are dependence on segmented objects and a specified concept language, remaining object-classification errors, pretrained parser resources, and narrow transfer evaluation. Combining the method with object localization and handling the much larger natural-language concept space remain future directions.
Acknowledgments.
The acknowledgments identify university, government, and industry support for the project.
Appendix A Domain-Specific Language
The supplementary tables retain every value type, primitive signature, semantic description, and implementation equation. Object selections are score vectors; category and relation arguments are concept names. Query operators return Boolean answers, integer counts, or labels. Existence and counting use threshold 0.8, selected on a separate question-answering set built from training scenes rather than on the fifty test questions.
Zero-shot transfer to 3D-QA.
New query functions reuse the learned category and relation classifiers. For example, querying an object type evaluates the category classifiers and selects the highest-scoring label. No new visual parameters are fitted, but the query definitions, parser examples, and calibrated threshold provide task-specific structure.
| Type | Representation | Semantics |
| object_set | Object score vectors. | Set of objects selected from a scene. |
| category | Concept names: table, chair, piano, etc. | Object-level properties. |
| relation | Concept names: near, left, behind, etc. | (Binary) Relationships between two objects. |
| t_relation | Concept names: between, anchor-left etc. | (Ternary) Relationships among three objects. |
| ∗boolean | Strings: yes, no. | Boolean values. |
| ∗integer | Integers: 0, 1, 2, etc. | Count of objects. |
| Signature & Implementation | Semantics | ||||
|
Return all objects in the 3D scene. | ||||
|
Return all objects satisfying a concept c. | ||||
|
Return all objects that satisfy the relationship rel between the object sets. | ||||
|
Return all objects that satisfy the ternary relationship trel between the object sets. | ||||
|
Return all objects that satisfy the relationship anchored on the first object set. | ||||
|
Return Yes/No corresponding to existence of object in the object set. | ||||
|
Return count of objects in the object set. | ||||
|
Return type of object in the object set. | ||||
|
Return relationship between the object sets. | ||||
|
Return ternary relationship between the object sets. |
Appendix B Experimental Details and Additional Results
The supplementary experiments document dataset construction, sparse-to-dense scene testing, a restricted natural-language subset, and additional visual examples. All supplementary figures and tables are preserved.
B.1 Dataset
The following dataset details separate the full benchmark, subsampled annotations, distribution-shift splits, and the new question set.
ReferIt3D datasets.
The full SR3D training setup uses 707 scenes with object-category annotations and 65,844 referring-expression examples. These annotations train both the category and grounding modules.
Data efficiency datasets.
Reduced-data experiments randomly retain 329, 987, 1,646, 3,292, or 6,584 expression examples, corresponding to 0.5%, 1.5%, 2.5%, 5%, and 10% of the training set. All use the same full SR3D test set.
PAIRS and SCENE generalization datasets.
PAIRS contains 16,200 training and 10,520 test examples split by frequent versus infrequent object–relation–object combinations. SCENE contains 57,125 training and 1,320 test examples split by the held-out living-room type. These are specified shifts within ReferIt3D rather than separate real-world deployments.
3D-QA dataset.
The fifty manually written test questions use ReferIt3D test scenes and a closed answer vocabulary. Templates cover presence of a related object, counts in a scene or relative to a reference, object categories identified by one or more references, and relations between objects. Anchored variants add a viewpoint reference. The scope is deliberately compositional and small.
B.2 Scene Complexity Generalization
Training on ten-object scenes and testing with as many as 88 objects yields 62.7% overall accuracy for NS3D. Under this sparse-training protocol, SAT reaches 44.4%, MVT 40.5%, and TransRefer 36.0%. BUTD-DETR is excluded because its full-scene input does not directly match the partial-scene setup. This demonstrates robustness to increased distractor sets within the tested benchmark.
B.3 NR3D Results
Parsing unrestricted NR3D natural language produces 91 function types and 5,892 concepts, creating a new long-tailed learning problem. The reported experiment therefore restricts examples to the functions and concepts already used by SR3D, leaving 3,659 training and 1,041 test examples. NS3D reaches 52.6% overall and 43.2% view-dependent accuracy, ahead of the listed baselines on this subset. These results are not a score for full NR3D.
B.4 Qualitative Examples
The supplementary figures preserve additional binary and ternary reference examples, view-dependent comparisons against four baselines, and question-answering successes and failures. Green and red annotations distinguish correct and incorrect selections or answers. These examples complement the aggregate metrics and expose errors that remain after compositional parsing.