Literature
Display
← All papers

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations

Joy Hsu · Jiayuan Mao · Jiajun Wu

· CVPR 2023 ↗

Reasoning8 min text · CondensedOriginal paper ↗
Contents

Overview

The complete edition includes the main method, benchmark experiments, the supplementary language specification, additional data splits, and all qualitative success and failure examples. The central design separates language structure from learned 3D object and relation classifiers.

At a glance

NS3D converts a referring expression into a symbolic program and executes it with neural modules over segmented object point clouds. Explicit three-object relations handle viewpoint anchoring and spatial relationships such as between. On the structured SR3D benchmark, the method is strong for view-dependent references, small training sets, and selected distribution shifts. Reusing the trained grounding modules also answers a small new question-answering set without further neural-network training, after supplying task-specific parser examples and calibrating a threshold.

1 Introduction

Grounding a sentence in a 3D scene often requires identifying reference objects before applying a spatial relation. NS3D represents this order explicitly, using programs whose intermediate outputs are soft selections of scene objects. Its key extension is a learned ternary relation module that can consider a target, a spatial reference, and a viewpoint anchor together.

The approach is evaluated on object sets supplied with the scene, rather than discovering all objects from raw scans. Its modular structure supports diagnosing object classification separately from relational reasoning and reusing those components for new output types.

Refer to caption
Figure 1: NS3D achieves grounding of 3D objects and relations in complex scenes, while showing state-of-the-art results in data efficiency, generalization, and zero-shot transfer.
Refer to caption
Figure 2: NS3D is composed of three main components. a) A semantic parser parses the input language into a symbolic program. b) A 3D object-centric encoder takes input objects and learns object, relation, and ternary relation features. c) A neural program executor executes the symbolic program with the learned features to retrieve the target referred object.

2 Related Work

Prior 3D grounding systems either operate on segmented object point clouds or model whole scenes and generate object proposals. NS3D belongs to the first group. It builds on neuro-symbolic concept learning [24], extending compositional neural execution to 3D relations of higher arity.

The comparison distinguishes object-centric methods from systems using full-scene modeling, pretrained detection, or additional image supervision. The evidence supports the chosen explicit relational representation; it does not establish that transformer architectures generally cannot represent multi-object reasoning.

3 NS3D

The input is a language expression and a set of colored object point clouds; the output is the uniquely referred object. A semantic parser creates the program, separate encoders extract object and spatial features, and a differentiable executor recursively applies learned predicates to select the target.

3.1 Semantic parser

A domain-specific language provides scene, filter, relate, ternary-relation, and viewpoint-anchor operations. Each operation consumes and returns object selections, allowing nested references to be expressed compositionally. Parsing identifies the reasoning structure, while learned visual modules determine which objects actually satisfy it.

Original figure
Figure 3: The NS3D semantic parser leverages Codex to parse input language into symbolic programs.

Semantic parsing with Codex.

The paper uses the historical Codex language-to-code model [11] with Synchromesh [27] to constrain generated programs to valid syntax. A small set of sentence–program examples supplies the desired grammar and structures, and the pretrained model can recognize category and relation words absent from those examples.

The authors report perfect parsing in the tested structured setup. That claim does not imply universal semantic correctness for arbitrary language, and syntax constraints alone cannot guarantee the intended meaning. The parser’s extensive pretraining is separate from the small task-specific annotation counts.

3.2 3D object-centric encoder

An object PointNet++ encoder uses 1,024 XYZRGB points to capture category information. A separate shallower PointNet++ encoder uses XYZ coordinates for spatial relations, preventing the object-classification and relation tasks from sharing all features.

Two-layer perceptrons combine two or three spatial embeddings into binary or ternary relation features. The binary and ternary branches share the underlying spatial encoder. Explicit triples improve representation of viewpoint-dependent relations, although enumerating higher-arity combinations can increase computation as object counts grow.

Refer to caption
Figure 4: The NS3D object-centric encoder learns object, relation, and ternary relation features from input object point clouds.
Equations
fiobj=ℰobj​(Oi),∀Oi∈𝒪.f^{\text{obj}}_{i}=\mathcal{E}^{\text{obj}}(O_{i}),\ \forall O_{i}\in\mathcal{O}.
fi,jrel=MLPbinary​(concat⁡(ℰrel​(Oipos),ℰrel​(Ojpos))),f^{\text{rel}}_{i,j}=\mathrm{MLP}^{\text{binary}}\left(\mathrm{concat}\left({\mathcal{E}}^{\text{rel}}(O^{\text{pos}}_{i}),{\mathcal{E}}^{\text{rel}}(O^{\text{pos}}_{j})\right)\right),
gi,jternary=\displaystyle g^{\text{ternary}}_{i,j}= MLPternary​(fi,jrel),\displaystyle\mathrm{MLP}^{\text{ternary}}\left(f^{\text{rel}}_{i,j}\right),
fi,j,kternary=\displaystyle f^{\text{ternary}}_{i,j,k}= concat⁡(gi,jternary,gj,kternary,gi,kternary),\displaystyle\mathrm{concat}\left(g^{\text{ternary}}_{i,j},g^{\text{ternary}}_{j,k},g^{\text{ternary}}_{i,k}\right),

3.3 Neural program executor

Execution represents an object selection by a score vector. The scene operator selects every object; filter intersects the input selection with a learned category score. Binary relate computes learned pairwise relation scores and softly weights them by the reference-object distribution.

The ternary operator similarly combines two reference selections with learned three-object relation scores. A viewpoint anchor is implemented through this operation, sometimes using the same object as both spatial reference and anchor. Recursively composing these modules yields the final target scores. The full original equations and execution diagrams preserve the exact soft-selection and intersection rules.

Refer to caption
Figure 5: The NS3D neural program executor executes the symbolic program recursively with the learned 3D features, and returns the target referred object 𝒯\mathcal{T}.
Equations
yi=min⁡(xi,probic)=min⁡(xi,MLPc​(fiobj)).y_{i}=\min\left(x_{i},\textit{prob}^{\text{c}}_{i}\right)=\min\left(x_{i},\mathrm{MLP}^{\text{c}}\left(f^{\text{obj}}_{i}\right)\right).
probi,jrel=\displaystyle\textit{prob}^{\text{rel}}_{i,j}= MLPrel​(fi,jrel)\displaystyle\mathrm{MLP}^{\text{rel}}(f^{\text{rel}}_{i,j})
yi=\displaystyle y_{i}= min⁡(xit,∑jsx​(xr)j⋅probi,jrel),\displaystyle\min\left(x^{t}_{i},\sum_{j}\textit{sx}(x^{r})_{j}\cdot\textit{prob}^{\text{rel}}_{i,j}\right),
probi,j,ktrel=MLPtrel​(fi,j,kternary)\displaystyle\textit{prob}^{\text{trel}}_{i,j,k}=\mathrm{MLP}^{\text{trel}}(f^{\text{ternary}}_{i,j,k})
yi=min⁡(xit,∑j∑ksx​(xr​1)j⋅sx​(xr​2)k⋅probi,j,ktrel).\displaystyle y_{i}=\min\left(x^{t}_{i},\sum_{j}\sum_{k}\textit{sx}(x^{r1})_{j}\cdot\textit{sx}(x^{r2})_{k}\cdot\textit{prob}^{\text{trel}}_{i,j,k}\right).

3.4 Training

Training uses object-category cross-entropy plus referred-target cross-entropy, with equal weights after initial object-classifier pretraining. Thus the experiments use object category labels as well as expression targets, despite the architecture also allowing target-only supervision in principle. Intermediate selections support diagnostic ablations. Training examples contain ten objects, while evaluation can include up to 88 objects.

4 Experiments

The principal benchmark is SR3D, the spatially structured variant of ReferIt3D [2]. The experiments separately study full-data referring expressions, reduced annotations, new combinations and room types, and a small question-answering transfer task.

4.1 3D referring expression comprehension

NS3D achieves 62.7% overall and 62.0% view-dependent accuracy. It leads the listed object-centric methods, while full-scene MVT and BUTD-DETR have higher overall scores, 64.5% and 67.0%. Its view-dependent score exceeds MVT’s 58.4% by 3.6 percentage points. BUTD-DETR also uses a more restricted 485-class setting, compared with the benchmark’s 607 classes.

Replacing learned object classification with ground-truth categories yields 96.9% overall but 82.3% view-dependent accuracy: categorization is a major bottleneck, while view-dependent reasoning remains imperfect. Sharing object and relation encoders reduces overall accuracy to 58.1%; removing the second reference argument reduces view-dependent accuracy to 56.2%. These ablations support separate features and explicit ternary structure.

Overall View-dep.
- Without 3D scene modeling
NS3D (ours) 0.627\mathbf{0.627} 0.620\mathbf{0.620}
SAT [37] 0.5790.579 0.4920.492
TransRefer [16] 0.5740.574 0.4990.499
LanguageRefer [30] 0.5600.560 0.4920.492
3DRefTransformer [1] 0.4700.470 0.4430.443
ReferIt3D [2] 0.4080.408 0.3920.392
+ With 3D scene modeling
BUTD-DETR [20] 0.670\mathbf{0.670}†† † We note that BUTD-DETR operates on a more constrained setting, assuming 485 classes instead of the full 607 classes in ReferIt3D. 0.5300.530
MVT [18] 0.6450.645 0.584\mathbf{0.584}
3DVG-Transformer [40] 0.5140.514 0.4460.446
InstanceRefer [39] 0.4800.480 0.4540.454
Text-Guided-GNNs [17] 0.4500.450 0.4580.458
Table 1: NS3D yields the highest overall accuracy on the SR3D task among object-centric methods, and state-of-the-art view-dependent accuracy across all methods.
Refer to caption
Figure 6: Example of NS3D’s execution trace on a view-dependent 3D-REC task. NS3D returns the correct target cabinet object.
0.5%0.5\% 1.5%1.5\% 2.5%2.5\% 5%5\% 10%10\%
All V-dep. All V-dep. All V-dep. All V-dep. All V-dep.
NS3D + full (Ours) 0.503\mathbf{0.503} 0.395\mathbf{0.395} 0.576\mathbf{0.576} 0.468\mathbf{0.468} 0.587\mathbf{0.587} 0.505\mathbf{0.505} 0.597\mathbf{0.597} 0.505\mathbf{0.505} 0.612\mathbf{0.612} 0.552\mathbf{0.552}
NS3D (Ours) 0.4260.426 0.3750.375 0.5200.520 0.4240.424 0.5560.556 0.4830.483 0.5910.591 0.4930.493 0.6000.600 0.5270.527
BUTD-DETR [20] 0.0830.083 0.0890.089 0.1580.158 0.1380.138 0.2590.259 0.2230.223 0.3950.395 0.3020.302 0.5280.528 0.4200.420
MVT [18] 0.1610.161 0.1180.118 0.2750.275 0.1990.199 0.3220.322 0.2700.270 0.3800.380 0.3750.375 0.4910.491 0.4260.426
SAT [37] 0.1720.172 0.1490.149 0.2600.260 0.2540.254 0.2980.298 0.2730.273 0.3300.330 0.3090.309 0.3620.362 0.3340.334
TransRefer [16] 0.1880.188 0.1520.152 0.2680.268 0.2330.233 0.3050.305 0.2780.278 0.3620.362 0.3800.380 0.3900.390 0.3780.378
Table 2: Data efficiency results of NS3D compared to prior works, with 0.5%, 1.5%, 2.5%, 5%, and 10% of train data. We report two variations of NS3D, with object classification pretrained on the full dataset and pretrained on the specified data-efficient train set.
Overall View-dep.
NS3D w/ gt obj. cls. 0.9690.969 0.8230.823
NS3D w/o sep. feat. 0.5810.581 0.5120.512
NS3D w/o ternary arg. 0.6090.609 0.5620.562
NS3D (Full) 0.6270.627 0.6200.620
Table 3: Ablation on NS3D with groundtruth object classification results in filter, without separation of object and relation features, and without leveraging the correct arity for ternary operations.

4.2 Data efficiency

At 1.5% of the expression training set, or 987 examples, NS3D reaches 52.0% accuracy versus the best listed baseline’s 27.5%, a 24.5-point gap. At 5%, it reaches 59.1%, compared with 62.7% at full data. A separately labeled NS3D + full variant pretrains object classification on the full training set and reaches 57.6% at 1.5% expression data. The two variants must remain distinct when interpreting annotation efficiency; baselines also differ in object pretraining.

Refer to caption
Figure 7: NS3D outperforms prior works by a large margin with 0.5%, 1.5%, 2.5%, 5%, and 10% of train data.

4.3 Generalization

PAIRS trains on the most frequent 5% of object–relation–object combinations and tests the remaining combinations. NS3D reaches 59.9% overall, or 61.2% with full-set category pretraining, compared with 44.0% for the strongest listed baseline. This tests new combinations of known concepts rather than unrestricted new visual categories.

SCENE holds out living-room examples and yields 54.4% overall, or 56.3% with full-set category pretraining, versus 51.5% for the strongest listed baseline. Only 36 test examples in this split are view-dependent, making the corresponding percentages unstable. The separate category-pretraining variant carries additional supervision and should not be conflated with the restricted-training result.

Pairs Scene
All V-dep. All V-dep
NS3D + full (Ours) 0.612\mathbf{0.612} 0.635\mathbf{0.635} 0.563\mathbf{0.563} 0.5830.583
NS3D (Ours) 0.5990.599 0.6200.620 0.5440.544 0.611\mathbf{0.611}
BUTD-DETR [20] 0.4400.440 0.4230.423 0.5150.515 0.5830.583
MVT [18] 0.4200.420 0.3530.353 0.5020.502 0.5000.500
SAT [37] 0.3590.359 0.3800.380 0.4510.451 0.5000.500
TransRefer [16] 0.3220.322 0.3440.344 0.3840.384 0.3610.361
Table 4: Generalization performance to unseen object co-occurrence pairs and scene. NS3D outperforms all baselines.

4.4 Zero-shot transfer

A manually constructed fifty-question set tests existence, counting, object labels, and relation labels, including anchored and ternary variants. Ten sentence–program examples adapt the parser. The learned visual modules are reused unchanged, with new deterministic query operators and an existence/count threshold selected on a separate training-scene question set.

Exact-match accuracy is 68% with Codex, 30% with T5 finetuned on the same parser examples, and 16% with randomly initialized visual modules. The result demonstrates reuse without further neural training, not transfer without any task-specific examples or calibration. The small, closed-vocabulary evaluation does not establish broad open-ended 3D question answering.

All Exist Count Obj Rel
NS3D + Codex 0.68\mathbf{0.68} 0.80\mathbf{0.80} 0.67\mathbf{0.67} 0.60\mathbf{0.60} 0.60\mathbf{0.60}
NS3D + T5 0.300.30 0.400.40 0.130.13 0.400.40 0.300.30
Random 0.160.16 0.400.40 0.070.07 0.000.00 0.100.10
Table 5: NS3D’s zero-shot transfer performance on the 3D-QA task, with comparison of semantic parsers (Codex vs T5), and with a randomly initialized model as baseline.
Refer to caption
Figure 8: Examples of 4 questions types from the 3D-QA task.

5 Conclusion

Explicit programs and specialized neural predicates improve data efficiency and selected generalization tests for 3D grounding. The main limitations are dependence on segmented objects and a specified concept language, remaining object-classification errors, pretrained parser resources, and narrow transfer evaluation. Combining the method with object localization and handling the much larger natural-language concept space remain future directions.

Acknowledgments.

The acknowledgments identify university, government, and industry support for the project.

Appendix A Domain-Specific Language

The supplementary tables retain every value type, primitive signature, semantic description, and implementation equation. Object selections are score vectors; category and relation arguments are concept names. Query operators return Boolean answers, integer counts, or labels. Existence and counting use threshold 0.8, selected on a separate question-answering set built from training scenes rather than on the fifty test questions.

Zero-shot transfer to 3D-QA.

New query functions reuse the learned category and relation classifiers. For example, querying an object type evaluates the category classifiers and selects the highest-scoring label. No new visual parameters are fitted, but the query definitions, parser examples, and calibrated threshold provide task-specific structure.

   Type       Representation       Semantics
   object_set       Object score vectors.       Set of objects selected from a scene.
   category       Concept names: table, chair, piano, etc.       Object-level properties.
   relation       Concept names: near, left, behind, etc.       (Binary) Relationships between two objects.
   t_relation       Concept names: between, anchor-left etc.       (Ternary) Relationships among three objects.
   ∗boolean       Strings: yes, no.       Boolean values.
   ∗integer       Integers: 0, 1, 2, etc.       Count of objects.
Table 6: Types in the NS3D domain-specific language. ∗: Types that are only used in the 3D-QA task.
Signature & Implementation Semantics
scene() ⟶\longrightarrow yy: object_set
Implementation: yi=0y_{i}=0, for all i∈{1,2,⋯,M}i\in\{1,2,\cdots,M\}
Return all objects in the 3D scene.
filter(xx: object_set, cc: category) ⟶\longrightarrow yy: object_set
Implementation: yi=min⁡(xi,probic)=min⁡(xi,MLPc​(fiobj))y_{i}=\min\left(x_{i},\textit{prob}^{\text{c}}_{i}\right)=\min\left(x_{i},\mathrm{MLP}^{\text{c}}\left(f^{\text{obj}}_{i}\right)\right)
Return all objects satisfying a concept c.
relate(xtx^{t}: object_set, xrx^{r}: object_set, rel: relation) ⟶\longrightarrow yy: object_set
Implementation: probi,jrel=MLPrel​(fi,jrel)\textit{prob}^{\text{rel}}_{i,j}=\mathrm{MLP}^{\text{rel}}(f^{\text{rel}}_{i,j})
    yi=min⁡(xit,∑jsx​(xr)j⋅probi,jrel)y_{i}=\min\left(x^{t}_{i},\sum_{j}\textit{sx}(x^{r})_{j}\cdot\textit{prob}^{\text{rel}}_{i,j}\right)
Return all objects that satisfy the relationship rel between the object sets.
ternary_relate(xtx^{t}: object_set, xr1x^{\textit{r1}}: object_set, xr2x^{\textit{r2}}: object_set,
        trel: t_relation) ⟶\longrightarrow yy: object_set
Implementation: probi,j,ktrel=MLPtrel​(fi,j,kternary)\textit{prob}^{\text{trel}}_{i,j,k}=\mathrm{MLP}^{\text{trel}}(f^{\text{ternary}}_{i,j,k})
    yi=min⁡(xit,∑j∑ksx​(xr​1)j⋅sx​(xr​2)k⋅probi,j,ktrel)y_{i}=\min\left(x^{t}_{i},\sum_{j}\sum_{k}\textit{sx}(x^{r1})_{j}\cdot\textit{sx}(x^{r2})_{k}\cdot\textit{prob}^{\text{trel}}_{i,j,k}\right)
Return all objects that satisfy the ternary relationship trel between the object sets.
anchor(object_set, object_set) ⟶\longrightarrow object_set
Implementation: internally handled using ternary_relate.
Return all objects that satisfy the relationship anchored on the first object set.
∗query_exist(xx: object_set) ⟶\longrightarrow yy: boolean
Implementation: y={yesif ​maxi⁡(σ⁡(xi))>tnoif ​maxi⁡(σ⁡(xi))≤ty=\begin{cases}\textit{yes}&\text{if }\max_{i}(\sigma(x_{i}))>t\\ \textit{no}&\text{if }\max_{i}(\sigma(x_{i}))\leq t\end{cases}
Return Yes/No corresponding to existence of object in the object set.
∗query_count(xx: object_set) ⟶\longrightarrow yy: integer
Implementation: y=∑i𝟙[σ(xi)>t]y=\sum_{i}\mathbbm{1}\left[\sigma(x_{i})>t\right]
Return count of objects in the object set.
∗query_object(xx: object_set) ⟶\longrightarrow cc: category
Implementation: c=arg⁡maxc⁡(∑is​x​(x)i⋅MLPc​(fiobj))c={\arg\max}_{\text{c}}\left(\sum_{i}sx(x)_{i}\cdot\mathrm{MLP}^{\text{c}}\left(f^{\text{obj}}_{i}\right)\right)
Return type of object in the object set.
∗query_relation(xtx^{t} object_set, xrx^{r}: object_set) ⟶\longrightarrow rel: relation
Implementation: rel=arg⁡maxrel⁡(∑i∑jsx​(xt)i⋅sx​(xr)j⋅MLPrel​(fi,jrel))\textit{rel}={\arg\max}_{\text{rel}}\left(\sum_{i}\sum_{j}\textit{sx}(x^{t})_{i}\cdot\textit{sx}(x^{r})_{j}\cdot\mathrm{MLP}^{\text{rel}}(f^{\text{rel}}_{i,j})\right)
Return relationship between the object sets.
∗query_t_relation(xtx^{t}:object_set, xr1x^{\textit{r1}}: object_set, xr2x^{\textit{r2}}: object_set) →\rightarrow trel: t_relation
Implementation: trel=\textit{trel}=
    arg⁡maxtrel⁡(∑i∑j∑ksx​(xt)i⋅sx​(xr1)j⋅sx​(xr2)k⋅MLPtrel​(fi,j,kternary)){\arg\max}_{\text{trel}}\left(\sum_{i}\sum_{j}\sum_{k}\textit{sx}(x^{t})_{i}\cdot\textit{sx}(x^{\textit{r1}})_{j}\cdot\textit{sx}(x^{\textit{r2}})_{k}\cdot\mathrm{MLP}^{\text{trel}}(f^{\text{ternary}}_{i,j,k})\right)
Return ternary relationship between the object sets.
Table 7: Primitive functions defined in the NS3D domain-specific language. ∗: Functions that are only used in the 3D-QA task. Here, sx​(⋅)\textit{sx}(\cdot) is the Softmax function, σ⁡(⋅)\sigma(\cdot) is the Sigmoid function, and 𝟙​[⋅]\mathbbm{1}[\cdot] is the indicator function which returns 1 when the expression inside the brackets evaluates to true, and 0 otherwise.

Appendix B Experimental Details and Additional Results

The supplementary experiments document dataset construction, sparse-to-dense scene testing, a restricted natural-language subset, and additional visual examples. All supplementary figures and tables are preserved.

B.1 Dataset

The following dataset details separate the full benchmark, subsampled annotations, distribution-shift splits, and the new question set.

ReferIt3D datasets.

The full SR3D training setup uses 707 scenes with object-category annotations and 65,844 referring-expression examples. These annotations train both the category and grounding modules.

Data efficiency datasets.

Reduced-data experiments randomly retain 329, 987, 1,646, 3,292, or 6,584 expression examples, corresponding to 0.5%, 1.5%, 2.5%, 5%, and 10% of the training set. All use the same full SR3D test set.

PAIRS and SCENE generalization datasets.

PAIRS contains 16,200 training and 10,520 test examples split by frequent versus infrequent object–relation–object combinations. SCENE contains 57,125 training and 1,320 test examples split by the held-out living-room type. These are specified shifts within ReferIt3D rather than separate real-world deployments.

3D-QA dataset.

The fifty manually written test questions use ReferIt3D test scenes and a closed answer vocabulary. Templates cover presence of a related object, counts in a scene or relative to a reference, object categories identified by one or more references, and relations between objects. Anchored variants add a viewpoint reference. The scope is deliberately compositional and small.

Overall View-dep.
NS3D (Ours) 0.627\mathbf{0.627} 0.620\mathbf{0.620}
MVT [18] 0.4050.405 0.3960.396
SAT [37] 0.4440.444 0.4150.415
TransRefer [16] 0.3600.360 0.3440.344
Table 8: Generalization results from sparse scenes to dense scenes.

B.2 Scene Complexity Generalization

Training on ten-object scenes and testing with as many as 88 objects yields 62.7% overall accuracy for NS3D. Under this sparse-training protocol, SAT reaches 44.4%, MVT 40.5%, and TransRefer 36.0%. BUTD-DETR is excluded because its full-scene input does not directly match the partial-scene setup. This demonstrates robustness to increased distractor sets within the tested benchmark.

B.3 NR3D Results

Parsing unrestricted NR3D natural language produces 91 function types and 5,892 concepts, creating a new long-tailed learning problem. The reported experiment therefore restricts examples to the functions and concepts already used by SR3D, leaving 3,659 training and 1,041 test examples. NS3D reaches 52.6% overall and 43.2% view-dependent accuracy, ahead of the listed baselines on this subset. These results are not a score for full NR3D.

Overall View-dep.
NS3D (Ours) 0.526\mathbf{0.526} 0.432\mathbf{0.432}
BUTD-DETR [20] 0.3820.382 0.3310.331
MVT [18] 0.4300.430 0.3240.324
SAT [37] 0.3290.329 0.2480.248
TransRefer [16] 0.3600.360 0.2860.286
Table 9: Results on a constrained version of NR3D from the ReferIt3D datasets.

B.4 Qualitative Examples

The supplementary figures preserve additional binary and ternary reference examples, view-dependent comparisons against four baselines, and question-answering successes and failures. Green and red annotations distinguish correct and incorrect selections or answers. These examples complement the aggregate metrics and expose errors that remain after compositional parsing.

Refer to caption
Figure 9: Additional examples of the input language instruction, scene, and the target output in the ReferIt3D 3D-REC task.
Refer to caption
Figure 10: Comparison between NS3D and baselines on the 3D-REC task, with the green outline indicating correct selection and red outline indicating incorrect selection.
Refer to caption
Figure 11: Qualitative examples of NS3D on the 3D-QA task. Examples of success cases are marked in green, while failure cases are marked in red.
Concept demonstration

Open image ↗