Literature
Display
← All papers

Visually Grounded Neural Syntax Acquisition

Haoyue Shi · Jiayuan Mao · Kevin Gimpel · Karen Livescu

· ACL 2019 ↗

Contents

Overview

The paper and complete supplement are included. The supplement compares baseline supervision, details implementation and model selection, checks parser-derived evaluation targets, and retains all eleven example parse trees.

Refer to caption
Figure 1: We propose to use image-caption pairs to extract constituents from text, based on the assumption that similar spans should be matched to similar visual objects and these concrete spans form constituents.

At a glance

VG-NSL learns binary constituency trees from images paired with captions, without training on labeled trees or part-of-speech tags. It rewards constituents whose embeddings match the associated image, coupling syntax induction with visually grounded representation learning. A language-specific head-initial bias improves results. In caption-domain experiments, the method is more stable across random seeds and more data-efficient than selected text-only baselines. These are grounded-caption results, not evidence of general syntax acquisition from arbitrary language.

1 Introduction

A visually meaningful phrase can provide a cue that its words form a constituent. VG-NSL repeatedly combines adjacent spans, represents the resulting constituents, and matches them to images. Training uses visual concreteness to guide these decisions; parsing at test time needs only text. The work evaluates tree agreement, seed stability, phrase types, data efficiency, and learned concreteness in English, with additional German and French caption experiments.

Refer to caption
Figure 2: VG-NSL consists of two modules: a textual module for inferring structures and representations for captions, and a visual-semantic module for matching constituents with images. VG-NSL induces constituency parse trees of captions by looking at images and reading paired captions.

2 Related Work

Linguistic structure induction from text.

Earlier latent-tree models learn from downstream tasks or language modeling. Many are unstable across random seeds or recover structures that differ from conventional syntax. Other unsupervised grammar methods rely on gold part-of-speech tags. VG-NSL replaces explicit syntactic annotations with paired visual evidence, while retaining pretrained visual features and structural inductive biases.

Grounded language acquisition.

Grounded language learning has used image, video, and reasoning tasks, often with labeled visual categories or hand-designed rules. This method borrows image-text alignment objectives but uses their signal to induce constituency structure.

Concreteness estimation.

Concreteness measures how readily a word or phrase corresponds to perceptible things, properties, or events. Unlike prior word-level estimates, VG-NSL scores composed spans and uses these scores during parsing.

3 Visually Grounded Neural Syntax Learner

A textual module constructs trees and recursively represents their constituents. A visual-semantic module aligns those representations with paired images. The two modules alternate optimization, so better grounding can guide structure and better structure can improve alignment.

3.1 Textual Representations and Structures

For a sentence of n tokens, the parser makes n−1 adjacent-span merges. A two-layer network scores every possible merge; training samples from the scores, while test-time inference chooses the maximum. Each merged representation is the normalized sum of its children. This deliberately simple composition discourages the encoder from ignoring all but a salient word. A complete binary tree contains 2n−1 constituents when individual words are included.

Equations
score (𝐗(t−1),Θ)j\displaystyle(\mathbf{X}^{(t-1)};\Theta)_{j}
≜score​([𝐱j(t−1),𝐱j+1(t−1)],Θ).\displaystyle\triangleq\textit{score}\left(\left[\mathbf{x}^{(t-1)}_{j},\mathbf{x}^{(t-1)}_{j+1}\right];\Theta\right).
Pr⁡[j∗]\displaystyle\Pr\left[j^{*}\right] =exp⁡(score​(𝐗(t−1),Θ)j∗)∑jexp⁡(score​(𝐗(t−1),Θ)j).\displaystyle=\frac{\exp\left(\textit{score}\left(\mathbf{X}^{(t-1)};\Theta\right)_{j^{*}}\right)}{\sum_{j}\exp\left(\textit{score}\left(\mathbf{X}^{(t-1)};\Theta\right)_{j}\right)}.
combine​(𝐱j∗(t−1),𝐱j∗+1(t−1))≜𝐱j∗(t−1)+𝐱j∗+1(t−1)‖𝐱j∗(t−1)+𝐱j∗+1(t−1)‖2.\displaystyle\textit{combine}\left(\mathbf{x}^{(t-1)}_{j^{*}},\mathbf{x}^{(t-1)}_{j^{*}+1}\right)\triangleq\frac{\mathbf{x}^{(t-1)}_{j^{*}}+\mathbf{x}^{(t-1)}_{j^{*}+1}}{\left\|\mathbf{x}^{(t-1)}_{j^{*}}+\mathbf{x}^{(t-1)}_{j^{*}+1}\right\|_{2}}.
Original figure
Figure 3: An illustration of how VG-NSL composes a constituency parse tree. At each step, the score function s​c​o​r​escore is evaluated on all pairs of consecutive constituents (dashed lines). Next, a pair of constituents is sampled from all pairs w.r.t. a distribution computed by the softmax of all predicted scores. The selected pair of constituents is combined into a larger one, while the other constituents remain unchanged (solid lines).

3.2 Visual-Semantic Embeddings

A learned projection maps image features into the same space as text constituents. Cosine similarity measures their match. The alignment stage treats a tree as a set of constituents rather than directly modeling its topology.

Equation
m⁡(𝐯(i),𝐜j(i),Φ)\displaystyle m(\mathbf{v}^{(i)},\mathbf{c}^{(i)}_{j};\Phi) ≜cos⁡(Φ​𝐯,𝐜),\displaystyle\triangleq\cos(\Phi\mathbf{v},\mathbf{c}),

3.3 Training

A triplet-ranking loss trains image and constituent representations to distinguish matched pairs from mismatches. REINFORCE updates the discrete merge policy. Its reward measures how concretely a newly formed span matches the paired image relative to alternatives. Images supervise training but are not needed when the learned parser is applied to new text. The complete matching, concreteness, and reward formulas are retained.

Equations 1–2
ℒ⁡(Φ,𝒱,𝒞)=∑i,k≠i,j,ℓ[m⁡(𝐜ℓ(k),𝐯(i))−m⁡(𝐜j(i),𝐯(i))+δ]++∑i,k≠i,j[m⁡(𝐜j(i),𝐯(k))−m⁡(𝐜j(i),𝐯(i))+δ]+,\displaystyle\begin{aligned} &\mathcal{L}(\Phi;\mathcal{V},\mathcal{C})=\\ &\sum_{\begin{subarray}{c}i,k\neq i,j,\ell\end{subarray}}\left[m(\mathbf{c}^{(k)}_{\ell},\mathbf{v}^{(i)})-m(\mathbf{c}^{(i)}_{j},\mathbf{v}^{(i)})+\delta\right]_{+}\\ +&\sum_{\begin{subarray}{c}i,k\neq i,j\end{subarray}}\left[m(\mathbf{c}^{(i)}_{j},\mathbf{v}^{(k)})-m(\mathbf{c}^{(i)}_{j},\mathbf{v}^{(i)})+\delta\right]_{+},\end{aligned}
concrete​(𝐳,𝐯(i))=\displaystyle\textit{concrete}(\mathbf{z},\mathbf{v}^{(i)})=
∑k≠i,p[m⁡(𝐳,𝐯(i))−m⁡(𝐜p(k),𝐯(i))−δ′]+\displaystyle\sum_{k\neq i,p}\left[m(\mathbf{z},\mathbf{v}^{(i)})-m(\mathbf{c}^{(k)}_{p},\mathbf{v}^{(i)})-\delta^{\prime}\right]_{+}
+\displaystyle+ ∑k≠i[m⁡(𝐳,𝐯(i))−m⁡(𝐳,𝐯(k))−δ′]+,\displaystyle\sum_{k\neq i}\left[m(\mathbf{z},\mathbf{v}^{(i)})-m(\mathbf{z},\mathbf{v}^{(k)})-\delta^{\prime}\right]_{+}, (1)
r⁡(𝐱j(t−1),𝐱j+1(t−1))=concrete​(𝐳,𝐯(i)),\displaystyle r(\mathbf{x}^{(t-1)}_{j},\mathbf{x}^{(t-1)}_{j+1})=\textit{concrete}(\mathbf{z},\mathbf{v}^{(i)}), (2)

3.4 The Head-Initial Inductive Bias

The head-initial variant penalizes merging an abstract right constituent with material to its left, encouraging heads such as prepositions to combine with following complements. This is an explicit language-specific bias, not structure learned entirely from data. The paper cautions that the same bias does not transfer directly to head-final languages such as Japanese.

Equations 3
r′(𝐱(t−1)j,\displaystyle r^{\prime}(\mathbf{x}^{(t-1)}_{j}, OPEN𝐱j+1(t−1))\displaystyle\mathbf{x}^{(t-1)}_{j+1}) (3)
=\displaystyle= r⁡(𝐱j(t−1),𝐱j+1(t−1))λ⋅abstract​(𝐱j+1(t−1),𝐯(i))+1,\displaystyle\frac{r(\mathbf{x}^{(t-1)}_{j},\mathbf{x}^{(t-1)}_{j+1})}{\lambda\cdot\textit{abstract}(\mathbf{x}^{(t-1)}_{j+1},\mathbf{v}^{(i)})+1}\ ,
abstract​(𝐳,𝐯(i))=\displaystyle\textit{abstract}(\mathbf{z},\mathbf{v}^{(i)})=
∑k≠i,p[m⁡(𝐜p(k),𝐯(i))−m⁡(𝐳,𝐯(i))+δ′]+\displaystyle\sum_{k\neq i,p}\left[m(\mathbf{c}^{(k)}_{p},\mathbf{v}^{(i)})-m(\mathbf{z},\mathbf{v}^{(i)})+\delta^{\prime}\right]_{+}
+\displaystyle+ ∑k≠i[m⁡(𝐳,𝐯(k))−m⁡(𝐳,𝐯(i))+δ′]+,\displaystyle\sum_{k\neq i}\left[m(\mathbf{z},\mathbf{v}^{(k)})-m(\mathbf{z},\mathbf{v}^{(i)})+\delta^{\prime}\right]_{+},

4 Experiments

Evaluation separates agreement with reference trees from self-agreement across independent training runs. It also examines individual phrase types, training-data requirements, concreteness estimates, and language transfer.

4.1 Data Sets and Metrics

MSCOCO contains 82,783 training images and 1,000 images each for development and test, with five captions per image. Test reference trees are generated by the supervised Benepar parser. A separate manual check on 50 captions finds 95.65 F1 agreement with Benepar. Reported constituency F1 aggregates spans across the dataset rather than averaging sentence-level F1. The evaluation stays within caption language; newspaper-domain parsing is left for future work.

4.2 Baselines

Baselines vary in objective and available information. The tables explicitly distinguish extra linguistic labels or corpora from paired-image supervision and pretrained visual features.

Model NP VP PP ADJP Avg. F1\textbf{F}_{1} Self F1\textbf{F}_{1}
Random 47.3 ±0.3 10.5 ±0.4 17.3 ±0.7 33.5 ±0.8 27.1 ±0.2 32.4
Left 51.4 1.8 0.2 16.0 23.323.3 N/A
Right 32.2 23.4 18.7 14.4 22.922.9 N/A
PMI 54.2 16.0 14.3 39.2 30.530.5 N/A
PRPN (Shen et al., 2018a)    72.8 ±9.7    33.0 ±9.1    61.6 ±9.9    35.4 ±4.3    52.5 ±2.6 60.3
ON-LSTM (Shen et al., 2019) 74.4 ±7.1 11.8 ±5.6 41.3 ±16.4 44.0 ±14.0 45.5 ±3.3 69.3
Gumbel (Choi et al., 2018)† 50.4 ±0.3 8.7 ±0.3 15.5 ±0.0 34.8 ±1.6 27.9 ±0.2 40.1
VG-NSL (ours)† 79.6 ±0.4 26.2 ±0.4 42.0 ±0.6 22.0 ±0.4 50.4 ±0.3 87.1
VG-NSL+HI (ours)† 74.6 ±0.5 32.5 ±1.5 66.5 ±1.2 21.7 ±1.1 53.3 ±0.2 90.2
VG-NSL+HI+FastText (ours)*† 78.8 ±0.5 24.4 ±0.9 65.6 ±1.1 22.0 ±0.7 54.4 ±0.4 89.8
Concreteness estimation–based models
Turney et al. (2011)* 65.5 30.8 35.3 30.4 42.5 N/A
Turney et al. (2011)+HI* 74.5 26.2 47.6 25.6 48.9 N/A
Brysbaert et al. (2014)* 54.1 27.8 27.0 33.1 34.1 N/A
Brysbaert et al. (2014)+HI* 73.4 23.9 50.0 26.1 47.9 N/A
Hessel et al. (2018)† 50.9 21.7 32.8 27.5 33.2 N/A
Hessel et al. (2018)+HI† 72.5 34.4 65.8 26.2 52.9 N/A
Table 1: Recall of specific typed phrases, and overall F1\text{F}_{1} score, evaluated on the MSCOCO test split, averaged over 5 runs with different random initializations. We also include self-agreement F1\text{F}_{1} score Williams et al. (2018) across the 5 runs. ±\pm denotes standard deviation. * denotes models requiring extra labels and/or corpus, and †\dagger denotes models requiring a pre-trained visual feature extractor. We highlight the best number in each column among all models that do not require extra data other than paired image-caption data, as well as the overall best number. The Left, Right, PMI, and concreteness estimation–based models have no standard deviation or self F1\text{F}_{1} (shown as N/A) as they are deterministic given the training and/or testing data.

Trivial tree structures.

Random binary, purely left-branching, and purely right-branching trees provide structural baselines without linguistic training.

Syntax acquisition by language modeling and statistics.

PRPN and ON-LSTM induce syntactic distances while optimizing language modeling. A separate statistical baseline uses negative pointwise mutual information between neighboring words as a distance. Each distance sequence is converted recursively into a binary tree.

Syntax acquisition from downstream tasks.

A Gumbel-softmax latent-tree caption encoder is trained for image-caption retrieval, testing whether a downstream alignment objective alone produces useful structure.

Syntax acquisition from concreteness estimation.

Additional baselines merge spans using externally estimated word concreteness. They include semi-supervised, crowdsourced, and multimodal estimates, with and without a head-initial bias. Their supervision differs, so they should not all be described as equally annotation-free.

4.3 Implementation Details

Word and constituent embeddings have 512 dimensions. A pretrained ResNet-101 supplies 2,048-dimensional image vectors, projected into the common space. The merge scorer has a 128-dimensional hidden layer. VG-NSL trains for 30 epochs with Adam, reducing the learning rate from 0.0005 to 0.00005 after 15 epochs. Hyperparameters are tuned through agreement among five random-seed runs.

4.4 Results: Unsupervised Constituency Parsing

VG-NSL with the head-initial bias performs best among the compared models without extra syntactic labels, particularly on noun and prepositional phrases. PRPN remains a competitive text-only baseline but is less stable. ON-LSTM performs poorly overall on these captions despite its strong newspaper-domain results, while remaining strong on adjective phrases. Adding pretrained FastText vectors raises mean F1 to 54.4%; that variant introduces additional text pretraining.

4.5 Results: Data Efficiency

Training fractions range from 1% to 100%. VG-NSL matches the fully trained PRPN baseline using about 20% of the caption data and becomes more consistent across seeds as data increases. The prose also describes this setting as 8K images with 40K captions, which does not equal 20% of the stated 82,783-image training set; Figure 4 preserves the source’s percentage-axis results without resolving that inconsistency.

Original figure
(a) The percent data-F1\text{F}_{1} curves.
Original figure
(b) The percent data-self F1\text{F}_{1} curves.
Figure 4: F1\text{F}_{1} score and self F1\text{F}_{1} score with respect to the amount of training data. All numbers are averaged over 5 runs with different random initialization.

4.6 Analysis: Consistency with Linguistic Concreteness

Word concreteness is averaged over associated images and compared with existing estimates or human judgments. Correlations are computed on the 100 most frequent test words, averaged over five runs. Agreement is substantial in this selected vocabulary, but does not establish equivalent concreteness judgments for rare words or abstract language.

Model/method VG-NSL (+HI)
Turney et al. (2011) 0.74 0.72
Brysbaert et al. (2014) 0.71 0.71
Hessel et al. (2018) 0.84 0.85
Table 2: Agreement between our concreteness estimates and existing models or labels, evaluated via the Pearson correlation coefficient computed over the most frequent 100 words in the MSCOCO test set, averaged over 5 runs with different random initialization.

4.7 Analysis: Self-Agreement F 1 Score as the Criterion for Model Selection

Model selection maximizes pairwise tree agreement across five independently initialized runs and nearby checkpoints. It yields 50.4 mean F1 for VG-NSL versus 47.7 with image-retrieval-based selection; the head-initial variant changes from 53.1 to 53.3. Self-agreement measures consistency, which alone need not imply correctness. PRPN and ON-LSTM were not tuned with this criterion because repeated evaluation was expensive, so the comparison does not isolate architecture from all selection differences.

Model Criterion Avg. F1\textbf{F}_{1} Self F1\textbf{F}_{1}
VG-NSL Self F1\text{F}_{1} 50.4 ±0.3 87.1
VG-NSL R@1 47.7 ±0.6 83.4
VG-NSL+HI Self F1\text{F}_{1} 53.3 ±0.2 90.2
VG-NSL+HI R@1 53.1 ±0.2 88.7
Table 3: Average F1\text{F}_{1} scores and Self F1\text{F}_{1} scores of VG-NSL and VG-NSL+HI with different model selection methods. R@1 denotes using recall at 1 Kiros et al. (2014) as the model selection criterion. All hyperparameters are tuned with respect to self-agreement F1\text{F}_{1} score. The numbers are comparable to those in Table 1.
Equations
∑1≤i<k≤N\displaystyle\sum_{1\leq i<k\leq N} max|ji−jk|<δ⁡F1​(ℳℋ(i,ji),ℳℋ(k,jk)),\displaystyle\max_{|j_{i}-j_{k}|<\delta}\textit{F}_{1}\left(\mathcal{M}_{\mathcal{H}}^{(i,j_{i})},\mathcal{M}_{\mathcal{H}}^{(k,j_{k})}\right),
arg​max{jℓ}ℓ=1N∑1≤i<k≤NF1(ℳℋ0(i,ji),ℳℋ0(k,jk)).\displaystyle\argmax_{\{j_{\ell}\}_{\ell=1}^{N}}\sum_{1\leq i<k\leq N}\textit{F}_{1}\left(\mathcal{M}_{\mathcal{H}_{0}}^{(i,j_{i})},\mathcal{M}_{\mathcal{H}_{0}}^{(k,j_{k})}\right).

4.8 Extension to Multiple Languages

Multi 30K provides 29,000 training images,1,014 development images, and 1,000 test images, with one caption per language per image. The head-initial variant performs best among the reported models in English, German, and French. These scores are not directly comparable with MSCOCO because the amount and distribution of caption data differ.

Model EN DE FR
PRPN   30.8 ±17.9   31.5 ±8.9   27.5 ±7.0
ON-LSTM   38.7 ±12.7   34.9 ±12.3   27.7 ±5.6
VG-NSL 33.5 ±0.2 36.3 ±0.2 34.3 ±0.6
VG-NSL+HI 38.7 ±0.2 38.3 ±0.2 38.1 ±0.6
Table 4: F1\text{F}_{1} scores on the Multi30K test split Young et al. (2014); Elliott et al. (2016); Elliott et al. (2017), averaged over 5 runs with different random initialization. ±\pm denotes the standard deviation.

5 Discussion

Visual grounding can make caption-domain latent-tree learning more accurate and reproducible in these experiments. Remaining limitations include static-scene supervision, whole-image alignment, shared syntax/semantics representations, reliance on a head-direction bias, and evaluation against mostly parser-generated targets. Structured visual representations, other modalities, and integration with text-only learning are proposed extensions.

Acknowledgement

The authors acknowledge Allyson Ettinger and the reviewers.

Appendix A Overview of Models for Constituency Parsing without Explicit Syntactic Supervision

Table 5 retains the complete comparison of objectives, extra labels, additional corpora, multimodal inputs, and stochastic behavior. All listed multimodal methods use pretrained visual feature extractors. Figure 5 retains the contrasting left- and right-branching examples.

Model Objective Extra Label Multi- Stochastic Extra
modal Corpus
CCM Klein and Manning (2002)* MAP POS ✗ ✓ ✗
DMV-CCM Klein and Manning (2005)* MAP POS ✗ ✓ ✗
U-DOP Bod (2006b)* Probability Estimation POS ✗ ✗ ✗
UML-DOP Bod (2006a)* MAP POS ✗ ✓ ✗
PMI N/A ✗ ✗ ✗ ✗
Random N/A ✗ ✗ ✓ ✗
Left N/A ✗ ✗ ✗ ✗
Right N/A ✗ ✗ ✗ ✗
PRPN Shen et al. (2018a) LM ✗ ✗ ✓ ✗
ON-LSTM Shen et al. (2019) LM ✗ ✗ ✓ ✗
Gumbel softmaxChoi et al. (2018) Cross-modal Retrieval ✗ ✓ ✓ ✗
VG-NSL (ours) Cross-modal Retrieval ✗ ✓ ✓ ✗
VG-NSL+HI (ours) Cross-modal Retrieval ✗ ✓ ✓ ✗
Concreteness estimation based models
Turney et al. (2011)* N/A
Concreteness
(Partial)
✗ ✗ ✓
Turney et al. (2011)+HI* N/A
Concreteness
(Partial)
✗ ✗ ✓
Brysbaert et al. (2014)* N/A
Concreteness
(Full)
✗ ✗ ✗
Brysbaert et al. (2014)+HI* N/A
Concreteness
(Full)
✗ ✗ ✗
Hessel et al. (2018) N/A ✗ ✓ ✗ ✗
Hessel et al. (2018)+HI N/A ✗ ✓ ✗ ✗
Table 5: Comparison of models for constituency parsing without explicit syntactic supervision. * denotes models requiring extra labels, such as POS tags or manually labeled concreteness scores. All multimodal methods listed in the table require a pretrained visual feature extractor (i.e., ResNet-101; He et al., 2016). A model is labeled as stochastic if for fixed training data and hyperparameters the model may produce different results (e.g., due to different choices of random initialization). To the best of our knowledge, results on concreteness estimation Turney et al. (2011); Brysbaert et al. (2014); Hessel et al. (2018) have not been applied to unsupervised parsing so far.
Original PDF tree panel: (a) Left-branching tree.
(a) Left-branching tree.
Original PDF tree panel: (b) Right-branching tree.
(b) Right-branching tree.
Figure 5: Examples of some trivial tree structures.

Appendix B Implementation Details for VG-NSL

The visual-semantic module follows the VSE++ implementation, with margin 0.2 and a shared vocabulary of 10,000 frequent MSCOCO words. Out-of-vocabulary words receive the same unseen-word treatment across applicable methods.

Hyperparameter tuning.

Hyperparameters are selected by self-agreement rather than labeled parsing accuracy. The head-initial penalty is searched over 1,2,5,10,20,50, and 100; the selected value is 20.

Appendix C Implementation Details for Baselines

Trivial tree structures.

Random trees repeatedly merge randomly chosen neighboring constituents; fixed left- and right-branching trees supply deterministic alternatives. The original examples remain in Figure 5.

Parsing-Reading-Predict Network (PRPN).

PRPN uses 512-dimensional hidden layers, Adam, and a search over one to three layers and three learning rates. Training budgets are 100 epochs on MSCOCO and 1,000 on Multi 30K, with language-model perplexity used for tuning and stopping.

Ordered Neurons (ON-LSTM).

ON-LSTM uses three 512-dimensional layers and chunk size 16; trees come from the second layer’s gates. Both Adam and stochastic gradient descent are explored with multiple learning rates. Training budgets and selection use language-model perplexity as for PRPN.

PMI based constituency parsing.

The statistical baseline estimates adjacent-word mutual information from training captions, negates it into a syntactic distance, and recursively splits at the largest distance. Algorithm 1 retains every step.

Algorithm 1 Constituency parsing based on given syntactic distance.
Input: text length mm, list of syntactic distances 𝒅=(d1,d2,…,dm−1)\bm{d}=(d_{1},d_{2},\ldots,d_{m-1})
Output: Boundaries of constituents B={(Li,Ri)}i=1,…,m−1B=\{(L_{i},R_{i})\}_{i=1,\ldots,m-1}
BB = parse(𝐝\bm{d}, 11, mm)
Function parse(𝐝\bm{d}, left, right)
if left=right\textit{left}=\textit{right} then
   return EmptySet
end if
p=arg​maxj∈[left,right-1]⁡djp=\argmax_{j\in[\textit{left},\textit{right-1}]}d_{j}
boundaries=\textit{boundaries}= union(
  {(left, right)},
  parse (𝒅\bm{d}, left, pp),
  parse (𝒅\bm{d}, p+1p+1, right)
)
return boundaries

Gumbel-softmax based latent tree.

The Gumbel-softmax baseline inserts a latent-tree text encoder into the visual-semantic retrieval system. Tables and figures retain its comparison with the proposed policy and concreteness baselines.

Turney et al. (2011) Brysbaert et al. (2014) Hessel et al. (2018) VG-NSL+HI
Turney et al. (2011) 1.00 0.84 0.58 0.72
Brysbaert et al. (2014) 0.84 1.00 0.55 0.71
Hessel et al. (2018) 0.58 0.55 1.00 0.85
VG-NSL+HI 0.72 0.71 0.85 1.00
Table 6: Pearson correlation coefficients between existing concreteness estimation methods, including baselines and VG-NSL+HI. In order to make a fair comparison, the correlation coefficients are evaluated on the 100 most frequent words on MSCOCO test set.
Original figure
Figure 6: Normalized concreteness scores of example words.

Concreteness estimation.

The semi-supervised concreteness baseline uses 4,295 manually scored words and Wikipedia word associations. Crowdsourced scores are taken from the released human-rating dataset; the multimodal baseline is estimated on MSCOCO. These sources are explicitly distinguished from VG-NSL’s paired-caption training signal.

Constituency parsing with concreteness scores.

Word-concreteness scores are normalized to[−1,1], with unseen words assigned−1. Adjacent spans with the largest average score are merged, and their new score is the children’s mean. The head-initial variant weights the right constituent by 20 for selection while leaving the composed score unweighted. Algorithm 2 preserves the complete procedure.

Equation
α′​(wi)=2​(α⁡(wi)−maxj⁡α⁡(wj)−minj⁡α⁡(wj)2)maxj⁡α⁡(wj)−minj⁡α⁡(wj).\displaystyle\alpha^{\prime}(w_{i})=\frac{2\left(\alpha(w_{i})-\frac{\max_{j}\alpha(w_{j})-\min_{j}\alpha(w_{j})}{2}\right)}{\max_{j}\alpha(w_{j})-\min_{j}\alpha(w_{j})}~.
Algorithm 2 Constituency parsing based on concreteness estimation.
Input: list of normalized concreteness scores 𝒂=(a1,a2,…,am)\bm{a}=(a_{1},a_{2},\ldots,a_{m}), hyperparameter τ\tau
Output: Boundaries of constituents B={(Li,Ri)}i=1,…,m−1B=\{(L_{i},R_{i})\}_{i=1,\ldots,m-1}
for j=1j=1 to mm do
   leftj=j\textit{left}_{j}=j
   rightj=j\textit{right}_{j}=j
end for
while len​(𝐚)>1\textit{len}(\bm{a})>1 do
   p=arg​maxj⁡aj+τ​aj+1p=\argmax_{j}a_{j}+\tau a_{j+1}
   add (leftp,rightp+1)(\textit{left}_{p},\textit{right}_{p+1}) to BB
   𝒂=𝒂<p\bm{a}=\bm{a}_{<p} + (ap+ap+12\frac{a_{p}+a_{p+1}}{2}) + 𝒂>p+1\bm{a}_{>p+1}
   left=left<p\textbf{{left}}=\textbf{{left}}_{<p} + (leftp\textit{left}_{p}) + left>p+1\textbf{{left}}_{>p+1}
   right=right<p\textbf{{right}}=\textbf{{right}}_{<p} + (rightp+1\textit{right}_{p+1}) + right>p+1\textbf{{right}}_{>p+1}
end while
Original PDF tree panel: (a) Constituency parse tree labeled by Benepar Kitaev and Klein (2018) .
(a) Constituency parse tree labeled by Benepar Kitaev and Klein (2018).
Original PDF tree panel: (b) Manually labeled constituency parse tree.
(b) Manually labeled constituency parse tree.
Figure 7: A failure example by Benepar, where it fails to parse the noun phrase “three white sinks in a bathroom under mirrors” – according to human commonsense, it is much more common for sinks, rather than a bathroom, to be under mirrors. However, most of the constituents (e.g., “three white sinks” and “under mirrors”) are still successfully extracted by Benepar.

Appendix D Details of Manual Ground Truth Evaluation

Fifty randomly selected test captions were manually parsed without looking at Benepar outputs or images. These annotations cover tree structure rather than constituent categories. Benepar agrees at 95.65 F1; remaining examples often involve commonsense-dependent attachment ambiguities. This supports its use as a reference on a small sample, while retaining the limits of that validation.

Appendix E Concreteness by Different Models

E.1 Correlation between Different Concreteness Estimations

The full correlation matrix compares the linguistic, human-rated, multimodal, and VG-NSL concreteness measures on the same 100 frequent test words. VG-NSL correlates more strongly with the linguistic and human-rated measures than the compared multimodal baseline does.

E.2 Concreteness Scores of Sample Words by Different Methods

Figure 6 preserves all normalized scores for the selected example words, illustrating both broad agreement and differences between methods.

Appendix F Sample Trees Generated by VG-NSL

Figure 8 retains all eleven example trees across panels(a)–(k), including every word and branch. The captions come from the MSCOCO test set; these qualitative examples show the learned parser’s outputs, including imperfect or debatable attachments.

Original PDF tree panel: (a) a kitchen with two windows and two metal sinks
(a) a kitchen with two windows and two metal sinks
Original PDF tree panel: (b) a blue small plane standing at the airstrip
(b) a blue small plane standing at the airstrip
Original PDF tree panel: (c) young boy sitting on top of a briefcase
(c) young boy sitting on top of a briefcase
Original PDF tree panel: (d) a small dog eating a plate of broccoli
(d) a small dog eating a plate of broccoli
Original PDF tree panel: (e) a building with a bunch of people standing around it
(e) a building with a bunch of people standing around it
Original PDF tree panel: (f) a horse walking by a tree in the woods
(f) a horse walking by a tree in the woods
Original PDF tree panel: (g) the golden waffle has a banana in it .
(g) the golden waffle has a banana in it .
Original PDF tree panel: (h) a bowl full of oranges that still have stems
(h) a bowl full of oranges that still have stems
Original PDF tree panel: (i) there is a person that is sitting in the boat on the water
(i) there is a person that is sitting in the boat on the water
Original PDF tree panel: (j) a sandwich and soup sit on a table
(j) a sandwich and soup sit on a table
Original PDF tree panel: (k) a big umbrella sitting on the beach
(k) a big umbrella sitting on the beach
Figure 8: Examples of parsing trees generated by VG-NSL.
Concept demonstration

Open image ↗