Literature
Display
← All papers

Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations

Hao Wu · Jiayuan Mao · Yufeng Zhang · Yuning Jiang · Lei Li · Weiwei Sun · Wei-Ying Ma

· CVPR 2019 ↗

Contents

Overview

This edition follows the CVPR version, arXiv v 1, and restores its eleven-page supplement. The later arXiv v 2 is a shorter workshop variant with a different title. Figure 1 introduces the contrastive image-caption examples.

Refer to caption
Figure 1: Two examplar image-caption pairs. Humans are able to establish accurate and generalizable alignments between vision and language, at different levels: objects, relations and full sentences. Pair A and B form a pair of contrastive example for the concepts clock and basin.

At a glance

Unified Visual-Semantic Embeddings align objects, attributes, relations, and complete captions with image regions or whole scenes in one representation space. Component-level contrastive learning establishes finer correspondences than sentence-level matching alone. An explicit aggregate of all parsed components helps a caption embedding remain sensitive to mismatched details. MS-COCO experiments show improved retrieval, robustness to specified text perturbations, and image-assisted recovery of word dependencies; these results rely on parsed caption structure and a fixed visual backbone.

1 Introduction

A caption describes several objects and relations, but a retrieval model can exploit co-occurrence and focus on only part of the sentence. Unified VSE addresses both uncertain object-to-region alignment and incomplete semantic coverage. Contrastive examples at several semantic levels teach what distinguishes a component, while a bag-of-components representation makes omitted or incompatible details affect the caption embedding. The same representation is subsequently tested as a visual cue for parsing ambiguous sentences.

Refer to caption
Figure 2: We build a visual-semantic embedding space, which unifies the embeddings for objects, attributes, relations and full scenes.

2 Related work

Earlier embedding systems combine image and sentence encoders with ranking losses, attention, or hard-negative mining. VSE-C adds manually constructed adversarial captions [40]. Unified VSE instead samples negatives in a factorized semantic space and aligns components at different scales. Its parser provides structural cues, but training does not require manually labeled scene graphs or object boxes. Existing parsing and pretrained visual/language models nevertheless supply substantial prior knowledge.

Refer to caption
Figure 3: Left: the architecture of Unified VSE. The semantic component alignment is learned from contrastive examples sampled from factorized semantic space. The model also learns a caption encoder that combines the semantic components and aligns the caption with the corresponding image. Right: An exemplar computation graph for retrieving images from texts. The presence of 𝐮c​o​m​p\mathbf{u}_{comp} in the caption encoding enforces the coverage of all semantic components. See Sec. 3.2 for details.

3 Unified Visual-Semantic Embeddings

The pipeline parses a caption into nouns, adjective-noun pairs, and subject-relation-object triples. Shared, type-aware encoders place these components and the sentence in a common space. A convolutional image encoder supplies regional and pooled scene representations. Multiple alignment objectives train the component representations and caption encoder together.

3.1 Visual-Semantic Embedding: A Revisit

The starting point is a bidirectional margin-ranking objective: matched image-caption pairs should be more similar than mismatches. Hard-negative mining selects difficult alternatives, and cosine similarity measures embedding agreement. Equation 1 retains the full loss.

Equation 1
ℓs​e​n​t=\displaystyle\ell_{sent}= ∑𝐮F𝐯−​(|δ+s⁡(𝐮,𝐯−)−s⁡(𝐮,𝐯+)|+)\displaystyle\sum_{\mathbf{u}}F_{\mathbf{v}^{-}}\left(|\delta+s(\mathbf{u},\mathbf{v}^{-})-s(\mathbf{u},\mathbf{v}^{+})|_{+}\right)
+\displaystyle+ ∑𝐯F𝐮−​(|δ+s⁡(𝐮−,𝐯)−s⁡(𝐮+,𝐯)|+)\displaystyle\sum_{\mathbf{v}}F_{\mathbf{u}^{-}}\left(|\delta+s(\mathbf{u}^{-},\mathbf{v})-s(\mathbf{u}^{+},\mathbf{v})|_{+}\right)\vskip-15.00002pt (1)

3.2 Semantic Encodings

A dependency parser and rules extract single-word object nouns, adjective modifiers, and relational dependencies. Object embeddings combine a pretrained basic word vector with a learned modifier vector; adjective-noun pairs replace the intrinsic noun modifier with the adjective modifier. Gated fusion produces a shared representation. A GRU then combines object-level embeddings into relation or sentence embeddings, with shared parameters across levels.

The final caption representation mixes the recurrent sentence embedding with an explicit aggregate of all extracted semantic components. The coefficient α controls this tradeoff at inference. Including every parsed component reduces the opportunity to ignore a mismatched object or relation, although it cannot recover components the parser failed to extract.

Equation
ϕ⁡(𝐰n)\displaystyle\small\phi(\mathbf{w}_{n}) OPEN=Norm⁡(σ⁡(𝐖1​𝐰n+𝐛1))​tanh⁡(𝐖2​𝐰n+𝐛2)),\displaystyle=\mathrm{Norm}(\sigma(\mathbf{W}_{1}\mathbf{w}_{n}+\mathbf{b}_{1}))\tanh(\mathbf{W}_{2}\mathbf{w}_{n}+\mathbf{b}_{2})),
ϕ⁡(𝐰a,n)\displaystyle\phi(\mathbf{w}_{a,n}) =Norm⁡(σ⁡(𝐖1​𝐰a,n+𝐛1)​tanh⁡(𝐖2​𝐰a,n+𝐛2)).\displaystyle=\mathrm{Norm}(\sigma(\mathbf{W}_{1}\mathbf{w}_{a,n}+\mathbf{b}_{1})\tanh(\mathbf{W}_{2}\mathbf{w}_{a,n}+\mathbf{b}_{2})).

3.3 Image Encodings

A fixed ImageNet-pretrained ResNet-152 produces a 7×7 spatial feature map. A learned 1×1 convolution maps each region to the joint embedding dimension. Spatial pooling then produces a whole-image representation. Regional features support object and attribute alignment; pooled features support broader semantics.

3.4 Learning Paradigm

Negative examples replace object words, attribute or noun components, and individual positions in relation triples; other captions and images supply additional negatives. Nouns and adjective-noun pairs align locally through relevance-weighted regional losses. Relations, component aggregates, and full sentences align globally with the image. Sentence and component representations are trained separately, so their inference-time mixing coefficient can be adjusted without retraining. The method uses component negatives even though it does not train on the complete adversarial captions used at evaluation.

Equations 2–3
𝐌i=\displaystyle\mathbf{M}_{i}= exp⁡(s⁡(𝐮o+,𝐕i))∑jexp⁡(s⁡(𝐮o+,𝐕j))\displaystyle\frac{\exp(s(\mathbf{u}_{o}^{+},\mathbf{V}_{i}))}{\sum_{j}\exp(s(\mathbf{u}_{o}^{+},\mathbf{V}_{j}))} (2)
ℓo​b​j=\displaystyle\ell_{obj}= ∑i∈7×7(𝐌i⋅|δ+s⁡(𝐮o−,𝐕i)−s⁡(𝐮o+,𝐕i)|+)\displaystyle\sum_{i\in 7\times 7}\left(\mathbf{M}_{i}\cdot\left|\delta+s(\mathbf{u}_{o}^{-},\mathbf{V}_{i})-s(\mathbf{u}_{o}^{+},\mathbf{V}_{i})\right|_{+}\right) (3)
Refer to caption
Figure 4: An illustration of our relevance-weighted alignment mechanism. The relevance map shows the similarity of each region with the object embedding 𝐮<c​l​o​c​k>\mathbf{u}_{<clock>}. We weight the alignment loss with the map to reinforce the correspondence between the 𝐮<c​l​o​c​k>\mathbf{u}_{<clock>} and its matched region.

3.5 Implementation details

The common embedding has 1,024 dimensions. The first two epochs emphasize object-level alignment; relation learning is then enabled. ResNet weights remain fixed for the reported comparisons. Adam begins at 0.001. The supplement supplies the exact sampling, aggregation, and optimization settings.

Equation 4
ℓ=ℓs​e​n​t+ηc​ℓc​o​m​p+ηo​ℓo​b​j+ηa​ℓa​t​t​r+ηr​ℓr​e​l\displaystyle\ell=\ell_{sent}+\eta_{c}\ell_{comp}+\eta_{o}\ell_{obj}+\eta_{a}\ell_{attr}+\eta_{r}\ell_{rel}\vskip-5.0pt (4)

4 Experiments

MS-COCO provides 82,783 training images, each with five captions. Evaluation uses standard 1K and 5K test settings. The study compares cross-modal retrieval, controlled adversarial retrieval, queries at several semantic levels, and visually assisted dependency recovery. Tables preserve the complete comparisons; methods that fine-tune the image encoder or add training data are excluded from the main retrieval comparison.

Task Image-to-sentence Retrieval Sentence-to-image Retrieval
Metric R@1 R@5 R@10 Med. r R@1 R@5 R@10 Med. r rsum
1K testing split (5,000 captions)
m-RNN [31] 41.0 73.0 83.5 2 29.0 42.2 77.0 3 345.7
DVSA [20] 38.4 69.9 80.5 1 27.4 60.2 74.8 3 351.2
MNLM [24] 43.4 75.7 85.8 - 31.0 66.7 79.9 - 382.5
m-CNN [30] 42.8 73.1 84.1 3 32.6 68.6 82.8 3 384.0
HM-LSTM[33] 43.9 - 87.8 2 36.1 - 86.7 3 -
Order-embedding [42] 46.7 - 88.9 2 37.9 - 85.9 2 -
VSE-C [40, 1] 48.0 81.0 89.2 2 39.7 72.9 83.2 2 414
DeepSP[44] 50.1 79.7 89.2 - 39.6 75.2 86.9 - 420.7
2WayNet [9] 55.8 75.2 - - 39.7 63.3 - - -
sm-LSTM [15] 53.2 83.1 91.5 1 40.7 75.8 87.4 2 431.8
RRF-Net[28] 56.4 85.3 91.5 - 43.9 78.1 88.6 - 443.8
VSE++ [11, 2] 57.7 86.0 94.0 1 42.8 77.2 87.4 2 445.1
CSE[49] 56.3 84.4 92.2 1 45.7 81.2 90.6 2 450.4
UniVSE (Ours) 64.3 89.2 94.8 1 48.3 81.7 91.2 2 469.5
5K testing split (25,000 captions)
Order-embedding [42] 23.3 - 65.0 5 18.0 - 57.6 7 -
VSE-C[11, 1] 22.3 51.1 65.1 5 18.7 43.8 56.7 7 257.7
CSE[49] 27.9 57.1 70.4 4 22.2 50.2 64.4 5 292.2
VSE++[11, 2] 31.7 60.9 72.7 3 22.1 49.0 62.7 6 299.1
UniVSE (Ours) 36.1 66.4 77.7 3 25.4 53.0 66.2 5 324.8
Table 1: Results of cross-modal retrieval task on MS-COCO dataset (1K and 5K testing split). All listed baselines and our models fix weights of the image encoders. For fair comparison, we do not include [10] and [16] that finetunes the image encoder or adds extra training data.
Object attack Attribute attack Relation attack
Metric R@1 R@5 R@10 rsum R@1 R@5 R@10 rsum R@1 R@5 R@10 rsum total sum
VSE++ 32.3 69.6 81.4 183.3 19.8 59.4 76.0 155.2 26.1 66.8 78.7 171.6 510.1
VSE-C 41.1 76.0 85.6 202.7 26.7 61.0 74.3 162.0 35.5 71.1 81.5 188.1 552.8
UniVSE (𝐮s​e​n​t\mathbf{u}_{sent}+𝐮c​o​m​p\mathbf{u}_{comp}) 45.3 78.3 87.3 210.9 35.3 71.5 83.1 189.9 39.0 76.5 86.7 202.2 603.0
UniVSE (𝐮s​e​n​t\mathbf{u}_{sent}) 40.7 76.4 85.5 202.6 30.0 70.5 80.6 181.1 32.6 72.6 83.5 188.7 572.4
UniVSE (𝐮s​e​n​t\mathbf{u}_{sent}+𝐮o​b​j\mathbf{u}_{obj}) 42.9 77.2 85.6 205.7 30.1 69.0 79.8 178.9 34.0 71.2 83.6 188.8 573.4
UniVSE (𝐮s​e​n​t\mathbf{u}_{sent}+𝐮a​t​t​r\mathbf{u}_{attr}) 40.1 73.9 83.3 197.3 37.4 72.0 81.9 191.3 30.5 70.0 81.9 182.4 571.0
UniVSE (𝐮s​e​n​t\mathbf{u}_{sent}+𝐮r​e​l\mathbf{u}_{rel}) 45.4 77.1 85.5 208.0 29.2 68.1 78.5 175.8 42.8 77.5 85.6 205.9 589.7
Table 2: Results on image-to-sentence retrieval task with text-domain adversarial attacks. For each caption, we generate 5 adversarial fake captions which do not match the images. Thus, the models need to retrieve 5 positive captions from 30,000 candidate captions.

4.1 Overall Evaluation on Cross-Modal Retrieval.

On the 1K setting, Unified VSE reaches 64.3% image-to-sentence recall at 1 and 48.3% sentence-to-image recall at 1, versus 57.7% and 42.8% for VSE++. On 5K images, these values are 36.1% and 25.4%, versus 31.7% and 22.1%. The reported recall sum aggregates recall at 1,5, and 10 across retrieval directions; it is not a separate accuracy measure.

4.2 Retrieval under text-domain adversarial attack

Evaluation adds five adversarial captions per original caption, producing 30,000 candidates. Attacks alter or append objects, attributes, or relations. Unified VSE improves over VSE++ and VSE-C across these specified attack types. Ablations show that explicitly aggregating a component particularly helps attacks on that component; relying only on the recurrent sentence representation weakens robustness. Mixing coefficients around 0.6–0.8 balance ordinary retrieval and attack resistance in these experiments. This is evidence about the defined perturbations, rather than unrestricted adversarial robustness.

Refer to caption
(a) Normal cross-modal retrieval (5,000 captions)
Refer to caption
(b) Adversarial attacked image-to-sentence retrieval (30,000 captions)
Figure 5: The performance of UniVSE on cross-modal retrieval tasks with different combination weight α\alpha. Our model can effective defending adversarial attacks, with no sacrifice for the performance on other tasks by choosing a reasonable α\alpha (thus we set α=0.75\alpha=0.75 in all other experiments).
Refer to caption
Figure 6: The relevance maps and grounded areas obtained from the retrieved images w.r.t. three queries. The temperature of the softmax for visualizing the relevance map is τ=0.1\tau=0.1. Pixels in white indicates a higher matching score. Note that the third image of the query “black dog” contains two dogs, while our model successfully locates the black one (on the left). It also succeeded in finding the white dog in the first image of “white dog”. Moreover, for the query “player swing bat”, although there are many players in the image, our model only attend to the man swinging the bat.
Refer to caption
Figure 7: Example showing that Unified VSE can leverage image to parse sentences with ambiguity. The matching score of “girl eat burger” is much higher than “sweater eat burger”, which resolves the ambiguity. Other components are also correctly inferred.
Task obj attr rel obj (det) sum
VSE++ 29.95 26.64 27.54 50.57 134.70
VSE-C 27.48 28.76 26.55 46.20 128.99
UniVSEall 39.49 33.43 39.13 58.37 170.42
UniVSEobj 39.71 33.37 34.38 56.84 164.30
UniVSEattr 31.31 37.51 34.73 52.26 155.81
UniVSErel 37.55 32.70 39.57 59.12 168.94
Table 3: The mAP performance on the unified text-to-image retrieval task. Please refer to the text for details.

4.3 Unified Text-to-Image Retrieval

Queries range from individual nouns through attributed objects and relation triples to full sentences. Caption-derived labels can miss visible objects, so the study also evaluates object queries using detection annotations. Component-specific training most directly improves corresponding queries, and the complete model performs best overall. Relevance maps provide qualitative evidence of regional alignment; they are not a substitute for bounding-box grounding accuracy.

Task attributed object relational phrase
Random 37.41 31.90
VSE++ 41.12 43.31
VSE-C 43.44 41.08
UniVSE 64.82 62.69
Table 4: The accuracy of different models on recovering word dependencies with visual cues. In the “Random” baseline, we randomly assign the word dependencies.

4.4 Semantic Parsing with Visual Cues

For each ambiguous modifier or relation, the method scores candidate dependencies against the image and chooses the best match. Reported attribute/relation recovery is 64.82%/62.69%, compared with 41.12%/43.31% for VSE++. The target dependencies come from a parser, and relation scoring gives half credit for a correct subject or object. These details matter when interpreting the headline accuracy: the evaluation is not fully human-annotated syntactic recovery. The supplement also shows cases where the visual evidence supports multiple valid readings.

5 Conclusion

The factorized embedding and explicit component coverage improve the tested retrieval and parsing tasks. Important limits are the parser’s vocabulary and recall, single-word object/attribute representation, noisy caption-derived labels, synthetic attack families, and dependence on pretrained fixed image features. The experiments support useful multilevel alignment without establishing general scene understanding or guaranteed compositional inference.

6 Acknowledgements

The authors acknowledge Haoyue Shi and support from the listed Chinese national research programs.

Supplement A.1. Negative sampling

Batches contain 128 image-caption pairs. Frequent nouns form a 1,205-word vocabulary;16 noun negatives are sampled per positive. Attribute replacements draw from 37 frequent attributes, and only the first modifier is kept when a noun has several. Attribute and noun replacements contribute separate one-directional losses without hard-negative mining.

Each relation receives eight component substitutions: four relation words, two subjects, and two objects. Additional relation negatives come from other captions in the batch. Bag-of-components and full-sentence losses instead use bidirectional, in-batch hard-negative mining. These choices define the actual weak-supervision and sampling assumptions.

Supplement A.2. Model settings

Removing 30% of parsed relations and attributes during training has limited effect in the reported ablation; removing them at test time particularly hurts adversarial discrimination. Semantic coverage depends on what is available to aggregate. Spatial max-k pooling averages the strongest k regional responses, with k=10 used here; the source’s written set formula uses a≤ comparison despite describing the k largest responses, a notation inconsistency retained in the source.

Average pooling is used for semantic-component aggregation. Alternatives trade ordinary retrieval, robustness, and computation differently: max pooling is strongest on the aggregated attack score, while a GRU slightly improves ordinary retrieval at greater cost. Learning α drives it to 0.93 and reduces attack robustness compared with the fixed 0.75 setting. The four restored tables retain every ablation value.

Train DropTest DropStandardObj. Atk.Attr. Atk.Rel. Atk.
469.5210.9189.9202.2
✓468.9213.7191.5199.4
✓✓468.7211.2182.2197.1
Supplementary Table 1. We evaluate the performance of UniVSE on the standard bidirectional retrieval task and the retrieval tasks with adversarial captions (object-typed, attribute-typed and relation-typed). We use rsum as the evaluation metric.
Spatial Aggregation MethodsStandardObj. Atk.Attr. Atk.Rel. Atk.
Avg452.9198.7184.2186.2
Max462.5209.6184.1193.6
Max-10469.5210.9189.9202.2
Max-10 (without ℓcomp)466.1202.9182.1192.2
Supplementary Table 2. We evaluate the performance of UniVSE under different spatial aggregation settings on the standard retrieval task, and the retrieval tasks with adversarial captions (we report the rsums).
Semantic Aggregation MethodsAvgSumMaxSelf-Att.GRU
Standard (rsum)469.5471.1465.8467.1472.0
Adversarial (rsum)603.0604.3628.4599.5603.7
Supplementary Table 3. We evaluate the performance of UniVSE under different semantic aggregation settings on the standard retrieval task, and the retrieval tasks with adversarial captions (we report the sum of rsums under three types of attacks).
ModelStandardObj. Atk.Attr. Atk.Rel. Atk.
Fixed α (0.75)469.5210.9189.9202.2
Learnable α (0.93)468.1204.1182.2190.4
Supplementary Table 4. The performance of UniVSE with a learnable or a fixed α on the standard retrieval task, and the retrieval tasks with adversarial captions (we report the rsums).

Supplement A.3. Hyperparameters

Basic word vectors have 300 dimensions, start from Common Crawl GloVe, and remain fixed. Modifier vectors have 100 dimensions and are learned from random initialization. Adam uses 0.001 for six epochs, then halves the learning rate each epoch down to 0.00001.

Supplement B.1. Cross-modal retrieval

The restored supplementary Figures 1 and 2 retain all selected top-five image-to-caption and caption-to-image examples, including similarity scores and color-coded correct matches. These are illustrative examples, not a representative error-rate estimate.

Supplementary Figure 1
Supplementary Figure 1. Examples showing the top-5 image-to-text retrieval results. We highlight the positive captions in blue. The score in the front of each sentence is the similarity score of the caption and image computed by different model. Best viewed in color.
Supplementary Figure 2
Supplementary Figure 2. Examples showing the top-5 sentence-to-image retrieval results. We highlight the correct images in green box.

Supplement B.2. Adversarial retrieval

Object attacks replace or append an irrelevant noun with equal probability. Attribute attacks replace an existing modifier or add one, avoiding listed near-synonym groups. Relation attacks replace one triple component or add a new relation when none is present. VSE-C is trained with all three of its own negative families for comparison. The attribute-only aggregation ablation additionally includes object embeddings when needed to make the representation defined. Supplementary Figure 3 preserves successful and unsuccessful ranked captions, including the source’s annotation typo.

Supplementary Figure 3
Supplementary Figure 3. Examples showing the top-5 image-to-sentence retrieval results with the presence of adversarial samples. We highlight the positive captions in blue. Captions with red words are adversarial samples generated from the original captions. Words in red indicates the irrelevant words in the adversarial captions. Best viewed in color.

Supplement B.3. Unified text-to-image retrieval

Component queries must appear in at least three test images, or five for noun queries. Baselines encode every query as a short sentence. Unified VSE selects its object or sentence encoder using validation performance where both apply. Detection-category queries use MS-COCO annotation labels. Supplementary Figure 4 retains all six sets of twenty retrieved images.

Supplementary Figure 4
Supplementary Figure 4. The top-20 retrieved image in the 1K test split set by queries different types: attribute-object pairs and relational triples.

Supplement B.4. Semantic parsing

Candidate adjective or relation dependencies are enumerated over nouns in each caption; semantically ambiguous relation words such as of and to are excluded. Attribute recovery receives exact-match credit, while relation recovery receives half credit per correct endpoint.

The twelve panels in Supplementary Figure 5 expose parser errors, visually ambiguous alternatives, a preference for spatially close objects, and two words referring to the same object. Several disagreements with parser-derived targets are therefore not unambiguous failures of visual grounding.

Supplementary Figure 5
Supplementary Figure 5. Examples showing the result of semantic parsing based on visual cues. The first and second rows visualize the examples of corrected dependency resolution and the last two rows are the failure cases (dependency resolutions differs from the one by our semantic parser). Words in italic are the content words whose dependency is to be recovered, and the words in red are wrong predictions. Fig. (g) is a failure case of our semantic parser: the word couple does not refer to a specific object in the scene. In Fig. (h), both dependencies cat-next-chair and cat-next-table are actually valid based only on visual cue. Similarly, Fig. (i), (j) and (k) are all cases where only visual cues can not recover the dependency. The result in Fig. (i) shows that our model has the tendency of linking spatially closer objects. In Fig. (l), hat and bananas actually refers to the same object. Best viewed in color.

Supplement B.5. Embedding visualization

Supplementary Figure 6 preserves the object/attribute, relation, and sentence t-SNE projections. Nearby components show qualitative semantic similarity; this visualization alone does not quantify generalization or prove compositional structure.

Supplementary Figure 6
Supplementary Figure 6. The visualization of the semantic embedding space of different semantic levels. The unified VSE space successfully recovers the similarities between semantic components at various levels.
Concept demonstration

Open image ↗