Literature
Display
← All papers

Learning Visually-Grounded Semantics from Contrastive Adversarial Samples

Haoyue Shi · Jiayuan Mao · Tete Xiao · Yuning Jiang · Jian Sun

· COLING 2018 ↗

Contents

Overview

At a glance

VSE-C learns stronger links between caption words and image concepts by constructing hard negative captions. It changes concrete nouns, numbers, noun-phrase order, or prepositions, then trains a visual-semantic embedding to distinguish these captions from the original image description. On MS-COCO, the resulting embeddings resist the same attack families and transfer to object-word retrieval and sentence completion. The evidence concerns this caption dataset and these synthetic attacks; it does not establish general counting or unrestricted compositional understanding.

1 Introduction

Caption collections cover only a small fraction of possible visual meanings. Standard image-caption retrieval can therefore succeed without distinguishing precisely which object, quantity, or relation a sentence describes. The paper exposes this gap by making small, meaningful caption changes that confuse VSE and VSE++ [16] [5]. VSE-C adds linguistically constrained negative captions during training, emphasizing the mismatch between a particular image and a particular altered concept. The aim is better grounding and transfer, rather than attack defense alone.

Refer to caption
Figure 1: An overview of our textual contrastive adversarial samples. For each caption, we have three paradigms to generate contrastive adversarial samples, i.e., noun, numeral and relation. For an given image, we expect the model to distinguish the real captions against the generated adversarial ones.

2 Related works

Joint embeddings

Joint embeddings map different modalities into a shared space. Here, the relevant alignment is between images and natural-language descriptions; learned word representations should retain useful visual meaning for later retrieval or prediction tasks.

Image-to-text translation

Earlier cross-modal approaches use correlated projections, neural image and sentence encoders, ranking losses, attention, or hard-negative mining. VSE++ emphasizes the hardest mismatched examples in a minibatch. VSE-C supplements these ordinary mismatches with targeted edits to the caption paired with the image.

Adversarial attack in text domain

Adversarial examples are used as a diagnostic of weak semantic grounding and as training data. Unlike arbitrary text corruption, the proposed negatives preserve much of a sentence while deliberately changing a visual concept.

3 Method

3.1 Preliminaries

Word embeddings

Each word has distributional and visually grounded components. GloVe supplies pretrained distributional information [30]; image-caption alignment learns the visual component.

Visual-semantic embeddings

A fixed ResNet-152 encodes images and a GRU encodes captions. Linear projections place both representations in a common space. A margin-based ranking loss favors matched image-caption pairs over mismatched pairs drawn from the batch. The original loss and similarity definition are retained below.

Equation 1
ℓV​S​E​(i,c)=∑c′[α+s⁡(i,c′)−s⁡(i,c)]++∑i′[α+s⁡(i′,c)−s⁡(i,c)]+,\ell^{VSE}(i,c)=\sum_{c^{\prime}}[\alpha+s(i,c^{\prime})-s(i,c)]_{+}+\sum_{i^{\prime}}[\alpha+s(i^{\prime},c)-s(i,c)]_{+}\,, (1)

3.2 Generating contrastive adversarial samples

Negatives target nouns, numerals, and relations. Table 1 retains all four concrete edit procedures: noun replacement, numeral replacement, noun-phrase shuffling, and preposition replacement.

Class Original Caption Contrastive Adversarial Example
Noun A person feeding a cat with a banana. A person feeding a dog with a banana.
Numeral A person feeding a cat with a banana. A person feeding five cats with a banana.
Relation-1 A person feeding a cat with a banana. A cat feeding a person with a banana.
Relation-2 A person feeding a cat with a banana. A person feeding a cat in a banana.
Table 1: Examples of contrastive adversarial samples generated with our heuristic rules and knowledge from WordNet. The samples can be classified into four types: noun replacement, numeral replacement, relation shuffling, and relation replacement.

Noun.

Candidate noun heads must occur more than 200 times and exceed a concreteness threshold of 0.6. Replacements come from similarly frequent, concrete nouns. WordNet excludes direct hypernym or hyponym pairs, reducing cases where the replacement remains a valid description: replacing cat with animal would not create a reliable negative.

Numeral.

Numbers are replaced with different quantities, with singular/plural agreement adjusted accordingly. The articles a and an count as one. This design changes grammatical form as well as quantity in some examples, motivating the separate plural-only evaluation.

Relation.

Relation edits either reorder noun phrases while preserving the bag of words or replace frequent prepositions. A manually annotated overlap table excludes some semantically compatible substitutions. SpaCy supplies noun-phrase, number, and preposition detection; these linguistic rules and lexical resources encode human prior knowledge.

3.3 Intra-pair hard negative mining

VSE++ mines the hardest image or caption mismatch within a minibatch. VSE-C adds a second margin loss for the hardest generated caption associated with the same image. Roughly 1,000 candidate negatives are available per caption, but only eight are sampled in each iteration. This limits computation while exposing more targeted alternatives than the current batch alone. The method still depends on the generated negatives being semantically valid.

Equations 2–3
ℓVSE++​(i,c)=maxc′≠c⁡[α+s⁡(i,c′)−s⁡(i,c)]+maxi′≠i⁡[α+s⁡(i′,c)−s⁡(i,c)].\ell^{\text{VSE++}}(i,c)=\max_{c^{\prime}\neq c}[\alpha+s(i,c^{\prime})-s(i,c)]+\max_{i^{\prime}\neq i}[\alpha+s(i^{\prime},c)-s(i,c)]. (2)
ℓVSE-C​(i,c)=ℓVSE++​(i,c)+maxc′′∈𝒞′​(c)⁡[α+s⁡(i,c′′)−s⁡(i,c)]+.\ell^{\text{VSE-C}}(i,c)=\ell^{\text{VSE++}}(i,c)+\max_{c^{\prime\prime}\in\mathcal{C}^{\prime}(c)}[\alpha+s(i,c^{\prime\prime})-s(i,c)]_{+}. (3)

4 Experiments

The experiments first test controlled caption attacks, then examine saliency, object-word alignment, and filling missing concept words. Image features remain fixed, so improvements primarily reflect the text representation and cross-modal alignment.

4.1 Adversarial attacks

Evaluation uses 1,000 selected MS-COCO test images with five positive captions each. The original pool contains 5,000 captions. For each image,60 edits per positive caption add 300 adversarial candidates, divided equally among noun, numeral, and relation attacks. Recall at rank k asks whether at least one true caption appears among the top k; median and mean rank provide complementary measures.

VSE-C are more robust to known-typed adversarial attacks than VSE and VSE++.

VSE and VSE++ deteriorate substantially under these attacks. VSE-C is more robust, with the strongest defense generally matching the negative type seen during training. Frequent object nouns are easier for the baselines than numbers and relations. This is evidence of robustness to the specified, known attack families, rather than arbitrary adversarial text.

Numeral-typed VSE-C improves the counting ability of models.

A plural-only subset contains 205 images and 1,025 captions. Changing one plural quantity to another tests more than singular-versus-plural grammar, and numeral-trained VSE-C improves over VSE++. Nevertheless,99.8% of plural counts in the test set are two through five. The experiment could largely be a small-class classification problem and does not fully assess counting.

Model MS-COCO Test MS-COCO Test (w/. adversarial)
R@1 R@10 Med r. Mean r. R@1 R@10 Med r. Mean r.
VSE 47.7 87.8 2.0 5.8 28.0 71.6 4.0 11.7
VSE++ 55.7 92.4 1.0 4.3 35.6 72.5 3.0 11.8
VSE-C (+n.) 50.7 90.7 1.0 5.2 40.3 80.2 2.0 9.2
VSE-C (+num.) 53.3 90.2 1.0 5.8 46.9 86.3 2.0 6.9
VSE-C (+rel.) 52.4 89.0 1.0 5.7 42.3 82.5 2.0 7.2
VSE-C (+all) 50.2 89.8 1.0 5.2 47.4 88.8 2.0 5.5
Table 2: Evaluation on image-to-caption retrieval. Although VSE++ [Faghri et al. (2017] obtains the best performance on original MS-COCO test set, it is more vulnerable to the caption-specific adversarial attack compared with the proposed VSE-C, and so does VSE [Kiros et al. (2014].
Model MS-COCO Test (+n.) MS-COCO Test (+num.) MS-COCO Test (+rel.)
R@1 R@10 Mean r. R@1 R@10 Mean r. R@1 R@10 Mean r.
VSE 37.6 85.8 6.9 38.5 82.3 7.7 30.7 76.7 8.8
VSE++ 45.7 89.1 5.5 45.9 82.3 7.2 42.3 80.0 7.6
VSE-C (+n.) 49.2 88.4 5.7 42.1 80.3 9.1 40.4 83.3 7.1
VSE-C (+num.) 51.0 89.5 6.1 53.3 90.2 5.8 49.0 87.0 6.6
VSE-C (+rel.) 48.0 88.8 5.3 45.4 83.9 6.7 50.1 90.2 4.9
VSE-C (+all.) 49.4 89.3 5.3 49.9 89.6 5.2 47.9 89.4 5.3
Table 3: Detailed results on each type of adversarial attack. Training VSE-C on one class gains the best performance on robustness against the adversarial attack of the class itself. In addition, training with numeral-typed adversarial samples helps improve the robustness against noun-typed and relation-typed attack. We hypothesis that this is attributed to the singularization or pluralization of the corresponding nouns in the process of numeral-typed adversarial sample generation.
Model MS-COCO Test (plural split, +plurals)
       R@1         R@10 Mean r.
VSE++ 43.7 78.3 9.1
VSE-C (+num.) 50.6 84.4 7.8
Table 4: Results on plural-typed adversarial attack to the plural split of MS-COCO test set. This split consists of 205 images, together with the 1,025 original captions. VSE-C outperforms VSE++ by a large margin on all the three considered metrics.

4.2 Saliency visualization

Input gradients show that VSE-C responds more strongly to the edited numeral or relation words than VSE++. Image saliency still concentrates on the dominant object. The authors attribute this limitation partly to ImageNet-pretrained ResNet features, which may represent secondary objects and relations less effectively. Figure 2 preserves both the image maps and the word-level saliency values.

Refer to caption
Figure 2: Saliency analysis on adversarial samples. The left column shows the saliency of VSE-C on the image (what is the difference between the image and the image you imagine from caption c′c^{\prime}), while the right column shows the saliency of both VSE++ and VSE-C on the caption (what is the difference between the caption and the caption you summarize from image ii). The magnitude of values indicates the level of saliency. For better visualization, the image saliency is LinfL_{\inf}-normalized and the caption saliency is L1L_{1}-normalized. For all the three classes of textual adversarial samples, the image encoding model (ResNet-152) almost only focuses on the main part of the image, i.e., elephant. For numeral-typed and relation-typed adversarial samples, VSE-C pays much more attention to the manipulated segments of the sentence than VSE++.
Equation 4
J=∇is​(i,c′)=∇iWiT​f​(i,θi)⋅WcT​g​(c′,θc),\textbf{J}=\nabla_{i}s(i,c^{\prime})=\nabla_{i}W_{i}^{T}f(i;\theta_{i})\cdot W_{c}^{T}g(c^{\prime};\theta_{c}), (4)

4.3 Correlate words and objects

A separate image-to-word task tests whether the learned embeddings support object grounding beyond memorizing caption co-occurrence.

Dataset

Noun-phrase heads in captions define positive objects for each image. Candidate object words must occur more than 200 times; direct hypernyms and hyponyms of positive objects are excluded from the negative set to reduce ambiguity. Table 5 retains the example image, captions, and object labels.

Image Captions
[Uncaptioned image] A table with a huge glass vase and fake flowers come out of it.
A plant in a vase sits at the end of a table.
A vase with flowers in it with long stems sitting on a table with candles.
A large centerpiece that is sitting on the edge of a dining table.
Flowers in a clear vase sitting on a table.
Positive Objects: table, plant, vase.
Negative Objects: screen, pickle, sandwich, toy, hill, coat, cat, etc.
Table 5: An example of the image-to-word retrieval dataset. We extract objects by detecting heads of noun phrases in captions. We only collect the “object” words with frequency higher than 200 in MS-COCO full dataset as available positive/negative objects for each image.

Training

An alignment network receives the outer product of the frozen image and word vectors. Only an MLP classifier is trained on image-word-positive/negative triples. Keeping both embeddings fixed makes this a transfer evaluation of their learned representations.

[Uncaptioned image]
Figure 3: Model structure for image-to-word retrieval. The network is trained though the interaction space. Note that only parameters in MLP are tuned during training.
Model MAP
GloVe 58.7
VSE 61.7
VSE++ 61.1
VSE-C (+all) 62.2
VSE-C (+n.) 62.8
VSE-C (+rel.) 62.3
VSE-C (+num.) 62.0
Table 6: Evaluation result (MAP in percentage) on image-to-word retrieval.

Testing

Average precision measures how well each image ranks its positive object words; MAP averages across images. Candidate object sets can differ between queries. The original metric definition is retained below.

Equation 5
A​P=∑k=1nP⁡(k)×p​o​s​i​t​i​v​e​(k)number​of​positive​objectsAP=\frac{\sum_{k=1}^{n}P(k)\times{positive(k)}}{\rm number~of~positive~objects} (5)

Results

Noun-trained VSE-C reaches 62.8% MAP, compared with 61.1% for VSE++,61.7% for VSE, and 58.7% for GloVe. Relation-trained VSE-C reaches 62.3%, suggesting that sentence-level relation edits can also improve object-word alignment. The full table preserves all model variants.

4.4 Concept to word

A second transfer task blanks a noun or relational word and asks a separately trained model to recover it from the image and remaining caption.

Dataset and implementation details

SpaCy identifies candidate concept words in MS-COCO captions. Training randomly masks one selected word and predicts its embedding from the image and the surrounding text.

Model

Separate GRUs encode words before and after the blank. Their final states are concatenated with fixed ResNet-152 image features, and a two-layer MLP predicts a word embedding using cosine similarity. Figure 4 preserves the full architecture.

Refer to caption
Figure 4: Model structure for fill-in-the-blank.

Results

Visually trained embeddings outperform ungrounded GloVe on this completion task, and the combined VSE-C variant improves over VSE++. Differences on preposition completion are small; the authors connect this to limited diversity of visual relations in the dataset. These results complement, rather than replace, the controlled attack evaluation.

Model Noun Filling Prep. Filling All (n. + prep.)
R@1 R@10 R@1 R@10 R@1 R@10
GloVe 23.2 58.8 23.3 79.9 23.3 66.6
VSE++ 25.0 61.7 34.9 84.9 28.4 68.1
VSE-C (ours) 27.3 62.9 35.2 85.2 30.0 70.98
Table 7: Evaluation result on the fill-in-the-blank task (in percentage). The word embeddings learned by VSE-C with all classes of contrastive adversarial samples help reach a better performance than those learned by VSE++ [Faghri et al. (2017].

5 Discussion and conclusion

Targeted caption negatives improve visual grounding in this experimental setting and transfer beyond the original retrieval objective. The work also exposes limits: caption data sparsely cover compositional meaning, pretrained image features can neglect secondary concepts, counting tests cover a narrow range, and rule-generated negatives depend on lexical knowledge and semantic-overlap judgments. The results should not be read as a general guarantee of visual reasoning.

Appendix A Semantic Overlap Table of Frequent Prepositions

The complete manually annotated table groups prepositions that can overlap in meaning. A preposition may belong to several groups. These groups constrain negative generation rather than defining universally interchangeable words.

Set # Words in a Semantical Set
1 towards, toward, beyond, to
2 behind, after, past
3 outside, out
4 underneath, under, beneath, down, below
5 on, upon, up, un, atop, onto, over, above, beyond
6 in, within, among, at, during, into, inside, from, between
7 if, while
8 with, by, beside
9 around, like
10 to, for, of
11 about, within
12 because, as, for
13 as, like
14 near, next, beside
15 though
16 thru, through
17 besides, along
18 against, next, to
19 along, during, across, while
20 off, out
21 without
22 than
23 before
Table 8: Manually annotated semantic overlap table.

Appendix B Training Details of VSE-C

Training uses Adam at 0.001 with batch size 128 and multiplies the learning rate by 0.1 every 15 epochs. No regularization or dropout is applied. Word vectors start from 300-dimensional GloVe; the single-layer bidirectional GRU has 512 units in each direction, and the joint embedding has 1,024 dimensions. The authors report little variation from different negative-sampling seeds when other settings are fixed.

Concept demonstration

Open image ↗