Overview
At a glance
VSE-C learns stronger links between caption words and image concepts by constructing hard negative captions. It changes concrete nouns, numbers, noun-phrase order, or prepositions, then trains a visual-semantic embedding to distinguish these captions from the original image description. On MS-COCO, the resulting embeddings resist the same attack families and transfer to object-word retrieval and sentence completion. The evidence concerns this caption dataset and these synthetic attacks; it does not establish general counting or unrestricted compositional understanding.
1 Introduction
Caption collections cover only a small fraction of possible visual meanings. Standard image-caption retrieval can therefore succeed without distinguishing precisely which object, quantity, or relation a sentence describes. The paper exposes this gap by making small, meaningful caption changes that confuse VSE and VSE++ [16] [5]. VSE-C adds linguistically constrained negative captions during training, emphasizing the mismatch between a particular image and a particular altered concept. The aim is better grounding and transfer, rather than attack defense alone.
2 Related works
Joint embeddings
Joint embeddings map different modalities into a shared space. Here, the relevant alignment is between images and natural-language descriptions; learned word representations should retain useful visual meaning for later retrieval or prediction tasks.
Image-to-text translation
Earlier cross-modal approaches use correlated projections, neural image and sentence encoders, ranking losses, attention, or hard-negative mining. VSE++ emphasizes the hardest mismatched examples in a minibatch. VSE-C supplements these ordinary mismatches with targeted edits to the caption paired with the image.
Adversarial attack in text domain
Adversarial examples are used as a diagnostic of weak semantic grounding and as training data. Unlike arbitrary text corruption, the proposed negatives preserve much of a sentence while deliberately changing a visual concept.
3 Method
3.1 Preliminaries
Word embeddings
Each word has distributional and visually grounded components. GloVe supplies pretrained distributional information [30]; image-caption alignment learns the visual component.
Visual-semantic embeddings
A fixed ResNet-152 encodes images and a GRU encodes captions. Linear projections place both representations in a common space. A margin-based ranking loss favors matched image-caption pairs over mismatched pairs drawn from the batch. The original loss and similarity definition are retained below.
Equation 1
| (1) |
3.2 Generating contrastive adversarial samples
Negatives target nouns, numerals, and relations. Table 1 retains all four concrete edit procedures: noun replacement, numeral replacement, noun-phrase shuffling, and preposition replacement.
| Class | Original Caption | Contrastive Adversarial Example |
|---|---|---|
| Noun | A person feeding a cat with a banana. | A person feeding a dog with a banana. |
| Numeral | A person feeding a cat with a banana. | A person feeding five cats with a banana. |
| Relation-1 | A person feeding a cat with a banana. | A cat feeding a person with a banana. |
| Relation-2 | A person feeding a cat with a banana. | A person feeding a cat in a banana. |
Noun.
Candidate noun heads must occur more than 200 times and exceed a concreteness threshold of 0.6. Replacements come from similarly frequent, concrete nouns. WordNet excludes direct hypernym or hyponym pairs, reducing cases where the replacement remains a valid description: replacing cat with animal would not create a reliable negative.
Numeral.
Numbers are replaced with different quantities, with singular/plural agreement adjusted accordingly. The articles a and an count as one. This design changes grammatical form as well as quantity in some examples, motivating the separate plural-only evaluation.
Relation.
Relation edits either reorder noun phrases while preserving the bag of words or replace frequent prepositions. A manually annotated overlap table excludes some semantically compatible substitutions. SpaCy supplies noun-phrase, number, and preposition detection; these linguistic rules and lexical resources encode human prior knowledge.
3.3 Intra-pair hard negative mining
VSE++ mines the hardest image or caption mismatch within a minibatch. VSE-C adds a second margin loss for the hardest generated caption associated with the same image. Roughly 1,000 candidate negatives are available per caption, but only eight are sampled in each iteration. This limits computation while exposing more targeted alternatives than the current batch alone. The method still depends on the generated negatives being semantically valid.
Equations 2–3
| (2) |
| (3) |
4 Experiments
The experiments first test controlled caption attacks, then examine saliency, object-word alignment, and filling missing concept words. Image features remain fixed, so improvements primarily reflect the text representation and cross-modal alignment.
4.1 Adversarial attacks
Evaluation uses 1,000 selected MS-COCO test images with five positive captions each. The original pool contains 5,000 captions. For each image,60 edits per positive caption add 300 adversarial candidates, divided equally among noun, numeral, and relation attacks. Recall at rank k asks whether at least one true caption appears among the top k; median and mean rank provide complementary measures.
VSE-C are more robust to known-typed adversarial attacks than VSE and VSE++.
VSE and VSE++ deteriorate substantially under these attacks. VSE-C is more robust, with the strongest defense generally matching the negative type seen during training. Frequent object nouns are easier for the baselines than numbers and relations. This is evidence of robustness to the specified, known attack families, rather than arbitrary adversarial text.
Numeral-typed VSE-C improves the counting ability of models.
A plural-only subset contains 205 images and 1,025 captions. Changing one plural quantity to another tests more than singular-versus-plural grammar, and numeral-trained VSE-C improves over VSE++. Nevertheless,99.8% of plural counts in the test set are two through five. The experiment could largely be a small-class classification problem and does not fully assess counting.
| Model | MS-COCO Test | MS-COCO Test (w/. adversarial) | ||||||
|---|---|---|---|---|---|---|---|---|
| R@1 | R@10 | Med r. | Mean r. | R@1 | R@10 | Med r. | Mean r. | |
| VSE | 47.7 | 87.8 | 2.0 | 5.8 | 28.0 | 71.6 | 4.0 | 11.7 |
| VSE++ | 55.7 | 92.4 | 1.0 | 4.3 | 35.6 | 72.5 | 3.0 | 11.8 |
| VSE-C (+n.) | 50.7 | 90.7 | 1.0 | 5.2 | 40.3 | 80.2 | 2.0 | 9.2 |
| VSE-C (+num.) | 53.3 | 90.2 | 1.0 | 5.8 | 46.9 | 86.3 | 2.0 | 6.9 |
| VSE-C (+rel.) | 52.4 | 89.0 | 1.0 | 5.7 | 42.3 | 82.5 | 2.0 | 7.2 |
| VSE-C (+all) | 50.2 | 89.8 | 1.0 | 5.2 | 47.4 | 88.8 | 2.0 | 5.5 |
| Model | MS-COCO Test (+n.) | MS-COCO Test (+num.) | MS-COCO Test (+rel.) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| R@1 | R@10 | Mean r. | R@1 | R@10 | Mean r. | R@1 | R@10 | Mean r. | |
| VSE | 37.6 | 85.8 | 6.9 | 38.5 | 82.3 | 7.7 | 30.7 | 76.7 | 8.8 |
| VSE++ | 45.7 | 89.1 | 5.5 | 45.9 | 82.3 | 7.2 | 42.3 | 80.0 | 7.6 |
| VSE-C (+n.) | 49.2 | 88.4 | 5.7 | 42.1 | 80.3 | 9.1 | 40.4 | 83.3 | 7.1 |
| VSE-C (+num.) | 51.0 | 89.5 | 6.1 | 53.3 | 90.2 | 5.8 | 49.0 | 87.0 | 6.6 |
| VSE-C (+rel.) | 48.0 | 88.8 | 5.3 | 45.4 | 83.9 | 6.7 | 50.1 | 90.2 | 4.9 |
| VSE-C (+all.) | 49.4 | 89.3 | 5.3 | 49.9 | 89.6 | 5.2 | 47.9 | 89.4 | 5.3 |
| Model | MS-COCO Test (plural split, +plurals) | ||
|---|---|---|---|
| R@1 | R@10 | Mean r. | |
| VSE++ | 43.7 | 78.3 | 9.1 |
| VSE-C (+num.) | 50.6 | 84.4 | 7.8 |
4.2 Saliency visualization
Input gradients show that VSE-C responds more strongly to the edited numeral or relation words than VSE++. Image saliency still concentrates on the dominant object. The authors attribute this limitation partly to ImageNet-pretrained ResNet features, which may represent secondary objects and relations less effectively. Figure 2 preserves both the image maps and the word-level saliency values.
Equation 4
| (4) |
4.3 Correlate words and objects
A separate image-to-word task tests whether the learned embeddings support object grounding beyond memorizing caption co-occurrence.
Dataset
Noun-phrase heads in captions define positive objects for each image. Candidate object words must occur more than 200 times; direct hypernyms and hyponyms of positive objects are excluded from the negative set to reduce ambiguity. Table 5 retains the example image, captions, and object labels.
| Image | Captions |
|---|---|
|
A table with a huge glass vase and fake flowers come out of it. |
| A plant in a vase sits at the end of a table. | |
| A vase with flowers in it with long stems sitting on a table with candles. | |
| A large centerpiece that is sitting on the edge of a dining table. | |
| Flowers in a clear vase sitting on a table. | |
| Positive Objects: table, plant, vase. | |
| Negative Objects: screen, pickle, sandwich, toy, hill, coat, cat, etc. | |
Training
An alignment network receives the outer product of the frozen image and word vectors. Only an MLP classifier is trained on image-word-positive/negative triples. Keeping both embeddings fixed makes this a transfer evaluation of their learned representations.
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/de/dee0a556367f2f5a3bf79a38a956500292f09e0163d2cc82672da5965f8aaa84.webp)
| Model | MAP |
|---|---|
| GloVe | 58.7 |
| VSE | 61.7 |
| VSE++ | 61.1 |
| VSE-C (+all) | 62.2 |
| VSE-C (+n.) | 62.8 |
| VSE-C (+rel.) | 62.3 |
| VSE-C (+num.) | 62.0 |
Testing
Average precision measures how well each image ranks its positive object words; MAP averages across images. Candidate object sets can differ between queries. The original metric definition is retained below.
Equation 5
| (5) |
Results
Noun-trained VSE-C reaches 62.8% MAP, compared with 61.1% for VSE++,61.7% for VSE, and 58.7% for GloVe. Relation-trained VSE-C reaches 62.3%, suggesting that sentence-level relation edits can also improve object-word alignment. The full table preserves all model variants.
4.4 Concept to word
A second transfer task blanks a noun or relational word and asks a separately trained model to recover it from the image and remaining caption.
Dataset and implementation details
SpaCy identifies candidate concept words in MS-COCO captions. Training randomly masks one selected word and predicts its embedding from the image and the surrounding text.
Model
Separate GRUs encode words before and after the blank. Their final states are concatenated with fixed ResNet-152 image features, and a two-layer MLP predicts a word embedding using cosine similarity. Figure 4 preserves the full architecture.
Results
Visually trained embeddings outperform ungrounded GloVe on this completion task, and the combined VSE-C variant improves over VSE++. Differences on preposition completion are small; the authors connect this to limited diversity of visual relations in the dataset. These results complement, rather than replace, the controlled attack evaluation.
| Model | Noun Filling | Prep. Filling | All (n. + prep.) | |||
|---|---|---|---|---|---|---|
| R@1 | R@10 | R@1 | R@10 | R@1 | R@10 | |
| GloVe | 23.2 | 58.8 | 23.3 | 79.9 | 23.3 | 66.6 |
| VSE++ | 25.0 | 61.7 | 34.9 | 84.9 | 28.4 | 68.1 |
| VSE-C (ours) | 27.3 | 62.9 | 35.2 | 85.2 | 30.0 | 70.98 |
5 Discussion and conclusion
Targeted caption negatives improve visual grounding in this experimental setting and transfer beyond the original retrieval objective. The work also exposes limits: caption data sparsely cover compositional meaning, pretrained image features can neglect secondary concepts, counting tests cover a narrow range, and rule-generated negatives depend on lexical knowledge and semantic-overlap judgments. The results should not be read as a general guarantee of visual reasoning.
Appendix A Semantic Overlap Table of Frequent Prepositions
The complete manually annotated table groups prepositions that can overlap in meaning. A preposition may belong to several groups. These groups constrain negative generation rather than defining universally interchangeable words.
| Set # | Words in a Semantical Set |
|---|---|
| 1 | towards, toward, beyond, to |
| 2 | behind, after, past |
| 3 | outside, out |
| 4 | underneath, under, beneath, down, below |
| 5 | on, upon, up, un, atop, onto, over, above, beyond |
| 6 | in, within, among, at, during, into, inside, from, between |
| 7 | if, while |
| 8 | with, by, beside |
| 9 | around, like |
| 10 | to, for, of |
| 11 | about, within |
| 12 | because, as, for |
| 13 | as, like |
| 14 | near, next, beside |
| 15 | though |
| 16 | thru, through |
| 17 | besides, along |
| 18 | against, next, to |
| 19 | along, during, across, while |
| 20 | off, out |
| 21 | without |
| 22 | than |
| 23 | before |
Appendix B Training Details of VSE-C
Training uses Adam at 0.001 with batch size 128 and multiplies the learning rate by 0.1 every 15 epochs. No regularization or dropout is applied. Word vectors start from 300-dimensional GloVe; the single-layer bidirectional GRU has 512 units in each direction, and the joint embedding has 1,024 dimensions. The authors report little variation from different negative-sampling seeds when other settings are fixed.