Literature
Display
← All papers

DisCo: Improving Compositional Generalization in Visual Reasoning through Distribution Coverage

Joy Hsu · Jiayuan Mao · Jiajun Wu

· TMLR 2023 ↗

Contents

At a glance

DisCo improves compositional visual question answering (VQA) by adapting a pretrained model to unlabeled images containing new or rare attribute combinations. It pairs these images with training questions, accepts low-entropy answer predictions as pseudo-labels, and trains on accepted examples together with labeled examples.

A confidence-based distribution-coverage score selects model checkpoints without target answer labels. Experiments use synthetic CLEVR datasets and three VQA backbones. The main setting exposes target-distribution images during adaptation and selection; a separate evaluation tests genuinely unseen images from that distribution.

1. Introduction

A model trained mostly on blue cubes and red cylinders may fail on red cubes even though it has learned the individual colors and shapes. DisCo uses unlabeled images of such combinations to extend a model’s learned associations.

The approach does not require additional human answers for those images, but does require suitable unlabeled visual data and a base model with enough correct, confident predictions to start self-training.

figure 1
Figure 1: The base VQA model in its original training paradigm is trained on labeled data, while DisCo leverages unlabeled, out-of-distribution images from the test distribution to improve compositional generalization performance.

3. Method

The framework has three stages: train a base VQA model, iteratively add confident image/question/answer pseudo-labels from an unlabeled pool, and choose a checkpoint using the fraction of candidate pairs accepted by the confidence rule.

3.1. Problem formulation

Labeled examples are image/question/answer triplets from a biased color–shape distribution. The unlabeled pool primarily supplies a different distribution of combinations. The target is object-level compositional generalization: recognizing familiar attributes when they occur in different combinations.

The experiments distinguish no labeled examples of the reversed combinations from settings with small proportions of those combinations. Thus some reported gains address rare combinations rather than entirely unseen labeled combinations.

3.2. Iterative pseudo-labeling

Algorithm 1 and Figure 2 preserve the complete procedure. A question sampled from labeled training data is paired with an unlabeled proposed image. The current model supplies an answer distribution; if its entropy is below the threshold, its highest-probability answer becomes a pseudo-label.

An accepted pseudo-labeled example and a labeled example receive equally weighted cross-entropy training. The labeled term is intended to anchor correctness while learning from the new distribution. Rejected candidate pairs do not supply pseudo-label updates. The original sixteen-step algorithm is retained without altering its printed loop notation.

Complete original Algorithm 1, including inputs, output and all sixteen steps
figure 2
Figure 2: Overview of the DisCo framework. At each pseudo-labeling step, labeled VQA triplets from the train distribution are used in an equal weighting to pseudo-labeled VQA triplets, which contain images from the test distribution of novel attribute compositions.

3.3. Image proposals

DisCo-S directly samples unlabeled target images. DisCo-G trains an unconditional StyleGAN2 on the same unlabeled pool and uses generated images as proposals [14]. The generator can vary scenes beyond the finite pool but introduces its own image-quality and coverage limitations.

Both routes use information from the unlabeled distribution. Generated proposals do not make the method independent of access to target-like images.

3.4. Entropy filtering

Random pairing creates two error sources: the question’s referred object may not exist, or the model may answer incorrectly despite a valid referent. Entropy, H(p) = −∑ pᵢ log pᵢ, is used as an empirical filter for both.

At the illustrated threshold near the thirtieth percentile, 85.06% of accepted pairs satisfy the question’s presupposition, and 82.07% both satisfy it and have the correct answer. Conditional accuracy among valid pairs is 96.48%. These are different denominators; low entropy does not eliminate erroneous pseudo-labels.

Figure 3 shows the largest change in presupposition adherence as the entropy threshold expands, while accuracy conditional on adherence changes less. This supports the filter’s usefulness in this dataset without establishing universal confidence calibration.

figure 3
Figure 3: Relationship between entropy and compositional generalization accuracy and count. The base VQA model is trained on a biased train dataset; entropy and prediction accuracy is evaluated on unlabeled images sampled from the test dataset paired with questions from the train set.

3.5. Model selection

Coverage is the fraction of proposed image/question pairs that pass entropy filtering. The checkpoint with greatest coverage is selected. This is confidence coverage under the sampling and threshold rule, not a direct measurement of all distinct semantic combinations covered.

In Figure 4, Pearson correlation with target accuracy is 0.6066 for coverage versus 0.2565 for ordinary validation accuracy. The validation set shares the biased training distribution and is close to saturation. Coverage is a stronger empirical selection signal here, but remains an imperfect proxy.

figure 4
Figure 4: Correlation between test set and validation set accuracy (left plot, with Pearson correlation coefficient of 0.2565) as well as between test set accuracy and our distribution coverage metric (right plot, with a coefficient of 0.6066).

4. Experiments

The method is tested with FiLM, Transparency by Design (TbD-net), and the Neuro-Symbolic Concept Learner (NS-CL) [30] [25] [24]. Baselines pretrain representations with a variational autoencoder (VAE) or SimCLR on exactly the same unlabeled images.

The comparison therefore evaluates different ways of using a shared unlabeled pool. It is not a comparison between DisCo with target data and baselines denied that data.

4.1. Datasets and implementation

CLEVR split A pairs cubes with gray/blue/brown/yellow and cylinders with red/green/purple/cyan; split B reverses these assignments. Spheres use all eight colors. Training includes either 0%, 0.1%, or 0.5% of objects from the reversed distribution.

The Ref dataset asks about attributes of referred objects, while OneHop adds relational questions. The original CLEVR compositional-generalization benchmark, CoGenT, contains more complex scenes and questions [13].

Backbones use their published implementations and hyperparameters. StyleGAN2 uses Adam at learning rate 0.002. The entropy threshold is chosen around the thirtieth percentile for the simpler datasets and the tenth percentile for CoGenT, where fewer random pairings satisfy presuppositions. Training uses one Titan RTX GPU.

4.2. Main results

On 0.5% Ref, FiLM improves from 0.9265 to 0.9621 with direct sampling and 0.9545 with generated proposals. On the separately held-out image set, scores are 0.9251, 0.9616, and 0.9510 respectively. These unseen-image results support gains beyond reusing the exact adapted image pool.

Direct sampling also improves OneHop from 0.9270 to 0.9469 and CoGenT from 0.7859 to 0.8004. However, the fully biased 0% Ref case is an exception to universal improvement: DisCo-S scores 0.7582 versus the 0.7589 base, while DisCo-G reaches 0.7760.

The “unknown” condition uses a mixed unlabeled pool containing both split A and split B while testing on B. It relaxes exact target-distribution matching but still includes target combinations; it is not an arbitrary unrelated unlabeled distribution.

table 1
Table 1: DisCo performance compared to the original training paradigm and baselines, where 0.5% Ref (unk) is performance on an unknown test distribution, with the unlabeled dataset not drawn from the specified test distribution, and 0.5% Ref (unseen) is performance on an unseen test set not exposed during training .

Other VQA backbones

Tables 2 and 3 preserve all original, unknown-distribution, and unseen-image results. On 0.5% Ref, direct-sampling DisCo raises TbD-net from 0.8993 to 0.9206 and NS-CL from 0.7622 to 0.8024. Generated proposals also improve both, with smaller gains in these comparisons.

table 2
Table 2: Comparison of DisCo with baselines on the TbD-net model. Performance reported on the original, unknown, and unseen test set of 0.5% Ref.
table 3
Table 3: Comparison of DisCo with baselines on the NS-CL model. Performance reported on the original, unknown, and unseen test set of 0.5% Ref.

4.3. Component ablations

Unfiltered pseudo-labeling lowers FiLM accuracy from 0.9265 to 0.6749. Adding entropy filtering raises it to 0.9605, and coverage-based selection gives 0.9621. The large failure without filtering is central evidence that adding unlabeled pairs alone is insufficient.

table 4
Table 4: Ablation of DisCo with direct sampler, trained on the 0.5% bias Ref. (PL = pseudo-labeling, EM = entropy measure, CS = coverage selection).

Selection-strategy comparison

Table 5 isolates checkpoint selection within each DisCo run. For direct sampling, replacing validation-accuracy selection with coverage changes FiLM from 0.9605 to 0.9621, TbD-net from 0.9000 to 0.9206, and NS-CL from 0.7887 to 0.8024.

The authors do not apply this metric to VAE or SimCLR pretraining because those baselines do not construct pseudo-labels. Selection gains vary substantially across backbones.

table 5
Table 5: Comparison of test accuracy with baselines on model selection strategy, trained on 0.5% Ref.

4.4. Qualitative analysis

Figure 5 preserves successful and unsuccessful answers. In a brown-cylinder example, the base model and DisCo-S answer incorrectly, while DisCo-G answers correctly. A related generated scene motivates the hypothesis that generator diversity helps connect attribute combinations.

These examples illustrate a possible explanation, not an isolated causal test of why generated proposals improve the fully biased case.

figure 5
Figure 5: Top row: two prediction examples from FiLM, DisCo-S, and DisCo-G on fully biased Ref. Bottom row: two closely aligned images from the train set and from the GAN proposal function.

Limitations

Bootstrapping relies on useful visual inductive biases and sufficiently correct initial predictions. Confidence can otherwise reinforce incorrect associations. The experiments use synthetic CLEVR scenes and do not establish performance on natural images, robot tasks, or much weaker initial models.

Coverage depends on the unlabeled pool itself. Missing relevant combinations cannot be repaired simply by maximizing confidence over a narrow pool, and the empirical correlation does not guarantee safe model selection under every shift.

5. Conclusion

The strongest result is a practical adaptation recipe: filter pseudo-labels for answerability and confidence, retain labeled supervision, and select using target-pool coverage. Across the evaluated backbones this improves several compositional benchmarks, while the fully biased direct-sampling exception and attribute-level regressions delimit the claim.

Broader impact

The paper emphasizes that an unlabeled dataset must itself cover relevant variation. Otherwise, self-training may inherit or amplify its biases despite requiring fewer labels.

Acknowledgments

The authors thank Eric Chan and acknowledge support from Stanford’s Institute for Human-Centered AI, Toyota Research Institute, the National Science Foundation, Office of Naval Research, and industrial sponsors. Joy Hsu acknowledges Knight-Hennessy and NSF graduate fellowship support.

Appendix A

All nine supplementary subsections are retained, with complete proposal examples, threshold tests, coverage stress tests, unknown-distribution comparisons, and per-attribute results.

A.1. Code

The released DisCo implementation builds on the FiLM codebase. The authors again caution that adequate unlabeled-data coverage is necessary.

A.2. Dataset construction

The generated Ref and OneHop data use two or three objects per scene. Split A has 8,000 training images and 2,000 validation images; split B has 2,000 target images. A further 8,000-image set evaluates unseen images from the same target distribution.

A.3. Proposal examples

Figure 6 retains sixteen example images from direct sampling and GAN generation, showing the range of proposed attribute combinations. Their visual diversity is illustrative rather than a quantitative guarantee of complete distribution coverage.

figure 6
Figure 6: Image proposals from direct sampling and GAN generation, used as unlabeled image input in DisCo.

A.4. Entropy thresholds

Table 6 reports 0.9612, 0.9616, 0.9667, and 0.9643 for entropy percentiles 30, 35, 40, and 45 on 0.5% Ref. Figure 7 retains the full accuracy trajectories and pseudo-labeling onset marker.

Runs vary both random seed and threshold. Their consistent improvements indicate some robustness, but the table does not isolate threshold effects from seed variability or provide repeated-run uncertainty for each threshold.

table 6
Table 6: Comparison of different percentiles of entropy thresholds on the 0.5% biased Ref dataset.
figure 7
Figure 7: Test accuracy curves of the base FiLM model (red), and DisCo-S at different percentiles of entropy thresholds.

A.5. Coverage under degradation

Figures 8 and 9 preserve cases where performance degrades. Coverage tracks accuracy with reported correlations 0.6585 and 0.7987, including a high-threshold run that fails substantially.

The proposed explanation is that conflicting labeled and pseudo-labeled supervision reduces confidence, thereby rejecting more candidate pairs. This is an empirical observation in the examples, not proof that erroneous labels cannot propagate.

figure 8
Figure 8: Test accuracy and distribution coverage curves of an experiment with large decrease in model performance, with Pearson correlation coefficient of 0.6585.
figure 9
Figure 9: Test accuracy and distribution coverage curves of an experiment with a slight decrease in model performance, with Pearson correlation coefficient of 0.7987.

A.6. Unknown distributions

The supplementary “unknown-subset” setting trains on labeled A, adapts with unlabeled B, and evaluates on the full A+B distribution. Tables 7–9 retain all comparisons. Direct sampling improves FiLM from 0.9634 to 0.9798, TbD-net from 0.9506 to 0.9582, and NS-CL from 0.8810 to 0.9011.

This tests an unlabeled pool narrower than the evaluation distribution, complementing the main unknown condition’s broader pool.

table 7
Table 7: DisCo on the FiLM model. Performance reported on a test set of 0.5% Ref from an unknown test distribution.
table 8
Table 8: DisCo on the TbD-net model. Performance reported on a test set of 0.5% Ref from an unknown test distribution.
table 9
Table 9: DisCo on the NS-CL model. Performance reported on a test set of 0.5% Ref from an unknown test distribution.

A.7. Baseline implementation

VAE and SimCLR baselines start from ResNet101 image features, add convolution, batch normalization, and LeakyReLU layers, and train with feature reconstruction or contrastive objectives. Their trained encoders then initialize additional visual encoding for the VQA models.

A.8. Attribute-level results

Table 10 retains every color–shape row. GAN proposals improve unseen cubes but lower several cylinder accuracies: blue cylinders fall from 0.6431 to 0.5752, and yellow cylinders from 0.6385 to 0.6106. Several sphere results also decrease slightly.

Thus aggregate improvement is not uniform compositional mastery. Different combinations benefit or regress, even in the best overall fully biased model.

table 10
Table 10: Comparison of test accuracy per color-shape combination of the referred object, trained on the fully biased Ref dataset.

A.9. Additional examples

Figure 10 retains five examples spanning low-entropy correct answers, valid but incorrectly answered questions, and a missing-referent failure. Figure 11 retains ten additional comparisons among the base model and both proposal variants, including successes unique to generated proposals.

figure 10
Figure 10: Qualitative examples VQA pairs at different levels of entropy.
figure 11
Figure 11: Qualitative examples of FiLM, DisCo-S, and DisCo-G on the fully biased Ref dataset.
Concept demonstration

Open image ↗