Literature
Display
← All papers

Object-Centric Diagnosis of Visual Reasoning

Jianwei Yang · Jiayuan Mao · Jiajun Wu · Devi Parikh · David D. Cox · Joshua B. Tenenbaum · Chuang Gan

Preprint posted · arXiv (preprint) ↗

Contents

At a glance

Final answer accuracy can conceal poor visual grounding and sensitivity to irrelevant objects. This paper uses GQA’s scene graphs and step-by-step question programs to compare where visual question-answering models attend and whether their answers change when selected object representations are removed. The two diagnostics reveal weaknesses that accuracy alone misses.

A Graph Reasoning Machine combines probabilistic scene graphs, generated programs, and a trainable executor with intermediate supervision. It improves accuracy, grounding, and background-removal stability over the purely symbolic comparison. The strongest ablation replaces predicted perception with annotated scene graphs, showing how much performance is limited by image understanding. The findings concern these models, annotation choices, and perturbation tests; attention alignment is not itself proof of faithful reasoning.

I Introduction

Figure 1 shows two motivating failures: a model retains an answer after relevant houses are covered, yet changes another answer when an irrelevant region is hidden. The study makes such concerns measurable using object boxes and reasoning annotations.

Its three contributions are object-level grounding and robustness diagnostics, comparisons across different reasoning architectures, and an interpretable Graph Reasoning Machine for analyzing component failures. The systematic robustness experiments remove object representations at the reasoning-module input; they should be distinguished from unrestricted adversarial changes to image pixels.

Refer to caption
Fig. 1: Two simple probing tests show that current VQA system fails to grounding on the correct objects and robust to perturbations: at top row, model gives the same answer after removing foreground objects “houses”; At bottom row, however, covering a background object inverts the answer.

II Related Work

Table I compares the visual representations, question representations, and reasoning mechanisms of the evaluated model families.

Method Visual Representation Question Representation Viusal Reasoning
Bottom-Up [10] Object Recurrent Neural Network Sequential attention
LCGN [12] Object Recurrent Neural Network Sequential attention
NSM [14] Scene graph Recurrent Neural Network Sequential attention
LXMERT [27] Object Self-attention Self-attention
NS-VQA [11] Scene graph Symbolic program Sequential execution
MMN [26] Object Neural program Sequential execution
TABLE I: An overview of VQA models on GQA by comparing their visual representations, question representations and reasoning method.

II-A Visual Question Answering

Visual question-answering systems combine image features, question encodings, and a multimodal prediction mechanism. Object features, contextual graph models, and large pretrained cross-modal networks improve accuracy, but accuracy cannot identify whether the answer depended on the intended objects. GQA’s scene graphs and programs supply richer diagnostic annotations than final answers alone.

II-B Grounding

Pixel activation maps and human-attention alignment offer one view of grounding, but are difficult to compare across purely neural and symbolic systems. The proposed diagnostic instead ranks detected object boxes and compares them with annotated target objects. This improves comparability while still requiring architecture-specific choices about which attention or execution scores to expose.

II-C Robustness

Prior robustness studies often alter questions, answer distributions, or image pixels. Object removal provides a semantic perturbation: an answer should remain stable when evidence irrelevant to the annotated reasoning chain is removed. The validity of that expectation depends on the completeness of the annotations and the definition of irrelevant evidence.

III Object-Centric Diagnosis

The diagnostics separate final object grounding from stability under visual-input perturbations, and assess both alongside answer accuracy.

III-A Preliminaries

Equations 1 and 2 summarize answer prediction and its decomposition into visual representation, question representation, and reasoning. Depending on the model, these components use feature vectors, scene graphs, attention, or explicit programs. Table II retains test-set accuracy and model-size comparisons; these scores use a different split from the later validation diagnostics.

Equations 1–2
𝒂∗=arg⁡max𝒂⁡p⁡(𝒂|𝑰,𝑸)\small\bm{a}^{*}=\arg\max_{\bm{a}}p(\bm{a}|\bm{I},\bm{Q}) (1)
𝒂∗=arg⁡max𝒂⁡p⁡(𝒂|𝑮,𝑹)​p​(𝑮|𝑰)​p​(𝑹|𝑸)\bm{a}^{*}=\arg\max_{\bm{a}}p(\bm{a}|\bm{G},\bm{R})p(\bm{G}|\bm{I})p(\bm{R}|\bm{Q}) (2)
Model Bottom-Up [10] LCGN [12] NSM [14] LXMERT [27] NS-VQA [11] MMN [26]
Size (MB) 19.63 10.63 n/a 202.13 12.81 36.27
Accuracy 49.74 56.10 63.17 60.30 28.81 59.33
TABLE II: Mode size and accuracy on the GQA test split. We cannot get model size for NSM [14] as it is not open-sourced.

III-B Measurements

Detected objects are the proposals available to a model. Referred objects are those needed at each annotated reasoning step; grounded objects are the final referred objects. Foreground proposals overlap an object needed anywhere in the reasoning chain, while background proposals do not overlap any such object. GQA scene graphs and programs provide the reference sets.

For sequential-attention models, grounding uses final-step attention. For LXMERT, the authors aggregate cross-modal attention by maximizing across heads and averaging across layers. Explicit executors expose step-wise object scores, but comparisons use only the final step because other architectures lack aligned intermediate steps.

Grounding is average precision over detected boxes ranked by model scores, with intersection-over-union above 0.5 defining a match to an annotated final object. It combines localization quality and ranking quality; it is not a direct test of whether those objects causally determine the answer.

Robustness removes background object representations before reasoning and reports two conditional answer-flip directions: correct to incorrect and incorrect to correct. Foreground removal and randomized object features provide additional probes. An improved perturbed accuracy does not mean all correct answers were preserved, and stable answers can still be wrong.

III-C Experiments

The balanced GQA data contain 72,140 training, 10,233 validation, and 398 testdev images. Validation images have scene graphs and roughly thirteen questions each. Five implemented model families are diagnosed; NSM appears in the accuracy comparison but is omitted from the detailed tests because an implementation was unavailable. NS-VQA receives a new scene-graph generator for natural images.

Table III shows final grounding AP of 52.26 for MMN, 36.24 for Bottom-Up, 33.00 for LCGN, 29.57 for NS-VQA, and 21.31 for LXMERT. MMN explicitly receives intermediate grounding supervision. LXMERT’s low score is based on the specified attention extraction; attributing it to spurious correlations is an interpretation rather than a controlled causal result. It had also seen GQA validation data during pretraining, and is excluded from the robustness table.

Table IV shows that removing background objects changes answers for every tested model. Correct-to-incorrect flip rates range from 4.82 to 6.27%. Several models improve aggregate accuracy because some initially incorrect answers also become correct. Foreground removal reduces accuracy, but contextual or language priors still support many answers. Randomized object features leave Bottom-Up, LCGN, and MMN at roughly 33–38% accuracy, whereas symbolic NS-VQA falls to 6.99%.

Figures 2 and 3 preserve the perturbation examples and detailed grounding comparisons. These diagnostics expose a mismatch between high answer accuracy, target-object alignment, and perturbation stability; no single measure substitutes for the others.

Model Question Type Semantic Type All
Open Binary Object Attribute Relation
Bottom-Up [10] 38.30 32.45 32.10 31.39 35.71 36.24
LCGN [12] 36.71 19.63 19.16 31.02 31.91 33.00
LXMERT [27] 26.62 12.53 27.39 16.32 22.66 21.31
NS-VQA [11] 25.25 36.78 29.59 44.70 20.99 29.57
MMN [26] 52.75 50.49 49.64 57.82 46.51 52.26
TABLE III: Object grounding scores of different models evaluated on the GQA validation set.
Method Val BG. Removal FG. Removal Randomized Obj.
Acc. Acc. C→\rightarrowI I→\rightarrowC Acc. C→\rightarrowI I→\rightarrowC Acc. C→\rightarrowI I→\rightarrowC
Buttom-Up [10] 53.73 56.30 5.92 12.71 46.99 42.10 7.65 32.81 64.17 21.92
LCGN [12] 63.87 65.79 4.82 10.11 44.94 33.74 7.14 37.71 51.81 19.15
NS-VQA [11] 40.49 38.64 5.77 1.23 25.41 59.38 1.10 6.99 85.38 0.00
MMN [26] 68.87 70.05 6.27 17.27 45.56 38.36 9.92 37.12 53.49 16.37
TABLE IV: Model accuracy and flipping rates after different type of image perturbations. The C→\rightarrowI denotes the percentage of questions that were answered correctly before the modification but wrong afterward, while I→\rightarrowC denotes the reversed direction. We perform three types of modifications: background removal, foreground removal and object randomization.
Refer to caption
Fig. 2: Examples showing answers flipped by removing a single irrelevant objects from images. From left to right, the methods are Bottom-Up [10], LCGN [12], MMN [26], and NS-VQA [11]. The red boxes show ground-truth grounded objects for the question below, while the green box is the one we remove from the image.
Refer to caption
Fig. 3: Interpreting grounded regions in different models. From top to bottom, we show ground-truth target objects, grounding results for different models. The red boxes in images represent grounded objects and the green boxes represent attended objects. We show at most five detected bounding boxes from the images for clarity. Below each image, we show the answer to the specific question for each model, with the corresponding grounding scores. The last row is from the proposed Graph Reasoning Machine, which we will introduce in Sec. IV.

IV Graph Reasoning Machine

The Graph Reasoning Machine combines the semantic transparency of NS-VQA with probabilistic visual predictions and intermediate training signals. Figure 4 shows its scene-graph generator, question-program generator, and neural executor. The purpose is component diagnosis, with improvement measured against the symbolic version.

Refer to caption
Fig. 4: Our graph reasoning machine consists of three components: a) probabilistic scene graph generation; b) question program generation and c) probabilistic program execution.

IV-A Scene Graph Generator

Nodes retain bounding boxes and full object/attribute classification scores; directed edges retain predicate scores for every object pair. Keeping score distributions allows execution to use plausible alternatives and allows loss gradients to improve the visual model. A hard symbolic graph would discard all but the highest-scoring label. Equation 3 specifies the prediction heads.

A ResNeXt-101/Faster R-CNN detector supplies object features, an added MLP predicts attributes, and Graph R-CNN with frequency priors predicts relations. The training vocabulary has 1,703 object, 618 attribute, and 311 relation categories. The generator uses 150,000 SGD iterations, batch size eight, initial learning rate 0.005, and scheduled reductions at 100,000 and 120,000 iterations.

Equation 3
soi=Φo​(𝒙oi);sai=Φa​(𝒙oi);spi​j=Φp​(𝒙pi​j)s^{i}_{o}=\mathrm{\Phi}_{o}(\bm{x}^{i}_{o});\quad s_{a}^{i}=\mathrm{\Phi}_{a}(\bm{x}_{o}^{i});\quad{s}_{p}^{ij}=\mathrm{\Phi}_{p}(\bm{x}_{p}^{ij}) (3)

IV-B Program Generator

Programs combine ten operators—filter, query, exist, verify, common, relate, choose, and, or, and not—with concept and category arguments. This yields 7,445 possible operation tokens. Dependencies connect them into executable reasoning chains.

A two-layer bidirectional LSTM encodes the question; a two-layer LSTM decoder predicts the operation sequence. Encoder and decoder hidden sizes are 256 and 512, with 300-dimensional token embeddings. The retained encoder formulas and Equation 4 specify state initialization and decoding. Ground-truth previous tokens are used during training, predicted tokens at inference.

The parser is trained on annotated programs for 70,000 Adam iterations, batch size 256, and then frozen. More than 95% answer accuracy with ground-truth scene graphs evaluates the combined parsing/execution pipeline, not exact program-string recovery.

Equations 4{𝒆f,𝒆b}=BiLSTM({𝒙q1:L})\{\bm{e}_{f},\bm{e}_{b}\}=\mathrm{BiLSTM}(\{\bm{x}_{q}^{1:L}\})𝒉r0=[𝒆f|𝒆b]\bm{h}^{0}_{r}=[\bm{e}_{f}|\bm{e}_{b}]
𝒉ri=LSTM⁡(𝒙ri,𝒉ri−1);yri∼Softmax⁡(𝑾T​𝒉ri)\bm{h}_{r}^{i}=\mathrm{LSTM}(\bm{x}_{r}^{i},\bm{h}_{r}^{i-1});\quad y_{r}^{i}\sim\mathrm{Softmax}(\bm{W}^{T}\bm{h}_{r}^{i})\\ (4)

IV-C Neural Program Executor

The executor traverses the probabilistic scene graph. Pointing operations such as filter and relate select object indices; query operations score answer categories; binary operations such as exist or verify produce calibrated scalar scores. Equations 5 and 6 retain the pointing and query interfaces. Boolean outputs can feed logical operations or provide final answers.

Teacher-forcing supervision is constructed by executing annotated programs on annotated scene graphs, then matching those objects to detected boxes at IoU above 0.5. Pointing and binary operations use binary cross-entropy, while answer queries use categorical cross-entropy. Summed intermediate losses backpropagate through the probabilistic representation into the scene-graph generator. This uses substantial object, relation, program, and step-level annotation beyond question-answer labels.

At inference, predicted graphs and programs replace the annotations. The reported search retains pointing candidates whose scores exceed 5.0 and propagates alternatives, with a slight reported benefit over greedy selection. This is the paper’s score threshold, not a probability threshold.

Equations 5–6
𝒁o​u​t=P​o​i​n​t​e​r​(𝑺l,𝒁i​n,c)\bm{Z}^{out}=Pointer(\bm{S}_{l};\bm{Z}^{in};c) (5)
𝒑lc=Q​u​e​r​y​(𝑺l,𝒁i​n,c)\bm{p}^{c}_{l}=Query({\bm{S}}_{l};\bm{Z}^{in};c) (6)
Model Val Grounding Robustness (w/o BG)
Acc. Correct Wrong All Acc. C→\rightarrowI I→\rightarrowC
Graph Reasoning Machine 58.49 61.89 41.56 52.64 60.78 2.83 8.24
-teacher-forcing training 43.94 47.11 41.01 42.16 40.83 4.75 3.24
-probabilistic scene graph 40.49 38.99 24.25 29.57 38.64 5.77 1.23
+ground-truth scene graph 95.26 88.18 50.55 86.51 94.40 1.23 87.96
TABLE V: Ablation study on Graph Reasoning Machine model. “-teacher-forcing training” means removing teacher-forcing training; “-probabilistic scene graph” means further replace probabilistic scene graph to symbolic ones; “+ ground-truth scene graph” means using ground-truth scene graph annotations.

IV-D Ablation Study

Table V isolates the design choices on validation data. The full model has 58.49% answer accuracy, 52.64 grounding AP, and a 2.83% correct-to-incorrect flip rate after background removal. Removing teacher forcing reduces accuracy to 43.94 and grounding to 42.16; additionally replacing probabilities with hard symbols reduces them to 40.49 and 29.57. The latter ablation is cumulative, so it is not an isolated factorial estimate of representation choice.

Supplying ground-truth scene graphs raises accuracy to 95.26 and grounding to 86.51, with a 1.23% harmful flip rate. This identifies a major perception bottleneck under an oracle-input condition. It does not demonstrate an obtainable detector improvement or perfect reasoning.

The full diagnostic model improves over the symbolic baseline, but its validation accuracy remains below MMN’s 68.87 in Table IV. Grounding is similar to MMN’s score. These results favor assessing accuracy, alignment, and stability separately rather than declaring a universal best model.

IV-E Failure Analysis

Figure 5 groups failure cases by final and intermediate operations. More than 75% of failures include a relation operation; this suggests a relation-perception bottleneck, but does not by itself give a frequency-normalized failure rate for that operator. Figure 6 preserves three successful and three unsuccessful reasoning traces.

Failures include nonexistent objects receiving high scores, selecting the wrong one of several similar objects, and misclassifying an attribute. The reported detector mAP is only 4.85 across the large vocabulary. Such errors can propagate through otherwise explicit execution.

The benchmark also introduces ambiguity: several objects may answer a question, while only one is annotated, and a single reference answer rejects plausible synonyms or hypernyms such as “child” for “boy.” These issues limit both answer evaluation and object-grounding diagnostics. Transparency makes these errors inspectable; it does not eliminate them.

Original figure
Fig. 5: Statistics of failure cases for the augmented NS-VQA model on validation set. Left is the distribution on last operator and right is on intermediate operation.
Refer to caption
Fig. 6: Examples showing step-wise visual reasoning for our graph reasoning machine. The top row shows three samples that our model predicts the correct answers while the bottom row shows three failure cases due to different reasons.

V Conclusion

Program-like reasoning and intermediate supervision can improve object alignment, while probabilistic visual representations reduce brittleness from hard scene-label errors. Grounding, answer accuracy, and stability under object removal remain distinct. Better perception and clearer question, object, and answer annotations are needed to interpret progress reliably.

Acknowledgments

The authors acknowledge support from CBMM, ONR, the MIT–IBM Watson AI Lab, and Mitsubishi Electric Research Laboratories.

Author biographies (as published)

The original author biographies and portraits are retained below as part of the published source. Their positions, affiliations, and career descriptions refer to the time of this version.

[Uncaptioned image] Jianwei Yang Jianwei Yang is a senior researcher in Deep Learning Group at Microsoft Research. Before joining Microsoft, he earned his Ph.D. in computer science from School of Interactive Computing at Georgia Institute of Technology, supervised by Prof. Devi Parikh. His primary research is in the intersection of computer vision, natural language processing (NLP) and embodiment. Specifically, his study focuses on how to leverage structure in image for effective visual understanding, generation and reasoning with natural language.
[Uncaptioned image] Jiayuan Mao Jiayuan Mao is a second-year PhD student in the department of Electrical Engineering and Computer Science at Massachusetts Institute of Technology, advised by Prof. Joshua B. Tenenbuam and Prof. Leslie P. Kaelbling. Mao’s research focuses on concept learning, neuro-symbolic reasoning, visual scene understanding, language acquisition, and robotic planning.
[Uncaptioned image] Jiajun Wu is an Assistant Professor of Computer Science at Stanford University, working on computer vision, machine learning, and computational cognitive science. Before joining Stanford, he was a Visiting Faculty Researcher at Google Research. He received his PhD in Electrical Engineering and Computer Science at Massachusetts Institute of Technology. Wu’s research has been recognized through the ACM Doctoral Dissertation Award Honorable Mention, the AAAI/ACM SIGAI Doctoral Dissertation Award, the MIT George M. Sprowls PhD Thesis Award in Artificial Intelligence and Decision-Making, the 2020 Samsung AI Researcher of the Year, the IROS Best Paper Award on Cognitive Robotics, and fellowships from Facebook, Nvidia, Samsung, and Adobe.
[Uncaptioned image] Devi Parikh is an Associate Professor in the School of Interactive Computing at Georgia Tech, and a Research Scientist at Facebook AI Research (FAIR). Her research interests are in computer vision, natural language processing, embodied AI, human-AI collaboration, and AI for creativity. She is a recipient of an NSF CAREER award, an IJCAI Computers and Thought award, a Sloan Research Fellowship, an Office of Naval Research (ONR) Young Investigator Program (YIP) award, an Army Research Office (ARO) Young Investigator Program (YIP) award, a Sigma Xi Young Faculty Award at Georgia Tech, an Allen Distinguished Investigator Award in Artificial Intelligence from the Paul G. Allen Family Foundation, four Google Faculty Research Awards, an Amazon Academic Research Award, a Lockheed Martin Inspirational Young Faculty Award at Georgia Tech, an Outstanding New Assistant Professor award from the College of Engineering at Virginia Tech, a Rowan University Medal of Excellence for Alumni Achievement, Rowan University’s 40 under 40 recognition, a Forbes’ list of 20 “Incredible Women Advancing A.I. Research” recognition, and a Marr Best Paper Prize awarded at the International Conference on Computer Vision (ICCV).
[Uncaptioned image] David D. Cox is the IBM Director of the MIT-IBM Watson AI Lab, a first of its kind industry-academic collaboration between IBM and MIT, focused on fundamental research in artificial intelligence. Prior to joining IBM, David was the John L. Loeb Associate Professor of the Natural Sciences and of Engineering and Applied Sciences at Harvard University, where he held appointments in Computer Science, the Department of Molecular and Cellular Biology and the Center for Brain Science. David’s ongoing research is primarily focused on bringing insights from neuroscience into AI research. His past work has spanned a variety of disciplines, from neuroscience experiments in living brains, to the development of machine learning and artificial intelligence methods, to applied machine learning and high performance computing methods.
[Uncaptioned image] Joshua B. Tenenbaum is Professor of Computational Cognitive Science at MIT in the Department of Brain and Cognitive Sciences, the Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Center for Brains, Minds and Machines (CBMM). He received his PhD from MIT in 1999, and taught at Stanford from 1999 to 2002. His long-term goal is to reverse-engineer intelligence in the human mind and brain, and use these insights to engineer more human-like machine intelligence. His current research focuses on the development of common sense in children and machines, the neural basis of common sense, and models of learning as Bayesian program synthesis. His work has been published in Science, Nature, PNAS, and many other leading journals, and recognized with awards at conferences in Cognitive Science, Computer Vision, Neural Information Processing Systems, Reinforcement Learning and Decision Making, and Robotics. He is the recipient of the Distinguished Scientific Award for Early Career Contributions in Psychology from the American Psychological Association (2008), the Troland Research Award from the National Academy of Sciences (2011), the Howard Crosby Warren Medal from the Society of Experimental Psychologists (2016), the R&D Magazine Innovator of the Year award (2018), and a MacArthur Fellowship (2019). He is a fellow of the Cognitive Science Society, the Society for Experimental Psychologists, and a member of the American Academy of Arts and Sciences.
[Uncaptioned image] Chuang Gan is a principal research staff member at MIT-IBM Watson AI Lab. He is also a visiting research scientist at MIT. Before that, he completed his Ph.D. with the highest honor at Tsinghua University, supervised by Prof. Andrew Chi-Chih Yao. His primary focuses are on machine learning and its applications to computer vision, natural language processing, and robotics. His research works have been recognized by Microsoft Fellowship, Baidu Fellowship, and media coverage from CNN, BBC, The New York Times, WIRED, Forbes, and MIT Tech Review. He has served as an area chair for ICLR, ICCV, AAAI, and IJCAI, and an associate editor for IEEE Transactions on Image Processing, ACM Transactions on Multimedia, IEEE Transactions on Circuits and Systems for Video Technology, and Pattern Recognition.
Concept demonstration

Open image ↗