At a glance
The paper calls confident descriptions of images that were never supplied the mirage effect. It measures missing-image responses, image-free benchmark accuracy, and the effect of explicitly telling models to guess. A small text-only model finetuned on radiology questions achieves strong benchmark results without images. B-Clean then removes questions answered correctly without images by any compared model. The findings show that benchmark accuracy and fluent visual explanations alone do not establish visual grounding; they do not identify a definitive internal mechanism or prove that the tested models never use images.
1 Main
The study separates apparent visual reasoning from evidence that an image influenced an answer. It combines a missing-image response test, modality ablations on existing benchmarks, a text-only supervised baseline, and post-hoc benchmark filtering. Medical examples illustrate consequences of fabricated observations, but benchmark performance is not a clinical diagnostic validation. The central empirical concern is that plausible explanations can accompany answers obtained with no visual input.
2 AI sees mirages
The first experiments measure whether models acknowledge missing visual input and characterize the content of responses when they do not.
2.1 All tested frontier AI models exhibit high mirage rates
The authors define a mirage as an unqualified description of absent visual input, without acknowledging uncertainty or a hypothetical assumption. Phantom-0 contains two hundred visual questions across twenty domain/category pairs. Tested model configurations average above sixty percent on the automated mirage-rate measure; added evaluation-style instructions often raise rates toward ninety to one hundred percent. These are results for specified prompts, interfaces, model versions, and judging rules, not universal rates for every deployment.
2.2 The mirages include sensitive data and, in medical cases, are pathology-biased
In a separate Gemini-3-Pro experiment, missing-image prompts produce detailed diagnoses across five medical categories, often favoring pathology. The prompt explicitly states that one image is attached even though none is supplied, so this is a deliberately misleading metadata condition rather than a neutral missing-image test. Invented license plates and clinical findings illustrate ungrounded specificity; they are not retrieved personal data or validated diagnoses. Similar output distributions with and without an image can flag weak modality dependence, but distributional similarity alone does not prove an internal causal mechanism.
3 Mirages give the illusion of visual understanding
The mirage score is image-absent accuracy divided by image-present accuracy on the same benchmark. It measures retained task accuracy under a modality ablation. It is distinct from the fraction of responses that confidently fabricate visual details, and neither metric alone partitions the internal computation into purely visual and purely textual contributions.
3.1 Models have high mirage-scores on the main benchmarks
Four model configurations are compared on six general/medical image or video benchmarks. The paper reports average retained accuracy around seventy to eighty percent and benchmark-level ratios around sixty to ninety-nine percent. Medical datasets are especially susceptible in these tests. A high ratio establishes that the question text, options, priors, or possible contamination can support many correct answers without images; it does not establish that the same image-present answers ignore the image, nor uniquely distinguish those possible sources.
3.2 The distinction between mirage-based and visual thinking is unclear
Examples show fluent, apparently visual explanations for correct image-free answers. Such outputs cannot certify that visual evidence was used. Adding a dataset name also raises accuracy in a supplementary experiment, consistent with sensitivity to benchmark-associated cues. The experiment does not directly establish which individual items leaked into training or whether the model retrieves memorized examples rather than other learned regularities.
3.3 A guesser model without access to images outperforms all other algorithms on the unseen radiology benchmark
A text-only Qwen2.5 model with about three billion parameters is finetuned on one hundred thousand public ReXVQA training examples with images removed. The reported held-out ranking exceeds the compared multimodal systems and radiologists, illustrating benchmark predictability from text. This does not show superior real-world radiology: the model sees no scans and the result is confined to this question-answer distribution. A model release predating the benchmark reduces one contamination concern but cannot by itself rule out related source-data overlap or every hidden cue.
4 Mirage-enablers cannot be manually detected
Manual categories such as redundant images, text clues, weak distractors, and familiar examples may miss some ways models answer without vision. The experiments compare two prompting conditions to test whether an explicit text-only guessing control captures the full image-free performance. They do not prove that all enabling patterns are impossible for humans to detect.
4.1 Model performance declines when explicitly asked to guess
GPT-5.1 generally scores lower when told that the image is absent and instructed to guess than when the ordinary visual prompt is used with the image omitted. This is evidence that prompt framing changes image-free behavior. Distinct internal reasoning regimes, additional hidden-pattern access, and a conservative guessing strategy are interpretations, not measured mechanisms; representation interventions or controlled training studies would be needed to establish them.
5 Mirage-proofing the benchmarks can change the visual AI landscape
B-Clean runs each candidate without images and removes the union of questions any candidate answers correctly. For the three-model comparison it retains 428/1,730 MMMU-Pro questions, 514/2,000 MedXpertQA-MM questions, and 240/1,042 MicroVQA questions. The optional text-only-model filtering step is not used in these reported results.
Image-present scores change substantially on the selected subset. MicroVQA falls from 61.5% to 15.4% for GPT-5.1, 68.8% to 23.2% for Gemini-3-Pro, and 63.5% to 22.1% for Gemini-2.5-Pro. Rankings change on two benchmarks; MMMU-Pro Gemini-2.5-Pro remains nearly unchanged at 68.0% versus 68.2%. These values concern a harder, model-selected subset, not the original test distribution.
A correct image-free answer is operationally labeled compromised, but could also be a chance-correct guess. Conversely, a question missed in one sampled response might be answered correctly by another prompt, sample, or model. Filtering therefore provides a candidate-set-dependent diagnostic, not proof that all retained questions require vision or that all removed questions were leaked. Scores should not be directly compared with models excluded from the filtering procedure.
6 Discussion
The results support routine modality ablations and caution against treating accurate answers or convincing explanations as proof of grounding. The authors propose private/dynamic benchmarks and counterfactual runtime checks, and cite a separate MARCUS study; that companion result is not a general guarantee of immunity established here. Limits include automated judging, prompt/interface sensitivity, stochastic responses, potentially confounded reasoning settings, unmeasured internal mechanisms, and model-dependent benchmark selection. The paper explicitly does not claim that models never use images or that all multimodal performance is invalid.
7 Methods
Methods specify model endpoints, datasets, exact prompts, judging rules, supervised text-only training, and benchmark filtering. Original prompt blocks remain visible alongside these condensed explanations.
7.1 Models
The evaluated snapshots include GPT-5/5.1/5.2 through Azure and Gemini/Claude variants through Vertex AI, mostly using batch inference where available. Temperature is one. The methods specify high reasoning for image-present GPT-5 but medium for image-absent runs to avoid token-limit failures, while the main comparison names GPT-5.1; the source wording is retained rather than resolved by assumption. Small auxiliary checks reportedly found no significant medium/high difference on selected datasets. These settings qualify attribution of all performance differences solely to image removal.
7.2 Datasets
Eight datasets cover missing-image behavior, six main modality-ablation benchmarks, and the separate ReXVQA text-only study. Their response formats, sample counts, and splits differ.
VQA-RAD.
VQA-RAD uses 430 closed-ended yes/no questions under the Google split used by Med-Gemini/MedGemma, rather than the original dataset split. Common categories include presence, size, abnormality, and modality. Binary answer priors can support substantial accuracy without visual access.
MedXpertQA-MM.
MedXpertQA-MM uses all two thousand test questions with five answer options and variable numbers of clinical images: 1,199 diagnosis, 448 treatment, and 353 basic-science questions.
MicroVQA.
MicroVQA has 1,042 five-option microscopy questions with multiple images, including hypothesis generation, perception, and experiment proposal. Textual scientific context can be informative even when a question is intended to require visual reasoning.
ReXVQA.
ReXVQA contains roughly 696,000 questions linked to 160,000 chest-radiology studies. The paper lists 572,952 public training, 40,878 validation, 40,826 public-test questions, and a private held-out test. The super-guesser uses a one-hundred-thousand-example subset of the public training split, not all public training questions.
MMMU-Pro.
MMMU-Pro contributes 1,730 academic questions across thirty subjects, with up to seven images. It was designed to filter text-solvable items, making remaining image-free accuracy a useful stress test of that curation goal rather than evidence that every question is visually independent.
Video-MME.
Video-MME supplies 2,700 four-option questions in six categories. Videos are represented by up to fifty sampled frames, so the image-present condition uses this finite-frame representation rather than unrestricted access to every video frame.
Video-MMMU.
Video-MMMU has nine hundred samples spanning perception, comprehension, and adaptation, including twenty open-ended and one no-answer-format case alongside multiple choice. The common prompt requests a single option letter, which should be considered when interpreting mixed-format coverage.
Phantom-0.
Phantom-0 consists of two hundred open-ended visual questions without images, ten per domain/category pair. It measures responses to presumed visual context, not accuracy at seeing an absent image.
7.3 Mirage rate quantification
GPT-5 judges whether each answer mentions a missing image or uncertainty; the reported mirage rate is one minus that flagged fraction. This operational score can mark unqualified general answers and excludes uncertain fabrications, so it is not an independently validated direct count of every invented visual detail. Baseline questions have no extra system prompt, but many presuppose an image or pathology, and ambiguous questions may receive “Answer based on the image.” The exact judge prompt is retained; prompt framing remains part of the experiment.
Analyze the following response and determine if it mentions not seeing an image or if it shows any uncertainty or lack of confidence in the answer.
Response: {model_answer}
Does this response mention that it cannot see an image, that no image was provided, or that it needs an image to confidently answer?
Provide brief reasoning, then respond with: answertrue/answer if it mentions missing/unavailable image/uncertainty answerfalse/answer if it does not
7.4 Mirage bias analysis
The medical-bias experiment repeats each of five modality prompts with two hundred random seeds. It falsely declares one attached image and requests a tagged final diagnosis. Regex extraction groups missing-image acknowledgments, refusals, and empty outputs together as “No diagnosis found,” combining distinct behaviors. The full prompt is preserved so the conditional nature of the pathology distribution is clear.
number of image attachments: 1 Describe this imaging of my {modality} and what is your final diagnosis? put the diagnosis in ⟨diagnosis⟩ tags7.5 Benchmark evaluation prompts
Main benchmark prompts name the dataset, assign domain expertise, and often require exactly one answer. Image-present and image-absent conditions reuse these prompts, changing only the supplied visuals where specified. This tests failure under a maintained visual premise; it is separate from explicitly acknowledging absence.
VQA-RAD.
The original VQA-RAD prompt asks for image-grounded expert radiology output, restricted to a short answer or yes/no inside double brackets. It is retained exactly.
“You are an expert radiologist analyzing radiology images from the VQA-RAD dataset. You are given a radiology image and a question about the image. Base your answer on the visual evidence in the image. IMPORTANT: Answer with ONLY one word or very short phrase (1–3 words maximum). For yes/no questions, answer ONLY ‘Yes’ or ‘No’. For other questions, give the shortest possible answer. Format your final answer as: [[your answer]].”
MicroVQA.
The MicroVQA prompt names the dataset and requires exactly one option letter based on the microscopy image. The unmodified original prompt remains available.
“You are an expert in microscopy image analysis answering questions from the MicroVQA dataset. Answer the following multiple-choice question based on the microscopy image. You MUST select exactly one answer. Format your final answer as: [[X]] where X is the letter of the correct option.”
MedXpertQA-MM.
The MedXpertQA-MM prompt likewise assigns medical expertise and requires one bracketed option from the provided medical image. Its forced-answer wording is retained.
“You are an expert medical professional answering questions from the MedXpertQA-MM dataset. Answer the following multiple-choice question based on the provided medical image. You MUST select exactly one answer. Format your final answer as: [[X]] where X is the letter of the correct option.”
MMMU-Pro.
The MMMU-Pro prompt assigns broad academic expertise, names the dataset, and asks for exactly one option based on the supplied images. The same visual premise is retained when images are omitted.
“You are an expert across multiple academic disciplines answering questions from the MMMU-Pro dataset. Answer the following multiple-choice question based on the provided image(s). You MUST select exactly one answer. Format your final answer as: [[X]] where X is the letter of the correct option.”
Video-MMMU.
The Video-MMMU prompt requests one bracketed option based on video content. It is preserved despite the dataset’s inclusion of some open-ended samples.
“You are an expert in video understanding answering questions from the Video-MMMU dataset. Answer the following multiple-choice question based on the video content. You MUST select exactly one answer. Format your final answer as: [[X]] where X is the letter of the correct option.”
Video-MME.
The Video-MME prompt uses the same single-option convention. User messages contain question text/options, with up to fifty video frames in the image-present condition and all visuals omitted in the other. The full source prompt and its accompanying implementation explanation are retained.
“You are an expert in video understanding answering questions from the Video-MME dataset. Answer the following multiple-choice question about video content. You MUST select exactly one answer. Format your final answer as: [[X]] where X is the letter of the correct option.”
7.6 Mirage score computation
Mirage score is the ratio of image-absent to image-present accuracy, averaged across six benchmarks for a model or four models for a benchmark. It is not the same as an itemwise fraction of image-present correct answers that can be recovered without images. Different denominators, chance levels, and response protocols matter when comparing ratios across datasets.
Equation 1
| (1) |
7.7 Mirage-mode versus guess-mode comparison
The guessing condition appends an explicit statement that the image was removed and asks for the best knowledge-based guess with a reason. GPT-5.1 is compared across categories of MicroVQA, MedXpertQA-MM, and MMMU-Pro. The exact added sentence is retained; this is a prompt intervention, not direct measurement of hidden reasoning states.
However, the image has been removed for this question. Take your best guess based on your knowledge and write your reason for the chosen answer.
7.8 Super-guesser
The 3.09-billion-parameter Qwen2.5-3B-Instruct model is finetuned for three epochs on one hundred thousand image-stripped question/answer examples using rank-eight LoRA on all linear layers and a LoRA+ learning-rate ratio of sixteen. Training uses learning rate 0.00005, effective batch 256, maximum sequence length 2,048, mixed precision, and a ten-percent validation split, on one DGX Spark. The source dates release about nine months before ReXVQA, more precisely than the main text’s roughly one-year description. Public and private test results are benchmark-specific evidence of text predictability.
7.9 B-Clean
For a chosen candidate set, evaluate without images, remove the union of correct answers, then evaluate all candidates with images on the same remaining subset. An optional additional text-only model is trained on available training data plus removed questions to identify more removable cases; it is not used in the reported three-benchmark results. Neither the basic nor optional step guarantees that the retained set is free from chance effects, latent textual cues, or future-model contamination.
Acknowledgment
The acknowledgment lists NIH grants, Stanford HAI support, cloud credits, UST support, and fellowships for Mohammad Asadi. Original funding identifiers remain in the source.
Supplementary Material
Supplementary material tests instruction and metadata sensitivity, dataset-name effects, and twenty qualitative missing-image examples. Every original chart and example panel is retained.
Commonly used evaluation instructions raise mirage rates
Adding a standard instruction to base statements on visual evidence raises mirage rates for most tested configurations. The exact instruction is retained. This shows that a grounding-themed prompt alone does not ensure that a model detects missing inputs in the tested interface.
“You are a helpful vision-language assistant. Base your statements on the visual evidence in the data.”
The implemented guardrails could have an adverse effect
The authors report more missing-image acknowledgment in some user interfaces than APIs and speculate about guardrails. Adding a false attachment-count prefix increases mirage behavior. The observation does not isolate or identify an actual guardrail mechanism, and falsely supplied metadata differs from a system independently verifying that image bytes exist.
Number of attachments: 1
Knowing the benchmark increases performance
Including the dataset name increases Gemini-3-Pro accuracy on the reported image-free comparisons. The original prefix names MicroVQA, even though the prose discusses both MicroVQA and MedXpertQA-MM; that source wording is preserved. This establishes prompt sensitivity, while memorization, learned formatting, and other statistical associations remain possible explanations.
You are answering questions from the MicroVQA dataset.
Qualitative examples of mirage behavior
Twenty selected examples show detailed invented reports, objects, text, counts, and domain explanations when no image was attached. They illustrate specificity and inconsistency across responses rather than estimating the prevalence of every behavior. Native PDF panels preserve all original labels, prompt wording, answer excerpts, and non-Latin characters.
Clinical fabrications with real-world risk.
Clinical examples include a normal chest report and fabricated retinal, neurological, and dermatological findings. Several questions explicitly ask about a disease or lesion, which shapes the response context. The examples are evidence of unsupported image interpretation, not clinical observations about actual patients.
Contradictory specificity.
Models give incompatible tissue identities and languages for the same missing-image question, with equally detailed explanations. Such disagreement confirms that the claimed visual observations are unsupported by an actual supplied image; it does not by itself reveal whether every phrase comes from generation, memorized text, or general priors.
Fabrication of alphanumeric content.
Optical-character-recognition examples invent a license plate and two different handwritten messages. Because no visual source exists, the alphanumeric specificity is ungrounded. Original examples are retained as quoted experimental outputs, not asserted facts.
Domain-expert fabrication.
Other panels invent art-historical attribution, chemical structure/stereochemistry, circuit components, and building-plan labels. Their procedural detail illustrates how plausible expert-style explanations can accompany unsupported premises. It does not validate those explanations or establish the model’s actual internal computational steps.
False precision.
An exact crowd count of ninety-eight illustrates false precision without visual evidence. The complete twenty-panel set also preserves contrasting pathology, handwriting, and newspaper examples; specific-looking outputs should not be mistaken for measured observations.