Literature
Display
← All papers

Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces

Jihan Yang · Shusheng Yang · Anjali W. Gupta · Rilyn Han · Li Fei-Fei · Saining Xie

· CVPR 2025 ↗

Computer vision13 min text · CondensedOriginal paper ↗
Contents

Overview

An indoor video can reveal individual objects without yielding an accurate memory of the room’s layout. This paper tests the distinction through questions about counts, distances, directions, dimensions, routes, and appearance order.

Original resources: project page, evaluation code, and dataset.

[Uncaptioned image]
Figure 1: Whether at home, in the workplace, or elsewhere, the ability to perceive a space, remember its layout, and retrieve this spatial information to answer questions on demand is a key aspect of visual-spatial intelligence. Recent Multimodal LLMs can understand general videos, but can they “think spatially” when presented with a video recording of an environment? Can they build an accurate, implicit “cognitive map” that allows them to answer questions about a space? What are the strengths and limitations of using MLLMs to enhance spatial intelligence? We dig into these questions by setting up video data for MLLMs to watch, building a VQA benchmark to check their recall, and examining what the MLLMs actually remember and understand.

At a glance

VSI-Bench contains over 5,000 question–answer pairs from 288 real indoor videos. Fifteen video-capable multimodal language models show useful but incomplete spatial understanding. On a shared 400-question subset, human evaluators score 79.2 versus 48.8 for Gemini-1.5 Pro, using a mixture of multiple-choice accuracy and numerical proximity scores.

The authors examine generated explanations and explicit grid maps. Tested verbal reasoning prompts reduce average performance, while generating a map improves Gemini’s relative-distance accuracy from 46% to 56%. These are behavioral results from specific models and prompts; post-hoc explanations and generated maps do not directly expose a model’s internal representations.

1 Introduction

Visual-spatial intelligence requires remembering objects and relations across changing viewpoints, then using that information for a new question. Accurate video recognition alone does not ensure a consistent room layout or a correct transformation between camera-centered and environment-centered coordinates.

VSI-Bench repurposes annotated indoor scans into video questions. Inspired by dual-coding theory, the analysis asks models to express spatial information through language and through maps. The key empirical contrast is that additional verbal deliberation does not reliably help these tasks, whereas an explicit spatial representation can help a particular distance query.

Original figure
Figure 2: A taxonomy of visual-spatial intelligence capabilities.

2 Visual-Spatial Intelligence

The taxonomy separates visual perception, linguistic reasoning, temporal processing, and spatial reasoning. Spatial reasoning includes relationships such as distance, direction, and relative size, plus transformations between egocentric and allocentric perspectives. Working memory and imagined viewpoints support those transformations.

The scope is real indoor environments rather than abstract mental-rotation puzzles. Tasks mix capabilities: appearance order needs temporal memory; object size can use familiar dimensions; route planning and relative direction strongly depend on perspective. The taxonomy organizes the authors’ error annotations, rather than independently measuring separate cognitive modules.

Refer to caption
Figure 3: Tasks demonstration of VSI-Bench. Note: the questions above are simplified slightly for clarity and brevity.

3 VSI-Bench

The benchmark combines video observations with questions derived from 3D scene annotations and human-authored routes.

3.1 Overview

The videos come from validation splits of ScanNet, ScanNet++, and ARKitScenes, spanning homes, professional spaces, and industrial settings. Configuration tasks ask object counts, relative distances, relative directions, and route plans; measurement tasks ask object size, room size, and absolute distance; appearance order tests spatiotemporal memory.

Seven tasks use templates and scan metadata; routes are human annotated. Video input exposes a sequence of partial views, while the source scans provide object categories, bounding boxes, and spatial measurements. These annotation-derived answers remain approximations of the physical environment, despite repeated human review.

Refer to caption
Figure 4: Benchmark curation pipeline. The pipeline first unifies diverse datasets into a standardized format and semantic space for consistent processing. QA pairs are then generated through both human annotation and question templates. To ensure quality, human verification is implemented at all key stages for filtering low-quality videos, annotations, and ambiguous QA pairs.
Refer to caption
Figure 5: Benchmark Statistics. Top: The distribution of tasks across three main categories. Bottom: The video length statistic.

3.2 Benchmark Construction

The pipeline standardizes videos and annotations, generates questions, and iteratively checks them with people. Reviewers filter incomplete reconstructions, correct metadata, and flag ambiguous questions without first seeing the expected answers. Corrections can change templates or generation rules so a recurring error is fixed at its source.

The retained evaluation figure includes the full results table and radar chart. Its human and dagger-marked model rows use the 400-question tiny subset; the other model rows use the full benchmark. They should not be treated as a single matched evaluation.

Obj. Count

Abs. Dist.

Obj. Size

Room Size

Rel. Dist.

Rel. Dir.

Route Plan

Appr. Order

Methods Rank Avg. Numerical Answer Multiple-Choice Answer
Baseline
Chance Level (Random) - - - - - - 25.0 36.1 28.3 25.0
Chance Level (Frequency) - 34.0 62.1 32.0 29.9 33.1 25.1 47.9 28.4 25.2
VSI-Bench (tiny) Perf.
†Human Level - 79.2 94.3 47.0 60.4 45.9 94.7 95.8 95.8 100.0
†Gemini-1.5 Flash - 45.7 50.8 33.6 56.5 45.2 48.0 39.8 32.7 59.2
†Gemini-1.5 Pro - 48.8 49.6 28.8 58.6 49.4 46.0 48.1 42.0 68.0
†Gemini-2.0 Flash - 45.4 52.4 30.6 66.7 31.8 56.0 46.3 24.5 55.1
Proprietary Models (API)
GPT-4o 3 34.0 46.2 5.3 43.8 38.2 37.0 41.3 31.5 28.5
Gemini-1.5 Flash 2 42.1 49.8 30.8 53.5 54.4 37.7 41.0 31.5 37.8
Gemini-1.5 Pro 1 45.4 56.2 30.9 64.1 43.6 51.3 46.3 36.0 34.6
Open-source Models
InternVL2-2B 11 26.5 25.7 24.0 20.0 29.2 32.1 44.1 30.4 6.3
InternVL2-8B 3 37.5 31.3 29.0 48.9 44.2 38.0 33.4 28.9 46.4
InternVL2-40B 4 37.0 41.3 26.2 48.2 27.5 47.6 32.7 27.8 44.7
LongVILA-8B 12 21.6 29.1 9.1 16.7 0.0 29.6 30.7 32.5 25.5
VILA-1.5-8B 9 28.9 17.4 21.8 50.3 18.8 32.1 34.8 31.0 24.8
VILA-1.5-40B 7 31.2 22.4 24.8 48.7 22.7 40.5 25.7 31.5 32.9
LongVA-7B 8 29.2 38.0 16.6 38.9 22.2 33.1 43.3 25.4 15.7
LLaVA-Video-7B 5 35.6 48.5 14.0 47.8 24.2 43.5 42.4 34.0 30.6
LLaVA-Video-72B 1 40.9 48.9 22.8 57.4 35.3 42.4 36.7 35.0 48.6
LLaVA-OneVision-0.5B 10 28.0 46.1 28.4 15.4 28.3 28.9 36.9 34.5 5.8
LLaVA-OneVision-7B 6 32.4 47.7 20.2 47.4 12.3 42.5 35.2 29.4 24.4
LLaVA-OneVision-72B 2 40.2 43.5 23.9 57.6 37.5 42.5 39.9 32.5 44.6
Original figure
Figure 6: Evaluation on VSI-Bench. Left: Dark gray indicates the best result among all models and light gray indicates the best result among open-source models. † indicates results on VSI-Bench (tiny) set. Right: Results including the top-3 open-source models.

4 Evaluation on VSI-Bench

Evaluation compares historical proprietary and open models, simple chance baselines, and a human subset. Input sampling and numerical-answer scoring matter when interpreting the reported averages.

4.1 Evaluation Setup

Models use their default input format with zero-shot prompts and greedy decoding. GPT-4o receives 16 sampled frames, open models generally receive 32, and Gemini receives the complete uploaded video through its API. This compares deployed model setups, not equal frame or compute budgets.

Multiple-choice answers are scored by exact or fuzzy matching. Numerical answers use mean relative accuracy: the retained equation averages ten indicators for relative error below 1−θ, with θ from 0.50 to 0.95. Thus the final eight-task average mixes categorical accuracy and a graded numerical score; it is not the fraction of all questions answered exactly.

The random baseline applies only to multiple-choice tasks. The frequency baseline always selects each task’s most common answer and scores 34.0 overall. Human evaluation uses 400 questions, fifty per task, with unlimited time and video replay. The same subset is used for the explicitly marked model comparisons.

Equation 1
ℳ​ℛ​𝒜=110​∑θ∈𝒞𝟙​(|y^−y|y<1−θ).\vskip-5.69046pt\small\mathcal{MRA}=\frac{1}{10}\sum_{\theta\in\mathcal{C}}\mathbbm{1}\left(\frac{|\hat{y}-y|}{y}<1-\theta\right). (1)
Refer to caption
Figure 7: Examples of how a MLLM thinks as seen in self-explanations. While a MLLM exhibits strong video understanding and linguistic reasoning capabilities, its spatial reasoning capabilities are still developing.

4.2 Main Results

On the full set, Gemini-1.5 Pro scores 45.4, Gemini-1.5 Flash 42.1, and GPT-4o 34.0. Leading open models are LLaVA-Video-72B at 40.9 and LLaVA-OneVision-72B at 40.2; seven of twelve open models fall below the frequency baseline. These results characterize the evaluated model versions, not current model rankings.

On the matched tiny subset, humans score 79.2 and Gemini-1.5 Pro 48.8, a 30.4-point gap. Comparing the human subset with the full-set model score, as the source’s approximately 33-point statement does, mixes evaluation sets. Humans excel on configuration and appearance order; differences are smaller for some measurements, with Gemini exceeding the human room-size score but remaining well below humans on absolute distance.

5 How MLLMs Think in Space Linguistically

The linguistic analysis distinguishes explanations elicited after an answer from prompting methods that request intermediate reasoning before the answer.

5.1 Probing via Self-Explanations

Gemini-1.5 Pro first answers a question, then explains its answer step by step. Human annotators classify 163 selected failures as perception, linguistic reasoning, relational spatial reasoning, or egocentric–allocentric transformation errors. A case with several causes contributes equal fractions to those categories.

About 71% of the attributed error mass is spatial. The appendix specifies all multiple-choice failures and the worst half of numerical-answer cases on the tiny set, whereas the main text calls the sample random. This is a selected error analysis, not a representative estimate of all model behavior. Generated post-hoc rationales may also differ from the actual causes of a prediction.

The retained examples contrast an apparently coherent global-coordinate explanation with a route error that confuses camera turning and the requested perspective. They illustrate the annotation scheme without establishing that the model literally maintains the internal world model described in its prose.

Refer to caption
Figure 8: Human-conducted analysis of errors by type. Over 70% of errors stem from faulty spatial reasoning capabilities.
Spatial reasoning is the primary bottleneck for MLLM performance on VSI-Bench.

5.2 Limits of CoT Methods in Visuospatial Tasks

The authors test zero-shot chain-of-thought, five-run self-consistency, and a tree-of-thoughts variant that proposes and votes on plans and answers. All three lower the reported overall score. The paper reports changes of about −4% for basic CoT and tree search and −1.1% for self-consistency; the plot labels these relative improvements without tabulating the corresponding raw scores. Appearance order and absolute distance improve in some settings, while object and room size often deteriorate.

This supports a limited conclusion about these prompts and Gemini on VSI-Bench, rather than a universal claim that linguistic reasoning harms spatial intelligence. The supplementary implementation differs from the main text on self-consistency temperature: 0.7 versus 1.0.

On a separate 500-question VideoMME subset, the retained table reports 77.2 without CoT and 79.8 with it. That is a 2.6-point gain; the source’s stated 1.6% improvement does not match those values.

Original figure
Figure 9: Relative improvements of CoT, self-consistency and Tree-of-Thought compared to the baseline. All three prevailing prompting techniques fail on average on our benchmark, and, in some cases, task performance becomes much worse after applying them. This implies that VSI-Bench cannot be solved by solely improving linguistic capabilities.
Linguistic prompting techniques, although effective in language reasoning and general visual tasks, are harmful for spatial reasoning.
Case Performance
Gemini-1.5 Pro (w/o CoT) 77.2
Gemini-1.5 Pro (w/   CoT) 79.8
Table 1: Gemini-1.5 Pro CoT performance on a 500-questions subset in VideoMME.

6 How MLLMs Think in Space Visually

The visual analysis asks a model to output object locations on a grid, then measures the quality and usefulness of those explicit maps.

6.1 Probing via Cognitive Maps

Gemini predicts every instance of specified object categories on a 10×10 grid. For locality analysis the prompt supplies all relevant categories, so the task emphasizes layout rather than category discovery. Inter-category distance is the shortest Euclidean distance between any pair of instances belonging to the two categories.

A predicted distance is correct when it differs from the ground-truth grid distance by at most one cell. Across eight distance bins, accuracy falls from 64% in the nearest bin to zero in the farthest. The fixed absolute tolerance and approximate discretized maps affect this trend. It shows stronger local than global accuracy in elicited outputs, not direct evidence of separate internal local world models.

Refer to caption
Figure 10: Visualization of cognitive maps from MLLM and GT.
Refer to caption
Figure 11: Locality of the MLLM’s predicted cognitive maps. The MLLM’s map-distance accuracy decreases dramatically with increasing object distance.
When remembering spaces, a MLLM forms a series of local world models in its mind from a given video, rather than a unified global model.

6.2 Better Distance Reasoning via Cognitive Maps

For relative-distance questions, Gemini first generates a map using only categories named in the question, then answers from it. On the fifty-question tiny-set task, accuracy rises from 46% to 56% with a 10×10 predicted map; a 20×20 predicted map scores 54%.

Ground-truth maps raise accuracy to 66% at 10×10 and 78% at 20×20. These are privileged spatial inputs, demonstrating remaining headroom rather than an ordinary video-only model improvement. The gains are 10 points for the best predicted-map setting and 20–32 points for supplied ground truth.

Case Rel. Dist Acc.
w/o Cog. map 46.0
w/   Cog. map 56.0
w/   Cog. map (GT) 66.0
(a) Cognitive map prompting.
Cog. Map Src. Size Rel. Dist Acc.
MLLM 10 ×\times 10 56.0
MLLM 20 ×\times 20 54.0
GT 10 ×\times 10 66.0
GT 20 ×\times 20 78.0
(b) Cognitive map canvas size.
Table 2: Relative distance task with cognitive map.

7 Related Works

The work builds on research in spatially grounded language models, embodied agents, world models, and cognitive maps. Compared with static-image or text-only spatial tasks, it asks models to integrate observations across a real video into a remembered layout.

General video benchmarks such as VideoMME and EgoSchema emphasize a broader range of content understanding. VSI-Bench isolates spatial questions whose answers may require relations across views, while still depending on perception, language, and temporal memory.

8 Discussion and Future Work

The results identify a substantial gap in spatial understanding despite useful video recognition. Explicit maps offer one intervention, but their benefits vary by model and task. Future work includes spatial training objectives, task-specific fine-tuning, and prompts that represent geometry more directly.

Limitations include annotation-derived measurements, a small human subset, unequal model input sampling, selected post-hoc error explanations, and imperfect grid maps. The paper’s strongest conclusions concern measured behavior, not the inaccessible mechanism of an internal cognitive map. Acknowledgments credit annotators, reviewers, Open Path AI Foundation, Google research compute programs, Intel, IITP, Amazon, and NSF support.

Appendix A Appendix Outline

The supplement documents benchmark construction, linguistic and map probes, evaluation settings, input-order ablations, qualitative examples, and complete results. All substantive material and media are retained below.

Appendix B Technical Details for VSI-Bench Construction and Analysis

The following implementation details explain how questions, explanations, and cognitive maps were produced.

B.1 VSI-Bench Construction Pipeline

The dataset includes 88 ScanNet, 50 ScanNet++, and 150 ARKitScenes videos. Frames are standardized to 640×480; ScanNet uses 24 FPS, while the other sources are subsampled to 30 FPS. ARKitScenes videos are rotated upright. Metadata unifies categories, counts, oriented boxes, room centers, and room areas estimated with an alpha shape.

Small and rare categories are excluded or remapped. Counting questions exclude answers of one. Relative-distance anchors must be unique, while candidate categories may have multiple instances and use their closest distance. Options are rejected when distances are too similar: 30 cm for rooms above 40 m² and 15 cm otherwise. Relative-direction questions filter ambiguous separations and angular boundaries.

Appearance order uses the first timestamp with enough visible object pixels and excludes near ties. Object size is the longest bounding-box dimension. Absolute distance is the minimum distance among sampled points inside the two boxes, an annotation-based approximation. Routes contain about two to five movements between unique landmarks; the question removes turn directions for the model to fill in.

The complete question templates are retained. Quality checks remove incomplete meshes, verify counts and metadata, and ask reviewers to answer without ground truth. Flagged errors trigger data correction, template changes, or removal. Questions are subsampled to limit overrepresentation of scenes and tasks, with multiple-choice letters distributed as uniformly as possible.

Task Question Template
Object Counting How many {category}(s) are in this room?
Relative Distance Measuring from the closest point of each object, which of these objects ({choice a}, {choice b}, {choice c}, {choice d}) is the closest to the {category}?
Relative Direction To create a comprehensive test of relative direction, three difficulty levels were created: • Easy: If I am standing by the {positioning object} and facing the {orienting object}, is the {querying object} to the left or the right of the {orienting object}? • Medium: If I am standing by the {positioning object} and facing the {orienting object}, is the {querying object} to my left, right, or back? An object is to my back if I would have to turn at least 135 degrees in order to face it. • Hard: If I am standing by the {positioning object} and facing the {orienting object}, is the {querying object} to my front-left, front-right, back-left, or back-right? Directions refer to the quadrants of a Cartesian plane (assuming I am at the origin and facing the positive y-axis).
Appearance Order What will be the first-time appearance order of the following categories in the video: {choice a}, {choice b}, {choice c}, {choice d}?
Object Size What is the length of the longest dimension (length, width, or height) of the {category}, measured in centimeters?
Absolute Distance Measuring from the closest point of each object, what is the direct distance between the {object 1} and the {object 2} (in meters)?
Room Size What is the size of this room (in square meters)? If multiple rooms are shown, estimate the size of the combined space.
Route Plan You are a robot beginning at {the bed facing the tv}. You want to navigate to {the toilet}. You will perform the following actions (Note: for each [please fill in], choose either ‘turn back,’ ‘turn left,’ or ‘turn right.’): {1. Go forward until the TV 2. [please fill in] 3. Go forward until the shower 4. [please fill in] 5. Go forward until the toilet.} You have reached the final destination.
Table 3: Question Templates for tasks in VSI-Bench. We replace the highlighted part in the question template from scene to scene to construct our benchmark. Note that a complete example question is provided for Route Plan.

B.2 Probing MLLM via Self-Explanations

Self-explanations are elicited after the model’s answer with a request to explain it step by step. This differs from CoT prompting, where the generated rationale precedes the final prediction.

The analyzed 163 failures comprise every multiple-choice error and the worst-performing half of numerical cases on the tiny subset. Annotators assign one or more of four error categories; a case assigned n categories contributes 1/n to each. The reported spatial share therefore describes this selected, human-labeled explanation sample.

B.3 Implementation Details of CoT Methods

Zero-shot CoT appends a step-by-step request, using temperature 0, top-p 1, and top-k 1. A follow-up dialogue extracts a single option letter or number, reducing answer-parsing errors.

Self-consistency runs the CoT procedure five times and takes the majority answer. The appendix specifies temperature 0.7, top-p 1, and top-k 40; the main text gives temperature 1.0. Tree-of-thoughts uses the appendix sampling settings: generate three plans, vote three times on the best plan, generate three candidate answers from it, and vote three times on the final answer. This is the particular search procedure evaluated, not an exhaustive exploration of possible prompts.

B.4 Cognitive Map

The exact map-generation prompt is preserved as selectable text. It requests all instances of listed categories, object-center coordinates, a 10×10 grid, and a dictionary response. Locality analysis supplies all potential categories; relative-distance QA supplies only categories already named in the question.

Distances are measured between categories using the closest instance pair, and compared with ground-truth map distances using an absolute tolerance of one grid cell. Eight bins summarize performance by distance. This definition is more specific than the main text’s description of all object pairs.

[Task]

This video captures an indoor scene. Your objective is to identify specific objects within the video, understand the spatial arrangement of the scene, and estimate the center point of each object, assuming the entire scene is represented by a 10x10 grid.

[Rule]

1. We provide the categories to care about in this scene: {categories_of_interest}. Focus ONLY on these categories. 

2. Estimate the center location of each instance within the provided categories, assuming the entire scene is represented by a 10x10 grid. 

3. If a category contains multiple instances, include all of them.

4. Each object’s estimated location should accurately reflect its real position in the scene, preserving the relative spatial relationships among all objects.

[Output]

Present the estimated center locations for each object as a list within a dictionary.
STRICTLY follow this JSON format:
{"category name": [(x_1, y_1), …], …}

B.5 Cognitive Map on More MLLMs

Two LLaVA-Video models also show generally better local than distant map accuracy, although their binwise curves are not strictly decreasing. The complete distance-bin table is retained.

On relative-distance QA, a generated map changes LLaVA-Video-72B from 36% to 42%, a six-point gain; the appendix’s eight-percent statement is inconsistent with the table. LLaVA-Video-7B falls from 40% to 32%. Producing a map is therefore not uniformly beneficial, and the proposed capacity explanation is a hypothesis rather than a controlled result.

Distance [1.0, 2.1] (2.1, 3.3] (3.3, 4.4] (4.4, 5.5] (5.5, 6.6] (6.6, 7.8] (7.8, 8.9] (8.9, 10.0]
Gemini-1.5 Pro 0.64 0.48 0.35 0.35 0.28 0.12 0.06 0.00
LLaVA-Video-72B 0.59 0.45 0.42 0.30 0.15 0.23 0.16 0.00
LLaVA-Video-7B 0.50 0.43 0.34 0.29 0.19 0.18 0.14 0.00
Table 4: Locality of cognitive maps.
Models LLaVA-Video-72B LLaVA-Video-7B
w/o. Cog. Map 36.0 40.0
w/.   Cog. Map 42.0 32.0
Table 5: Rel. dist. task with cognitive maps.

Appendix C Evaluation Details

Evaluation details specify prompts, human access to video, frame sampling, and the additional model ablations.

C.1 General Evaluation Setup

Evaluation uses the LMMs-Eval framework with greedy decoding unless a prompting experiment specifies otherwise. Input order is video frames, pre-prompt, question/options, and post-prompt. Open numerical-answer models are asked for a single word or phrase; proprietary models for a single number; multiple-choice models for an option letter. The full prompt table is retained.

C.2 Human Evaluation Setup

Human evaluators receive question and video together and may replay the video without a time limit. This estimates performance with flexible human inspection, rather than under the models’ fixed input constraints.

A further 560-question subset was selected to keep model averages close to full-set averages, with reported mean discrepancy 0.5 points and maximum 2.9. Although the prose mentions human evaluation on this subset, the associated table shows model rows only; no additional human score can be inferred from it.

C.3 Number of Frames Setup

Open models and GPT-4o receive evenly sampled frames over the entire video; Gemini receives an uploaded video with API-controlled processing. The frame-count table preserves each model’s setting. Increasing frames from 8 to 64 gives modest or inconsistent changes for the selected models, including a decline for InternVL2-2B. This ablation does not prove frame selection or visual coverage is unimportant for every model and task.

Refer to caption
Figure 12: Analysis of different # sampled frames.

C.4 More Evaluation Results

Blind evaluation removes visual input. Video improves most overall scores, especially object count, room size, relative distance, and appearance order, but some individual tasks or models worsen. Object size already benefits from language-based priors without video. The smallest OneVision model has an overall visual-minus-blind difference of −0.5 points.

A Socratic variant uses LLaVA-Video-72B frame captions as input to GPT-4o. It scores 29.3 versus 34.0 for direct GPT-4o and 14.5 for blind GPT-4o. Captions convey useful information but lose information available to the direct visual model.

Complete tiny, blind, and visual-minus-blind results are retained. Tiny-set rankings are broadly similar, not identical: LLaVA-Video-72B beats OneVision-72B on the full set but trails it on tiny. The six-model average in the blind-evaluation plot should not be mistaken for every individual model’s behavior.

Original figure
Figure 13: Performance comparisons between Vision Enabled (w/ video), Vision Disabled (w/o video) and Chance Level (Freq.). Enabled−-Disabled indicates the gap between Vision Enabled and Vision Disabled, and Disabled−-Chance betokens the gap between Vision Disabled and Chance Level (Freq.). Tasks are sorted by Enable−-Disable for better understanding.
GPT-4o Standard Socratic Blind
Avg. 34.0 29.3 14.5
Table 6: Socratic LLMs with Frame Captions.

Appendix D Input Sequencing and Repetition Analysis

On VSI-Bench tiny, video-first input scores 48.8 and question-first input 46.3. The table therefore supports a 2.5-point decline from moving the question first. One prose sentence reverses that direction; the retained table is the basis for this interpretation.

Repeating the same video around intervening context raises the score to 50.9, a 2.1-point increase over one presentation. This is an observed effect of input sequencing and repetition. It does not establish that an autoregressive model can freely revisit its visual input in the same way a person can replay a video.

Order Avg.
Video first 48.8
Question first 46.3
(a) Input Sequence
# Times Avg.
1 48.8
2 50.9
(b) Video Repetition Times
Table 7: Ablations on the video input sequence and repetition.

Appendix E Visualization Results

The remaining media preserve benchmark examples, annotated errors, complete prompting demonstrations, additional predicted maps, and all supplementary evaluation tables.

E.1 VSI-Bench Examples

Two full-page figures show eight task types in four scenes, including cases without a valid question. They preserve the exact options, units, ground-truth answers, and video-frame strips used to illustrate the benchmark format.

E.2 Error Analysis Examples

The error figure provides eight annotated examples across perception, relational reasoning, linguistic reasoning, and perspective transformation. Highlighted response excerpts show the authors’ basis for each diagnosis; these remain qualitative interpretations of generated explanations. The adjacent frame-count table preserves the model input settings.

Methods # of Frames
Proprietary Models (API)
GPT-4o 16
Gemini-1.5 Flash -
Gemini-1.5 Pro -
Open-source Models
InternVL2-2B 32
InternVL2-8B 32
InternVL2-40B 32
LongVILA-8B 32
VILA-1.5-8B 32
VILA-1.5-40B 32
LongVA-7B 32
LLaVA-Video-7B 32
LLaVA-Video-72B 32
LLaVA-OneVision-0.5B 32
LLaVA-OneVision-7B 32
LLaVA-OneVision-72B 32
Table 8: Number of frames used in evaluation.

E.3 Linguistic Prompting Examples

The CoT, self-consistency, and tree-search figures retain complete examples for object count, object size, and room size. They include the candidate responses, votes, selected plans, and final answers, exposing both useful deductions and unsupported geometric assumptions. Native PDF graphics preserve the small text for zooming.

E.4 Cognitive Map Examples

Ten additional predicted maps are paired with ground truth, retaining all object-category legends and layout comparisons. The final tables give evaluation prompts, the 560-question subset, complete tiny-set scores, blind scores, and visual-minus-blind differences. They include negative effects and all reported model variants.

Models QA. Type Prompt
Pre-Prompt - - These are frames of a video.
Post-Prompt Open-source Models NA Please answer the question using a single word or phrase.
MCA Answer with the option’s letter from the given choices directly.
Proprietary Models NA Do not respond with anything other than a single number!
MCA Answer with the option’s letter from the given choices directly.
Table 9: Prompts used in evaluation. NA and MAC indicates questions with Numerical Answer and Multiple Choice Answer respectively.

Obj. Count

Abs. Dist.

Obj. Size

Room Size

Rel. Dist.

Rel. Dir.

Route Plan

Appr. Order

Methods Avg. Numerical Answer Multiple-Choice Answer
Proprietary Models (API)
Gemini-1.5 Flash 41.6 49.1 30.3 52.7 53.7 37.1 40.8 31.4 37.1
Gemini-1.5 Pro 44.9 55.1 30.3 63.1 43.3 50.0 45.9 35.7 35.7
Open-source Models
InternVL2-2B 27.0 22.4 24.9 21.1 34.1 32.9 43.5 30.0 7.1
InternVL2-8B 34.1 22.6 28.3 47.6 39.6 35.7 30.4 30.0 38.6
InternVL2-40B 35.5 34.4 26.9 45.6 31.3 41.4 31.7 32.9 40.0
LongVILA-8B 21.0 28.7 8.6 16.3 0.0 28.6 30.5 31.4 24.3
VILA-1.5-8B 28.4 17.3 21.6 49.9 18.6 31.4 34.4 30.0 24.3
VILA-1.5-40B 30.8 21.4 24.4 48.3 21.9 40.0 25.0 30.0 35.7
LongVA-7B 29.0 38.1 16.9 38.1 21.7 32.9 42.8 25.7 15.7
LLaVA-Video-7B 34.9 47.9 13.4 46.7 23.9 42.9 41.9 32.9 30.0
LLaVA-Video-72B 40.5 48.3 22.6 56.7 34.6 41.4 36.5 35.7 48.6
LLaVA-OneVision-0.5B 27.6 45.1 27.9 14.7 27.9 28.6 37.0 34.3 5.7
LLaVA-OneVision-7B 32.1 46.9 19.9 46.9 12.1 41.4 35.1 30.0 24.3
LLaVA-OneVision-72B 39.6 42.7 23.7 56.7 36.9 41.4 39.5 31.4 44.3
Table 10: Evaluation results on VSI-Bench 560 samples subset.

Obj. Count

Abs. Dist.

Obj. Size

Room Size

Rel. Dist.

Rel. Dir.

Route Plan

Appr. Order

Methods Avg. Numerical Answer Multiple-Choice Answer
Proprietary Models (API)
GPT-4o 35.6 36.2 4.6 47.2 40.4 40.0 46.2 32.0 38.0
Gemini-1.5 Flash 45.7 50.8 33.6 56.5 45.2 48.0 39.8 32.7 59.2
Gemini-1.5 Pro 48.8 49.6 28.8 58.6 49.4 46.0 48.1 42.0 68.0
Gemini-2.0 Flash 45.4 52.4 30.6 66.7 31.8 56.0 46.3 24.5 55.1
Open-source Models
InternVL2-2B 25.5 30.6 20.4 26.0 29.6 28.0 39.2 28.0 2.0
InternVL2-8B 32.9 26.4 25.4 43.8 41.6 30.0 32.2 20.0 44.0
InternVL2-40B 37.6 40.8 23.8 48.0 26.0 46.0 30.1 42.0 44.0
LongVILA-8B 19.1 23.4 10.8 11.4 0.0 20.0 33.1 28.0 26.0
VILA-1.5-8B 31.4 12.2 23.4 51.4 18.6 36.0 41.5 42.0 26.0
VILA-1.5-40B 32.3 14.6 21.0 48.0 20.6 42.0 22.0 40.0 50.0
LongVA-7B 31.8 41.2 17.4 39.6 25.4 30.0 52.8 34.0 14.0
LLaVA-Video-7B 35.7 49.0 12.8 48.6 21.4 40.0 43.5 34.0 36.0
LLaVA-Video-72B 39.3 41.4 26.6 55.6 31.6 36.0 25.6 42.0 56.0
LLaVA-OneVision-0.5B 27.7 44.0 23.0 18.8 28.4 30.0 33.4 36.0 8.0
LLaVA-OneVision-7B 33.8 48.2 22.0 44.4 14.0 44.0 31.9 34.0 32.0
LLaVA-OneVision-72B 41.6 38.0 31.6 54.4 35.2 44.0 39.7 32.0 58.0
Table 11: Complete VSI-Bench (tiny) evaluation results.

Obj. Count

Abs. Dist.

Obj. Size

Room Size

Rel. Dist.

Rel. Dir.

Route Plan

Appr. Order

Methods Avg. Numerical Answer Multiple-Choice Answer
Proprietary Models (API)
GPT-4o 14.5 0.1 5.2 36.7 0.0 10.8 23.2 26.9 13.1
Gemini-1.5 Flash 19.9 25.0 30.3 52.5 0.0 0.0 21.2 29.9 0.2
Gemini-1.5 Pro 32.3 30.6 11.5 51.5 33.1 33.8 44.6 33.5 20.2
Open-source Models
InternVL2-2B 17.8 5.4 23.7 9.2 0.0 26.9 41.2 27.9 7.9
InternVL2-8B 27.6 31.9 26.8 38.3 0.7 27.1 39.2 33.0 23.6
InternVL2-40B 24.4 5.4 29.1 39.2 0.7 30.3 37.7 27.9 24.7
LongVILA-8B 20.2 47.4 12.6 8.7 0.6 24.3 27.0 27.4 13.9
VILA-1.5-8B 21.5 7.4 7.6 45.7 0.0 25.4 39.1 29.4 17.6
VILA-1.5-40B 25.5 5.3 27.6 46.5 0.7 30.2 37.1 31.5 25.0
LongVA-7B 21.9 5.1 18.1 27.4 26.1 23.4 39.8 26.9 8.7
LLaVA-Video-7B 25.2 14.8 14.6 32.5 26.1 26.8 45.0 33.0 8.5
LLaVA-Video-72B 29.1 19.0 25.4 46.3 26.1 29.0 38.8 33.0 15.5
LLaVA-OneVision-0.5B 28.6 38.4 30.1 32.0 24.3 22.0 41.8 34.5 5.4
LLaVA-OneVision-7B 25.3 13.8 8.5 45.5 26.1 28.6 41.2 27.9 11.1
LLaVA-OneVision-72B 28.9 8.2 23.8 54.1 26.1 30.4 38.1 33.0 17.1
Table 12: Complete blind evaluation results.

Obj. Count

Abs. Dist.

Obj. Size

Room Size

Rel. Dist.

Rel. Dir.

Route Plan

Appr. Order

Methods Avg. Numerical Answer Multiple-Choice Answer
Proprietary Models (API)
GPT-4o 19.5 46.1 0.1 7.1 38.2 26.2 18.0 4.6 15.4
Gemini-1.5 Flash 22.2 24.9 0.5 1.0 54.4 37.7 19.9 1.5 37.7
Gemini-1.5 Pro 13.0 25.5 19.5 12.6 10.6 17.5 1.7 2.5 14.4
Open-source Models
InternVL2-2B 8.7 20.3 0.3 10.8 29.2 5.2 2.9 2.5 -1.6
InternVL2-8B 9.9 -0.6 2.2 10.6 43.5 10.9 -5.8 -4.1 22.8
InternVL2-40B 12.6 35.9 -2.9 9.0 26.8 17.3 -5.0 9.9 20.0
LongVILA-8B 1.4 -18.2 -3.5 7.9 -0.6 5.3 3.7 5.1 11.5
VILA-1.5-8B 7.3 10.0 14.2 4.6 18.8 6.7 -4.4 1.5 7.2
VILA-1.5-40B 5.7 17.1 -2.8 2.2 22.0 10.4 -11.4 0.0 7.9
LongVA-7B 7.2 32.9 -1.5 11.5 -3.9 9.7 3.5 -1.5 7.1
LLaVA-Video-7B 10.5 33.8 -0.6 15.2 -1.9 16.7 -2.7 1.0 22.1
LLaVA-Video-72B 11.7 29.9 -2.6 11.1 9.2 13.3 -2.0 2.0 33.0
LLaVA-OneVision-0.5B -0.5 7.8 -1.7 -16.6 4.0 6.9 -5.0 0.0 0.3
LLaVA-OneVision-7B 7.0 33.9 11.7 1.9 -13.9 13.9 -6.0 1.5 13.3
LLaVA-OneVision-72B 11.4 35.4 0.1 3.5 11.4 12.1 1.8 -0.5 27.4
Table 13: Results of Vision Enabled −- Vision Disabled.
Refer to caption
Figure 14: VSI-Bench Examples (Part 1).
Refer to caption
Figure 15: VSI-Bench Examples (Part 2).
Refer to caption
Figure 16: Additional Error Analysis Examples.
Refer to caption
Figure 17: Zero-Shot CoT Examples.
Refer to caption
Figure 18: Self-Consistency w/ CoT Examples.
Refer to caption
Figure 19: Tree-of-Thought Examples.
Original figure
Figure 20: Additional predicted cognitive map examples.
Concept demonstration

Open image ↗