Overview
An indoor video can reveal individual objects without yielding an accurate memory of the room’s layout. This paper tests the distinction through questions about counts, distances, directions, dimensions, routes, and appearance order.
Original resources: project page, evaluation code, and dataset.
At a glance
VSI-Bench contains over 5,000 question–answer pairs from 288 real indoor videos. Fifteen video-capable multimodal language models show useful but incomplete spatial understanding. On a shared 400-question subset, human evaluators score 79.2 versus 48.8 for Gemini-1.5 Pro, using a mixture of multiple-choice accuracy and numerical proximity scores.
The authors examine generated explanations and explicit grid maps. Tested verbal reasoning prompts reduce average performance, while generating a map improves Gemini’s relative-distance accuracy from 46% to 56%. These are behavioral results from specific models and prompts; post-hoc explanations and generated maps do not directly expose a model’s internal representations.
1 Introduction
Visual-spatial intelligence requires remembering objects and relations across changing viewpoints, then using that information for a new question. Accurate video recognition alone does not ensure a consistent room layout or a correct transformation between camera-centered and environment-centered coordinates.
VSI-Bench repurposes annotated indoor scans into video questions. Inspired by dual-coding theory, the analysis asks models to express spatial information through language and through maps. The key empirical contrast is that additional verbal deliberation does not reliably help these tasks, whereas an explicit spatial representation can help a particular distance query.
2 Visual-Spatial Intelligence
The taxonomy separates visual perception, linguistic reasoning, temporal processing, and spatial reasoning. Spatial reasoning includes relationships such as distance, direction, and relative size, plus transformations between egocentric and allocentric perspectives. Working memory and imagined viewpoints support those transformations.
The scope is real indoor environments rather than abstract mental-rotation puzzles. Tasks mix capabilities: appearance order needs temporal memory; object size can use familiar dimensions; route planning and relative direction strongly depend on perspective. The taxonomy organizes the authors’ error annotations, rather than independently measuring separate cognitive modules.
3 VSI-Bench
The benchmark combines video observations with questions derived from 3D scene annotations and human-authored routes.
3.1 Overview
The videos come from validation splits of ScanNet, ScanNet++, and ARKitScenes, spanning homes, professional spaces, and industrial settings. Configuration tasks ask object counts, relative distances, relative directions, and route plans; measurement tasks ask object size, room size, and absolute distance; appearance order tests spatiotemporal memory.
Seven tasks use templates and scan metadata; routes are human annotated. Video input exposes a sequence of partial views, while the source scans provide object categories, bounding boxes, and spatial measurements. These annotation-derived answers remain approximations of the physical environment, despite repeated human review.
3.2 Benchmark Construction
The pipeline standardizes videos and annotations, generates questions, and iteratively checks them with people. Reviewers filter incomplete reconstructions, correct metadata, and flag ambiguous questions without first seeing the expected answers. Corrections can change templates or generation rules so a recurring error is fixed at its source.
The retained evaluation figure includes the full results table and radar chart. Its human and dagger-marked model rows use the 400-question tiny subset; the other model rows use the full benchmark. They should not be treated as a single matched evaluation.
|
Obj. Count |
Abs. Dist. |
Obj. Size |
Room Size |
Rel. Dist. |
Rel. Dir. |
Route Plan |
Appr. Order | |||
| Methods | Rank | Avg. | Numerical Answer | Multiple-Choice Answer | ||||||
| Baseline | ||||||||||
| Chance Level (Random) | - | - | - | - | - | - | 25.0 | 36.1 | 28.3 | 25.0 |
| Chance Level (Frequency) | - | 34.0 | 62.1 | 32.0 | 29.9 | 33.1 | 25.1 | 47.9 | 28.4 | 25.2 |
| VSI-Bench (tiny) Perf. | ||||||||||
| †Human Level | - | 79.2 | 94.3 | 47.0 | 60.4 | 45.9 | 94.7 | 95.8 | 95.8 | 100.0 |
| †Gemini-1.5 Flash | - | 45.7 | 50.8 | 33.6 | 56.5 | 45.2 | 48.0 | 39.8 | 32.7 | 59.2 |
| †Gemini-1.5 Pro | - | 48.8 | 49.6 | 28.8 | 58.6 | 49.4 | 46.0 | 48.1 | 42.0 | 68.0 |
| †Gemini-2.0 Flash | - | 45.4 | 52.4 | 30.6 | 66.7 | 31.8 | 56.0 | 46.3 | 24.5 | 55.1 |
| Proprietary Models (API) | ||||||||||
| GPT-4o | 3 | 34.0 | 46.2 | 5.3 | 43.8 | 38.2 | 37.0 | 41.3 | 31.5 | 28.5 |
| Gemini-1.5 Flash | 2 | 42.1 | 49.8 | 30.8 | 53.5 | 54.4 | 37.7 | 41.0 | 31.5 | 37.8 |
| Gemini-1.5 Pro | 1 | 45.4 | 56.2 | 30.9 | 64.1 | 43.6 | 51.3 | 46.3 | 36.0 | 34.6 |
| Open-source Models | ||||||||||
| InternVL2-2B | 11 | 26.5 | 25.7 | 24.0 | 20.0 | 29.2 | 32.1 | 44.1 | 30.4 | 6.3 |
| InternVL2-8B | 3 | 37.5 | 31.3 | 29.0 | 48.9 | 44.2 | 38.0 | 33.4 | 28.9 | 46.4 |
| InternVL2-40B | 4 | 37.0 | 41.3 | 26.2 | 48.2 | 27.5 | 47.6 | 32.7 | 27.8 | 44.7 |
| LongVILA-8B | 12 | 21.6 | 29.1 | 9.1 | 16.7 | 0.0 | 29.6 | 30.7 | 32.5 | 25.5 |
| VILA-1.5-8B | 9 | 28.9 | 17.4 | 21.8 | 50.3 | 18.8 | 32.1 | 34.8 | 31.0 | 24.8 |
| VILA-1.5-40B | 7 | 31.2 | 22.4 | 24.8 | 48.7 | 22.7 | 40.5 | 25.7 | 31.5 | 32.9 |
| LongVA-7B | 8 | 29.2 | 38.0 | 16.6 | 38.9 | 22.2 | 33.1 | 43.3 | 25.4 | 15.7 |
| LLaVA-Video-7B | 5 | 35.6 | 48.5 | 14.0 | 47.8 | 24.2 | 43.5 | 42.4 | 34.0 | 30.6 |
| LLaVA-Video-72B | 1 | 40.9 | 48.9 | 22.8 | 57.4 | 35.3 | 42.4 | 36.7 | 35.0 | 48.6 |
| LLaVA-OneVision-0.5B | 10 | 28.0 | 46.1 | 28.4 | 15.4 | 28.3 | 28.9 | 36.9 | 34.5 | 5.8 |
| LLaVA-OneVision-7B | 6 | 32.4 | 47.7 | 20.2 | 47.4 | 12.3 | 42.5 | 35.2 | 29.4 | 24.4 |
| LLaVA-OneVision-72B | 2 | 40.2 | 43.5 | 23.9 | 57.6 | 37.5 | 42.5 | 39.9 | 32.5 | 44.6 |
4 Evaluation on VSI-Bench
Evaluation compares historical proprietary and open models, simple chance baselines, and a human subset. Input sampling and numerical-answer scoring matter when interpreting the reported averages.
4.1 Evaluation Setup
Models use their default input format with zero-shot prompts and greedy decoding. GPT-4o receives 16 sampled frames, open models generally receive 32, and Gemini receives the complete uploaded video through its API. This compares deployed model setups, not equal frame or compute budgets.
Multiple-choice answers are scored by exact or fuzzy matching. Numerical answers use mean relative accuracy: the retained equation averages ten indicators for relative error below 1−θ, with θ from 0.50 to 0.95. Thus the final eight-task average mixes categorical accuracy and a graded numerical score; it is not the fraction of all questions answered exactly.
The random baseline applies only to multiple-choice tasks. The frequency baseline always selects each task’s most common answer and scores 34.0 overall. Human evaluation uses 400 questions, fifty per task, with unlimited time and video replay. The same subset is used for the explicitly marked model comparisons.
Equation 1
| (1) |
4.2 Main Results
On the full set, Gemini-1.5 Pro scores 45.4, Gemini-1.5 Flash 42.1, and GPT-4o 34.0. Leading open models are LLaVA-Video-72B at 40.9 and LLaVA-OneVision-72B at 40.2; seven of twelve open models fall below the frequency baseline. These results characterize the evaluated model versions, not current model rankings.
On the matched tiny subset, humans score 79.2 and Gemini-1.5 Pro 48.8, a 30.4-point gap. Comparing the human subset with the full-set model score, as the source’s approximately 33-point statement does, mixes evaluation sets. Humans excel on configuration and appearance order; differences are smaller for some measurements, with Gemini exceeding the human room-size score but remaining well below humans on absolute distance.
5 How MLLMs Think in Space Linguistically
The linguistic analysis distinguishes explanations elicited after an answer from prompting methods that request intermediate reasoning before the answer.
5.1 Probing via Self-Explanations
Gemini-1.5 Pro first answers a question, then explains its answer step by step. Human annotators classify 163 selected failures as perception, linguistic reasoning, relational spatial reasoning, or egocentric–allocentric transformation errors. A case with several causes contributes equal fractions to those categories.
About 71% of the attributed error mass is spatial. The appendix specifies all multiple-choice failures and the worst half of numerical-answer cases on the tiny set, whereas the main text calls the sample random. This is a selected error analysis, not a representative estimate of all model behavior. Generated post-hoc rationales may also differ from the actual causes of a prediction.
The retained examples contrast an apparently coherent global-coordinate explanation with a route error that confuses camera turning and the requested perspective. They illustrate the annotation scheme without establishing that the model literally maintains the internal world model described in its prose.
5.2 Limits of CoT Methods in Visuospatial Tasks
The authors test zero-shot chain-of-thought, five-run self-consistency, and a tree-of-thoughts variant that proposes and votes on plans and answers. All three lower the reported overall score. The paper reports changes of about −4% for basic CoT and tree search and −1.1% for self-consistency; the plot labels these relative improvements without tabulating the corresponding raw scores. Appearance order and absolute distance improve in some settings, while object and room size often deteriorate.
This supports a limited conclusion about these prompts and Gemini on VSI-Bench, rather than a universal claim that linguistic reasoning harms spatial intelligence. The supplementary implementation differs from the main text on self-consistency temperature: 0.7 versus 1.0.
On a separate 500-question VideoMME subset, the retained table reports 77.2 without CoT and 79.8 with it. That is a 2.6-point gain; the source’s stated 1.6% improvement does not match those values.
6 How MLLMs Think in Space Visually
The visual analysis asks a model to output object locations on a grid, then measures the quality and usefulness of those explicit maps.
6.1 Probing via Cognitive Maps
Gemini predicts every instance of specified object categories on a 10×10 grid. For locality analysis the prompt supplies all relevant categories, so the task emphasizes layout rather than category discovery. Inter-category distance is the shortest Euclidean distance between any pair of instances belonging to the two categories.
A predicted distance is correct when it differs from the ground-truth grid distance by at most one cell. Across eight distance bins, accuracy falls from 64% in the nearest bin to zero in the farthest. The fixed absolute tolerance and approximate discretized maps affect this trend. It shows stronger local than global accuracy in elicited outputs, not direct evidence of separate internal local world models.
6.2 Better Distance Reasoning via Cognitive Maps
For relative-distance questions, Gemini first generates a map using only categories named in the question, then answers from it. On the fifty-question tiny-set task, accuracy rises from 46% to 56% with a 10×10 predicted map; a 20×20 predicted map scores 54%.
Ground-truth maps raise accuracy to 66% at 10×10 and 78% at 20×20. These are privileged spatial inputs, demonstrating remaining headroom rather than an ordinary video-only model improvement. The gains are 10 points for the best predicted-map setting and 20–32 points for supplied ground truth.
7 Related Works
The work builds on research in spatially grounded language models, embodied agents, world models, and cognitive maps. Compared with static-image or text-only spatial tasks, it asks models to integrate observations across a real video into a remembered layout.
General video benchmarks such as VideoMME and EgoSchema emphasize a broader range of content understanding. VSI-Bench isolates spatial questions whose answers may require relations across views, while still depending on perception, language, and temporal memory.
8 Discussion and Future Work
The results identify a substantial gap in spatial understanding despite useful video recognition. Explicit maps offer one intervention, but their benefits vary by model and task. Future work includes spatial training objectives, task-specific fine-tuning, and prompts that represent geometry more directly.
Limitations include annotation-derived measurements, a small human subset, unequal model input sampling, selected post-hoc error explanations, and imperfect grid maps. The paper’s strongest conclusions concern measured behavior, not the inaccessible mechanism of an internal cognitive map. Acknowledgments credit annotators, reviewers, Open Path AI Foundation, Google research compute programs, Intel, IITP, Amazon, and NSF support.
Appendix A Appendix Outline
The supplement documents benchmark construction, linguistic and map probes, evaluation settings, input-order ablations, qualitative examples, and complete results. All substantive material and media are retained below.
Appendix B Technical Details for VSI-Bench Construction and Analysis
The following implementation details explain how questions, explanations, and cognitive maps were produced.
B.1 VSI-Bench Construction Pipeline
The dataset includes 88 ScanNet, 50 ScanNet++, and 150 ARKitScenes videos. Frames are standardized to 640×480; ScanNet uses 24 FPS, while the other sources are subsampled to 30 FPS. ARKitScenes videos are rotated upright. Metadata unifies categories, counts, oriented boxes, room centers, and room areas estimated with an alpha shape.
Small and rare categories are excluded or remapped. Counting questions exclude answers of one. Relative-distance anchors must be unique, while candidate categories may have multiple instances and use their closest distance. Options are rejected when distances are too similar: 30 cm for rooms above 40 m² and 15 cm otherwise. Relative-direction questions filter ambiguous separations and angular boundaries.
Appearance order uses the first timestamp with enough visible object pixels and excludes near ties. Object size is the longest bounding-box dimension. Absolute distance is the minimum distance among sampled points inside the two boxes, an annotation-based approximation. Routes contain about two to five movements between unique landmarks; the question removes turn directions for the model to fill in.
The complete question templates are retained. Quality checks remove incomplete meshes, verify counts and metadata, and ask reviewers to answer without ground truth. Flagged errors trigger data correction, template changes, or removal. Questions are subsampled to limit overrepresentation of scenes and tasks, with multiple-choice letters distributed as uniformly as possible.
| Task | Question Template |
| Object Counting | How many {category}(s) are in this room? |
| Relative Distance | Measuring from the closest point of each object, which of these objects ({choice a}, {choice b}, {choice c}, {choice d}) is the closest to the {category}? |
| Relative Direction | To create a comprehensive test of relative direction, three difficulty levels were created: • Easy: If I am standing by the {positioning object} and facing the {orienting object}, is the {querying object} to the left or the right of the {orienting object}? • Medium: If I am standing by the {positioning object} and facing the {orienting object}, is the {querying object} to my left, right, or back? An object is to my back if I would have to turn at least 135 degrees in order to face it. • Hard: If I am standing by the {positioning object} and facing the {orienting object}, is the {querying object} to my front-left, front-right, back-left, or back-right? Directions refer to the quadrants of a Cartesian plane (assuming I am at the origin and facing the positive y-axis). |
| Appearance Order | What will be the first-time appearance order of the following categories in the video: {choice a}, {choice b}, {choice c}, {choice d}? |
| Object Size | What is the length of the longest dimension (length, width, or height) of the {category}, measured in centimeters? |
| Absolute Distance | Measuring from the closest point of each object, what is the direct distance between the {object 1} and the {object 2} (in meters)? |
| Room Size | What is the size of this room (in square meters)? If multiple rooms are shown, estimate the size of the combined space. |
| Route Plan | You are a robot beginning at {the bed facing the tv}. You want to navigate to {the toilet}. You will perform the following actions (Note: for each [please fill in], choose either ‘turn back,’ ‘turn left,’ or ‘turn right.’): {1. Go forward until the TV 2. [please fill in] 3. Go forward until the shower 4. [please fill in] 5. Go forward until the toilet.} You have reached the final destination. |
B.2 Probing MLLM via Self-Explanations
Self-explanations are elicited after the model’s answer with a request to explain it step by step. This differs from CoT prompting, where the generated rationale precedes the final prediction.
The analyzed 163 failures comprise every multiple-choice error and the worst-performing half of numerical cases on the tiny subset. Annotators assign one or more of four error categories; a case assigned n categories contributes 1/n to each. The reported spatial share therefore describes this selected, human-labeled explanation sample.
B.3 Implementation Details of CoT Methods
Zero-shot CoT appends a step-by-step request, using temperature 0, top-p 1, and top-k 1. A follow-up dialogue extracts a single option letter or number, reducing answer-parsing errors.
Self-consistency runs the CoT procedure five times and takes the majority answer. The appendix specifies temperature 0.7, top-p 1, and top-k 40; the main text gives temperature 1.0. Tree-of-thoughts uses the appendix sampling settings: generate three plans, vote three times on the best plan, generate three candidate answers from it, and vote three times on the final answer. This is the particular search procedure evaluated, not an exhaustive exploration of possible prompts.
B.4 Cognitive Map
The exact map-generation prompt is preserved as selectable text. It requests all instances of listed categories, object-center coordinates, a 10×10 grid, and a dictionary response. Locality analysis supplies all potential categories; relative-distance QA supplies only categories already named in the question.
Distances are measured between categories using the closest instance pair, and compared with ground-truth map distances using an absolute tolerance of one grid cell. Eight bins summarize performance by distance. This definition is more specific than the main text’s description of all object pairs.
[Task]
This video captures an indoor scene. Your objective is to identify specific objects within the video, understand the spatial arrangement of the scene, and estimate the center point of each object, assuming the entire scene is represented by a 10x10 grid.
[Rule]
1. We provide the categories to care about in this scene: {categories_of_interest}. Focus ONLY on these categories.
2. Estimate the center location of each instance within the provided categories, assuming the entire scene is represented by a 10x10 grid.
3. If a category contains multiple instances, include all of them.
4. Each object’s estimated location should accurately reflect its real position in the scene, preserving the relative spatial relationships among all objects.
[Output]
Present the estimated center locations for each object as a list within a dictionary.
STRICTLY follow this JSON format:
{"category name": [(x_1, y_1), …], …}B.5 Cognitive Map on More MLLMs
Two LLaVA-Video models also show generally better local than distant map accuracy, although their binwise curves are not strictly decreasing. The complete distance-bin table is retained.
On relative-distance QA, a generated map changes LLaVA-Video-72B from 36% to 42%, a six-point gain; the appendix’s eight-percent statement is inconsistent with the table. LLaVA-Video-7B falls from 40% to 32%. Producing a map is therefore not uniformly beneficial, and the proposed capacity explanation is a hypothesis rather than a controlled result.
| Distance | [1.0, 2.1] | (2.1, 3.3] | (3.3, 4.4] | (4.4, 5.5] | (5.5, 6.6] | (6.6, 7.8] | (7.8, 8.9] | (8.9, 10.0] |
| Gemini-1.5 Pro | 0.64 | 0.48 | 0.35 | 0.35 | 0.28 | 0.12 | 0.06 | 0.00 |
| LLaVA-Video-72B | 0.59 | 0.45 | 0.42 | 0.30 | 0.15 | 0.23 | 0.16 | 0.00 |
| LLaVA-Video-7B | 0.50 | 0.43 | 0.34 | 0.29 | 0.19 | 0.18 | 0.14 | 0.00 |
| Models | LLaVA-Video-72B | LLaVA-Video-7B |
| w/o. Cog. Map | 36.0 | 40.0 |
| w/. Cog. Map | 42.0 | 32.0 |
Appendix C Evaluation Details
Evaluation details specify prompts, human access to video, frame sampling, and the additional model ablations.
C.1 General Evaluation Setup
Evaluation uses the LMMs-Eval framework with greedy decoding unless a prompting experiment specifies otherwise. Input order is video frames, pre-prompt, question/options, and post-prompt. Open numerical-answer models are asked for a single word or phrase; proprietary models for a single number; multiple-choice models for an option letter. The full prompt table is retained.
C.2 Human Evaluation Setup
Human evaluators receive question and video together and may replay the video without a time limit. This estimates performance with flexible human inspection, rather than under the models’ fixed input constraints.
A further 560-question subset was selected to keep model averages close to full-set averages, with reported mean discrepancy 0.5 points and maximum 2.9. Although the prose mentions human evaluation on this subset, the associated table shows model rows only; no additional human score can be inferred from it.
C.3 Number of Frames Setup
Open models and GPT-4o receive evenly sampled frames over the entire video; Gemini receives an uploaded video with API-controlled processing. The frame-count table preserves each model’s setting. Increasing frames from 8 to 64 gives modest or inconsistent changes for the selected models, including a decline for InternVL2-2B. This ablation does not prove frame selection or visual coverage is unimportant for every model and task.
C.4 More Evaluation Results
Blind evaluation removes visual input. Video improves most overall scores, especially object count, room size, relative distance, and appearance order, but some individual tasks or models worsen. Object size already benefits from language-based priors without video. The smallest OneVision model has an overall visual-minus-blind difference of −0.5 points.
A Socratic variant uses LLaVA-Video-72B frame captions as input to GPT-4o. It scores 29.3 versus 34.0 for direct GPT-4o and 14.5 for blind GPT-4o. Captions convey useful information but lose information available to the direct visual model.
Complete tiny, blind, and visual-minus-blind results are retained. Tiny-set rankings are broadly similar, not identical: LLaVA-Video-72B beats OneVision-72B on the full set but trails it on tiny. The six-model average in the blind-evaluation plot should not be mistaken for every individual model’s behavior.
Appendix D Input Sequencing and Repetition Analysis
On VSI-Bench tiny, video-first input scores 48.8 and question-first input 46.3. The table therefore supports a 2.5-point decline from moving the question first. One prose sentence reverses that direction; the retained table is the basis for this interpretation.
Repeating the same video around intervening context raises the score to 50.9, a 2.1-point increase over one presentation. This is an observed effect of input sequencing and repetition. It does not establish that an autoregressive model can freely revisit its visual input in the same way a person can replay a video.
| Order | Avg. |
| Video first | 48.8 |
| Question first | 46.3 |
| # Times | Avg. |
| 1 | 48.8 |
| 2 | 50.9 |
Appendix E Visualization Results
The remaining media preserve benchmark examples, annotated errors, complete prompting demonstrations, additional predicted maps, and all supplementary evaluation tables.
E.1 VSI-Bench Examples
Two full-page figures show eight task types in four scenes, including cases without a valid question. They preserve the exact options, units, ground-truth answers, and video-frame strips used to illustrate the benchmark format.
E.2 Error Analysis Examples
The error figure provides eight annotated examples across perception, relational reasoning, linguistic reasoning, and perspective transformation. Highlighted response excerpts show the authors’ basis for each diagnosis; these remain qualitative interpretations of generated explanations. The adjacent frame-count table preserves the model input settings.
| Methods | # of Frames |
| Proprietary Models (API) | |
| GPT-4o | 16 |
| Gemini-1.5 Flash | - |
| Gemini-1.5 Pro | - |
| Open-source Models | |
| InternVL2-2B | 32 |
| InternVL2-8B | 32 |
| InternVL2-40B | 32 |
| LongVILA-8B | 32 |
| VILA-1.5-8B | 32 |
| VILA-1.5-40B | 32 |
| LongVA-7B | 32 |
| LLaVA-Video-7B | 32 |
| LLaVA-Video-72B | 32 |
| LLaVA-OneVision-0.5B | 32 |
| LLaVA-OneVision-7B | 32 |
| LLaVA-OneVision-72B | 32 |
E.3 Linguistic Prompting Examples
The CoT, self-consistency, and tree-search figures retain complete examples for object count, object size, and room size. They include the candidate responses, votes, selected plans, and final answers, exposing both useful deductions and unsupported geometric assumptions. Native PDF graphics preserve the small text for zooming.
E.4 Cognitive Map Examples
Ten additional predicted maps are paired with ground truth, retaining all object-category legends and layout comparisons. The final tables give evaluation prompts, the 560-question subset, complete tiny-set scores, blind scores, and visual-minus-blind differences. They include negative effects and all reported model variants.
| Models | QA. Type | Prompt | |
| Pre-Prompt | - | - | These are frames of a video. |
| Post-Prompt | Open-source Models | NA | Please answer the question using a single word or phrase. |
| MCA | Answer with the option’s letter from the given choices directly. | ||
| Proprietary Models | NA | Do not respond with anything other than a single number! | |
| MCA | Answer with the option’s letter from the given choices directly. |
|
Obj. Count |
Abs. Dist. |
Obj. Size |
Room Size |
Rel. Dist. |
Rel. Dir. |
Route Plan |
Appr. Order | ||
| Methods | Avg. | Numerical Answer | Multiple-Choice Answer | ||||||
| Proprietary Models (API) | |||||||||
| Gemini-1.5 Flash | 41.6 | 49.1 | 30.3 | 52.7 | 53.7 | 37.1 | 40.8 | 31.4 | 37.1 |
| Gemini-1.5 Pro | 44.9 | 55.1 | 30.3 | 63.1 | 43.3 | 50.0 | 45.9 | 35.7 | 35.7 |
| Open-source Models | |||||||||
| InternVL2-2B | 27.0 | 22.4 | 24.9 | 21.1 | 34.1 | 32.9 | 43.5 | 30.0 | 7.1 |
| InternVL2-8B | 34.1 | 22.6 | 28.3 | 47.6 | 39.6 | 35.7 | 30.4 | 30.0 | 38.6 |
| InternVL2-40B | 35.5 | 34.4 | 26.9 | 45.6 | 31.3 | 41.4 | 31.7 | 32.9 | 40.0 |
| LongVILA-8B | 21.0 | 28.7 | 8.6 | 16.3 | 0.0 | 28.6 | 30.5 | 31.4 | 24.3 |
| VILA-1.5-8B | 28.4 | 17.3 | 21.6 | 49.9 | 18.6 | 31.4 | 34.4 | 30.0 | 24.3 |
| VILA-1.5-40B | 30.8 | 21.4 | 24.4 | 48.3 | 21.9 | 40.0 | 25.0 | 30.0 | 35.7 |
| LongVA-7B | 29.0 | 38.1 | 16.9 | 38.1 | 21.7 | 32.9 | 42.8 | 25.7 | 15.7 |
| LLaVA-Video-7B | 34.9 | 47.9 | 13.4 | 46.7 | 23.9 | 42.9 | 41.9 | 32.9 | 30.0 |
| LLaVA-Video-72B | 40.5 | 48.3 | 22.6 | 56.7 | 34.6 | 41.4 | 36.5 | 35.7 | 48.6 |
| LLaVA-OneVision-0.5B | 27.6 | 45.1 | 27.9 | 14.7 | 27.9 | 28.6 | 37.0 | 34.3 | 5.7 |
| LLaVA-OneVision-7B | 32.1 | 46.9 | 19.9 | 46.9 | 12.1 | 41.4 | 35.1 | 30.0 | 24.3 |
| LLaVA-OneVision-72B | 39.6 | 42.7 | 23.7 | 56.7 | 36.9 | 41.4 | 39.5 | 31.4 | 44.3 |
|
Obj. Count |
Abs. Dist. |
Obj. Size |
Room Size |
Rel. Dist. |
Rel. Dir. |
Route Plan |
Appr. Order | ||
| Methods | Avg. | Numerical Answer | Multiple-Choice Answer | ||||||
| Proprietary Models (API) | |||||||||
| GPT-4o | 35.6 | 36.2 | 4.6 | 47.2 | 40.4 | 40.0 | 46.2 | 32.0 | 38.0 |
| Gemini-1.5 Flash | 45.7 | 50.8 | 33.6 | 56.5 | 45.2 | 48.0 | 39.8 | 32.7 | 59.2 |
| Gemini-1.5 Pro | 48.8 | 49.6 | 28.8 | 58.6 | 49.4 | 46.0 | 48.1 | 42.0 | 68.0 |
| Gemini-2.0 Flash | 45.4 | 52.4 | 30.6 | 66.7 | 31.8 | 56.0 | 46.3 | 24.5 | 55.1 |
| Open-source Models | |||||||||
| InternVL2-2B | 25.5 | 30.6 | 20.4 | 26.0 | 29.6 | 28.0 | 39.2 | 28.0 | 2.0 |
| InternVL2-8B | 32.9 | 26.4 | 25.4 | 43.8 | 41.6 | 30.0 | 32.2 | 20.0 | 44.0 |
| InternVL2-40B | 37.6 | 40.8 | 23.8 | 48.0 | 26.0 | 46.0 | 30.1 | 42.0 | 44.0 |
| LongVILA-8B | 19.1 | 23.4 | 10.8 | 11.4 | 0.0 | 20.0 | 33.1 | 28.0 | 26.0 |
| VILA-1.5-8B | 31.4 | 12.2 | 23.4 | 51.4 | 18.6 | 36.0 | 41.5 | 42.0 | 26.0 |
| VILA-1.5-40B | 32.3 | 14.6 | 21.0 | 48.0 | 20.6 | 42.0 | 22.0 | 40.0 | 50.0 |
| LongVA-7B | 31.8 | 41.2 | 17.4 | 39.6 | 25.4 | 30.0 | 52.8 | 34.0 | 14.0 |
| LLaVA-Video-7B | 35.7 | 49.0 | 12.8 | 48.6 | 21.4 | 40.0 | 43.5 | 34.0 | 36.0 |
| LLaVA-Video-72B | 39.3 | 41.4 | 26.6 | 55.6 | 31.6 | 36.0 | 25.6 | 42.0 | 56.0 |
| LLaVA-OneVision-0.5B | 27.7 | 44.0 | 23.0 | 18.8 | 28.4 | 30.0 | 33.4 | 36.0 | 8.0 |
| LLaVA-OneVision-7B | 33.8 | 48.2 | 22.0 | 44.4 | 14.0 | 44.0 | 31.9 | 34.0 | 32.0 |
| LLaVA-OneVision-72B | 41.6 | 38.0 | 31.6 | 54.4 | 35.2 | 44.0 | 39.7 | 32.0 | 58.0 |
|
Obj. Count |
Abs. Dist. |
Obj. Size |
Room Size |
Rel. Dist. |
Rel. Dir. |
Route Plan |
Appr. Order | ||
| Methods | Avg. | Numerical Answer | Multiple-Choice Answer | ||||||
| Proprietary Models (API) | |||||||||
| GPT-4o | 14.5 | 0.1 | 5.2 | 36.7 | 0.0 | 10.8 | 23.2 | 26.9 | 13.1 |
| Gemini-1.5 Flash | 19.9 | 25.0 | 30.3 | 52.5 | 0.0 | 0.0 | 21.2 | 29.9 | 0.2 |
| Gemini-1.5 Pro | 32.3 | 30.6 | 11.5 | 51.5 | 33.1 | 33.8 | 44.6 | 33.5 | 20.2 |
| Open-source Models | |||||||||
| InternVL2-2B | 17.8 | 5.4 | 23.7 | 9.2 | 0.0 | 26.9 | 41.2 | 27.9 | 7.9 |
| InternVL2-8B | 27.6 | 31.9 | 26.8 | 38.3 | 0.7 | 27.1 | 39.2 | 33.0 | 23.6 |
| InternVL2-40B | 24.4 | 5.4 | 29.1 | 39.2 | 0.7 | 30.3 | 37.7 | 27.9 | 24.7 |
| LongVILA-8B | 20.2 | 47.4 | 12.6 | 8.7 | 0.6 | 24.3 | 27.0 | 27.4 | 13.9 |
| VILA-1.5-8B | 21.5 | 7.4 | 7.6 | 45.7 | 0.0 | 25.4 | 39.1 | 29.4 | 17.6 |
| VILA-1.5-40B | 25.5 | 5.3 | 27.6 | 46.5 | 0.7 | 30.2 | 37.1 | 31.5 | 25.0 |
| LongVA-7B | 21.9 | 5.1 | 18.1 | 27.4 | 26.1 | 23.4 | 39.8 | 26.9 | 8.7 |
| LLaVA-Video-7B | 25.2 | 14.8 | 14.6 | 32.5 | 26.1 | 26.8 | 45.0 | 33.0 | 8.5 |
| LLaVA-Video-72B | 29.1 | 19.0 | 25.4 | 46.3 | 26.1 | 29.0 | 38.8 | 33.0 | 15.5 |
| LLaVA-OneVision-0.5B | 28.6 | 38.4 | 30.1 | 32.0 | 24.3 | 22.0 | 41.8 | 34.5 | 5.4 |
| LLaVA-OneVision-7B | 25.3 | 13.8 | 8.5 | 45.5 | 26.1 | 28.6 | 41.2 | 27.9 | 11.1 |
| LLaVA-OneVision-72B | 28.9 | 8.2 | 23.8 | 54.1 | 26.1 | 30.4 | 38.1 | 33.0 | 17.1 |
|
Obj. Count |
Abs. Dist. |
Obj. Size |
Room Size |
Rel. Dist. |
Rel. Dir. |
Route Plan |
Appr. Order | ||
| Methods | Avg. | Numerical Answer | Multiple-Choice Answer | ||||||
| Proprietary Models (API) | |||||||||
| GPT-4o | 19.5 | 46.1 | 0.1 | 7.1 | 38.2 | 26.2 | 18.0 | 4.6 | 15.4 |
| Gemini-1.5 Flash | 22.2 | 24.9 | 0.5 | 1.0 | 54.4 | 37.7 | 19.9 | 1.5 | 37.7 |
| Gemini-1.5 Pro | 13.0 | 25.5 | 19.5 | 12.6 | 10.6 | 17.5 | 1.7 | 2.5 | 14.4 |
| Open-source Models | |||||||||
| InternVL2-2B | 8.7 | 20.3 | 0.3 | 10.8 | 29.2 | 5.2 | 2.9 | 2.5 | -1.6 |
| InternVL2-8B | 9.9 | -0.6 | 2.2 | 10.6 | 43.5 | 10.9 | -5.8 | -4.1 | 22.8 |
| InternVL2-40B | 12.6 | 35.9 | -2.9 | 9.0 | 26.8 | 17.3 | -5.0 | 9.9 | 20.0 |
| LongVILA-8B | 1.4 | -18.2 | -3.5 | 7.9 | -0.6 | 5.3 | 3.7 | 5.1 | 11.5 |
| VILA-1.5-8B | 7.3 | 10.0 | 14.2 | 4.6 | 18.8 | 6.7 | -4.4 | 1.5 | 7.2 |
| VILA-1.5-40B | 5.7 | 17.1 | -2.8 | 2.2 | 22.0 | 10.4 | -11.4 | 0.0 | 7.9 |
| LongVA-7B | 7.2 | 32.9 | -1.5 | 11.5 | -3.9 | 9.7 | 3.5 | -1.5 | 7.1 |
| LLaVA-Video-7B | 10.5 | 33.8 | -0.6 | 15.2 | -1.9 | 16.7 | -2.7 | 1.0 | 22.1 |
| LLaVA-Video-72B | 11.7 | 29.9 | -2.6 | 11.1 | 9.2 | 13.3 | -2.0 | 2.0 | 33.0 |
| LLaVA-OneVision-0.5B | -0.5 | 7.8 | -1.7 | -16.6 | 4.0 | 6.9 | -5.0 | 0.0 | 0.3 |
| LLaVA-OneVision-7B | 7.0 | 33.9 | 11.7 | 1.9 | -13.9 | 13.9 | -6.0 | 1.5 | 13.3 |
| LLaVA-OneVision-72B | 11.4 | 35.4 | 0.1 | 3.5 | 11.4 | 12.1 | 1.8 | -0.5 | 27.4 |