Literature
Display
← All papers

What Makes a Maze Look Like a Maze?

Joy Hsu · Jiayuan Mao · Joshua B. Tenenbaum · Noah D. Goodman · Jiajun Wu

· ICLR 2025 ↗

Contents

Overview

The paper studies how explicit concept schemas can help interpret unconventional visual representations of familiar abstractions.

At a glance

Deep Schema Grounding uses a language model to decompose a named abstract concept into dependent components, then queries a vision-language model to ground those components in an image before answering questions. The Visual Abstractions Benchmark contains 540 questions over 180 images and twelve concepts. With GPT-4o, the method improves multiple-choice exact-match accuracy from 66.4% to 73.0%. It is an inference-time prompting framework, not a newly trained vision model, and it remains limited by ambiguous schemas and visual grounding errors.

1 Introduction

A maze can be made from drawn lines, food, or household objects. Recognizing the material alone does not identify its role as walls, layout, or entry and exit. The proposed framework supplies a structured decomposition of an already named concept and uses its dependencies to guide visual interpretation. It aims to improve reasoning across different instantiations; discovering the intended concept from an unlabeled image is outside the evaluated task.

Refer to caption
Figure 1: There exist abstract concepts, such as “maze”, which are defined by lifted symbols and patterns, instead of concrete visual features. We propose Deep Schema Grounding (DSG), a framework for visual reasoning over such abstract concepts, which uses schemas to structure models’ interpretation of images. DSG hierarchically grounds conceptual schemas on images and uses them to provide holistic context to VLMs, improving performance across diverse downstream queries.

2 Related Works

The work connects visual question answering, cognitive accounts of concepts, and hierarchical visual reasoning.

Vision-language grounding and reasoning benchmarks.

Existing benchmarks examine objects, relations, commonsense questions, and scene descriptions. This benchmark focuses on visual instantiations of abstract concepts, where low-level entities must be interpreted according to a higher-level pattern.

Concept representations in minds and machines.

The schema view draws on cognitive theories treating concepts as structured explanatory knowledge, including Schank and Abelson’s scripts and plans. Here, that inspiration becomes a dependency graph used to guide model queries; the experiments do not establish a mechanistic match to human cognition.

Hierarchical grounding of visual concepts.

Prior methods use intermediate reasoning, class descriptions, hierarchies, and executable programs. This method instead grounds a language-generated concept graph through repeated visual-model queries. Its inference remains model-based natural-language interpretation, rather than a symbolic engine that guarantees logically correct results.

3 Deep Schema Grounding

The input is an image and a question that explicitly reveals its intended abstract concept to every method. The pipeline extracts a schema, grounds its components in dependency order, and supplies the resolved schema to the final answering model. No task-specific parameter training is performed.

Refer to caption
Figure 2: Deep Schema Grounding consists of three main stages: (1) extracting schemas of abstract concepts with large language models, (2) hierarchically grounding schemas on images with vision-language models, and (3) conducting visual question-answering augmented with resolved schemas.

3.1 Visual Abstraction Schema

A schema is a concise program-like description of a directed acyclic graph. Nodes denote components such as maze layout, walls, and entry/exit; edges determine which earlier interpretations condition a later one. The schema is intended to generalize across materials and scenes, but its claimed universality is an intended property, not formally verified.

Refer to caption
Figure 3: Examples of schemas for concepts and the visual features that they may be grounded to.

3.2 Extracting Schemas of Abstract Concepts

A language model receives a simple schema-generation instruction and one example based on academia. It proposes a decomposition from the concept’s name without seeing the individual target image. The appendix preserves the exact prompt and all twelve resulting schemas, including the instruction to use at most four necessary visually mappable components.

3.3 Hierarchically Grounding Schemas on Images

The vision-language model first describes components without prerequisites, then grounds dependent components using those earlier descriptions. For a maze, the identified layout can inform the interpretation of entries and exits. Outputs are short language descriptions rather than verified object masks or geometric constraints. Incorrect early descriptions can therefore influence later grounding.

Original figure
Figure 4: The schema grounding process.

3.4 Visual Question-Answering Augmented with Grounded Schemas

The final model receives the original image and question, component mappings, and the grounding conversation history. This gives it richer image-derived context than the concept name alone. The approach trades extra model queries and context for more explicit reasoning; the paper does not supply a measured latency or monetary-cost comparison.

4 Visual Abstractions Benchmark

The Visual Abstractions Benchmark contains fifteen images per concept, twelve concepts, and three questions per image: 180 images and 540 question–image pairs. It is a small, manually curated test set, with additional free-response evaluation.

Refer to caption
Figure 5: The Visual Abstractions Benchmark comprises diverse, real-world images that represent 1212 different abstract concepts across 44 categories.

Abstract concepts.

Four concept groups are represented: strategic games such as mazes and tic-tac-toe; physical constructions illustrating scientific ideas such as atoms and cells; social situations such as helping or deceiving; and household goals such as setting a table for two. These groups require different forms of relational interpretation and are not interchangeable difficulty levels.

Images and question-answers.

Images come from the internet and are chosen for varied real-world instantiations. Questions cover binary choices, counts, and more general content. In the main evaluation, all question categories—including those called open-ended—use multiple-choice answers scored against the modal response of five annotators. The separate free-response variant uses different metrics.

5 Results

Baselines include visual models and language models that call visual tools. Every method receives the intended concept. Reported main results average five runs on the same benchmark; these repetitions measure run variability, not performance across five independent datasets.

Table 1: DSG outperforms prior works across all questions and different question types in the Visual Abstractions Benchmark. Results are averaged over 55 runs.
All Counting Binary Open
ViperGPT (Surís et al., 2023) 0.2600.260 0.1910.191 0.4260.426 0.1410.141
VisProg (Gupta & Kembhavi, 2023) 0.3040.304 0.2490.249 0.3850.385 0.2550.255
LLaVA (Liu et al., 2024) 0.4150.415 0.3440.344 0.5520.552 0.3210.321
InstructBLIP (Dai et al., 2024) 0.4370.437 0.3220.322 0.5380.538 0.3920.392
GPT-4o (OpenAI, 2024) 0.6640.664 0.6040.604 0.6550.655 0.6930.693
DSG (ours) 0.730\mathbf{0.730} 0.704\mathbf{0.704} 0.690\mathbf{0.690} 0.776\mathbf{0.776}
Human 0.8460.846 0.8160.816 0.8330.833 0.8680.868
Table 2: DSG improves the performance of GPT-4o across categories of abstract concepts.
Strategic Scientific Social Domestic
GPT-4o (OpenAI, 2024) 0.6530.653 ±0.052\pm 0.052 0.6460.646 ±0.074\pm 0.074 0.6470.647 ±0.163\pm 0.163 0.7080.708 ±0.082\pm 0.082
DSG (ours) 0.751\mathbf{0.751} ±0.088\pm 0.088 0.693\mathbf{0.693} ±0.126\pm 0.126 0.723\mathbf{0.723} ±0.027\pm 0.027 0.754\mathbf{0.754} ±0.104\pm 0.104

VLMs benefit from decomposing abstract concepts into primitive-level components.

GPT-4o accuracy rises from 66.4% to 73.0%, a 6.6-percentage-point gain. Counting rises from 60.4% to 70.4%, binary questions from 65.5% to 69.0%, and the open-content multiple-choice category from 69.3% to 77.6%. Gains occur in all four concept groups, with the largest in strategic concepts. A randomly chosen human annotator agrees with the modal label 84.6% of the time, showing both remaining model headroom and imperfect annotator agreement; this is not an independently adjudicated upper bound. The older GPT-4v also improves from 65.3% to 70.7% under the framework.

Table 3: Summarization of DSG ablations; the full DSG framework shows strongest performance.
schema grounding hierarchy context All Counting Binary Open
+ schema ✓ ✗ ✗ ✗ 0.6810.681 0.6290.629 0.6720.672 0.7090.709
+ grounding ✓ ✓ ✗ ✗ 0.6930.693 0.7270.727 0.6600.660 0.7100.710
+ hierarchy ✓ ✓ ✓ ✗ 0.7010.701 0.7130.713 0.6710.671 0.7230.723
DSG (ours) ✓ ✓ ✓ ✓ 0.730\mathbf{0.730} 0.704\mathbf{0.704} 0.690\mathbf{0.690} 0.776\mathbf{0.776}

Explicit hierarchical grounding of schemas is better than implicit and sequential grounding.

Adding a schema in one prompt reaches 68.1%; sequential explicit grounding reaches 69.3%; hierarchical grounding without full conversation history reaches 70.1%; the full pipeline reaches 73.0%. These ablations support the combined design on average, but individual categories do not improve monotonically: sequential grounding has higher counting accuracy than the full method. The comparisons do not isolate structured reasoning from the additional inference work in a cost-matched experiment.

DSG improves other (open-source) VLMs, too.

With LLaVA as the visual model, accuracy increases from 41.5% to 44.4%, or about 7% relative. Counting remains unchanged at 34.4%. The method can help more than one model, but cannot compensate fully for a base model’s weak counting or perception.

Table 4: DSG improves the performance of the open-source LLaVA model.
All Counting Binary Open
LLaVA (Liu et al., 2024) 0.4150.415 0.344\mathbf{0.344} 0.5520.552 0.3210.321
DSG (ours) 0.444\mathbf{0.444} 0.344\mathbf{0.344} 0.590\mathbf{0.590} 0.354\mathbf{0.354}
Table 5: Comparison of LLaVA and DSG across categories; DSG outperforms the base VLM.
Strategic Scientific Social Domestic
LLaVA (Liu et al., 2024) 0.4070.407 ±0.038\pm 0.038 0.3330.333 ±0.101\pm 0.101 0.3780.378 ±0.126\pm 0.126 0.5410.541 ±0.091\pm 0.091
DSG (ours) 0.430\mathbf{0.430} ±0.046\pm 0.046 0.378\mathbf{0.378} ±0.119\pm 0.119 0.393\mathbf{0.393} ±0.154\pm 0.154 0.578\mathbf{0.578} ±0.096\pm 0.096

6 Discussion

Generated schemas can be biased, ambiguous, or unsuitable for particular images. Precise spatial components are especially difficult: naming an unobstructed maze path does not equip the visual model to verify that path. The flexible language interface avoids a fixed vocabulary but provides no formal grounding guarantee. Better layouts, component grounding, and more efficient inference remain open problems; the method is not a general solution to abstraction or spatial reasoning.

7 Conclusion

Explicit concept decomposition and dependency-aware visual grounding improve the tested question-answering systems. The main contribution is a reusable inference procedure and benchmark, with evidence of benefits across several concepts and models while substantial errors remain.

Acknowledgments

The authors acknowledge feedback and support from university, government, and industry research programs, plus individual fellowship support.

Supplementary for: What Makes a Maze Look Like a Maze?

The supplement adds evaluation metrics and ablations, qualitative grounding examples, benchmark construction details, the full prompts and schemas, and a human assessment of schema quality.

Appendix A Additional Results

Additional results examine answer ambiguity, free responses, query structure, image access, and schema complexity.

A.1 Graded Accuracy Comparison

The graded metric awards the fraction of five annotators who agree with a prediction, rather than requiring agreement with the modal answer. GPT-4o scores 0.622 and the proposed method 0.684. This preserves the main trend while acknowledging label disagreement; it measures alignment with this annotator set, not objective correctness independent of interpretation.

Table 6: Comparison of DSG to prior works with a graded accuracy metric based on degree of alignment to human judgements.
All Counting Binary Open
ViperGPT (Surís et al., 2023) 0.2540.254 0.2000.200 0.4130.413 0.1360.136
VisProg (Gupta & Kembhavi, 2023) 0.2930.293 0.2270.227 0.3800.380 0.2420.242
LLaVA (Liu et al., 2024) 0.4000.400 0.3270.327 0.5480.548 0.2980.298
InstructBLIP (Dai et al., 2024) 0.4200.420 0.3070.307 0.5280.528 0.3680.368
GPT-4o (OpenAI, 2024) 0.6220.622 0.5190.519 0.6550.655 0.6320.632
DSG (ours) 0.684\mathbf{0.684} 0.627\mathbf{0.627} 0.681\mathbf{0.681} 0.708\mathbf{0.708}
Human 0.7890.789 0.7280.728 0.7840.784 0.8160.816

A.2 Comparison on Free Response Generation

Free-response BERTScore increases from 0.900 to 0.908, a small change in text similarity rather than an accuracy percentage. Using the GPT-4 answer-evaluation protocol, overall scores rise from 36.67 to 37.78. Counting and open-response scores improve, while binary scores fall from 55.24 to 49.52. Model-judged answers and similarity metrics have different limitations from exact-match evaluation.

Table 7: Performance on free response generation; DSG outperforms the base VLM.
All Counting Binary Open
GPT-4o (OpenAI, 2024) 0.9000.900 0.9170.917 0.9420.942 0.8580.858
DSG (ours) 0.908\mathbf{0.908} 0.949\mathbf{0.949} 0.946\mathbf{0.946} 0.859\mathbf{0.859}
Table 8: Performance on free response generation with  Kamalloo et al. (2023)’s GPT-4-based metric for evaluating open-domain QA models.
All Counting Binary Open
GPT-4o 36.6736.67 45.5645.56 55.24\mathbf{55.24} 14.5814.58
DSG (ours) 37.78\mathbf{37.78} 53.33\mathbf{53.33} 49.5249.52 21.25\mathbf{21.25}

A.3 Results on Single-round vs. Multi-round Reasoning

One-query prompts containing schema components or the full hierarchy obtain 66.7% and 68.1%, compared with 73.0% for explicit multi-round grounding. The result supports separate grounding steps for the tested models, while leaving their additional query cost and whether other single-query prompts could close the gap unresolved.

Table 9: Comparison of DSG with single- and multi-round grounding of the schema, with results averaged over 55 runs.
All Counting Binary Open
GPT-4o 0.6640.664 0.6040.604 0.6550.655 0.6930.693
Single-round with schema components 0.6670.667 0.6070.607 0.6610.661 0.6960.696
Single-round with hierarchical schema 0.6810.681 0.6290.629 0.6720.672 0.7090.709
Multi-round with hierarchical schema (DSG) 0.730\mathbf{0.730} 0.704\mathbf{0.704} 0.690\mathbf{0.690} 0.776\mathbf{0.776}

A.4 Language-only Baselines

Removing image access entirely gives 39.7% accuracy. Supplying a previously image-grounded schema to the final language-only answer step raises this to 52.2%. The latter is not a purely text-only pipeline: its schema already contains visual information. Full visual access and grounding together remain better.

Table 10: Comparison with language-only baseline and baseline with DSG’s grounded schema, with results averaged over 55 runs.
All Counting Binary Open
Language-only 0.3970.397 0.3000.300 0.5100.510 0.3350.335
Language w/ DSG schema 0.522\mathbf{0.522} 0.440\mathbf{0.440} 0.530\mathbf{0.530} 0.546\mathbf{0.546}

A.5 Schema Complexity Ablation

Examples with three, five, seven, or nine components produce overall accuracies of 73.0%, 69.1%, 68.5%, and 67.6%. The more detailed examples yield average generated schema lengths of 3.00, 4.66, 5.75, and 7.00 components. Longer schemas can introduce harder or less general components; the evidence does not show that future stronger models will necessarily benefit from adding detail. All four original example programs are retained with their source indentation.

With 3 components:
gen(concept=academia) =
    gen(faculty | concept=academia)
    gen(students | concept=academia)
    gen(research-output | concept=academia, faculty, students)
With 5 components:
gen(concept=academia) =
    gen(faculty | concept=academia)
    gen(students | concept=academia)
    gen(courses | concept=academia, faculty, students)
    gen(research-output | concept=academia, faculty, students, courses)
    gen(funding | concept=academia, research-output)
With 7 components:
gen(concept=academia) =
    gen(faculty | concept=academia)
    gen(students | concept=academia)
    gen(departments | concept=academia, faculty, students)
    gen(courses | concept=academia, departments, faculty, students)
    gen(research-output | concept=academia, departments, faculty, students)
    gen(funding | concept=academia, research-output, departments)
    gen(infrastructure | concept=academia, funding, departments, courses)
With 9 components:
gen(concept=academia) =
    gen(faculty | concept=academia)
    gen(students | concept=academia)
    gen(departments | concept=academia, faculty, students)
    gen(courses | concept=academia, departments, faculty, students)
    gen(research-output | concept=academia, departments, faculty, students)
    gen(funding | concept=academia, research-output, departments)
    gen(infrastructure | concept=academia, funding, departments, courses)
    gen(administration | concept=academia, infrastructure, departments, faculty, students)
    gen(community-engagement | concept=academia, research-output, administration, students, faculty)
Table 11: Ablation with varying number of components in the example prompt.
All Counting Binary Open
DSG w/ 3 comp. 0.730\mathbf{0.730} 0.689\mathbf{0.689} 0.700\mathbf{0.700} 0.771\mathbf{0.771}
DSG w/ 5 comp. 0.6910.691 0.6220.622 0.6710.671 0.7330.733
DSG w/ 7 comp. 0.6850.685 0.6670.667 0.6620.662 0.7130.713
DSG w/ 9 comp. 0.6760.676 0.7000.700 0.6570.657 0.6830.683
Table 12: Number of components in schema and the resulting average number of components in the generated schemas.
Generated comp. #
3 comp. 3.003.00
5 comp. 4.664.66
7 comp. 5.755.75
9 comp. 7.007.00

A.6 Qualitative Examples

The retained examples show grounded schemas across concept groups, with incorrect component interpretations highlighted in red. They illustrate how errors can persist despite a sensible high-level decomposition and should be read alongside the quantitative averages.

Refer to caption
Figure 6: Examples of DSG’s resolved schemas across different abstract concepts; in red we highlight failure cases where inference of individual components errors.

A.7 Hierarchical Grounding Example

An atom example contrasts hierarchical grounding with a sequential variant. Conditioning energy-level interpretation on previously grounded nucleus and electron descriptions produces a better interpretation in that example. This is qualitative support for dependency ordering, not a separate broad accuracy estimate.

Refer to caption
Figure 7: In comparison to DSG, which hierarchically grounds each component, a sequential variant errors when grounding a more fine-grained symbol, energy-levels, without conditioning on prior predictions of more concrete symbols, nucleus and electrons.

A.8 Computing Resources

Closed-model experiments use the OpenAI interface. Open models and visual-tool components run inference on one A40 accelerator. No model training is required by the framework, but multiple calls per example still contribute inference cost.

Appendix B Visual Abstractions Benchmark

The benchmark appendix documents images, questions, selection criteria, annotations, and release organization.

B.1 Benchmark Examples

The complete retained montage provides question–image pairs across all concept and question categories, complementing the main overview of the twelve concepts.

B.2 Benchmark Construction

Researchers manually search for diverse physical instantiations of each concept and select faithful, high-quality images. They write broadly applicable questions and send the resulting pairs for annotation. This selection process is not procedurally reproducible and can shape which kinds of abstraction the benchmark tests.

B.3 Annotation

Five Prolific annotators label each question–image pair through the illustrated interface, with compensation stated as twelve US dollars per hour. Their modal response defines the primary evaluation label; disagreement motivates the graded supplementary metric.

Refer to caption
Figure 8: Example of our annotation interface shown to Prolific annotators.

B.4 Release

The released benchmark groups images by concept. The paper reports manual auditing of the release. This is the benchmark evaluated here, rather than an automatically sampled distribution of all possible visual abstractions.

Refer to caption
Figure 9: Examples of visual question-answering pairs in the Visual Abstractions Benchmark.

Appendix C LLM-Generated Schemas

The final appendix makes the schema-generation prompt, all twelve extracted schemas, and the schema-quality study available for inspection.

C.1 Prompt

The exact original prompt asks for a simple program-like concept schema, supplies an academia example, requests the same format without explanation, and limits the output to four or fewer necessary components that can map to image objects. Its examples and formatting are preserved below.

Can you give me a program representing the schema for a concept? For example,
gen(concept=academia) =
    gen(faculty | concept=academia)
    gen(students | concept=academia)
    gen(research-output | concept=academia, faculty, students)
Please do the same for gen(concept=[abstract-concept]) in the same format without explanation. Keep the program simple with four or less components. Use only the most necessary parts of the schema that can be mapped to objects in an image.

C.2 All Schemas

All original schemas are retained by strategic, scientific, social, and domestic category. Their component names and conditional dependencies define the structure actually supplied to the grounding procedure, making omissions and potentially ambiguous concepts visible.

gen(concept=tic-tac-toe) =
    gen(board | concept=tic-tac-toe)
    gen(symbols | concept=tic-tac-toe)
    gen(strategy | concept=tic-tac-toe, symbols)

gen(concept=maze) =
    gen(layout | concept=maze)
    gen(walls | concept=maze)
    gen(entry-exit | concept=maze, layout)

gen(concept=treasure-map) =
    gen(map | concept=treasure-map)
    gen(x-marks-the-spot | concept=treasure-map)
    gen(path | concept=treasure-map, map)
gen(concept=solar-system) =
    gen(sun | concept=solar-system)
    gen(planets | concept=solar-system)
    gen(orbits | concept=solar-system, sun, planets)

gen(concept=atom) =
    gen(nucleus | concept=atom)
    gen(electrons | concept=atom)
    gen(energy-levels | concept=atom, nucleus, electrons)

gen(concept=cell) =
    gen(membrane | concept=cell)
    gen(nucleus | concept=cell)
    gen(organelles | concept=cell)
gen(concept=helping) =
    gen(helper | concept=helping)
    gen(recipient | concept=helping)
    gen(task | concept=helping, helper, recipient)

gen(concept=deceiving) =
    gen(deceiver | concept=deceiving)
    gen(victim | concept=deceiving)
    gen(deceptive-object | concept=deceiving, deceiver, victim)

gen(concept=negotiating) =
    gen(participants | concept=negotiating)
    gen(setting | concept=negotiating)
    gen(objects | concept=negotiating, participants)
gen(concept=setting-up-table-for-two) =
    gen(table | concept=setting-up-table-for-two)
    gen(two-chairs | concept=setting-up-table-for-two)
    gen(table-setting | concept=setting-up-table-for-two, table)

gen(concept=tidying-up-guest-room) =
    gen(bed | concept=tidying-up-guest-room)
    gen(storage | concept=tidying-up-guest-room)
    gen(cleanliness | concept=tidying-up-guest-room, bed, storage)

gen(concept=putting-up-decorations-on-door) =
    gen(decoration-type | concept=putting-up-decorations-on-door)
    gen(door-type | concept=putting-up-decorations-on-door)
    gen(tools | concept=putting-up-decorations-on-door, decoration-type)

C.3 Human Study of Schema Quality

Twenty participants judge twelve schemas on representativeness, components to keep, and components to add, producing 720 answers. Mean representativeness is 5.74 out of seven; participants retain 2.59 of the three components on average and suggest 1.59 additions. Negotiating receives the lowest mean representativeness and component-retention scores. These judgments support reasonable initial schemas while showing incompleteness; they do not verify every grounding or prove universal concept definitions.

Here you are given a concept, maze, and the core components underlying it: layout, walls, entry-exit. How well do these core components represent this concept on a scale of 1-7?
Here you are given a concept, maze, and the core components underlying it: layout, walls, entry-exit. How many components out of the ones listed accurately represent part of this concept?
Here you are given a concept, maze, and the core components underlying it: layout, walls, entry-exit. How many more components, if any, would you add to accurately represent this concept?
Table 13: Prolific study on quality of generated schemas, with results averaged over 2020 participants.
Representativeness (1-7) # comp. to keep # comp. to add
Participant responses 5.745.74 2.592.59 1.591.59
Concept demonstration

Open image ↗