Literature
Display
← All papers

VISTA: A Visual Harness for Reasoning in an Interactive World

Qiushi Han · Keya Hu · Linlu Qiu · Cathy Wu · Kaiming He

Preprint posted · arXiv (preprint) ↗

AI agents10 min text · CondensedOriginal paper ↗
Contents

At a glance

VISTA lets a multimodal agent revisit its visual evidence: it archives every returned frame, retrieves past observations, enlarges regions, and reads exact pixels. The harness adds no trained model; reasoning remains in the underlying model.

Claude Opus 5.0 completes all 25 public ARC-AGI-3 games with 100.00 RHAE, using 7,302 actions versus a 17,135-action human reference. GPT-5.6 Sol scores 99.00. These are public-game action-efficiency results; different reasoning settings and budgets limit baseline comparisons, and training-data overlap cannot be ruled out.

1 Introduction

Interactive games require discovering hidden rules and goals [5]. One-time visual encoding can lose detail, while context truncation discards earlier evidence [29] [27].

VISTA keeps original observations outside the bounded context. The agent reasons in language, predicts outcomes, and retrieves frames or regions as needed [55] [59]. Tests cover public ARC-AGI-3 games and selected tasks from three further benchmarks.

Original figure
Figure 1: Comparison of different agent designs. (a) Language agents reason over textual observations. (b) Multimodal agents encode images into visual representations only once, but these representations may be compressed, lossy, and insufficient for subsequent reasoning, potentially limiting the models’ reasoning capabilities. (c) VISTA allows the multimodal model to iteratively call visual tools to retrieve past visual observations and reorganize its visual input as it reasons. For simplicity, we show the VLM as a vision transformer (ViT) encoder followed by an LLM. The two VLM blocks represent the same underlying model.
Refer to caption
Figure 2: VISTA trajectories on ARC-AGI-3 (left) and GameWorld (right). The harness allows the agent to directly observe the visual states, reason in natural language, and inspect its lossless visual memory at the original level of detail when needed.

2 Related Work

ARC-AGI benchmarks.

ARC-AGI-1 and ARC-AGI-2 test static grid reasoning; ARC-AGI-3 adds exploration and action [16] [14] [5]. Many strong interactive systems construct executable symbolic models [62] [25] [11]. VISTA instead uses images and natural-language reasoning, avoiding an explicit programmatic model of the environment.

Computer-use agents.

Computer-use agents either consume structured interfaces such as DOM trees or act directly from screenshots [17] [64] [13] [57]. VISTA follows the visual approach and concentrates on retaining and revisiting evidence during interaction.

Game-playing agents.

Earlier game agents range from separately trained policies to generalist foundation models [31] [21] [51] [53] [46] [49] [54]. VISTA instead emphasizes a complete visual archive accessible independently of current context.

3 VISTA: A Visual Harness

Agents cannot know initially which visual details will matter later. Preserving original frames lets them recover evidence once a relevant question becomes clear.

Refer to caption
Figure 3: VISTA’s key design components and per-turn pipeline. We use ARC-AGI-3 as an example. (a) Visual observations: The model receives rendered visual states. (b) Lossless visual memory: Every returned frame is stored in its original form. (c) Model-directed visual inspection: The model selects frames and regions to examine during reasoning. (d) Per-turn pipeline: The agent observes, reasons, and acts, while each returned frame is preserved in visual memory. Revisiting past observations and updating notes are optional steps directed by the model.

3.1 Visual Harness Components

Three components give the model explicit control over which parts of its visual history to examine.

Visual observation.

Rendered observations retain object appearance and spatial relationships for the model’s visual reasoning.

Lossless visual memory.

Every returned frame, including intermediate animation frames, is stored in its original form and indexed by turn and frame. This archive survives context truncation or compaction.

Visual inspection.

The agent can retrieve arbitrary earlier frames, compare states across time, or crop and enlarge rectangular regions. Inspection is optional and directed by the model’s current question.

3.2 Pipeline

A single agent observes, reasons, optionally inspects frames or pixels, then calls play. The mechanical harness executes tools and reports invalid requests; each turn can include multiple inspections.

GUIDE.md stores reusable knowledge; WORKING.md holds current plans. The agent predicts action outcomes and updates its understanding afterward. Near the context limit it writes a handoff summary and resumes with notes, history, and the visual archive intact.

4 VISTA on ARC-AGI-3

4.1 Setup

Dataset.

Evaluation uses the 25 public ARC-AGI-3 games [5]. Rules and goals are initially hidden. Success requires discovering mechanics through interaction and transferring that knowledge across increasingly difficult levels within each game.

Evaluation metrics.

RHAE compares actions with first-time human players, weights later levels more heavily, and caps games by weighted completion [4]. Unfinished levels score zero. It measures action efficiency, not inference cost or time. Appendix A.3 gives the equations.

Models.

The main backends are Claude Opus 5.0 at extra-high reasoning effort and GPT-5.6 Sol at maximum effort, accessed through their respective command-line interfaces [3] [33]. Each runs independently under the VISTA interaction protocol.

4.2 Results

Opus scores 100.00 RHAE with 7,302 actions, 57.4% below the 17,135-action human reference. GPT scores 99.00 with 9,126 actions. Both complete every public game. Table 1 compares VISTA with concurrent systems, many of which use program synthesis.

The official Opus baseline scores 40.68 at high effort, whereas VISTA uses extra-high. Official GPT scores 13.33. These are system-level comparisons with interface and stopping-budget differences; they do not isolate the visual harness alone. The controlled incremental configurations below provide more specific evidence.

Table 1: System-level comparison on 25 public games of ARC-AGI-3. VISTA is, to our knowledge, the first system to achieve perfect or near-perfect performance without program synthesis.
System Program Model Effort RHAE
Official impl. [5] No GPT-5.6 Sol [34] max 13.33
Opus 5.0† high 40.68
Schema [62] Yes GPT-5.6 Sol xhigh →\rightarrow max 95.35
Opus 4.8 →\rightarrow Fable 5 max 98.98
ewma_sv_v1.6 [39] Yes GPT-5.6 Sol xhigh 98.97
Retrodict [11] Yes GPT-5.6 Sol max 99.86
Tycho [25] Yes GPT-5.6 Sol max 100.00
Opus 5.0 xhigh 100.00
VISTA No GPT-5.6 Sol max 99.00
Opus 5.0 xhigh 100.00

†Calculated from the officially reported per-game scores for the 25 public games [6].

Refer to caption
Figure 4: Example trajectory in ARC-AGI-3. Each frame is labeled with its turn and action; ×N\times N denotes repeated actions. Boxes mark targets, and rings mark the agent’s clicks. The agent must infer the rules: black pieces move together with their reflections across the mirror line, and all yellow targets must be covered. Level 3 introduces a movable horizontal mirror line and selectable pieces.

4.3 Ablations

Unless varied, ablations use GPT-5.6 Sol at maximum effort, a 200K-token context limit, and 512 × 512 images obtained by 8× nearest-neighbor enlargement of the original 64 × 64 frames.

Original figure
Figure 5: Harness configurations on ARC-AGI-3. GPT-5.6 Sol under progressively richer harness configurations, evaluated using (a) RHAE, (b) the total number of actions taken across the 25 games, and (c) the percentage of the 25 games completed. For reference, the official implementation using textual grids (hatched gray bars) achieves a reported RHAE score of 13.33 with 10,619 actions [7, 34]. (i) The official implementation with PNG images replacing its textual grid observations. (ii) Increases the action and time limits to match our evaluation protocol (up to 2,000 actions per game) instead of the official stopping criteria. (iii) Adds a continuous conversation, using the model’s native compaction to continue past its context limit. (iv) Adds the two notes, GUIDE.md and WORKING.md, and encourages the agent to build a compact and revisable model of the game. (v) Adds lossless visual memory and inspection. (vi) Adds exact pixel readout, completing VISTA. Configurations (i)–(iv) automatically receive up to seven frames per action, following the official protocol; configurations (v)–(vi) introduce VISTA’s key visual components: the agent automatically receives only the final frame and can actively inspect its visual memory as it reasons. See Appendix A.2 for details.

Effects of VISTA components.

Incremental RHAE: image observations 47.32 → larger action/time budgets 51.66 → continuous conversation with compaction 65.82 → persistent notes 70.05 → visual memory and inspection 94.10 → exact pixel readout 99.00.

The official textual baseline’s 13.33 is an external reference using a different model-access interface. Configurations (i)–(vi) share the experimental interface. Early configurations automatically receive up to seven frames per action; the final two receive only the last frame and retrieve others as needed. Fewer actions in weak configurations can reflect early termination, rather than efficient completion.

Refer to caption
Figure 6: Comparison of observation representations on ARC-AGI-3. (a) An example game state shown as a flattened text grid and a 2D image; the displayed text grid is cropped from the full 64×6464\times 64 grid. (b) RHAE, total actions across the 25 games, and average token usage per game for the two observation representations.

Reasoning directly from images.

Replacing images with textual grids inside the otherwise unchanged harness preserves comparable RHAE but requires slightly more actions. A frame costs roughly 4,000 text tokens versus 308 image tokens at the default scale. Average token usage is 71.9M per game with text and 30.7M with images.

Original figure
Figure 7: Ablations on ARC-AGI-3. Left column: RHAE. Middle column: the total number of actions taken across the 25 games. Right column: the average number of tokens per game. (a): maximum context size, which sets when context compaction occurs. (b): input image scale relative to the official 64×6464\times 64 frame; scales of 2×2\times, 4×4\times, 8×8\times, and 16×16\times correspond to 128, 256, 512, and 1,024 pixels per side and roughly 20, 77, 308, and 1,229 image tokens per frame, respectively.

Effect of maximum context size.

The default 200K context achieves 99.00 RHAE. A 100K limit loses only 0.4 points while roughly halving token usage. Increasing to 780K costs 105.7M tokens per game versus 30.7M at 200K, without improving the best score.

Effect of input image scale.

A 4× enlargement gives comparable performance to the default 8× with 75% fewer image tokens per frame. At 2× performance falls; 16× brings no gain and increases cost. More image tokens are therefore not uniformly useful.

Original figure
Figure 8: Results on ARC-AGI-3 using the open-weight GLM-5.3 Flash 320B model. We compare VISTA against the official implementation using a minimal harness, as well as a VISTA variant with inspect and read_pixels removed. VISTA’s visual harness leads to a significant performance improvement.

Results with an open-weight model.

The open-weight GLM-5.3 Flash 320B backend reaches 66.93 RHAE with VISTA [20] [63]. Figure 8 also compares the official minimal harness and a VISTA variant without inspection or pixel readout. Gains extend beyond the two closed-weight backends, although their near-perfect performance is not reproduced.

Refer to caption
Figure 9: 3D renderings of ARC-AGI-3. (a) Two games rendered in 3D. (b) RHAE scores and total action counts for VISTA using 2D images and 3D renderings.

Effects of 3D rendering.

Rendering the same games in 3D yields 84.12 RHAE and 20,640 actions. This tests a changed visual representation of the game mechanics; the substantial action increase shows that richer visuals remain harder.

4.4 Qualitative Analysis

Figure 10 illustrates reconstructing a map from earlier frames after a reset, and enlarging small blocks to identify orientation markers: retrieval across time and local inspection.

Refer to caption
Figure 10: Model-directed use of visual memory. (a) Inspect across space and time: after the board resets, the model retrieves four frames from the preceding 22 turns in a single inspect call to reconstruct the map. (b) Inspect local visual details: the model uses inspect to zoom in on three small blocks by about 13×13\times. The enlarged views reveal which side of each block carries a purple marker, allowing the model to identify each block’s orientation.

5 General Visual Environments

The same visual harness is adapted to GameWorld and AI GameStore, plus static BabyVision tracking questions [35] [60] [12]. Comparisons use GPT-5.6 Sol with each benchmark’s official minimal harness; human references follow each benchmark’s own protocol.

Refer to caption
Original figure
Figure 11: Evaluation environments and results. Top: examples from the three benchmarks. GameWorld and AI GameStore are interactive, while BabyVision is static. Bottom: VISTA with GPT-5.6 Sol compared with human players and the official implementation using a minimal harness. We report success rate and progress for GameWorld, the geometric mean of human-normalized scores (human median =100=100) for AI GameStore, and accuracy for BabyVision. The dashed line marks the best previously reported result.

5.1 Setup

GameWorld.

GameWorld comprises 170 tasks across 34 browser games. The agent sees screenshots, controls, and the official task objective, and acts through keyboard and mouse input. Official success rate measures task completion; progress also credits partial completion.

AI GameStore.

AI GameStore evaluation uses ten public games. Each score is normalized to a published human median of 100, then the ten normalized scores are combined geometrically.

BabyVision.

BabyVision evaluation covers 39 of its 388 questions: 20 mazes and 19 connecting-lines tasks. The agent can inspect regions and read pixels in a single static image; accuracy is measured on this visual-tracking subset.

5.2 Results

GameWorld mean success rises from 40.0% to 63.3%, and progress from 62.9% to 79.3%. AI GameStore’s aggregate score rises from 47.3 to 140.3. VISTA exceeds GameWorld’s novice human reference and AI GameStore’s human-median-normalized reference.

BabyVision subset accuracy rises from 41.0% to 63.2%, below the 92.3% human reference. These static tasks demonstrate image inspection’s usefulness without an interaction archive. Human comparisons are specific to each benchmark’s population and protocol; Appendix B.1 reports repeated-run variation.

6 Limitations

Generalization remains unverified on private or genuinely new games. The models were released after the public benchmarks, so the authors cannot exclude training-data overlap. Performance depends on the underlying models, and the reported action-efficiency gains do not establish low inference cost. Private ARC-AGI-3 evaluation would provide a stronger test.

7 Conclusion

Original-frame storage and model-directed inspection improve performance on these benchmarks. Extension to realistic embodied tasks remains future work.

Acknowledgments

The authors thank Itamar Pres, Alan Sun, Yung-Sung Chuang, and Zhaofeng Wu for feedback and support.

Appendix A Experimental Details

A.1 Baselines: Official Implementations

Baselines use published results where available and otherwise rerun official implementations. Experiments use subscription command-line interfaces, including GLM as an alternative backend. Access differs from some published baselines.

ARC-AGI-3.

The official ARC harness supplies textual grids and up to seven frames per action, then discards the earliest turns beyond a 175K-token context. The paper uses up to 2,000 actions per game for the two closed-weight models and 3,000 for GLM. Main-table closed-weight baselines are published results; the GLM baseline is rerun.

GameWorld.

The reproduced GameWorld baseline includes screenshots, reasoning, and actions from the previous two steps. Each task allows 100 actions.

AI GameStore.

The AI GameStore baseline makes 120 decisions, one per game second. Each decision receives five current frames sampled 0.2 seconds apart and the previous scratchpad/action/result. Games pause during inference.

BabyVision.

The BabyVision baseline answers from one image-and-question request, without iterative inspection. Its final answer is extracted using the benchmark’s reference implementation.

A.2 Ablation Details

The six incremental configurations share one model interface; the published official result uses another and is an external reference. Images reduce seven-frame input from roughly 28K text tokens to 2.2K image tokens.

Budget changes: five times human actions per section and 12 hours per game become 2,000 actions and 48 hours. Subsequent steps add continuous conversation with compaction, notes, then archived-frame inspection and pixel readout. Only the final two configurations retrieve intermediate frames on demand.

A.3 Evaluation Metrics

ARC-AGI-3.

Completed-level credit is the squared human/agent action ratio, clipped at 1.15; unfinished levels receive zero. Human actions use the upper median among first-time players who completed that level. Later levels have linearly increasing weights, and each game is capped by weighted completion. The reported score averages the 25 games and multiplies by 100. Total actions distinguish systems whose clipped scores tie.

Equations 1–2
Sℓ,g=min⁡(1.15,(hℓ,gaℓ,g)2),S_{\ell,g}=\min\left(1.15,\left(\frac{h_{\ell,g}}{a_{\ell,g}}\right)^{2}\right), (1)
Sg=min⁡(∑ℓ=1kwℓ∑ℓ=1nwℓ,∑ℓ=1nwℓ​Sℓ,g∑ℓ=1nwℓ).S_{g}=\min\left(\frac{\sum_{\ell=1}^{k}w_{\ell}}{\sum_{\ell=1}^{n}w_{\ell}},\frac{\sum_{\ell=1}^{n}w_{\ell}S_{\ell,g}}{\sum_{\ell=1}^{n}w_{\ell}}\right). (2)

GameWorld.

Success rate averages successful completions over 170 tasks. Progress averages each task’s highest normalized progress, as determined from serialized game state by the official evaluator.

Equations 3–4
SR=1N​∑i=1N𝕀⁡[task ​i​ succeeded],\mathrm{SR}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\left[\text{task }i\text{ succeeded}\right], (3)
PG=1N​∑i=1Npi.\mathrm{PG}=\frac{1}{N}\sum_{i=1}^{N}p_{i}. (4)

AI GameStore.

Each raw game score is divided by the published human median and multiplied by 100, then clipped to [1, 10,000]. The final score is the geometric mean over ten games.

Equations 5–6
ng=clip⁡(100×rgmg, 1, 104).n_{g}=\operatorname{clip}\!\left(\frac{100\times r_{g}}{m_{g}},\,1,\,10^{4}\right). (5)
Score=(∏g=110ng)1/10.\mathrm{Score}=\left(\prod_{g=1}^{10}n_{g}\right)^{1/10}. (6)

BabyVision.

BabyVision answers are assessed through exact matching and an LLM judge. Failed runs count as incorrect.

A.4 Agent Tools

A host controller executes structured calls, stores frames and notes, and alone advances the game, sequentially. The agent is sandboxed without environment credentials; Claude’s built-in shell/filesystem tools are disabled.

The complete interface below includes play, inspect, read_pixels, history, notes, and compaction checkpoints. Only play changes the game. Frames carry turn/frame labels. The controller requires a saved checkpoint before compaction, then restores notes, history, and the current frame.

Algorithm 1 The VISTA tool interface in pseudocode. play is the only tool that changes the game environment, note-writing tools update only the agent’s stored notes, and all other tools are read-only.
# ---------------- Action ----------------
play(action, x=None, y=None) -> (frame, n_frames, available_actions)
    # Execute exactly one game action. The action set is defined by each
    # benchmark and game (e.g. ARC-AGI-3: RESET, ACTION1..7, where ACTION6
    # is a click at (x, y); GameWorld and AI GameStore use their own inputs).
    # Returns the final frame. All intermediate frames are archived.

# ------------- Visual memory -------------
inspect(question, views) -> images
    # question: the visual question the views should answer
    # view = {label, turn, frame=final, region=full},
    # region = {x, y, w, h}, each view returns one image
    # Retrieve any archived frame, crop the region, and enlarge it

read_pixels(question, views) -> (palette, samples[])
    # question: the visual question the views should answer
    # view = {label, turn, frame=final, region=full},
    # region = {x, y, w, h}, each view returns one image
    # Split the region into rows x columns cells and read the pixel value
    # of each cell. Return the values as a list of numbers.

# ---------------- History ----------------
history(view, start_turn=None, end_turn=None, limit=64) -> records[]
    # view = "events":   exact action/result records for turns [start, end)
    # view = "attempts": one summary for each attempt in the current level. An
    # attempt starts when the level is entered or RESET is
    # issued, and ends at GAME_OVER, WIN, or TERMINAL.

# ----------------- Notes -----------------
# GUIDE.md stores reusable knowledge about the game across levels and sessions.
# WORKING.md stores temporary state and plans for the current level.
# It is retained across RESETs and context restarts, cleared when the level advances

read_guide() -> text        # read GUIDE.md
write_guide(content)        # replace GUIDE.md
read_working() -> text      # read WORKING.md
write_working(content)      # replace WORKING.md

save_compact_checkpoint(guide, working_memory)
    # Atomically save the state needed to resume after context compaction.
    # Write WORKING.md with working_memory. If guide is not null, also
    # write GUIDE.md; otherwise leave it unchanged.

A.5 Agent Prompts

The complete prompts below encourage revisable understanding and evidence gathering. Interactive prompts additionally request outcome predictions and comparisons with observed results.

ARC-AGI-3.

Every ARC game uses the same prompt: complete it with few actions, predict each action’s result, record all visible changes, and keep durable understanding in the two notes.

Prompt 1 ARC-AGI-3.
# Visual game task

Complete the game with as few game actions as possible.

Build and use a compact, revisable model of the game and its current state. Update
it as new evidence changes what is supported.

Before each `play`, briefly state what you expect to see. Afterward, briefly state
all visible changes, expected or not.

Keep concise, durable, revisable game understanding in `GUIDE.md`; use `WORKING.md`
as a scratchpad when useful.

GameWorld and AI GameStore.

Both interactive browser-game benchmarks use an environment-neutral variant with act and a supplied objective. GameWorld includes its official task instruction verbatim.

Prompt 2 GameWorld and AI GameStore.
# Interactive world task

Pursue the supplied objective.

Build and use a compact, revisable model of the environment and its current state.
Update it as new evidence changes what is supported.

Before each `act`, briefly state what you expect to observe. Afterward, briefly
state all observable changes, expected or not.

Keep concise, durable, revisable understanding in `GUIDE.md`; use `WORKING.md` as a
scratchpad when useful.

BabyVision.

The static-image prompt requests targeted inspection and pixel measurements before answering. It omits persistent notes because questions are independent.

Prompt 3 BabyVision.
# Visual question task

Before answering, gather the evidence the question actually needs from the supplied
image. Use `inspect` to view enlarged regions where detail matters, and
`read_pixels` to measure exact colors where a judgement depends on them. Say what
each view is meant to settle, then check whether it did.

Build a short, revisable account of what the image shows and update it as the
evidence changes what is supported. Answer only when the evidence supports one
answer.

A.6 Human References

ARC-AGI-3.

The ARC study recruited 486 people; each game had ten first-time players and was retained only if at least two completed every level. Sessions lasted 90 minutes, with 20-minute soft and 30-minute hard limits per game. RHAE references the upper-median actions among successful players at each level.

GameWorld.

GameWorld’s human study involved two computer-science graduate students, each attempting all 170 tasks with the same 100-action budget. The paper compares against the novice, who had not seen the games; the other participant studied and practiced them.

AI GameStore.

AI GameStore recruited 106 participants, randomly assigning ten games each and allowing two minutes per game. Each game’s human median supplies the normalization reference.

BabyVision.

Sixteen adults with at least bachelor’s degrees completed all 388 BabyVision questions. Maze and connecting-lines accuracy were 95.0% and 89.5%; weighting these by the evaluated subset’s 20 and 19 questions gives 92.3%.

Appendix B Additional Results

B.1 Run-to-run Variation

Across three runs, GPT’s ARC RHAE is 99.00, 98.90, and 99.23: 99.04 ± 0.17. GameWorld averages 63.33 ± 2.37% success and 79.30 ± 2.14% progress. BabyVision averages 63.23 ± 3.00%, from 61.5%, 61.5%, and 66.7%.

AI GameStore’s three aggregate run scores are 119.99, 148.58, and 140.06. Its reported multi-run result first averages raw scores within each game, then normalizes and aggregates; it is not the arithmetic mean of those three final scores.

B.2 Per-game Results on ARC-AGI-3

Table 2 preserves all 25 games. Opus scores 100.00 throughout. GPT completes everything but scores 75.03 on ls20, yielding a 99.00 mean. Total actions are 0.43× and 0.53× the summed human reference.

Table 2: Per-game results on the 25 public ARC-AGI-3 games. We evaluate VISTA with Claude Opus 5.0 and GPT-5.6 Sol. For each game, human actions are the sum of the per-level human baselines, each defined as the upper-median action count among first-time human players who completed that level. The action ratio is the agent’s total actions divided by this reference. Claude Opus 5.0 achieves an RHAE score of 100.00 on every game. GPT-5.6 Sol completes every game and achieves a mean RHAE score of 99.00.
Claude Opus 5.0 GPT-5.6 Sol
Game Human actions Score Actions Ratio Score Actions Ratio
ar25 748 100.00 256 0.34×\times 100.00 260 0.35×\times
bp35 651 100.00 521 0.80×\times 100.00 534 0.82×\times
cd82 171 100.00 112 0.65×\times 100.00 100 0.58×\times
cn04 789 100.00 225 0.29×\times 100.00 200 0.25×\times
dc22 1,228 100.00 547 0.45×\times 100.00 560 0.46×\times
ft09 208 100.00 75 0.36×\times 100.00 77 0.37×\times
g50t 879 100.00 338 0.38×\times 100.00 486 0.55×\times
ka59 730 100.00 303 0.42×\times 100.00 396 0.54×\times
lf52 1,339 100.00 1,008 0.75×\times 100.00 936 0.70×\times
lp85 388 100.00 104 0.27×\times 100.00 97 0.25×\times
ls20 776 100.00 458 0.59×\times 75.03 1,135 1.46×\times
m0r0 1,107 100.00 219 0.20×\times 100.00 224 0.20×\times
r11l 233 100.00 75 0.32×\times 100.00 93 0.40×\times
re86 1,255 100.00 628 0.50×\times 100.00 582 0.46×\times
s5i5 638 100.00 254 0.40×\times 100.00 293 0.46×\times
sb26 213 100.00 125 0.59×\times 100.00 124 0.58×\times
sc25 350 100.00 153 0.44×\times 100.00 228 0.65×\times
sk48 1,070 100.00 413 0.39×\times 100.00 605 0.57×\times
sp80 518 100.00 119 0.23×\times 100.00 184 0.36×\times
su15 361 100.00 99 0.27×\times 100.00 123 0.34×\times
tn36 317 100.00 143 0.45×\times 100.00 225 0.71×\times
tr87 414 100.00 140 0.34×\times 100.00 209 0.50×\times
tu93 462 100.00 193 0.42×\times 100.00 197 0.43×\times
vc33 447 100.00 182 0.41×\times 100.00 272 0.61×\times
wa30 1,843 100.00 612 0.33×\times 100.00 986 0.53×\times
Total (25 games) 17,135 100.00 7,302 0.43×\times 99.00 9,126 0.53×\times

B.3 Interaction trajectories

Figures 12–13 preserve full trajectories for two gravity-and-block BP35 levels. Figure 14 shows inspection across time and space in Minecraft Clone and Pac-Man, plus line tracing and maze analysis in BabyVision.

Refer to caption
Figure 12: A full VISTA trajectory on an ARC-AGI-3 task (ID BP35), one frame per action (level 1). The player begins under upward gravity and can move left or right. Clicking a green block removes it, allowing the player to clear obstacles and navigate through otherwise blocked passages. The goal is to reach the pink object. Each frame shows the state after the labeled action; rings indicate clicks. Read left to right, top to bottom.
Refer to caption
Figure 13: A full VISTA trajectory on an ARC-AGI-3 task (ID BP35), one frame per action (level 3). Level 3 introduces an orange block that disappears when clicked and reappears when clicked again. The agent can remove it to pass through a blocked path, or restore it to prevent the player from rising into the deadly purple spikes.
Refer to caption
Figure 14: Four VISTA trajectories beyond ARC-AGI-3 with GPT-5.6 Sol. The top row shows GameWorld games, and the bottom row shows BabyVision tasks. Top left: in Minecraft Clone, the agent inspects across space, magnifying the hotbar and ore vein to distinguish iron from stone. Top right: in Pac-Man, it inspects across time, revisiting the current and initial frames to view the full maze. Bottom left: in Connect the Lines, the agent magnifies two regions to trace a path that is difficult to follow at the supplied image resolution, then reads exact pixel values before answering. Bottom right: in Maze, it renders the board at 2×2\times scale and then reads the wall grid as pixel values.
Concept demonstration

Open image ↗