At a glance
VISTA lets a multimodal agent revisit its visual evidence: it archives every returned frame, retrieves past observations, enlarges regions, and reads exact pixels. The harness adds no trained model; reasoning remains in the underlying model.
Claude Opus 5.0 completes all 25 public ARC-AGI-3 games with 100.00 RHAE, using 7,302 actions versus a 17,135-action human reference. GPT-5.6 Sol scores 99.00. These are public-game action-efficiency results; different reasoning settings and budgets limit baseline comparisons, and training-data overlap cannot be ruled out.
1 Introduction
Interactive games require discovering hidden rules and goals [5]. One-time visual encoding can lose detail, while context truncation discards earlier evidence [29] [27].
VISTA keeps original observations outside the bounded context. The agent reasons in language, predicts outcomes, and retrieves frames or regions as needed [55] [59]. Tests cover public ARC-AGI-3 games and selected tasks from three further benchmarks.
2 Related Work
ARC-AGI benchmarks.
ARC-AGI-1 and ARC-AGI-2 test static grid reasoning; ARC-AGI-3 adds exploration and action [16] [14] [5]. Many strong interactive systems construct executable symbolic models [62] [25] [11]. VISTA instead uses images and natural-language reasoning, avoiding an explicit programmatic model of the environment.
Computer-use agents.
Computer-use agents either consume structured interfaces such as DOM trees or act directly from screenshots [17] [64] [13] [57]. VISTA follows the visual approach and concentrates on retaining and revisiting evidence during interaction.
Game-playing agents.
Earlier game agents range from separately trained policies to generalist foundation models [31] [21] [51] [53] [46] [49] [54]. VISTA instead emphasizes a complete visual archive accessible independently of current context.
3 VISTA: A Visual Harness
Agents cannot know initially which visual details will matter later. Preserving original frames lets them recover evidence once a relevant question becomes clear.
3.1 Visual Harness Components
Three components give the model explicit control over which parts of its visual history to examine.
Visual observation.
Rendered observations retain object appearance and spatial relationships for the model’s visual reasoning.
Lossless visual memory.
Every returned frame, including intermediate animation frames, is stored in its original form and indexed by turn and frame. This archive survives context truncation or compaction.
Visual inspection.
The agent can retrieve arbitrary earlier frames, compare states across time, or crop and enlarge rectangular regions. Inspection is optional and directed by the model’s current question.
3.2 Pipeline
A single agent observes, reasons, optionally inspects frames or pixels, then calls play. The mechanical harness executes tools and reports invalid requests; each turn can include multiple inspections.
GUIDE.md stores reusable knowledge; WORKING.md holds current plans. The agent predicts action outcomes and updates its understanding afterward. Near the context limit it writes a handoff summary and resumes with notes, history, and the visual archive intact.
4 VISTA on ARC-AGI-3
4.1 Setup
Dataset.
Evaluation uses the 25 public ARC-AGI-3 games [5]. Rules and goals are initially hidden. Success requires discovering mechanics through interaction and transferring that knowledge across increasingly difficult levels within each game.
Evaluation metrics.
RHAE compares actions with first-time human players, weights later levels more heavily, and caps games by weighted completion [4]. Unfinished levels score zero. It measures action efficiency, not inference cost or time. Appendix A.3 gives the equations.
Models.
The main backends are Claude Opus 5.0 at extra-high reasoning effort and GPT-5.6 Sol at maximum effort, accessed through their respective command-line interfaces [3] [33]. Each runs independently under the VISTA interaction protocol.
4.2 Results
Opus scores 100.00 RHAE with 7,302 actions, 57.4% below the 17,135-action human reference. GPT scores 99.00 with 9,126 actions. Both complete every public game. Table 1 compares VISTA with concurrent systems, many of which use program synthesis.
The official Opus baseline scores 40.68 at high effort, whereas VISTA uses extra-high. Official GPT scores 13.33. These are system-level comparisons with interface and stopping-budget differences; they do not isolate the visual harness alone. The controlled incremental configurations below provide more specific evidence.
| System | Program | Model | Effort | RHAE |
|---|---|---|---|---|
| Official impl. [5] | No | GPT-5.6 Sol [34] | max | 13.33 |
| Opus 5.0† | high | 40.68 | ||
| Schema [62] | Yes | GPT-5.6 Sol | xhigh max | 95.35 |
| Opus 4.8 Fable 5 | max | 98.98 | ||
| ewma_sv_v1.6 [39] | Yes | GPT-5.6 Sol | xhigh | 98.97 |
| Retrodict [11] | Yes | GPT-5.6 Sol | max | 99.86 |
| Tycho [25] | Yes | GPT-5.6 Sol | max | 100.00 |
| Opus 5.0 | xhigh | 100.00 | ||
| VISTA | No | GPT-5.6 Sol | max | 99.00 |
| Opus 5.0 | xhigh | 100.00 |
†Calculated from the officially reported per-game scores for the 25 public games [6].
4.3 Ablations
Unless varied, ablations use GPT-5.6 Sol at maximum effort, a 200K-token context limit, and 512 × 512 images obtained by 8× nearest-neighbor enlargement of the original 64 × 64 frames.
Effects of VISTA components.
Incremental RHAE: image observations 47.32 → larger action/time budgets 51.66 → continuous conversation with compaction 65.82 → persistent notes 70.05 → visual memory and inspection 94.10 → exact pixel readout 99.00.
The official textual baseline’s 13.33 is an external reference using a different model-access interface. Configurations (i)–(vi) share the experimental interface. Early configurations automatically receive up to seven frames per action; the final two receive only the last frame and retrieve others as needed. Fewer actions in weak configurations can reflect early termination, rather than efficient completion.
Reasoning directly from images.
Replacing images with textual grids inside the otherwise unchanged harness preserves comparable RHAE but requires slightly more actions. A frame costs roughly 4,000 text tokens versus 308 image tokens at the default scale. Average token usage is 71.9M per game with text and 30.7M with images.
Effect of maximum context size.
The default 200K context achieves 99.00 RHAE. A 100K limit loses only 0.4 points while roughly halving token usage. Increasing to 780K costs 105.7M tokens per game versus 30.7M at 200K, without improving the best score.
Effect of input image scale.
A 4× enlargement gives comparable performance to the default 8× with 75% fewer image tokens per frame. At 2× performance falls; 16× brings no gain and increases cost. More image tokens are therefore not uniformly useful.
Results with an open-weight model.
The open-weight GLM-5.3 Flash 320B backend reaches 66.93 RHAE with VISTA [20] [63]. Figure 8 also compares the official minimal harness and a VISTA variant without inspection or pixel readout. Gains extend beyond the two closed-weight backends, although their near-perfect performance is not reproduced.
Effects of 3D rendering.
Rendering the same games in 3D yields 84.12 RHAE and 20,640 actions. This tests a changed visual representation of the game mechanics; the substantial action increase shows that richer visuals remain harder.
4.4 Qualitative Analysis
Figure 10 illustrates reconstructing a map from earlier frames after a reset, and enlarging small blocks to identify orientation markers: retrieval across time and local inspection.
5 General Visual Environments
The same visual harness is adapted to GameWorld and AI GameStore, plus static BabyVision tracking questions [35] [60] [12]. Comparisons use GPT-5.6 Sol with each benchmark’s official minimal harness; human references follow each benchmark’s own protocol.

5.1 Setup
GameWorld.
GameWorld comprises 170 tasks across 34 browser games. The agent sees screenshots, controls, and the official task objective, and acts through keyboard and mouse input. Official success rate measures task completion; progress also credits partial completion.
AI GameStore.
AI GameStore evaluation uses ten public games. Each score is normalized to a published human median of 100, then the ten normalized scores are combined geometrically.
BabyVision.
BabyVision evaluation covers 39 of its 388 questions: 20 mazes and 19 connecting-lines tasks. The agent can inspect regions and read pixels in a single static image; accuracy is measured on this visual-tracking subset.
5.2 Results
GameWorld mean success rises from 40.0% to 63.3%, and progress from 62.9% to 79.3%. AI GameStore’s aggregate score rises from 47.3 to 140.3. VISTA exceeds GameWorld’s novice human reference and AI GameStore’s human-median-normalized reference.
BabyVision subset accuracy rises from 41.0% to 63.2%, below the 92.3% human reference. These static tasks demonstrate image inspection’s usefulness without an interaction archive. Human comparisons are specific to each benchmark’s population and protocol; Appendix B.1 reports repeated-run variation.
6 Limitations
Generalization remains unverified on private or genuinely new games. The models were released after the public benchmarks, so the authors cannot exclude training-data overlap. Performance depends on the underlying models, and the reported action-efficiency gains do not establish low inference cost. Private ARC-AGI-3 evaluation would provide a stronger test.
7 Conclusion
Original-frame storage and model-directed inspection improve performance on these benchmarks. Extension to realistic embodied tasks remains future work.
Acknowledgments
The authors thank Itamar Pres, Alan Sun, Yung-Sung Chuang, and Zhaofeng Wu for feedback and support.
Appendix A Experimental Details
A.1 Baselines: Official Implementations
Baselines use published results where available and otherwise rerun official implementations. Experiments use subscription command-line interfaces, including GLM as an alternative backend. Access differs from some published baselines.
ARC-AGI-3.
The official ARC harness supplies textual grids and up to seven frames per action, then discards the earliest turns beyond a 175K-token context. The paper uses up to 2,000 actions per game for the two closed-weight models and 3,000 for GLM. Main-table closed-weight baselines are published results; the GLM baseline is rerun.
GameWorld.
The reproduced GameWorld baseline includes screenshots, reasoning, and actions from the previous two steps. Each task allows 100 actions.
AI GameStore.
The AI GameStore baseline makes 120 decisions, one per game second. Each decision receives five current frames sampled 0.2 seconds apart and the previous scratchpad/action/result. Games pause during inference.
BabyVision.
The BabyVision baseline answers from one image-and-question request, without iterative inspection. Its final answer is extracted using the benchmark’s reference implementation.
A.2 Ablation Details
The six incremental configurations share one model interface; the published official result uses another and is an external reference. Images reduce seven-frame input from roughly 28K text tokens to 2.2K image tokens.
Budget changes: five times human actions per section and 12 hours per game become 2,000 actions and 48 hours. Subsequent steps add continuous conversation with compaction, notes, then archived-frame inspection and pixel readout. Only the final two configurations retrieve intermediate frames on demand.
A.3 Evaluation Metrics
ARC-AGI-3.
Completed-level credit is the squared human/agent action ratio, clipped at 1.15; unfinished levels receive zero. Human actions use the upper median among first-time players who completed that level. Later levels have linearly increasing weights, and each game is capped by weighted completion. The reported score averages the 25 games and multiplies by 100. Total actions distinguish systems whose clipped scores tie.
Equations 1–2
| (1) |
| (2) |
GameWorld.
Success rate averages successful completions over 170 tasks. Progress averages each task’s highest normalized progress, as determined from serialized game state by the official evaluator.
Equations 3–4
| (3) |
| (4) |
AI GameStore.
Each raw game score is divided by the published human median and multiplied by 100, then clipped to [1, 10,000]. The final score is the geometric mean over ten games.
Equations 5–6
| (5) |
| (6) |
BabyVision.
BabyVision answers are assessed through exact matching and an LLM judge. Failed runs count as incorrect.
A.4 Agent Tools
A host controller executes structured calls, stores frames and notes, and alone advances the game, sequentially. The agent is sandboxed without environment credentials; Claude’s built-in shell/filesystem tools are disabled.
The complete interface below includes play, inspect, read_pixels, history, notes, and compaction checkpoints. Only play changes the game. Frames carry turn/frame labels. The controller requires a saved checkpoint before compaction, then restores notes, history, and the current frame.
# ---------------- Action ----------------
play(action, x=None, y=None) -> (frame, n_frames, available_actions)
# Execute exactly one game action. The action set is defined by each
# benchmark and game (e.g. ARC-AGI-3: RESET, ACTION1..7, where ACTION6
# is a click at (x, y); GameWorld and AI GameStore use their own inputs).
# Returns the final frame. All intermediate frames are archived.
# ------------- Visual memory -------------
inspect(question, views) -> images
# question: the visual question the views should answer
# view = {label, turn, frame=final, region=full},
# region = {x, y, w, h}, each view returns one image
# Retrieve any archived frame, crop the region, and enlarge it
read_pixels(question, views) -> (palette, samples[])
# question: the visual question the views should answer
# view = {label, turn, frame=final, region=full},
# region = {x, y, w, h}, each view returns one image
# Split the region into rows x columns cells and read the pixel value
# of each cell. Return the values as a list of numbers.
# ---------------- History ----------------
history(view, start_turn=None, end_turn=None, limit=64) -> records[]
# view = "events": exact action/result records for turns [start, end)
# view = "attempts": one summary for each attempt in the current level. An
# attempt starts when the level is entered or RESET is
# issued, and ends at GAME_OVER, WIN, or TERMINAL.
# ----------------- Notes -----------------
# GUIDE.md stores reusable knowledge about the game across levels and sessions.
# WORKING.md stores temporary state and plans for the current level.
# It is retained across RESETs and context restarts, cleared when the level advances
read_guide() -> text # read GUIDE.md
write_guide(content) # replace GUIDE.md
read_working() -> text # read WORKING.md
write_working(content) # replace WORKING.md
save_compact_checkpoint(guide, working_memory)
# Atomically save the state needed to resume after context compaction.
# Write WORKING.md with working_memory. If guide is not null, also
# write GUIDE.md; otherwise leave it unchanged.
A.5 Agent Prompts
The complete prompts below encourage revisable understanding and evidence gathering. Interactive prompts additionally request outcome predictions and comparisons with observed results.
ARC-AGI-3.
Every ARC game uses the same prompt: complete it with few actions, predict each action’s result, record all visible changes, and keep durable understanding in the two notes.
# Visual game task Complete the game with as few game actions as possible. Build and use a compact, revisable model of the game and its current state. Update it as new evidence changes what is supported. Before each `play`, briefly state what you expect to see. Afterward, briefly state all visible changes, expected or not. Keep concise, durable, revisable game understanding in `GUIDE.md`; use `WORKING.md` as a scratchpad when useful.
GameWorld and AI GameStore.
Both interactive browser-game benchmarks use an environment-neutral variant with act and a supplied objective. GameWorld includes its official task instruction verbatim.
# Interactive world task Pursue the supplied objective. Build and use a compact, revisable model of the environment and its current state. Update it as new evidence changes what is supported. Before each `act`, briefly state what you expect to observe. Afterward, briefly state all observable changes, expected or not. Keep concise, durable, revisable understanding in `GUIDE.md`; use `WORKING.md` as a scratchpad when useful.
BabyVision.
The static-image prompt requests targeted inspection and pixel measurements before answering. It omits persistent notes because questions are independent.
# Visual question task Before answering, gather the evidence the question actually needs from the supplied image. Use `inspect` to view enlarged regions where detail matters, and `read_pixels` to measure exact colors where a judgement depends on them. Say what each view is meant to settle, then check whether it did. Build a short, revisable account of what the image shows and update it as the evidence changes what is supported. Answer only when the evidence supports one answer.
A.6 Human References
ARC-AGI-3.
The ARC study recruited 486 people; each game had ten first-time players and was retained only if at least two completed every level. Sessions lasted 90 minutes, with 20-minute soft and 30-minute hard limits per game. RHAE references the upper-median actions among successful players at each level.
GameWorld.
GameWorld’s human study involved two computer-science graduate students, each attempting all 170 tasks with the same 100-action budget. The paper compares against the novice, who had not seen the games; the other participant studied and practiced them.
AI GameStore.
AI GameStore recruited 106 participants, randomly assigning ten games each and allowing two minutes per game. Each game’s human median supplies the normalization reference.
BabyVision.
Sixteen adults with at least bachelor’s degrees completed all 388 BabyVision questions. Maze and connecting-lines accuracy were 95.0% and 89.5%; weighting these by the evaluated subset’s 20 and 19 questions gives 92.3%.
Appendix B Additional Results
B.1 Run-to-run Variation
Across three runs, GPT’s ARC RHAE is 99.00, 98.90, and 99.23: 99.04 ± 0.17. GameWorld averages 63.33 ± 2.37% success and 79.30 ± 2.14% progress. BabyVision averages 63.23 ± 3.00%, from 61.5%, 61.5%, and 66.7%.
AI GameStore’s three aggregate run scores are 119.99, 148.58, and 140.06. Its reported multi-run result first averages raw scores within each game, then normalizes and aggregates; it is not the arithmetic mean of those three final scores.
B.2 Per-game Results on ARC-AGI-3
Table 2 preserves all 25 games. Opus scores 100.00 throughout. GPT completes everything but scores 75.03 on ls20, yielding a 99.00 mean. Total actions are 0.43× and 0.53× the summed human reference.
| Claude Opus 5.0 | GPT-5.6 Sol | ||||||
|---|---|---|---|---|---|---|---|
| Game | Human actions | Score | Actions | Ratio | Score | Actions | Ratio |
| ar25 | 748 | 100.00 | 256 | 0.34 | 100.00 | 260 | 0.35 |
| bp35 | 651 | 100.00 | 521 | 0.80 | 100.00 | 534 | 0.82 |
| cd82 | 171 | 100.00 | 112 | 0.65 | 100.00 | 100 | 0.58 |
| cn04 | 789 | 100.00 | 225 | 0.29 | 100.00 | 200 | 0.25 |
| dc22 | 1,228 | 100.00 | 547 | 0.45 | 100.00 | 560 | 0.46 |
| ft09 | 208 | 100.00 | 75 | 0.36 | 100.00 | 77 | 0.37 |
| g50t | 879 | 100.00 | 338 | 0.38 | 100.00 | 486 | 0.55 |
| ka59 | 730 | 100.00 | 303 | 0.42 | 100.00 | 396 | 0.54 |
| lf52 | 1,339 | 100.00 | 1,008 | 0.75 | 100.00 | 936 | 0.70 |
| lp85 | 388 | 100.00 | 104 | 0.27 | 100.00 | 97 | 0.25 |
| ls20 | 776 | 100.00 | 458 | 0.59 | 75.03 | 1,135 | 1.46 |
| m0r0 | 1,107 | 100.00 | 219 | 0.20 | 100.00 | 224 | 0.20 |
| r11l | 233 | 100.00 | 75 | 0.32 | 100.00 | 93 | 0.40 |
| re86 | 1,255 | 100.00 | 628 | 0.50 | 100.00 | 582 | 0.46 |
| s5i5 | 638 | 100.00 | 254 | 0.40 | 100.00 | 293 | 0.46 |
| sb26 | 213 | 100.00 | 125 | 0.59 | 100.00 | 124 | 0.58 |
| sc25 | 350 | 100.00 | 153 | 0.44 | 100.00 | 228 | 0.65 |
| sk48 | 1,070 | 100.00 | 413 | 0.39 | 100.00 | 605 | 0.57 |
| sp80 | 518 | 100.00 | 119 | 0.23 | 100.00 | 184 | 0.36 |
| su15 | 361 | 100.00 | 99 | 0.27 | 100.00 | 123 | 0.34 |
| tn36 | 317 | 100.00 | 143 | 0.45 | 100.00 | 225 | 0.71 |
| tr87 | 414 | 100.00 | 140 | 0.34 | 100.00 | 209 | 0.50 |
| tu93 | 462 | 100.00 | 193 | 0.42 | 100.00 | 197 | 0.43 |
| vc33 | 447 | 100.00 | 182 | 0.41 | 100.00 | 272 | 0.61 |
| wa30 | 1,843 | 100.00 | 612 | 0.33 | 100.00 | 986 | 0.53 |
| Total (25 games) | 17,135 | 100.00 | 7,302 | 0.43 | 99.00 | 9,126 | 0.53 |
B.3 Interaction trajectories
Figures 12–13 preserve full trajectories for two gravity-and-block BP35 levels. Figure 14 shows inspection across time and space in Minecraft Clone and Pac-Man, plus line tracing and maze analysis in BabyVision.