Literature
Display
← All papers

GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

Yiran Wang · Xingyilang Yin · Junfu Pu · Guangzhi Wang · Kaifeng Li · Mingyu Ouyang · Huiqiang Sun · Lingen Li · Cheng Cheng · Wangbo Yu · Honghao Chen · Xiaodong Cun · Chi-Man Pun · Zhiguo Cao · Ying Shan

· arXiv (preprint) ↗

AI agents6 min text · CondensedOriginal paper ↗
Contents

Overview

Refer to caption
Figure 1: GameHorizon Suite spans diverse games with a pyramid of primitive actions (e.g., Left Mouse Button as LMB), short-horizon operations, medium-horizon goals, and long-horizon strategies, providing a unified yardstick for testing gameplay capabilities across models and temporal horizons. Red text highlights key actions or objects in video frames.

At a glance

GameHorizon combines 5,000 hours of synchronized human gameplay from 21 titles with an automatic instruction annotator and a benchmark. Instructions describe immediate operations, medium-term goals, and longer strategies. The offline track tests 47 models with standardized questions; an online Minecraft track diagnoses individual failures within longer tasks. Future planning and top-down goal decomposition remain difficult, and offline success does not establish reliable open-ended gameplay.

1 Introduction

Fast game-action policies and general-purpose reasoning models are usually tested with different interfaces. GameHorizon places observations, instructions, and executable controls in a common evaluation format. It aims to separate recognizing an ongoing action, choosing a future action, and decomposing a broad objective. [1], [2], [4], [9], [15].

The contribution is a dataset and diagnostic evaluation framework, rather than a newly trained game agent. Its breadth comes mainly from the offline corpus; the online experiments use Minecraft, where underlying states can be checked and reset.

2 Related Work

Earlier corpora often cover a single game, infer controls from video, or omit dense language annotations. Bespoke online agent harnesses and small rollout counts also make comparisons difficult. GameHorizon records controls directly across multiple games and evaluates several model families through common questions, while retaining an online track for execution. [1], [9], [22], [23], [25], [26], [36], [38].

3 GameHorizon Suite

The suite links three stages: an annotator that builds instruction hierarchies, a synchronized gameplay dataset, and an offline/online benchmark.

Refer to caption
Figure 2: Overview of GameHorizon Suite. GameHorizon-Annotator automatically produces a three-level pyramid of short-horizon operations L1L_{1}, medium-horizon goals L2L_{2}, and long-horizon strategies L3L_{3}. GameHorizon-Data contains 5​@​0005{,}000 hours of gameplay across 2121 game titles, with temporally aligned videos, actions, and multi-horizon instructions. GameHorizon-Bench provides reproducible offline and stepwise online testing. The offline track contains thousands of standardized questions across three primary tasks T1T_{1}–T3T_{3} and diagnostic variants T∗T^{*}. The online track tests order-dependent causal and order-flexible thematic tasks through verifiable subtasks and game resets for failure localization.

3.1 Overview

One hundred experienced players record 2K video at 60 frames per second alongside keyboard and mouse events. A bottom-up annotation pipeline groups immediate operations into goals and then strategies. The nominal horizons are 1–5 seconds, 1–2 minutes, and 5–8 minutes. Unlike inverse dynamics labels, the recorded controls reflect actual player inputs; the textual instructions are model-generated interpretations of those inputs.

Refer to caption
Figure 3: Workflow of GameHorizon-Annotator. Videos and actions are first processed by action-aware segmentation to produce short-horizon clips, with key actions determining their temporal boundaries. A VLM annotates each clip with an L1L_{1} operation. Lower-level clips with their instructions are progressively merged into medium- and long-horizon clips based on action continuity and semantic coherence, from which the VLM derives L2L_{2} goals and L3L_{3} strategies. The bottom-up procedure abstracts fine-grained video-action trajectories into a pyramid of multi-horizon text instructions.

3.2 GameHorizon-Annotator

Game-specific keybinds map raw input events to candidate actions. Rules identify sustained actions and discrete boundaries; a VLM uses the video to resolve ambiguous controls and refine segments. Adjacent clips are merged according to action continuity and semantic coherence, with dynamic programming enforcing duration constraints. [41], [42], [43].

For immediate operations, the annotator sees frames, synchronized controls, and keybinds. Medium-horizon goals add the constituent operation instructions. Long-horizon strategies use frames and medium-level goals, omitting raw controls and local details. This reduces annotation cost, but agreement with automatically generated labels should not be mistaken for independently verified human intent.

Table 1: Statistics of GameHorizon-Data across 𝟐𝟏\bm{21} game titles. We report the duration and share of recordings, counts of actions and instructions, and the average temporal span of each distinct instruction. Games are sorted by duration. Valid rate denotes the fraction retained for instruction annotation after filtering. Actions are reported in units of 10410^{4}.
Game Title Recording Duration and Share Annotation Counts Avg. Instruction Span (s)
Total (h) Share (%) Valid Rate (%) Valid (h) Actions (𝟏𝟎𝟒\bm{10^{4}}) 𝑳𝟏\bm{L_{1}} 𝑳𝟐\bm{L_{2}} 𝑳𝟑\bm{L_{3}} 𝑳𝟏\bm{L_{1}} 𝑳𝟐\bm{L_{2}} 𝑳𝟑\bm{L_{3}}
Valorant 617.5617.5 12.3512.35 86.486.4 533.8533.8 5​@​8115{,}811 705​@​380705{,}380 23​@​11323{,}113 5​@​7785{,}778 2.722.72 83.183.1 332.5332.5
Minecraft 589.8589.8 11.8011.80 84.584.5 498.3498.3 5​@​0875{,}087 448​@​166448{,}166 21​@​71521{,}715 5​@​4295{,}429 4.004.00 82.682.6 330.4330.4
Grand Theft Auto V 566.1566.1 11.3211.32 86.586.5 489.9489.9 3​@​5853{,}585 382​@​302382{,}302 21​@​64421{,}644 5​@​4115{,}411 4.614.61 81.581.5 325.9325.9
Palworld 438.7438.7 8.778.77 83.583.5 366.5366.5 3​@​7413{,}741 329​@​615329{,}615 15​@​97115{,}971 3​@​9933{,}993 4.004.00 82.682.6 330.4330.4
Delta Force 337.0337.0 6.746.74 85.185.1 286.9286.9 3​@​1233{,}123 379​@​091379{,}091 12​@​42212{,}422 3​@​1053{,}105 2.722.72 83.183.1 332.5332.5
Roco Kingdom: World 329.2329.2 6.586.58 91.991.9 302.4302.4 2​@​6772{,}677 786​@​446786{,}446 13​@​05413{,}054 3​@​2633{,}263 1.381.38 83.483.4 333.6333.6
Red Dead Redemption 2 304.4304.4 6.096.09 85.985.9 261.6261.6 1​@​9921{,}992 202​@​428202{,}428 11​@​55311{,}553 2​@​8882{,}888 4.654.65 81.581.5 326.1326.1
Cyberpunk 2077 301.2301.2 6.026.02 87.987.9 264.7264.7 2​@​5742{,}574 192​@​222192{,}222 11​@​64611{,}646 2​@​9122{,}912 4.964.96 81.881.8 327.3327.3
Genshin Impact 232.6232.6 4.654.65 90.490.4 210.4210.4 1​@​8621{,}862 547​@​134547{,}134 9​@​0829{,}082 2​@​2702{,}270 1.381.38 83.483.4 333.6333.6
PUBG: Battlegrounds 197.4197.4 3.953.95 84.684.6 167.0167.0 1​@​8181{,}818 220​@​726220{,}726 7​@​2337{,}233 1​@​8081{,}808 2.722.72 83.183.1 332.5332.5
Elden Ring Nightreign 168.9168.9 3.383.38 90.090.0 152.0152.0 1​@​3481{,}348 250​@​877250{,}877 6​@​6306{,}630 1​@​6581{,}658 2.182.18 82.682.6 330.2330.2
The Witcher 3: Wild Hunt 158.7158.7 3.173.17 89.489.4 141.8141.8 1​@​3231{,}323 287​@​433287{,}433 6​@​2906{,}290 1​@​5731{,}573 1.781.78 81.281.2 324.6324.6
Elden Ring 135.9135.9 2.722.72 92.692.6 125.9125.9 1​@​1161{,}116 207​@​706207{,}706 5​@​4895{,}489 1​@​3721{,}372 2.182.18 82.682.6 330.2330.2
Escape from Tarkov 119.8119.8 2.402.40 81.781.7 97.997.9 1​@​0651{,}065 129​@​320129{,}320 4​@​2374{,}237 1​@​0591{,}059 2.722.72 83.183.1 332.5332.5
Apex Legends 116.5116.5 2.332.33 85.385.3 99.499.4 1​@​0821{,}082 131​@​381131{,}381 4​@​3054{,}305 1​@​0761{,}076 2.722.72 83.183.1 332.5332.5
Wuthering Waves 108.4108.4 2.172.17 89.189.1 96.596.5 819819 294​@​088294{,}088 4​@​0774{,}077 1​@​0191{,}019 1.181.18 85.285.2 341.0341.0
Neverness to Everness 101.3101.3 2.032.03 89.389.3 90.490.4 800800 235​@​145235{,}145 3​@​9033{,}903 976976 1.381.38 83.483.4 333.6333.6
Assassin’s Creed 51.151.1 1.021.02 87.087.0 44.544.5 338338 34​@​40134{,}401 1​@​9631{,}963 491491 4.654.65 81.581.5 326.1326.1
Black Myth: Wukong 51.151.1 1.021.02 91.391.3 46.646.6 414414 76​@​97176{,}971 2​@​0342{,}034 509509 2.182.18 82.682.6 330.2330.2
Watch Dogs 2 38.538.5 0.770.77 85.085.0 32.732.7 249249 25​@​32425{,}324 1​@​4451{,}445 361361 4.654.65 81.581.5 326.1326.1
Honor of Kings: World 35.935.9 0.720.72 87.387.3 31.331.3 277277 81​@​42981{,}429 1​@​3521{,}352 338338 1.381.38 83.483.4 333.6333.6
Total / Avg. 5​@​0005{,}000 100.00100.00 86.886.8 4​@​3414{,}341 41​@​10341{,}103 5​@​947​@​5885{,}947{,}588 189​@​158189{,}158 47​@​29047{,}290 2.632.63 82.682.6 330.4330.4
Table 2: Comparison with existing gameplay datasets. Large-scale denotes at least 3​@​0003{,}000 hours of gameplay. AAA-focused indicates that more than half of the included game titles are AAA games. Direct human actions refer to cases where the majority of action labels are recorded directly from human players rather than inferred, extracted, or generated.
Dataset Data Scope Annotation Properties Notes
Large Scale AAA- Focused Direct Human Actions Instruction Annotations Dense Instructions Multi- Horizon
Single-Game Datasets
MineRL (Guss et al., 2019) ✗ ✗ ✓ ✗ ✗ ✗ Minecraft; human actions; no textual instructions.
MineDojo (Fan et al., 2022) ✓ ✗ ✗ ✗ ✗ ✗ Web Minecraft videos; no actions or instructions.
VPT (Baker et al., 2022) ✓ ✗ ✗ ✗ ✗ ✗ Web Minecraft videos; IDM actions; no instructions.
STEVE-1 (Lifshitz et al., 2023) ✗ ✗ ✓ ✓ ✗ ✗ Minecraft; only 10K short-term text instructions.
PLAICraft (He et al., 2025) ✓ ✗ ✓ ✗ ✗ ✗ Minecraft; human actions; no textual instructions.
WildWorld (Li et al., 2026) ✗ ✓ ✗ ✗ ✗ ✗ Monster Hunter Wilds; AI actions; no instructions.
EgoCS-400K (Guo et al., 2026) ✓ ✓ ✗ ✗ ✗ ✗ CS; replay-derived action labels; no instructions.
Multi-Game Datasets
NitroGen (Magne et al., 2026) ✓ ✗ ✗ ✗ ✗ ✗ Gamepad-overlay actions without text instructions.
Open-P2P (Yue et al., 2026) ✓ ✗ ✓ ✓ ✗ ✗ Majority non-AAA games; sparse text instructions.
D2E (Choi et al., 2026) ✗ ✗ ✗ ✗ ✗ ✗ Mostly IDM-inferred actions; no text instructions.
Gaming-500-hours (Markov AI, 2026) ✗ ✗ ✓ ✗ ✗ ✗ Majority non-AAA games; no textual instructions.
GameHorizon-Data (Ours) ✓ ✓ ✓ ✓ ✓ ✓ AAA; human actions; dense multi-horizon instructions.

3.3 GameHorizon-Data

The corpus includes 411.03 million action events. Filtering removes uninformative or low-quality footage before language annotation, leaving 4,341 hours, or 86.8% of the recordings; controls remain available for all 5,000 hours. The retained footage has 5,947,588 operation instructions, 189,158 goals, and 47,290 strategies. Mean spans are 2.63, 82.6, and 330.4 seconds, respectively.

Game representation is uneven: Valorant contributes 617.5 hours, while the smallest title contributes 35.9 hours. Each retained frame shares an instruction at each horizon, so millions of frame-aligned labels are not millions of independent human annotations.

Refer to caption
Figure 4: The offline track of GameHorizon-Bench. It comprises three primary tasks and ten variant tasks. Given sampled frames and the short-horizon instruction L1L_{1}, single-horizon action T1T_{1} requires models to determine the correct action sequence. Multi-horizon decomposition T2T_{2} evaluates whether models can decompose a medium-horizon goal L2L_{2} into the correct sequence of short-horizon operations L1L_{1}. Cross-horizon consistency T3T_{3} assesses the overall consistency across frames, L1L_{1}–L3L_{3} instructions, and actions. Variant tasks T∗T^{*} provide additional diagnostics and insights into different gameplay abilities. Due to space constraints, L1L_{1} instructions are abridged in T2T_{2}. Only the correct option appears in T3T_{3}.

3.4 GameHorizon-Bench

The offline track has three four-choice tasks: match observations and an immediate instruction to controls; select the operation sequence for a medium-level goal; and identify a consistent combination of video, instructions, and actions. Ten diagnostic variants vary the time horizon, available language, abstraction direction, and inclusion of controls.

The online track contains ten causal tasks with ordered steps and ten thematic tasks with flexible step order, comprising 62 subtasks in Minecraft. After a failed step, the environment is reset to its success state so later steps can still be diagnosed. A long task passes only if every step succeeds without such assistance; subtask scores and end-to-end success therefore measure different things.

Refer to caption
Figure 5: The online track of GameHorizon-Bench. It contains long-horizon causal and thematic tasks, each comprising multiple verifiable short-horizon subtasks. The causal tasks can only be completed in a prescribed order because of dependencies between successive subtasks. The thematic tasks contain subtasks that share a common theme but can be performed in any order. The figure shows results from Gemini 3.6 Flash (Google DeepMind, 2026b), with arrows indicating its actual execution order. After a subtask fails, the environment is reset to the corresponding success state so that evaluation can continue. A long-horizon task is considered passed only when all constituent subtasks succeed.
Table 3: Comparison with existing gameplay benchmarks. Model diversity denotes evaluation across at least three model paradigms, e.g., VLMs, UMMs, GUI, coding, and game agents. Our GameHorizon-Bench tests 4747 models of five types.
Benchmark Benchmark Scope Annotation Properties Evaluation Settings Notes
Model Diversity AAA- Focused Direct Human Actions Multi- Horizon Reproducible Offline Stepwise Online
Single-Game Benchmarks
MineDojo (Fan et al., 2022) ✗ ✗ ✗ ✗ ✗ ✗ Online success rates in Minecraft; the game agents only.
MCU (Zheng et al., 2025) ✗ ✗ ✗ ✗ ✗ ✗ Minecraft; VLM-based online scores; game agents only.
MineExplorer (Ju et al., 2026) ✗ ✗ ✗ ✗ ✗ ✓ Online milestone rates in Minecraft; the VLMs only.
Multi-Game Benchmarks
BALROG (Paglieri et al., 2025) ✗ ✗ ✗ ✗ ✗ ✓ Non-AAA; the online progress scores; LLMs and VLMs.
VideoGameBench (Zhang et al., 2025) ✗ ✗ ✗ ✗ ✗ ✗ Retro and classic games; online scores and VLMs only.
Orak (Park et al., 2026) ✗ ✗ ✗ ✗ ✗ ✗ Majority non-AAA; the online scores; LLMs and VLMs.
GameWorld (Ouyang et al., 2026) ✗ ✗ ✗ ✗ ✗ ✗ Browser mini-games only; online scores; VLM agents.
GameVerse (Zhang et al., 2026) ✗ ✗ ✗ ✗ ✗ ✓ Mostly non-AAA; online milestone scores; VLM agents.
GameHorizon-Bench (Ours) ✓ ✓ ✓ ✓ ✓ ✓ AAA; reproducible offline; stepwise online; diverse models.

4 Experiments

The experiments compare question-answering models, specialist game policies, reasoning settings, and online execution. All reported rankings concern the versions and configurations tested in this paper.

4.1 Evaluation Protocols and Details

Each primary task has 1,000 questions, and each of ten variants has 200: 5,000 questions altogether. Of 47 models, 44 answer questions directly. Three specialist game agents are evaluated only on action matching: their predicted control sequences are aligned to the four options, and predictions matching none are marked wrong. This interface differs from direct multiple-choice answering. [1], [2], [15], [50].

The online study selects three models from each of four offline performance tiers. Every subtask receives up to 400 model calls, with automatic success checks against Minecraft state. Diagnostic variants use four models, limiting conclusions about the wider model population.

4.2 Offline Primary Tasks

Across the 44 question-answering models, mean accuracy is 64.7%, ranging from 44.6% to 80.2%; chance is 25%. Mean scores are 57.3% for action matching, 65.1% for goal decomposition, and 71.6% for cross-horizon consistency. GPT-6-Astra leads the reported aggregate ranking. These are recognition tasks with supplied alternatives, not free-form action rollouts.

Specialist game agents score 22.0–28.1% on unseen games; a Minecraft-tuned JARVIS-VLA reaches 54.2% on its in-domain comparison. Predictions with no option overlap can score below chance. Differences in training domain, model size, and output interface prevent attributing all gaps to one architectural choice.

Thinking improves two tested configurations by 5.6 and 9.3 percentage points but reduces two smaller configurations by 8.2 and 5.9 points. The authors associate some declines with hallucinated reasoning. Comparisons suggesting interference from multimodal unification or computer-use tuning are observational model comparisons, rather than controlled causal ablations.

Table 4: Offline results of the primary tasks T𝟏\bm{T_{1}}–T𝟑\bm{T_{3}}. The table reports results for general VLMs, UMMs, GUI agents, and coding agents with question-answering abilities. Accuracies are reported as percentages. Overall denotes the mean accuracy across three tasks. Models are ordered from higher to lower performance and divided into four tiers. The best and second-best metrics are marked in bold and underlined. The final row reports the average accuracy for all models.
Tier 1 Tier 2
Rank Model 𝑻𝟏\bm{T_{1}} 𝑻𝟐\bm{T_{2}} 𝑻𝟑\bm{T_{3}} Overall Rank Model 𝑻𝟏\bm{T_{1}} 𝑻𝟐\bm{T_{2}} 𝑻𝟑\bm{T_{3}} Overall
 1 GPT-6-Astra 69.4 79.6 91.5 80.2 12 Doubao-Seed-2.0-Pro 59.2 74.9 80.0 71.4
 2 Gemini 3.8 Flash 65.9 81.2 84.8 77.3 13 Claude Fable 5 62.9 72.7 78.1 71.2
 3 Gemini 3.7 Flash 66.2 80.1 83.9 76.7 14 GPT-5.6 Terra 62.1 75.6 74.4 70.7
4 Gemini 3.6 Flash 62.3 79.1 84.6 75.3 15 Qwen3.7-Plus 58.8 75.9 71.5 68.7
5 GPT-5.6 Sol 65.3 77.5 81.5 74.8 16 Qwen3.8-27B 59.9 77.4 68.1 68.5
6 Kimi-K3 64.4 79.3 79.7 74.5 17 GPT-5.6 Luna 61.6 70.7 71.1 67.8
7 Gemini 3.5 Flash 64.9 77.4 80.8 74.4 17 Kimi-K2.6 61.2 73.2 69.0 67.8
8 GPT-5.5 64.6 78.9 76.2 73.2 19 Doubao-Seed-2.0-Lite 55.2 69.0 76.3 66.8
9 Gemini 3.1 Pro 64.5 75.8 78.2 72.8 20 Gemma 4 31B-IT 53.8 68.6 76.6 66.3
10 Doubao-Seed-2.1-Turbo 63.1 79.4 75.5 72.7 21 MiniMax-M3 57.8 68.5 70.6 65.6
11 Doubao-Seed-2.1-Pro 62.9 76.7 76.6 72.1 22 Qwen3-VL-235B-A22B-Thinking 58.2 71.6 65.9 65.2
Tier 3 Tier 4
Rank Model 𝑻𝟏\bm{T_{1}} 𝑻𝟐\bm{T_{2}} 𝑻𝟑\bm{T_{3}} Overall Rank Model 𝑻𝟏\bm{T_{1}} 𝑻𝟐\bm{T_{2}} 𝑻𝟑\bm{T_{3}} Overall
23 Claude Opus 4.8 59.1 62.1 73.1 64.8 34 Qwen3.6-35B-A3B 53.3 52.9 66.3 57.5
24 Claude Sonnet 5 54.4 57.7 82.0 64.7 35 Qwen3-Omni-30B-A3B-Instruct 52.5 55.6 63.7 57.3
25 Step-3.7-Flash 57.5 71.3 63.4 64.1 36 GELab-Zero-4B-Preview 51.8 46.8 72.7 57.1
26 Qwen3-VL-235B-A22B-Instruct 55.1 62.1 73.1 63.4 37 Qwen2.5-VL-7B-Instruct 52.0 54.3 63.7 56.7
27 GLM-5V-Turbo 53.4 65.4 70.8 63.2 38 InternVL3.5-8B 51.8 50.2 65.5 55.8
28 GPT-5.2 58.9 58.9 70.6 62.8 39 GLM-4.1V-9B-Thinking 54.9 54.8 55.3 55.0
29 Qwen3.5-397B-A17B 53.3 60.1 73.5 62.3 40 GPT-4o 55.7 46.3 61.3 54.4
30 BAGEL-7B-MoT 51.7 57.3 75.1 61.4 41 UI-TARS-1.5-7B 49.1 47.1 62.9 53.0
31 Qwen2.5-VL-32B-Instruct 54.8 58.0 69.7 60.8 42 Ovis-U1-3B 44.7 39.6 59.2 47.8
32 Step3-VL-10B 55.4 64.7 61.7 60.6 43 InternVL3.5-2B 45.1 40.3 52.7 46.0
33 SenseNova-U1-8B-MoT 48.2 59.2 67.5 58.3 44 InternVL-U-4B 46.1 37.9 49.8 44.6
Average over all models: 𝑻𝟏=57.3\bm{T_{1}=57.3}  𝑻𝟐=65.1\bm{T_{2}=65.1}  𝑻𝟑=71.6\bm{T_{3}=71.6}  Overall =64.7\bm{=64.7}
Table 5: Offline results of game agents. We evaluate NitroGen (Magne et al., 2026), Open-P2P (Yue et al., 2026), and JARVIS-VLA (Li et al., 2025), with Gemma 4 31B-IT as a reference. Game agents are trained to predict actions without question answering or reasoning abilities. We test them only on the single-horizon action task T1T_{1}. In-domain denotes fine-tuning on the dataset, e.g., JARVIS-VLA on VPT (Baker et al., 2022). Zero-shot indicates the opposite.
Model 𝑻𝟏\bm{T_{1}} In-Domain Zero-Shot
NitroGen 22.0 ✗ ✓
Open-P2P 24.4 ✗ ✓
JARVIS-VLA 28.1 ✗ ✓
(a) GameHorizon-Bench (AAA games)
Model 𝑻𝟏\bm{T_{1}} In-Domain Zero-Shot
Open-P2P 26.8 ✗ ✓
Gemma 4 31B-IT 54.1 ✗ ✓
JARVIS-VLA 54.2 ✓ ✗
(b) VPT (Minecraft)
Table 6: Effects of thinking on offline results. Δ\Delta denotes the change in overall accuracy. For lightweight models, thinking often induces hallucinations, leading to performance degradation. More capable models can leverage thinking to plan and decompose complex goals in gameplay, thus improving the accuracy, e.g., GLM-5V-Turbo and Doubao-Seed-2.1-Pro.
Model Thinking 𝑻𝟏\bm{T_{1}} 𝑻𝟐\bm{T_{2}} 𝑻𝟑\bm{T_{3}} Overall 𝚫\bm{\Delta}
UI-TARS-1.5-7B ✗ 49.1 47.1 62.9 53.0 –
✓ 45.2 45.4 43.9 44.8 -8.2
Qwen2.5-VL-7B ✗ 52.0 54.3 63.7 56.7 –
✓ 44.4 52.2 55.7 50.8 -5.9
(a) Degradation with Thinking
Model Thinking 𝑻𝟏\bm{T_{1}} 𝑻𝟐\bm{T_{2}} 𝑻𝟑\bm{T_{3}} Overall 𝚫\bm{\Delta}
GLM-5V-Turbo ✗ 53.2 56.4 63.1 57.6 –
✓ 53.4 65.4 70.8 63.2 +5.6
Doubao-Seed-2.1-Pro ✗ 56.4 56.8 75.2 62.8 –
✓ 62.9 76.7 76.6 72.1 +9.3
(b) Improvement with Thinking

4.3 Offline Variant Tasks

Across four models, current-action recognition exceeds future-action planning. Short instructions raise recognition from 44.5% to 58.6%; adding higher-level instructions reaches 60.0%. Future planning changes from 34.6% with frames alone to 34.8% with short instructions and 41.8% with all horizons: a 7.2-point gain over frames alone, or 7.0 points beyond short instructions.

Bottom-up abstraction scores 98.4%, compared with 69.5% for top-down decomposition. Rich low-level evidence makes selecting a higher-level goal easier, but near-ceiling accuracy limits this variant’s discrimination. Adding action sequences to cross-horizon options changes accuracy from 74.0% to 73.4%; this small effect is specific to the tested questions and models.

Table 7: Results of the offline variant tasks T𝟏∗\bm{T_{1}^{*}}. The table presents six variants of the primary task T1T_{1}, including current-action perception and future-action planning under three input settings, i.e., frames only, frames with short-horizon instructions, and frames with multi-horizon instructions. Considering the evaluation costs, we test four models using 200200 questions for each variant. Values outside parentheses denote accuracy, while parenthesized values denote changes in percentage points relative to the preceding column. The final row reports the average results across all four models.
Model Current-Action Perception Future-Action Planning
Frames Only + Short + Multi-Horizon Frames Only + Short + Multi-Horizon
Qwen2.5-VL-32B-Instruct 39.0 53.5 (+14.5+14.5) 54.5 (+1.0+1.0) 33.5 29.5 (−4.0-4.0) 33.5 (+4.0+4.0)
Qwen3-VL-235B-A22B-Instruct 41.0 52.0 (+11.0+11.0) 54.5 (+2.5+2.5) 37.0 34.0 (−3.0-3.0) 39.5 (+5.5+5.5)
Gemini 3.1 Pro 50.0 67.0 (+17.0+17.0) 67.0 (+0.0+0.0) 34.0 36.5 (+2.5+2.5) 46.5 (+10.0+10.0)
Gemini 3.5 Flash 48.0 62.0 (+14.0+14.0) 64.0 (+2.0+2.0) 34.0 39.0 (+5.0+5.0) 47.5 (+8.5+8.5)
Overall 44.5 58.6 (+14.1+14.1) 60.0 (+1.4+1.4) 34.6 34.8 (+0.2+0.2) 41.8 (+7.0+7.0)
Table 8: Offline variant tasks T𝟐∗\bm{T_{2}^{*}} and T𝟑∗\bm{T_{3}^{*}}. The left table compares top-down decomposition from L2L_{2} to L1L_{1} with bottom-up abstraction from L1L_{1} to L2L_{2}. The right table quantifies the accuracy loss from action decoding in cross-horizon consistency.
Model Top-Down Decomposition Bottom-Up Abstraction
Qwen2.5-VL-32B-Instruct 61.0 98.0 (+37.0+37.0)
Qwen3-VL-235B-A22B-Instruct 61.5 98.0 (+36.5+36.5)
Gemini 3.1 Pro 76.5 99.0 (+22.5+22.5)
Gemini 3.5 Flash 79.0 98.5 (+19.5+19.5)
Overall 69.5 98.4 (+28.9+28.9)
(a) Goal Decomposition and Abstraction (T𝟐∗\bm{T_{2}^{*}})
Model 𝑳𝟑−𝑳𝟐−𝑳𝟏\bm{L_{3}\!-\!L_{2}\!-\!L_{1}} w/o Actions 𝑳𝟑−𝑳𝟐−𝑳𝟏\bm{L_{3}\!-\!L_{2}\!-\!L_{1}} w/ Actions
Qwen2.5-VL-32B-Instruct 69.0 68.5 (−0.5-0.5)
Qwen3-VL-235B-A22B-Instruct 72.0 71.0 (−1.0-1.0)
Gemini 3.1 Pro 77.5 77.5 (+0.0+0.0)
Gemini 3.5 Flash 77.5 76.5 (−1.0-1.0)
Overall 74.0 73.4 (−0.6-0.6)
(b) Cross-Horizon Consistency (T𝟑∗\bm{T_{3}^{*}})

4.4 Stepwise Online Track

The twelve sampled models preserve their broad offline tier ordering online. The highest reported end-to-end score is 45%, or nine of twenty tasks; ten models complete at most two tasks. Subtask resets expose where failures occur without inflating the all-steps-success measure.

This association supports offline scores as a useful diagnostic for the tested Minecraft tasks. It does not establish general validity across the 21 offline titles, arbitrary agent harnesses, or unrestricted long-horizon gameplay; the online task set is small and models were sampled by offline tier.

Table 9: Online results of GameHorizon-Bench. Due to the high evaluation costs, we test 1212 models in our online track, with three models per offline performance tier. Each entry reports the success rate as a percentage, followed by the number of passed tasks out of the total in parentheses. The offline and online rankings show a clear positive association.
Tier 1 Tier 2
Online Model Offline Short-Horizon Long-Horizon Causal Thematic Online Model Offline Short-Horizon Long-Horizon Causal Thematic
Rank Rank Subtasks Tasks Rank Rank Subtasks Tasks
 1 GPT-6-Astra 1 66.1 (41/62) 45.0 (9/20) 40.0 (4/10) 50.0 (5/10) 4 GPT-5.6 Terra 14 37.1 (23/62) 10.0 (2/20) 20.0 (2/10) 0.0 (0/10)
 2 Gemini 3.6 Flash 4 56.5 (35/62) 30.0 (6/20) 30.0 (3/10) 30.0 (3/10) 5 GPT-5.6 Luna 17 33.9 (21/62) 10.0 (2/20) 10.0 (1/10) 10.0 (1/10)
 3 Kimi-K3 6 46.8 (29/62) 10.0 (2/20) 20.0 (2/10) 0.0 (0/10) 6 MiniMax-M3 21 29.0 (18/62) 10.0 (2/20) 10.0 (1/10) 10.0 (1/10)
Tier 3 Tier 4
Online Model Offline Short-Horizon Long-Horizon Causal Thematic Online Model Offline Short-Horizon Long-Horizon Causal Thematic
Rank Rank Subtasks Tasks Rank Rank Subtasks Tasks
7 GLM-5V-Turbo 27 27.4 (17/62) 5.0 (1/20) 10.0 (1/10) 0.0 (0/10) 10 Qwen3.6-35B-A3B 34 11.3 (7/62) 5.0 (1/20) 10.0 (1/10) 0.0 (0/10)
8 Qwen3.5-397B-A17B 29 19.4 (12/62) 5.0 (1/20) 10.0 (1/10) 0.0 (0/10) 11 InternVL3.5-8B 38 3.2 (2/62) 0.0 (0/20) 0.0 (0/10) 0.0 (0/10)
9 Step3-VL-10B 32 14.5 (9/62) 5.0 (1/20) 10.0 (1/10) 0.0 (0/10) 12 UI-TARS-1.5-7B 41 1.6 (1/62) 0.0 (0/20) 0.0 (0/10) 0.0 (0/10)

5 Conclusion

GameHorizon provides aligned controls and multi-horizon language plus complementary recognition and execution tests. Its results motivate better future planning and goal decomposition. Remaining evaluation limits include model-generated instruction targets, multiple-choice recognition, restricted diagnostic model samples, and online coverage confined to Minecraft. The paper states an intention to release the resources. Project · Repository.

Appendix A More Details on GameHorizon-Annotator

A.1 Annotation Prompts

The three original prompts are preserved below. All include video metadata and require a concise instruction as output. Immediate prompts retain operational details; medium-level prompts combine lower-level instructions into a sustained goal; long-level prompts omit controls and keybinds to emphasize strategy.

Refer to caption
Figure 6: The prompt for short-horizon operations. Key differences across horizons are highlighted in red, such as the inputs, outputs, and required levels of detail in the instructions. Blue text denotes the code variables in the prompt. The short-horizon prompt takes sampled frames and temporally aligned keyboard-mouse actions as inputs. It requires the VLM to describe the current operation with sufficient details, such as the object descriptions and spatial relations.
Refer to caption
Figure 7: The prompt for medium-horizon goals. Key differences across the three horizons are highlighted in red, such as the inputs, outputs, and required levels of detail in the instructions. Blue text denotes the code variables in the prompt. The medium-horizon prompt incorporates video frames, aligned actions, and the constituent short-horizon instructions, requiring the VLM to summarize the sustained goal of the player while avoiding excessive local details.
Refer to caption
Figure 8: The prompt for long-horizon strategies. Key differences across three horizons are highlighted in red, such as the inputs, outputs, and required levels of detail in the instructions. Blue text denotes the code variables in the prompt. The long-horizon prompt takes sampled frames and the constituent medium-horizon goals as inputs while omitting actions and keybinds. It requires the VLM to capture the overall gameplay strategy without local operational details.

A.2 Action-Aware Video Segmentation

Segmentation calibrates input semantics, classifies actions as deterministic, ambiguous, or visually dependent, and reconstructs frame-aligned held keys, triggers, mouse states, and camera motion. Priority rules choose a primary action while retaining concurrent evidence. It merges repeated labels, smooths brief noise, splits long segments, and uses a VLM to refine ambiguous boundaries and remove temporal gaps or overlaps.

A.3 Game-Specific Keybinds

The original Apex Legends and Cyberpunk 2077 keybind tables retain all event-to-action mappings, including inputs with multiple meanings. Context-dependent mappings explain why raw control labels alone cannot always determine the intended action.

Table 10: The keybind for Apex Legends. Keyboard and mouse inputs are mapped to their corresponding in-game action semantics via game-specific keybinds. Multiple candidate semantics for an ambiguous input are separated by slashes.
Input Event In-Game Action Semantics
KEY_W Move Forward
KEY_S Move Back
KEY_A Move Left
KEY_D Move Right
SPACE Jump
LSHIFT Sprint
KEY_C Crouch – Toggle
LCONTROL Crouch – Hold
RI_MOUSE_LEFT_BUTTON_DOWN Attack
RI_MOUSE_RIGHT_BUTTON_DOWN Aim Down Sight
KEY_R Reload
KEY_V Melee
KEY_1 Equip Weapon 1
KEY_2 Equip Weapon 2
KEY_3 Holster Weapons
RI_MOUSE_WHEEL Cycle Weapon
KEY_G Equip Grenade
KEY_N Inspect Weapon
KEY_Q Tactical Ability
KEY_Z Ultimate Ability
KEY_E Interact / Pickup / Open Door / Revive / Use Zipline
KEY_4 Use Selected Health Item
KEY_H Character Utility Action
RI_MOUSE_MIDDLE_BUTTON_DOWN Smart Tag / Middle Mouse Button
TAB Open/Close Inventory
KEY_M Open/Close Map
Table 11: The keybind for Cyberpunk 2077. Keyboard and mouse inputs are mapped to their corresponding in-game action semantics via game-specific keybinds. Multiple candidate semantics for an ambiguous input are separated by slashes.
Input Event In-Game Action Semantics
KEY_W Move Forward / Vehicle Accelerate
KEY_S Move Back / Vehicle Brake or Reverse
KEY_A Move Left / Vehicle Turn Left
KEY_D Move Right / Vehicle Turn Right
SPACE Jump / Vehicle Handbrake / Pause Braindance
LSHIFT Sprint – Toggle / Change Braindance Layer
RSHIFT Sprint – Hold
KEY_C Crouch – Toggle / Skip Dialogue
LCONTROL Crouch – Hold / Vehicle Horn / Panzer Smokescreen
RI_MOUSE_LEFT_BUTTON_DOWN Shoot / Fast Attack / Panzer Primary Cannon
RI_MOUSE_RIGHT_BUTTON_DOWN Aim / Block / Panzer Missile Launcher
RI_MOUSE_MIDDLE_BUTTON_DOWN Use Combat Gadget / Tag / Reverse Camera / Braindance Zoom
RI_MOUSE_WHEEL Cycle Weapon / Next or Previous Weapon / Braindance Zoom
KEY_R Reload / Vehicle Radio / Restart Braindance
KEY_Q Quick Attack / Vehicle Cycle Camera / Rewind Braindance
KEY_E Cyberware Systems / Fast-Forward Braindance
KEY_F Interact / Pickup / Enter or Exit Vehicle
KEY_X Use Consumable / Exit Braindance
KEY_Y Skip Dialogue / Dialogue Confirm
KEY_Z Open Notifications / Quickhack Details
KEY_T Open Phone
TAB Scan – Hold / Braindance Analysis Mode
KEY_1 Select First Weapon / Previous Item
KEY_2 Select Second Weapon
KEY_3 Select Third Weapon / Next Item
LMENU Cycle Weapon / Weapon Wheel / Vehicle Cycle Lights
KEY_V Call Vehicle
KEY_B Draw or Withdraw Weapon
KEY_I Open Main Menu / Inventory
KEY_M Open Map
KEY_J Open Journal
KEY_K Open Crafting
ESCAPE Pause Menu / Close Menu

Appendix B More Results on GameHorizon-Bench

The appendix preserves additional online examples and all twenty task definitions with their 62 subtasks. Causal tasks require the prescribed sequence; thematic tasks allow flexible order. The examples show actual execution order and resets after failures, while a task counts as completed only when all its steps succeed.

Refer to caption
Figure 9: The online track of GameHorizon-Bench. It contains long-horizon causal and thematic tasks, each comprising multiple verifiable short-horizon subtasks. The causal tasks can only be completed in a prescribed order because of dependencies between successive subtasks. The thematic tasks contain subtasks that share a common theme but can be performed in any order. The figure shows results from Gemini 3.6 Flash [Google DeepMind, 2026b], with arrows indicating its actual execution order. After a subtask fails, the environment is reset to the corresponding success state so that evaluation can continue. A long-horizon task is considered passed only when all constituent subtasks succeed.
Table 12: Long-horizon online tasks and the constituent short-horizon subtasks. The online track contains 2020 long-horizon tasks and 6262 short-horizon subtasks. The causal tasks impose a fixed order, whereas thematic tasks allow any order.
No. Long-Horizon Task Steps Short-Horizon Subtasks
Causal Tasks
1 Craft a wooden axe. 2 (1)Chop down trees and gather wood logs outside. (2)Use the crafting table to craft a wooden axe and retrieve it.
2 Craft an armor stand from workshop materials. 5 (1)Craft four oak planks from an oak log. (2)Craft eight sticks from the oak planks. (3)Smelt three stone blocks into smooth stone. (4)Craft six smooth stone slabs. (5)Craft an armor stand from the sticks and a smooth stone slab.
3 Craft an item frame from workshop materials. 3 (1)Craft eight oak planks from two oak logs. (2)Craft eight sticks from four oak planks. (3)Craft an item frame from eight sticks and one leather.
4 Retrieve shears, shear a sheep, and store the wool in the chest. 3 (1)Retrieve the shears from the chest. (2)Shear the sheep in the nearby pen. (3)Store the wool in the chest.
5 Fill the remaining bucket with milk, bake a cake, and place it on the dining table. 3 (1)Fill the remaining empty bucket with milk from the cow. (2)Craft a cake at the crafting table. (3)Place the cake on the marked dining table.
6 Repair the gap in the animal pen. 2 (1)Select the oak fence from the hotbar. (2)Place the oak fence to close the gap in the pen.
7 Mine the missing obsidian block, repair the ruined portal, and ignite it. 3 (1)Mine the loose obsidian block using the diamond pickaxe. (2)Place the obsidian in the missing portal-frame position. (3)Ignite the completed Nether portal with flint and steel.
8 Collect lava using a bucket. 2 (1)Select the bucket from the hotbar. (2)Scoop up lava from the pool using the empty bucket.
9 Mine iron, forge an iron sword, and use it to defeat the zombie. 4 (1)Mine two iron ore blocks using the stone pickaxe. (2)Smelt the iron ore into two iron ingots. (3)Craft an iron sword. (4)Kill the zombie using the iron sword.
10 Select the golden sword and use it to kill the spider. 2 (1)Select the golden sword from the hotbar. (2)Kill the spider using the golden sword.
Thematic Tasks
1 Complete the target house’s exterior wall using the finished house beside it as a reference. 6 (1)Place a stripped oak log in the ground-row left gap. (2)Place a stone brick block in the ground-row center gap. (3)Place a birch plank in the ground-row right gap. (4)Place a dark oak plank in the second-row left gap. (5)Place a glass block in the second-row center gap. (6)Place a spruce plank in the second-row right gap.
2 Eliminate the zombie and shear the sheep. 2 (1)Kill the zombie inside the animal pen. (2)Shear the sheep in the animal pen.
3 Store items from the inventory into the chest. 2 (1)Store the iron ingots in the chest. (2)Store the bread in the chest.
4 Assemble a mixed-material toolkit at the workshop. 3 (1)Craft an iron pickaxe. (2)Craft a wooden axe. (3)Craft a diamond sword.
5 Secure and light the tunnel to mine coal and iron. 4 (1)Fill a gap in the passage using cobblestone. (2)Place a torch in the mine. (3)Mine coal ore. (4)Mine iron ore.
6 Complete routine farm chores by gathering food, planting crops, and managing livestock. 5 (1)Milk the cow using an empty bucket. (2)Pick up the egg lying on the ground. (3)Plant wheat seeds on the farmland. (4)Kill the sheep using the diamond sword. (5)Kill the chicken using the iron sword.
7 Collect raw blocks for building. 2 (1)Dig dirt blocks using an iron shovel. (2)Mine stone blocks using a diamond pickaxe.
8 Process building materials and food at the shelter. 2 (1)Smelt cobblestone in the furnace. (2)Cook raw mutton in the smoker.
9 Assemble a wooden toolkit at the workshop. 3 (1)Craft a wooden pickaxe. (2)Craft a wooden axe. (3)Craft a wooden shovel.
10 Prepare the travel campsite by setting up the sleeping area, lighting the entrance, cooking food, and storing navigation equipment. 4 (1)Place the red bed on the wooden sleeping pad inside the tent. (2)Place a torch on the marked block at the campsite entrance. (3)Cook the raw cod in the smoker. (4)Store the compass in the supply chest.
Concept demonstration

Open image ↗