Literature
Display
← All papers

Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents

Suman Navaratnarajah · Taehyoung Kim · Jona Ruthardt · Ishaan Bhimwal · Ryousuke Yamada · Yannik Blei · Wolfram Burgard · Yuki M Asano

Preprint posted · arXiv (preprint) ↗

Robotics14 min text · CondensedOriginal paper ↗
Contents

Overview

Complete original PDF block S0.F1
Figure 1: MissionBench evaluates MLLMs on mission-level aerial reasoning across four task families: Reporting, Inspection, Manipulation, and Patrol, and five types of environments. Each episode begins with a single natural-language directive. The agent must then iteratively decide where to go, how to position itself, and what results to report to complete the mission.

At a glance

This edition covers the complete twenty-five-page v1, including all five figures, eleven numbered tables, the patrol-overlap equation and four complete prompts. MissionBench contains 120 simulated aerial missions; the main experiment tests twenty-two frozen multimodal models on thirty missions, repeated three times.

Gemini 3.1 Pro leads at the reported 34.8% success, versus 70.0% for two humans using the model interface and 84.4% for three humans using continuous keyboard control. These small simulated evaluations show a substantial execution gap. They do not establish real flight readiness or evaluate all 120 missions with every model.

The benchmark couples visual navigation, viewpoint selection, reporting and stopping. Its detailed prompts also supply current pose and task-dependent movement limits; “RGB-only” therefore does not mean images are the agent’s only state information.

1 Introduction

A high-level mission such as reading a license plate requires the agent to decide how to approach, orient, extract information and stop. Route-following accuracy alone cannot measure that complete sequence. MissionBench targets frozen general-purpose models without aerial fine-tuning, using continuous action magnitudes within a restricted primitive vocabulary.

Four task families broaden evaluation beyond navigation. Comparisons across model sizes and versions are observational: training data, architecture, inference settings and prompting also vary, so the results do not isolate scaling as a causal mechanism.

2 Related Work

Complete original PDF block S2.T1
Table 1: Comparison of aerial navigation and mission-execution benchmarks. Paradigm: Route (step-by-step instructions), Goal (destination only), ObjNav (find object/SAR), Mission (task-level objective). RGB-only: Agent relies only on RGB input (no depth or additional sensors). Action: Disc. (discrete primitives), Wpt. (waypoints), Cont. (motion actions with continuously variable magnitude). DOF: Degrees of freedom in control. Pose: Agent must select a viewpoint to complete the task. Success: Dist. (agent/target distance to goal), Hierarch. (hierarchical step-, loop-, and task-level evaluation), Mission (task completion). Tasks: Rep (reporting), Insp (inspection), Manip (manipulation), Patrol.
Original selectable table

Benchmark Paradigm RGB-only Action DOF Pose Success Rep Insp Manip Patrol Simulator
AerialVLN Route ✗ Disc. 4 ✗ Dist. ✗ ✗ ✗ ✗ AirSim + UE4
AVDN Route ✓ Wpt. 3 ✗ Dist. ✗ ✗ ✗ ✗ xView
OpenUAV Route ✗ Wpt. 6 ✗ Dist. ✗ ✗ ✗ ✗ AirSim + UE4
OpenFly Route ✓ Disc. 4 ✗ Dist. ✗ ✗ ✗ ✗
AirSim + UE, GTAV,
GEarth, 3DGS
CityNav Goal ✗ Disc. 4 ✗ Dist. ✗ ✗ ✗ ✗ SensatUrban + Potree
FlySearch ObjNav ✓ Cont. 3 ✗ Dist. ✗ ✗ ✗ ✗ UnrealCV + UE5
UAV-ON ObjNav ✗ Cont. 4 ✗ Dist. ✗ ✗ ✗ ✗ AirSim + UE
ESARBench
Mission
✗ Disc./Wpt. 4 ✗ Mission ✗ ✓ ✗ ✗ AirSim + UE5
BEDI
Mission
✓ Disc./Wpt. 4 ✓ Hierarch. ✗ ✓ ✓ ✗ AirSim + UE4
MissionBench
(Ours)
Mission ✓ Cont. 4 ✓ Mission ✓ ✓ ✓ ✓ UE5 + CosysAirSim

Frozen MLLMs as Embodied Reasoning Agents.

Systems such as SayCan, PaLM-E and See-Point-Fly ground language or visual reasoning in robot actions. MissionBench contributes a shared evaluation of frozen aerial agents, rather than a new trained controller or a claim that prior planners cannot perform missions.

Aerial Vision-Language Benchmarks.

AerialVLN, CityNav and OpenFly emphasize navigation, while ESARBench and BEDI already consider broader mission structures. Table 1 positions this benchmark by jointly comparing observation assumptions, control degrees of freedom, task types and success criteria.

Beyond aerial navigation.

Static spatial tests reveal localization and depth errors but do not measure how errors compound over a feedback loop. MissionBench adds a sequential test, while its own ten-image perception proxy remains too limited to comprehensively diagnose every perceptual bottleneck.

3 MissionBench

3.1 Task Formulation

The agent receives a mission, current RGB frame, recent frames and action history, then produces a target box, explanation, relative action with magnitude, optional report and completion signal. Eight primitives cover forward/backward, lateral/vertical translation and left/right yaw; a simulator executes each command as a waypoint. This is four-degree-of-freedom high-level control, not direct motor control.

The appendix’s actual prompt also supplies current x/y/z, yaw, camera geometry and remaining actions. Boxes and explanations guide generation but are not directly scored. The loop stops at completion or the step budget; collisions are recorded without automatically ending it.

3.2 Mission Types

Reporting requires extracting information, inspection requires reaching a suitable view, manipulation adds a task-specific completion action, and patrol requires covering a route. Starts reveal at least part of the relevant object or region, so the benchmark focuses on approach and completion rather than unconstrained target discovery.

For manipulation, the supplied prompt marks an adequately positioned delivery with task_gt: drop and navigation done. The paper does not establish a separate payload-manipulation controller’s physical reliability; this task family should be interpreted within its simulator/interface definition.

Complete original PDF block S3.F2
Figure 2: The MissionBench closed-loop evaluation pipeline. The MLLM receives a mission instruction and egocentric images and outputs a structured response; actions are parsed into executable waypoints until the agent signals completion. The evaluation metrics Mission Progress (MP), Success Rate (SR), Oracle Success Rate (OSR), Collision Rate (CR) and Step Efficiency (Eff) are described in 3.4

3.3 Benchmark Construction

Simulation Suite.

The suite combines Unreal Engine 5 with Cosys-AirSim across Neighborhood, City, Forest, Savannah and AirSimNH. Natural and urban imagery vary, but the principal thirty-mission test covers only Neighborhood, Forest and City.

Mission Definition.

Authors manually design missions, place relevant assets and choose a fixed partly informative start. Reference route lengths span 13–4,171 m, with median 93 m and mean 260 m. Several reviewers check solvability and plausible operator instructions. These curated starts and hand-built objectives limit conclusions about arbitrary new missions or search from an unseen target location.

Ground-Truth Acquisition.

A human pilot records a feasible trajectory and terminal pose for each mission, with expected textual reports annotated afterward. References are executable examples rather than proven optimal routes. Position-based thresholds remain tied to a selected reference endpoint even when alternative viewpoints could also be useful.

3.4 Evaluation Metrics

Inspection requires final position within 5 m and yaw within 15° of the reference; manipulation adds the task-specific action. Reporting uses semantic report matching and can succeed outside 5 m. Patrol succeeds when overlap of 20 m-buffered planar routes exceeds 0.5. Thus success is deliberately heterogeneous across task families.

Oracle success records ever entering the 5 m vicinity; it is not an upper bound on reporting success. Mission progress for non-patrol tasks is max(0,Σ(d[t−1]−d[t])/d[0]), which telescopes to max(0,1−d[final]/d[0]). It measures terminal net approach, not cumulative route efficiency or proof of understanding.

Step efficiency is 1−used/budget, so early failure can score highly. Collision rate is the fraction of steps with collisions; stationary or prematurely stopped behavior can have low exposure. These metrics need joint interpretation rather than an isolated safety or competence ranking.

4 Experiments

Complete original PDF block S4.T2
Table 2: Dataset splits. All splits are disjoint except Ablation, which is a subset of Proxy.
Original selectable table

N=Neighborhood, F=Forest, C=City, S=Savannah, NH=AirSimNH. R=Reporting, I=Inspection, M=Manipulation, P=Patrol.

Split Missions Environments Composition Purpose
Proxy 10 N Mixed types Spatial perception (§4.3)
Ablation 5 N Easiest ⊂\subset Proxy Framework ablation (§4.5)
Test 30 N, F, C 17 R / 7 I / 3 M / 3 P Main evaluation (§4.2)
Held-out 80 N, F, C, S, NH Remaining Future fine-tuning
Total 120 N, F, C, S, NH 57 R / 38 I / 16 M / 9 P Overall dataset

4.1 Experimental Setup

Model setup.

The default interface uses three recent 1920 × 1080 images, fixed−45° pitch and stated temperature 0.7. Two humans use the same constrained interface; three use continuous keyboard control. These are small participant groups with different interfaces.

Appendix D gives provider/model-specific settings, including Qwen 3.6-35B-A3B-Q8 temperature 1.0 and differing thinking levels. The experiment therefore does not enforce exactly identical decoding settings across every model.

Benchmark setup.

Patrol receives fifty actions; other missions receive twenty. Maximum translation is one quarter of the distance from start to reference final pose, and nominal rotation limit is three times that magnitude in degrees. This uses reference geometry to configure the task and communicates the resulting cap in the prompt. It is privileged benchmark setup information, not wholly unassisted visual range estimation.

Three runs are planned per model/mission. Reaching a reference endpoint quickly is not guaranteed solely by this distance-based cap when obstacles, alignment and route coverage also matter.

Dataset splits.

The 120 missions divide into ten Proxy, thirty Test and eighty Held-out missions; five easiest Proxy missions also form Ablation. Main Test contains 17 reporting,7 inspection,3 manipulation and 3 patrol tasks. Held-out is reserved for future fine-tuning rather than reported as a full evaluation here.

4.2 Performance on MissionBench

Table 3 retains all twenty-two model results and human/random baselines. Some reported binary success percentages, including 34.8% and 7.9%, do not equal a rounded integer count out of 90 equally weighted trials. The paper does not specify exclusions or alternate denominators that resolve this, so these are reported percentages rather than reconstructed success counts.

Complete original PDF block S4.T3
Table 3: Results on MissionBench. All metrics are averaged across 3 runs on the 30-mission test split. All metrics are averaged across 3 runs. (SR=Success Rate, MP=Mission Progress, OSR=Oracle Success Rate, CR=Collision Rate, Eff=Step Efficiency). For human baselines, participants (1) followed the same input/output format as models (N=2N=2) or (2) were allowed to use the keyboard to continuously control the drone (N=3N=3). The random baseline samples a random action primitive and magnitude at every step.
Original selectable table
Model SR (%)↑\uparrow MP↑\uparrow OSR (%)↑\uparrow CR (%)↓\downarrow Eff (%)↑\uparrow
Random baseline 0.00.0 5.75.7 3.33.3 0.10.1 64.864.8
Gemini 3.1 Pro 34.834.8 73.373.3 17.2 4.34.3 53.6¯\underline{53.6}
Gemini 2.5 Pro 8.98.9 54.254.2 14.414.4 2.92.9 18.218.2
Gemini 3 Flash 7.97.9 54.054.0 20.220.2 4.94.9 30.830.8
Gemini 2.5 Flash 5.65.6 39.039.0 12.412.4 4.14.1 24.724.7
Gemini 3.1 Flash Lite 6.96.9 24.024.0 8.18.1 2.52.5 29.929.9
Gemini Robotics 1.5 0.00.0 35.935.9 6.96.9 5.85.8 32.132.1
Gemini Robotics 1.6 32.2¯\underline{32.2} 70.3¯\underline{70.3} 17.8¯\underline{17.8} 0.1¯\underline{0.1} 24.224.2
Gemma-4-31B-IT 26.726.7 59.359.3 11.111.1 1.61.6 13.613.6
Claude Opus 4.6 3.33.3 18.918.9 10.010.0 4.74.7 16.616.6
Claude Sonnet 4.6 4.74.7 31.231.2 7.17.1 2.02.0 30.230.2
GPT-5.4 2.02.0 45.045.0 0.00.0 4.04.0 17.917.9
GPT-5.4 Mini 1.01.0 20.020.0 0.00.0 5.05.0 25.725.7
Nova Premier 1.11.1 15.015.0 5.65.6 1.71.7 17.017.0
Nova Pro 0.00.0 15.215.2 3.33.3 2.02.0 59.059.0
Qwen 3.5 2B 0.00.0 18.118.1 5.65.6 0.00.0 48.348.3
Qwen 3.5 4B 1.11.1 28.628.6 5.65.6 0.00.0 36.836.8
Qwen 3.5 9B 3.33.3 34.834.8 7.87.8 6.26.2 29.029.0
Qwen 3.5 27B 10.010.0 53.253.2 8.98.9 0.00.0 16.716.7
Qwen 3.6 27B 5.65.6 32.532.5 8.98.9 3.23.2 10.610.6
Qwen 3.5 35B-A3B 2.22.2 39.139.1 7.87.8 7.57.5 32.132.1
Qwen 3.6 35B-A3B-Q8 7.97.9 41.541.5 7.97.9 0.20.2 29.529.5
InternVL 3.5 14B 1.11.1 29.229.2 6.76.7 0.20.2 11.411.4
Human baseline (w/ VLM interface) 70.070.0 79.079.0 36.736.7 0.10.1 36.036.0
Human baseline (w/ keyboard control) 84.484.4 94.294.2 51.151.1 0.00.0 −-

Overall performance.

Gemini 3.1 Pro reports 34.8% success/73.3 progress, Gemini Robotics 1.6 32.2%/70.3 and Gemma 4-31B26.7%/59.3. Human success remains higher through either interface. Most other models finish few missions despite using much of their budget. These rankings describe the tested prompts, starts and API configurations.

Oracle gap.

A report can be legible from beyond 5 m, explaining success above oracle proximity. Conversely, reaching the vicinity does not guarantee the right orientation, identity, report or stopping decision. Gemini 3 Flash has 20.2% oracle versus 7.9% success; these aggregate gaps suggest possible failure modes but do not by themselves identify each episode’s cause.

Scaling behavior.

Within Qwen 3.5 dense models,2B → 27B improves progress 18.1 → 53.2 and success 0 → 10%. Other comparisons are not monotonic: Sonnet 4.6 exceeds Opus 4.6 and Qwen 3.6 does not uniformly exceed 3.5. Model family, training, quantization and inference differences prevent treating size alone as the explanatory variable.

Spatial awareness.

Task success and collision frequency differ: the best-success model records 4.3% collided steps, while Qwen 3.5-27B records zero in this sample. A per-step collision average depends on the chosen route and actions; zero observed collisions is not a general safety guarantee.

4.3 Analysis of Perception Abilities

A separate ten-mission initial-image proxy asks for a target box and distance. Across models, localization mIoU correlates with mission progress at 0.463; distance RMSE correlates at−0.315. Opus has the lowest distance error but poor mission completion, while some models progress despite poor boxes.

These cross-model correlations support evaluating complete behavior in addition to static perception. They do not causally show that perception is unimportant or cannot be the main bottleneck: the proxy has only ten starts in one environment and differs from the thirty-mission test distribution.

4.4 Qualitative Failure Mode Analysis

Selected trajectories show early stopping, circling/drift and arriving near a target without fulfilling the request. Low progress with high step efficiency suggests early termination; low progress with low efficiency suggests wasted actions. These signatures are useful diagnostics, not unique causal explanations of internal model behavior.

Complete original PDF block S4.F3
Figure 3: Example trajectories. The human ground truth (green) is shown together with the trajectories of three MLLMs. The end point of each trajectory is indicated by a green circle (successful mission) or a red cross (failed mission). We examine the causes of these failure cases in Section B.3.

4.5 Framework Ablation

On five easier Proxy missions with Gemini 3 Flash, the default temperature 0.7 gives 57.1% reported success versus 40.0% at 0 and 46.7% at 1. Higher resolution increases completion while lower resolution gives higher net progress. Three-image context performs better than one or five on success.

Removing explanation reduces success 57.1 → 20.0%; removing bounding boxes gives 42.9%. These interventions alter the whole output/prompt behavior, so they do not prove a particular internal reasoning mechanism. The tiny selected split, substantial run variability and unresolved percentage denominators limit generalization of the preferred settings.

Complete original PDF block S4.T4
Table 4: Ablation of key design choices on the 5-mission Ablation split using Gemini 3 Flash. One aspect is ablated at a time and default settings are highlighted in gray.
Original selectable table
Temperature TT SR MP
0.0 40.0 68.1
0.7 57.1 68.4
1.0 46.7 59.8
(a) Sampling temperature.
Resolution SR MP
640×480640\times 480 (SD) 42.9 75.3
1280×7201280\times 720 (HD) 50.0 72.8
1920×10801920\times 1080 (FHD) 57.1 68.4
(b) Input resolution.
# Past Images SR MP
1 28.6 33.1
3 57.1 68.4
5 30.8 69.6
(c) History length.
Model output SR MP
Default 57.1 68.4
w/o BBox 42.9 70.2
w/o Reasoning 20.0 58.9
(d) Required structured output fields.

5 Conclusion

The benchmark’s main contribution is exposing failures that require closed-loop mission evaluation: movement, viewpoint selection, reporting and stopping must work together. The source supports a substantial simulated human–model gap, not real-world autonomous flight certification or a universal model ranking.

6 Acknowledgements

The work acknowledges Erlangen high-performance computing resources and BayernKI funding.

Supplementary Appendix

The supplement supplies limitations, variability and task breakdowns, simulator/reference construction, report-judge validation, model settings and four complete original prompts. Its details qualify the main paper’s observation and evaluation descriptions.

Appendix A Further Discussion

A.1 Limitations and Future Work

Only 120 curated simulated missions are available, skewed toward reporting 57 and inspection 38, with 16 manipulation and 9 patrol. Sensor noise, dynamic obstacles, adversarial conditions and real-flight transfer are not covered comprehensively. Standard prompts aid comparison but may disadvantage models that need different formatting.

The eighty held-out missions and proposed real-world evaluation are future work. More within-family task variation does not remove the imbalance or give equal statistical confidence for the small task families.

A.2 Ethical Considerations and Broader Impacts

Dual-use and deployment risks.

The authors discuss risks from deploying general models in drones before testing reliability under distribution shift and long feedback loops. These are prospective concerns rather than measured misuse or field incidents.

Practical deployment constraints.

Cloud inference and large models introduce latency, connectivity and compute constraints. The reported evaluation does not measure an onboard real-time implementation, so benchmark success alone cannot establish deployability.

Uncontrolled capability emergence.

Improvement without aerial fine-tuning motivates continued assessment of capabilities gained through general pretraining. The observed model comparisons do not isolate an emergence threshold or prove that future scaling will reliably solve control.

Evaluation limitations and over-reliance on benchmarks.

Simulation benchmarks should be complemented with real-world and whole-system validation. Their fixed task distributions and reference criteria provide a controlled measurement, with limited coverage of deployment conditions.

Potential benefits.

Possible benefits include easier prototyping and reduced task-specific data requirements. The paper does not quantify development savings or establish that deployment requires no additional engineering.

A.3 Disclosure of LLM Use

Authors disclose coding and language-editing assistance. Most evaluation is rule-based; textual report correctness uses a language-model judge validated on 150 sampled cases. That validation measures agreement on this sample rather than universal semantic correctness.

Appendix B Extended Results

B.1 Perception Proxy vs. Mission-Level Performance

Table B1 preserves localization and distance errors for all twenty-two models alongside mission scores. Gemini Robotics 1.6 has highest box mIoU 0.66; Opus 4.6 has lowest RMSE 10.7m. Different leaders and modest correlations show that this proxy is an incomplete predictor, with no intervention establishing which subskill causes failure.

Complete original PDF block A2.T1
Table B1: Perception results and key mission metrics. Perception metrics (mIoU, RMSE) are measured on the single-frame proxy task over the 10-mission set. Mission metrics (MP, SR) are measured on the 30-mission test split. All results are averaged over 3 runs.
Original selectable table
Perception Proxy MissionBench
Model mIoU↑\uparrow RMSE (m)↓\downarrow MP↑\uparrow SR (%)↑\uparrow
Gemini 3.1 Pro 0.51 27.7 73.3 34.8
Gemini 2.5 Pro 0.01 53.4 54.2 8.9
Gemini 3 Flash 0.58 72.3 54.0 7.9
Gemini 2.5 Flash 0.00 69.3 39.0 5.6
Gemini 3.1 Flash Lite 0.45 71.4 24.0 6.9
Gemini Robotics 1.5 0.03 20.0 35.9 0.0
Gemma-4-31B-IT 0.47 12.5 59.3 26.7
Gemini Robotics 1.6 0.66 61.1 70.3 32.2
Claude Opus 4.6 0.02 10.7 18.9 3.3
Claude Sonnet 4.6 0.16 17.1 31.2 4.7
GPT-5.4 0.29 73.1 45.0 2.0
GPT-5.4 Mini 0.21 71.8 20.0 1.0
Nova Premier 0.33 73.7 15.0 1.1
Nova Pro 0.28 59.5 15.2 0.0
Qwen 3.5 35B-A3B 0.34 65.5 39.1 2.2
Qwen 3.5 2B 0.18 97.5 18.1 0.0
Qwen 3.5 4B 0.36 79.3 28.6 1.1
Qwen 3.5 9B 0.46 69.2 34.8 3.3
Qwen 3.5 27B 0.53 44.3 53.2 10
Qwen 3.6 27B 0.55 77.7 32.5 5.6
Qwen 3.6 35B-A3B-Q8 0.38 24.2 41.5 7.9
InternVL 3.5 14B 0.01 113.3 29.2 1.1
Pearson correlation coefficients rmIoU,MP=0.463r_{\text{mIoU,MP}}=0.463
rRMSE,MP=−0.315r_{\text{RMSE,MP}}=-0.315

B.2 Performance Variability Across Mission Runs

Table B2 reports standard deviations for selected models over three main-test runs. Table B3 reports five ablation runs with Gemini 3 Flash success 54.2±17.9 and progress 73.4±14.9, distinct from Table 4’s default 57.1/68.4. These are run standard deviations, not confidence intervals, and demonstrate sensitivity to stochastic execution.

Complete original PDF block A2.T2
Table B2: This table reports standard deviations of selected models from Table 3. Results are from 3 identical runs with varying random seeds on the 30-mission test split.
Original selectable table
Model SR (%)↑\uparrow MP↑\uparrow OSR (%)↑\uparrow CR (%)↓\downarrow Eff (%)↑\uparrow
Gemini 3.1 Pro 34.8±13.534.8\pm 13.5 73.3±10.673.3\pm 10.6 17.2±7.7¯\underline{17.2\pm 7.7} 4.3±1.54.3\pm\text{1.5} 53.6±10.7¯\underline{53.6\pm 10.7}
Gemini 2.5 Pro 8.9±11.5¯\underline{8.9\pm 11.5} 54.2±20.0¯\underline{54.2\pm 20.0} 14.4±11.514.4\pm 11.5 2.9±1.32.9\pm\text{1.3} 18.2±15.718.2\pm\text{15.7}
Gemini 3 Flash 7.9±9.67.9\pm\text{9.6} 54.0±19.654.0\pm\text{19.6} 20.2±15.820.2\pm 15.8 4.9±4.84.9\pm\text{4.8} 30.8±14.230.8\pm\text{14.2}
Gemini 2.5 Flash 5.6±9.65.6\pm\text{9.6} 39.0±22.739.0\pm\text{22.7} 12.4±11.512.4\pm\text{11.5} 4.1±1.64.1\pm\text{1.6} 24.7±12.724.7\pm\text{12.7}
Gemini 3.1 Flash Lite 6.9±0.06.9\pm\text{0.0} 24.0±12.324.0\pm\text{12.3} 8.1±2.08.1\pm\text{2.0} 2.5±1.22.5\pm\text{1.2} 29.9±10.329.9\pm\text{10.3}
Gemini Robotics 1.5 0.0±0.00.0\pm\text{0.0} 35.9±14.635.9\pm\text{14.6} 6.9±4.16.9\pm\text{4.1} 5.8±4.55.8\pm\text{4.5} 32.1±16.332.1\pm\text{16.3}
Claude Opus 4.6 3.3±0.03.3\pm\text{0.0} 18.9±11.518.9\pm\text{11.5} 10.0±2.110.0\pm\text{2.1} 4.7±1.34.7\pm\text{1.3} 16.6±5.816.6\pm\text{5.8}
Claude Sonnet 4.6 4.7±6.64.7\pm\text{6.6} 31.2±14.331.2\pm\text{14.3} 7.1±4.47.1\pm\text{4.4} 2.0±0.6¯\underline{2.0\pm 0.6} 30.2±13.630.2\pm\text{13.6}
GPT-5.4 2.0±0.02.0\pm\text{0.0} 45.0±0.145.0\pm\text{0.1} 0.0±0.00.0\pm\text{0.0} 4.0±0.04.0\pm\text{0.0} 17.9±9.317.9\pm\text{9.3}
GPT-5.4 Mini 1.0±2.21.0\pm\text{2.2} 20.0±8.620.0\pm\text{8.6} 0.0±0.00.0\pm\text{0.0} 5.0±1.75.0\pm\text{1.7} 25.7±9.525.7\pm\text{9.5}
Nova Premier 1.1±2.21.1\pm\text{2.2} 15.0±6.115.0\pm\text{6.1} 5.6±4.45.6\pm\text{4.4} 1.7±1.61.7\pm 1.6 17.0±13.417.0\pm\text{13.4}
Nova Pro 0.0±0.00.0\pm\text{0.0} 15.2±7.615.2\pm\text{7.6} 3.3±0.03.3\pm\text{0.0} 2.0±3.02.0\pm\text{3.0} 59.0±16.559.0\pm 16.5
Qwen 3.5 35B-A3B 2.2±0.02.2\pm\text{0.0} 39.1±12.439.1\pm\text{12.4} 7.8±2.27.8\pm\text{2.2} 7.5±2.97.5\pm\text{2.9} 32.1±14.032.1\pm\text{14.0}
Complete original PDF block A2.T3
Table B3: Standard deviation of mission metrics across 5 runs on Gemini 3 Flash on the 5-Ablation test split.
Original selectable table
Model SR (%)↑\uparrow MP↑\uparrow OSR (%)↑\uparrow CR (%)↓\downarrow Eff (%)↑\uparrow
Gemini 3 Flash 54.2±17.954.2\pm\text{17.9} 73.4±14.973.4\pm\text{14.9} 45.5±11.045.5\pm 11.0 0.0±0.00.0\pm\text{0.0} 29.0±11.029.0\pm\text{11.0}

B.3 Quantitative Failure Mode Analysis

Rules assign unsuccessful episodes to primary categories using trajectory, terminal pose and outputs. Table B4 lists model-specific patterns, but its displayed percentages sum to 75.6 for GPT-5.4 and 83.3 for Gemini 3 Flash rather than 100. Denominators, uncovered cases and tie-breaking are not fully stated; preserve these as listed frequencies rather than an exhaustive normalized distribution.

Complete original PDF block A2.T4
Table B4: Breakdown of the failure case distribution for two models. The type of mistakes leading to non-successful mission completion and their frequency are highly model-specific.
Original selectable table
Failure Type GPT 5.4 Gemini 3 Flash
Drift and Oscillation 36.6 6.7
Unmet Mission Objectives 15.9 28.9
Altitude / Align Failure 8.5 24.4
Premature Termination 7.3 14.4
Continuous Collision 7.3 8.9

Continuous Collision

At least three collisions in one run triggers continuous-collision categorization. This operational threshold measures repeated contacts; it does not determine the agent’s intent or prove disregard for safety.

Drift and Oscillation.

Low progress combined with low step efficiency identifies drift/oscillation. Exact numeric cutoffs are not specified in the prose, limiting independent reproduction of the category assignment.

Premature Termination.

High efficiency with low progress identifies premature completion. The category distinguishes quick failure from prolonged unsuccessful navigation, subject to unspecified threshold choices.

Unmet Mission Objectives.

Unmet objectives describe reaching the target vicinity but misreporting, misidentifying, omitting an interaction or failing to terminate. An aggregate oracle–success gap can motivate this check but is not equivalent to every individual category assignment across heterogeneous task criteria.

Altitude or Align Failures.

Altitude/alignment failures use terminal distance or orientation outside the allowed threshold. This can overlap conceptually with other failures; the source does not give a complete rule priority. GPT’s listed drift 36.6% exceeds Gemini Flash 6.7%, while unmet objectives 15.9% versus 28.9% reverse the pattern.

B.4 Task-Specific Results

Gemini 3.1 Pro succeeds on 48.0% reporting,19.0% inspection,0% manipulation and 33.3% patrol in Table B5. Manipulation progress 86.4 alongside zero success illustrates why approach is not completion. Only three manipulation and three patrol missions appear in the main test, so per-family results are particularly sparse.

Complete original PDF block A2.T5
Table B5: Breakdown of task-specific performance. The table shows the results for visual inspection, manipulation, and patrol tasks separately, with the number of missions for each in parentheses, presented as SR / MP format.
Original selectable table

Agent Overall Reporting (17) Inspection (7) Manipulation (3) Patrol (3)
Gemini 3.1 Pro 34.8 / 73.3 48.0 / 79.0 19.0 / 67.4 0.0 / 86.4 33.3 / 42.5
Qwen 3.5 27B 10.0 / 53.2 17.6/56.8 0.0/57.1 0.0 / 60.6 0.0 / 16.4
Human Baseline 70.0 / 79.0 88.9/83.9 46.2/77.0 60.0 / 96.6 16.7 / 43.8
Human Baseline (w/ keyboard control) 84.4/ 94.2 92.13/98.5 71.4/95.0 66.7/95.5 88.9/67.23

Appendix C Additional Benchmark Details

C.1 Simulation Environment.

Cosys-AirSim extends AirSim with UE5 support. Forest uses free assets; Neighborhood and City extend existing scenes; Savannah and AirSimNH use distributed binaries. This mixture supplies visual variety without constituting a random sample of real aerial environments.

C.2 Mission Definition.

Human authors write mildly underspecified operator commands and deliberately place relevant scene assets. Multiple reviewers check consistency and solvability. Fixed starts keep targets partly visible and are reused for references and model runs; the setup tests viewpoint refinement more directly than wide-area discovery.

C.3 Reference Data Collection

Manual simulator flights provide timestamped poses and images for 120 missions. Reporting tasks receive expected captions. Reference collection uses free piloting rather than the constrained model interface, so those reference trajectories should not be conflated with the separate interface-matched human baseline.

C.4 Quality Assurance and Iterative Refinement.

Authors iterate scene design, packaging and test flights to remove visual artifacts and simulator inconsistencies. The paper describes this review process without reporting a residual defect rate or independent environment audit.

C.5 Evaluation Metrics for Patrol

Patrol routes are resampled into 100 segments and expanded by a20 m radius in the plane; equation 1 measures intersection area divided by union area. Success requires IoU>0.5. Buffering tolerates lateral deviations, while bypassing much of a route lowers overlap.

The metric intentionally ignores time alignment. It also does not directly encode traversal order, revisit counts or flight altitude, so equal covered planar regions can score alike despite different operational patrol behavior.

Complete original PDF block A3.F1
Figure C1: Patrol IoU evaluation. The ground-truth route (green) and predicted route (red) are each dilated by a 20​m20\,\text{m} buffer. The shaded overlap region (purple) determines IoUpatrol\text{IoU}_{\text{patrol}}; here the agent exhausts its step budget before completing the intended route, resulting in a low mIoU.
Equation 1
IoUpatrol=|ℬ⁡(τ∗,r)∩ℬ⁡(τ^,r)||ℬ⁡(τ∗,r)∪ℬ⁡(τ^,r)|\text{IoU}_{\text{patrol}}=\frac{|\,\mathcal{B}(\tau^{*},\,r)\;\cap\;\mathcal{B}(\hat{\tau},\,r)\,|}{|\,\mathcal{B}(\tau^{*},\,r)\;\cup\;\mathcal{B}(\hat{\tau},\,r)\,|} (1)

C.6 LLM-as-judge validation

Gemini 3 Flash Preview compares reports with expected answers. Against 150 human-annotated cases, the table gives precision 1.00/recall 0.94 versus exact-match 1.00/0.44, with three stricter judge disagreements described. Gemini 3.1 Pro’s reported success falls 34.8 → 19.1% under exact string matching.

The actual judge prompt permits 90% character similarity and even one or two differing letters/numbers. This is more permissive than formatting equivalence, particularly for license plates. The sample agreement supports this chosen rubric, not exact identifier transcription or error-free judging on unseen cases.

Complete original PDF block A3.T6
Table C6: Validation of LLM-as-judge against exact matching across 150 human-annotated report evaluation cases. Exact match artificially suppresses success rates due to minor formatting discrepancies, whereas the LLM-as-judge closely mirrors human assessment of semantic correctness.
Original selectable table
Evaluator Precision Recall
LLM-as-judge 1.00 0.94
Exact match 1.00 0.44

Appendix D Model Details

D.1 Inference Parameters

The supplement uses provider defaults/recommendations. Qwen 3.6-35B-A3B-Q8 specifies 8-bit quantization, temperature 1.0, top-p0.95, top-k20, min-p0, presence penalty 1.5 and repetition penalty 1.0. Gemini 2.5 uses dynamic defaults; Gemini 3 and Robotics 1.6 use high thinking. These exceptions qualify the main temperature 0.7 description.

D.2 Inference Cost

The paper estimates API costs for thirty missions × three runs from about 12.8 million input and 0.3 million output tokens. Its dated table lists $19.2 Nova Pro, $36.7 GPT-5.4, $56.9 Gemini 3.1 Pro and $71.9 Opus 4.6; these are source estimates, not current price quotes. Local open-model compute is excluded, and costs do not measure control latency.

Complete original PDF block A4.T7
Table D7: Per-model API cost for the full Test split evaluation.
Original selectable table
Model Name Input ($/M tokens) Output ($/M tokens) Total Cost ($)
Gemini 3.1 Pro 4 18 56.9
Claude Opus 4.6 5 25 71.9
GPT-5.4 2.5 15 36.7
Nova Pro 1.25 10 19.2
Complete original PDF block A4.F2
Figure D2: Structure of model input (prompts)

D.3 Model Prompts

The four complete prompt listings remain below, including every continuation page and placeholder. They handcraft target retention, step sizing, collision avoidance and stopping guidance. The evaluated system therefore combines a frozen model with a substantial handcrafted interface, rather than a bare high-level instruction alone.

System Prompt.

The system prompt demands relative commands and XML explanation/action fields, emphasizes target visibility, discourages oscillation and scales movements with estimated distance. Extracting a specific report value forces completion. These rules are instructions to the model, not an external verified safety controller.

Complete original PDF block source-system-prompt.part1Complete original PDF block source-system-prompt.part2
Original system prompt
Original selectable prompt
1   You are a skilled UAV navigation assistant with a camera providing egocentric bird’s eye view images.
2  
3   YOUR CORE RESPONSIBILITIES:
4   1. Analyze the current drone state and camera view
5   2. Predict the next immediate navigation action to progress toward the mission goal
6   3. Determine if the mission is complete or should continue
7  
8   OUTPUT FORMAT - You must ALWAYS respond with these XML tags:
9   <Reasoning>
10   - Analyze current image and drone state
11   - Explain why you chose this action or why mission is complete
12   - For inspection missions: if target details are readable, state what you see
13   - Include distance/angle estimates when relevant
14   - End with: task_gt: <value if mission requires specific info, otherwise ’N/A’>
15   </Reasoning>
16   <Action>
17   *** CRITICAL: ONLY output relative movement commands. NEVER use absolute coordinates or fly_to(). ***
18   - If mission incomplete: output ONLY command-style actions with numeric magnitude.
19   Allowed commands (ONLY these):
20   - move_forward(<meters>)
21   - move_backward(<meters>)
22   - strafe_left(<meters>)
23   - strafe_right(<meters>)
24   - move_up(<meters>)
25   - move_down(<meters>)
26   - turn_left(<degrees>)
27   - turn_right(<degrees>)
28   Output exactly ONE command per step.
29   DO NOT output: fly_to, GPS coordinates, absolute positions, or any command with (x,y,z) format.
30   - If mission complete: "navigation done"
31   </Action>
32  
33   AVAILABLE ACTIONS:
34   - Fly forward/backward distance meters
35   - Turn left/right angle degrees
36   - Ascend/descend distance meters
37   - Strafe left/right distance meters
38  
39   CRITICAL RULES:
40   1. ALWAYS keep target object in view - losing sight is the worst outcome.
41   2. Review action history before deciding; avoid repeating failed patterns.
42   3. NO oscillating opposite movements (forward/backward, ascend/descend, left/right turns).
43   4. Commit to a direction for 1-2 steps unless target visibility degrades.
44   5. For ground targets: FREQUENTLY combine forward progress with gradual descent - this is essential preparation to read ground-level details and inspect targets properly.
45   6. Respect maximum step size, angle limits, and action budget.
46   7. Maintain safe altitude (at least 1m above ground) and avoid collisions.
47   8. Keep target near frame center using turn/strafe corrections.
48   9. If target leaves frame, immediately prioritize recovery action (do not continue blindly).
49   10. Avoid repeating the exact same action more than 2 consecutive steps without reassessment.
50   11. NEVER output absolute GPS coordinates or fly_to commands - use ONLY relative movement commands (move_forward, move_down, etc).
51  
52   MISSION COMPLETION CHECKLIST:
53   - If you can successfully extract the required information (e.g., license plate text, sign text, object details), the mission is COMPLETE.
54   - CRITICAL: If you include "task_gt: <specific_value>" (not "N/A"), you MUST output "navigation done" - do not continue navigation.
55   - For text reading tasks: if text is readable enough to confidently extract it, mission is complete even if not pixel-perfect.
56   - For visual inspection: if target object details are identifiable and requirements are met, mission is complete.
57   - Do NOT continue navigating after successfully extracting required information - this wastes action budget.
58   - If objective is not clearly satisfied and you cannot extract task_gt, continue with one navigation action.
59   - If action budget is exhausted, output navigation done.
60  
61   DISTANCE-BASED STRATEGY:
62   - Far (>10m): mostly forward, occasional descend.
63   - Medium (5-10m): mix forward and descend.
64   - Close (<5m): mostly descend, small forward/heading adjustments.
65  
66   ADAPTIVE STEP SIZES - CRITICAL FOR PREVENTING OVERSHOOT:
67   - When target is far (>15m): larger steps allowed (up to max step size).
68   - When target is medium distance (5-15m): moderate steps (50-70% of max).
69   - When target is close (<5m): SMALL steps only (1-2 meters max for translation, 10-15° for rotation).
70   - When target fills >50% of frame: STOP forward movement, focus only on alignment/descent with tiny steps (<1m).
71   - Final approach (<3m or details becoming readable): Use 0.5-1.0m steps maximum.
72  
73   OVERSHOOT PREVENTION:
74   - Before each forward movement, estimate if it will overshoot the target.
75   - If target is already large in frame or close, reduce step size proportionally.
76   - Never move forward more than 50% of estimated remaining distance to target.
77   - If unsure about distance, use conservative smaller steps.
78  
79   RECOVERY MECHANISMS:
80   - If target was visible but now missing from frame: STOP forward progress immediately.
81   - Recovery priority 1: Turn toward last known target direction (check left/right edges of previous frame).
82   - Recovery priority 2: If target might be below, descend slightly while turning.
83   - Recovery priority 3: If target might be above or behind, move backward slowly while scanning.
84   - After recovery action, reassess before continuing approach.
85   - If target lost for 2 consecutive steps, use smaller exploratory movements until reacquired.
86  
87   Oscillation Prevention - CRITICAL:
88   - WRONG pattern: Ascend 8m →\rightarrow Descend 8m →\rightarrow Ascend 5m (cancels out, wastes actions)
89   - WRONG pattern: Forward 10m →\rightarrow Backward 10m (no net progress)
90   - WRONG pattern: Turn right 20° →\rightarrow Turn left 20° (wasted budget)
91   - If you commit to a direction (forward/up/down/turn), continue 1-2 steps before reassessing.
92   - Review history: if last 3 actions alternate opposites, STOP and take ONE corrective action.
93  
94   Recovery from Lost Target:
95   - If target was visible but now missing: analyze which action caused loss.
96   - Recovery priority 1: Turn toward last known target direction (check left/right edges).
97   - Recovery priority 2: Descend slightly while turning if target might be below.
98   - Recovery priority 3: Move backward slowly while scanning if target might be behind.
99   - After recovery, reassess before continuing approach.
100  
101   STRICT OUTPUT REQUIREMENTS:
102   - Always wrap final answer in <Reasoning> and <Action>.
103   - Do not output markdown, bullets, or prose outside these tags.
104   - In <Action>, output exactly one command with explicit numeric magnitude, or navigation done.
105   - Preferred command style: move_forward(2.0), turn_left(15), move_down(1.5), etc.
106   - Backward-compatible fallback (natural language) is tolerated, but command-style is strongly preferred.
107  
108   ALWAYS output in the XML format - no additional text outside tags.

Per-Step Prompt.

The per-step prompt includes simulator position/altitude/yaw, camera parameters, action counts/history and images. It later adds a normalized 0–1000 target box, depth and object name ahead of explanation/action output. Manipulation and patrol use textual completion markers.

Several source instructions conflict: a60° rotation cap appears alongside 3 × maximum-step rules; image1 is called primary but another clause requests reasoning from image2; two XML tags are demanded before a third bounding-box tag is added. These exact formulations are retained rather than silently repaired, and may affect formatting/behavior without establishing a particular implementation failure.

Complete original PDF block source-step-prompt.part1Complete original PDF block source-step-prompt.part2Complete original PDF block source-step-prompt.part3
Original per-step prompt
Original selectable prompt
1   Mission: <textual mission instruction>
2  
3   Camera: <resolution>, FOV: 90°, Pitch: -45.0°
4  
5   Current State:
6   - Position: (x=<x position>, y=<y position>, z(-
7   Altitude)=<altitude>)
8   - Note the AirSim uses a NED coordinate system where z is negative when above ground level, so altitude is -z.
9   - If the z is positive, then it means it crashed into the ground, so be careful with descending actions and always consider the current altitude when planning descends to avoid crashing.
10   - Yaw: <yaw>° (facing negative y-direction)
11   - Action count: <used actions>/<available actions>
12  
13   Constraints:
14   - Max step size: <max step size>m (angles: 60°)
15   - Action budget: <action budget> actions
16  
17   Action History: ’<action (T-1)>’, ’<action (T-2)>’, ’<action (T-3)>’
18  
19   Images:
20   - <image (T-1)>: Most recent frame (latest/current view)
21   - <image2>: Intermediate recent frame
22   - <image3>: Oldest frame in this recent history window
23  
24   Use <image1> as the primary decision frame and use <image2> ... <image3> as short-term history context. Determine the next action OR declare mission complete.
25  
26   - the most important thing is to remember the target object, it is the same throughout the mission and it is
27   important to alway keep it in view and not get distracted by other objects in the scene.
28   - CRITICAL: the action MUST include a numeric magnitude and unit:
29   - Distance moves use meters ("Fly forward 2 meters", "Strafe left 1 meter", "Ascend 0.5 meters").
30   - Turns use degrees ("Turn right 20 degrees").
31   - INVALID outputs: "fly_forward", "forward", "turn_left", "descend" (missing magnitude/unit).
32   <Instructions>
33   Analyse the image <image2> and the target objects and the required mission. Orientation and positioning of objects should be inferred from the <image2>.
34   Based on the current state, determine what the NEXT IMMEDIATE action should be.
35   The action MUST INCLUDE approximate DISTANCES and ANGLES based on visual reasoning and provided information.
36   Look carefully at the facing directions of the objects of interest, as well as the drone’s facing direction.
37   In this mission, the agent needs to precisely view the target object from the specified viewpoint.
38  
39   <Output Format>
40   Your response MUST contain two XML tags:
41   1. <Reasoning> tag: Explain your analysis of the current state and why you chose this action
42   2. <Action> tag: The specific action to execute
43  
44   Hovering, capturing images, landing and taking off DO NOT need to be included.
45   Just focus on the navigation action required to progress towards the mission goal.
46   Respect constraints: no single move longer than {max_step_size} meters and max_angle_step = 3 x {max_step_size}; stay within the action budget of {max_actions} (current count {current_action_index}).
47   The action should be concise and actionable. Choose actions such as: forward, backwards, turn left, ascend, descend, turn right, strafe right, strafe left.
48   CRITICAL ACTION FORMAT RULE:
49   - Every navigation action MUST include a numeric magnitude unit.
50   - Distance actions MUST include meters (e.g., "Fly forward 2 meters", "Strafe right 1.5 meters", "Descend 1 meter").
51   - Rotation actions MUST include degrees (e.g., "Turn left 15 degrees").
52   - NEVER output bare action tokens without magnitude (invalid examples: "fly_forward", "forward", "turn_left", "descend").
53   Important: when specifying distances/angles, use approximate values based on the image analysis and spatial reasoning information provided.
54   The distances should be in meters and angles in degrees.
55  
56   Guidelines:
57   0. Think in termns of left and right, instead of compass directions (north, south, etc.) since the drone heading can change
58   1. **TARGET VISIBILITY IS PARAMOUNT**: ALWAYS keep the object of interest in view - if it leaves frame, STOP and immediately correct direction
59   2. **USE HISTORY INTELLIGENTLY**: Before each action, review the action history and <image1> sequence to understand:
60   - What movements brought you to the current state
61   - Whether previous actions successfully maintained target visibility
62   - If you’re repeating a pattern that previously lost the target
63   3. **AVOID OSCILLATING MOVEMENTS**: Do NOT alternate between opposite actions (e.g., ascend then descend repeatedly):
64   - Example of WRONG: "Ascend 8m" →\rightarrow "Descend 8m" →\rightarrow "Ascend 5m" →\rightarrow "Descend 5m" (cancels out, wastes actions)
65   - Example of WRONG: "Forward 10m" →\rightarrow "Backward 10m" (no net progress, action budget wasted)
66   - Example of WRONG: "Turn right 20°" →\rightarrow "Turn left 20°" (ends at same heading, wasted action)
67   - If you need to adjust altitude, commit to it - use ONE descent or ONE ascent, not alternating
68   - If you need forward progress, move forward decisively - don’t second-guess with backward movements
69   - otherwise you can decrease the step size to make finer adjustments
70  
71   4. **COMMIT TO DECISIONS**: Once you choose a direction (forward/backward/up/down), continue in that direction for 1-2 more steps before changing:
72   - If target is visible ahead: move forward decisively, don’t retreat
73   - If target is above: ascend decisively, then assess before descending
74   - If target is to the side: strafe in that direction, don’t oscillate left-right
75   5. **ACTION BUDGET IS LIMITED**: Every action counts toward your total ({max_actions} max):
76   - Wasting 10 actions on oscillation leaves you only {max_actions - 10} for actual progress
77   - CRITICAL: Cancel out any recent opposite movements (if last action was "ascend 5m" and you want to descend, just descend less)
78   6. **RECOGNIZE STUCK PATTERNS**: If you see 3 alternating opposite movements in history, STOP and reassess:
79   - This means the current strategy is failing
80   - Look at where the target actually is and take ONE clear corrective action
81   - Don’t keep wiggling in the same bad pattern
82   7. **RECOVER FROM LOSSES**: If the object of interest is not visible in current <image1>:
83   - Analyze which recent action(s) caused the loss of view
84   - Execute reverse/corrective movements to return to last known good position
85   - Do NOT continue blindly forward when target is not visible
86   8. **LEARN FROM MISTAKES**: If an action pattern failed (e.g., multiple forward movements lost the target), try alternative approach:
87   - Adjust heading before moving forward
88   - Use slower, smaller movements
89   - Strafe instead of moving forward/backward
90   9. Use precise distance and angle estimates based on visual cues and spatial reasoning data
91   10. Avoid unnecessary movements that do not contribute to mission progress
92   11. Ensure each action is feasible given the drone’s current position and orientation
93   12. Do not exceed the maximum single-step movement of {max_step_size} meters but for angel max_angle_step = 3 x {max_step_size}
94   13. Analyse the current drone position in the context and beware that the height MUST always be at least -1 above the ground level
95   So when decending actions, consider the current drone pose and plan accordingly.
96   13. AVOID COLLISIONS at all costs
97   14. Avoid aggressive altitude changes - descend/ascend gradually (max {max_step_size} meters per step, max_angle_step = 3 x {max_step_size}) to maintain visibility of target
98   15. Do not repeat the same action more than 2 consecutive times without reassessing - if stuck in a pattern, consult action history and try a different direction
99   16. ONLY provide the answer in the specified XML format - no additional text outside the tags
100   17. For patrol missions, ensure to cover the entire area methodically without missing sections or overlapping excessively and dont get distracted by non-target objects or intermediate routes.
101   </Instructions>
102  
103  
104   Format reminder:
105   - Return only <Reasoning> and <Action> tags.
106   - End <Reasoning> with: task_gt: <value> (e.g., license plate text if readable, otherwise N/A).
107   - If task_gt has a specific value (not N/A), you MUST output <Action>navigation done</Action> - do NOT continue.
108   - If the task can be solved, you can direclty ouput the task_gt value and end the mission with <Action>navigation done</Action>.
109   You’re not required to keep achieving the most perfect viewpoint, as long as the task can be solved with the current view, you can end the mission.
110   - If not done, output exactly ONE action command in <Action>.
111   - If mission is complete (target details readable or budget exhausted), output <Action>navigation done</Action>.
112  
113   For manipulation missions like package delivery, sample collection, etc.:
114   - Assess whether the drone is in the correct position to complete the task (e.g., above the drop zone, close enough to the target, etc.)
115   - If YES: output with the following for e.g a package delivery mission:
116   <Reasoning> explain how the drone is in the correct position and the task is complete: task_gt: drop </Reasoning>
117   <Action>navigation done</Action>
118  
119   For patrol/surveillance missions:
120   - Assess whether the drone has covered the required area and observed the necessary viewpoints
121   - If YES: output with the following:
122   <Reasoning> explain how the drone has successfully patrolled the area and observed the required viewpoints: task_gt: patrol complete </Reasoning>
123   <Action>navigation done</Action>
124  
125  
126   ---
127   BOUNDING BOX PREDICTION REQUIREMENT
128   You MUST predict the location of the target object, its depth in m and the target_object_name
129   in the CURRENT image using a bounding box.
130   Use coordinates normalised to 0-1000 where:
131   - (0, 0) →\rightarrow top-left corner of the image
132   - (1000, 1000) →\rightarrow bottom-right corner of the image
133   - Format: x_min, y_min, x_max, y_max, depth, target_object_name
134  
135   Include this prediction as the FIRST XML tag in your response:
136   <BoundingBox>400, 450, 600, 550, 10.0, ’car’</BoundingBox>
137  
138   If the target is NOT visible in the current image use:
139  
140   <BoundingBox>-1,-1,-1,-1, -1, ’unknown’</BoundingBox>
141  
142   FULL output format (in this exact order):
143  
144   <BoundingBox>x_min, y_min, x_max, y_max, depth, target_object_name</BoundingBox>
145   <Reasoning>Your analysis here. End with: task_gt: <value></Reasoning>
146   <Action>Your single navigation action OR "navigation done"</Action>

Perception Proxy Task Prompt.

The perception proxy prompt supplies camera resolution,90° field of view and−45° pitch, then requests normalized boxes and metric depth for the mission object. The provided example is a license-plate approach. This isolates initial-image localization/ranging, not the entire perception burden encountered during navigation.

Complete original PDF block source-proxy-prompt.part1Complete original PDF block source-proxy-prompt.part2
Original perception-proxy prompt
Original selectable prompt
1   <Context>
2   You are in command of a UAV equipped with a camera of resolution 1920x1080 and a horizontal field of view of 90 degrees.
3   The camera has a pitch of -45.0 degrees. So the images captured by the camera resemble a bird’s eye view.
4  
5   The ultimate mission is: Fly towards and down close to a position where the license plate of the grey car is centered in the camera view and clearly legible. Read and report all the characters/numbers visible on the plate., for this purpose you need to identify and locate target objects in the image by predicting their bounding boxes and estimating their depth from the drone.
6  
7   </Context>
8  
9   <Objective>
10   Your goal is to analyse the current image and determine the bounding boxes around the target objects as well as estimate their depth from the drone.
11   Output only the bounding box coordinates in the format (x_min, y_min, x_max, y_max) along with the estimated depth in meters for each detected object.
12   </Objective>
13  
14   <Instructions>
15   Analyse the image and the target objects and the required mission. Orientation and positioning of objects should be inferred from the image.
16   Return bounding boxes for the main object relevant to the mission.
17   Provide the bounding box coordinate as integers normalised in the range (0-1000) and depth in meters as a float rounded to two decimal places.
18   </Instructions>
19  
20   <Output Format>
21   Only provide the output in the following format:
22   <reasoning>
23   Provide your reasoning here if necessary.
24   </reasoning>
25   <bounding_boxes_and_depth>
26   " a list.
27   xmin, ymin, xmax, ymax, depth, object_name.
28   </bounding_boxes_and_depth>
29   e.g.,
30   <reasoning>
31   The car is located near the center of the image, slightly towards the bottom right. The tree is on the left side of the image.
32   </reasoning>
33  
34   <bounding_boxes_and_depth>
35   400, 450, 600, 550, 10.0, ’car’
36   </bounding_boxes_and_depth>
37  
38   </Output Format>

LLM as a judge prompt

The exact report-judge prompt asks for success or failed based on semantic similarity, explicitly tolerating some character mismatches. Its outcome is the benchmark’s selected reporting criterion; users needing exact text should not treat it as character-perfect recognition.

Complete original PDF block source-judge-prompt.part1
Original report-judge prompt
Original selectable prompt
1   "You are an evaluator. Compare EXPECTED task_gt with MODEL task_gt."
2   " Determine if they match in meaning, allowing for minor variations"
3   " in wording."
4   " If the MODEL task_gt conveys the same intent and meaning as the"
5   " EXPECTED task_gt,"
6   " even if the wording is different, consider it a match."
7   " If they differ significantly in intent or details, consider it"
8   " not a match."
9   " For letters or numbers, a 90 percentage similarity threshold"
10   " can be used"
11   " to determine a match; if one or two numbers/letters differ,"
12   " it can still be considered success."
13   " Respond with exactly one word: success or failed.\n\n"
14   f"EXPECTED task_gt:\n{task_gt_expected}\n\n"
15   f"MODEL task_gt:\n{task_gt_pred}\n"
Concept demonstration

Open image ↗