Overview
At a glance
EvoSkill-GUI improves deployed interface agents by editing reusable skill files instead of model weights. An executor revises skills during a task; a separate session of the same model diagnoses failed trajectories and guides later edits. Across the evaluated backbones, maximum absolute success-rate gains are 16.2 percentage points on MobileWorld, 6.0 on AndroidWorld, and 10.5 on OSWorld. Reuse and extra inference cost are evaluated separately.
1 Introduction
Pop-ups, delayed loads, and changing widget positions make fixed plans fragile. The method stores planning, localization, recovery, and failure knowledge separately so observable mistakes can produce targeted, reusable file edits. Its claim is inference-time skill improvement without additional model training.
Related references: [24], [18], [35], [6], [11], [14], [23], [12], [39], [2], [37], [30], [33], [38], [5], [27], [29], [7], [1], [22], [34], [44], [17].
2 Related Work
2.1 Agent Skills
Prior skill systems package reusable workflows, construct structured memories, or distill procedures into model weights. This framework instead revises deployed GUI skill packages from execution feedback.
Related references: [2], [43], [33], [5], [27], [38], [9], [41], [44], [30], [25], [16], [15], [37].
2.2 Self-evolving Agents
Experience-based agents update policies, memories, or reflection modules. EvoSkill-GUI specifically updates the procedural files that condition future execution.
Related references: [40], [31], [28], [22], [34], [13], [12], [29], [7], [1], [8], [41], [17].
3 Method
3.1 Problem Formulation
Tasks are partially observed control problems: screenshots and accessibility trees expose only part of the application state. A persistent skill conditions the agent’s policy, and revisions seek higher completion rates across related tasks.
Related references: [37].
Equations 1–3
| (1) |
| (2) |
| (3) |
3.2 Skill Package Representation
Each package separates retrieval metadata, accessibility utilities, executable plans, backup localization, recovery rules, and failure examples. This maps a diagnosed failure to a specific component rather than rewriting one long prompt.
Equation 4
| (4) |
Tool interface.
Restricted file tools expose only the predefined skill schema. A separate read-only accessibility tool augments observations without modifying the skill.
Equation 5
| (5) | ||||
3.3 Reflect-Revise Loop
Execution, isolated diagnosis, and targeted revision alternate until success or the iteration limit.
Rollout and instant revision.
During execution, the agent can immediately edit a skill when observed interface behavior contradicts its plan, reducing cascades from local mismatches.
Equation 6
| (6) |
Reflection.
The same model acts as critic in a separate session using only the instruction and executed trajectory. It identifies failure steps, causes, and actionable fixes without access to privileged answers or the executor’s hidden reasoning.
Equations 7–8c
| (7) |
| (8a) | ||||
| (8b) | ||||
| (8c) | ||||
Revision.
The executor applies the critique to specific files and records failed cases. Experiments allow the initial attempt and at most two post-failure revision rounds.
Equation 9
| (9) |
3.4 Skill Retrieval and Reuse
Retrieval uses structured intent, application, platform, keywords, argument slots, history, and status. Avoiding full procedural text reduces misleading lexical overlap between unrelated tasks.
Equation 10
| (10) | ||||
Retrieval score.
The score combines lexical coverage and semantic tightness with application and keyword bonuses, subtracting a penalty for missing query-specific concepts. The original equations preserve the chosen weights.
Equations 11–12
| (11) |
| (12) |
Reuse.
A candidate above the retrieval threshold is reused; otherwise a new package is generated. Packages are retained after successful execution or revision through the evolution loop.
4 Experiments
4.1 Experimental Setup
Benchmarks
Evaluation covers AndroidWorld, the GUI-only MobileWorld subset, and OSWorld desktop tasks. AndroidWorld uses 116 tasks across 20 applications; related task variants support the reuse study.
Models
The study includes general multimodal and GUI-specialized models, testing whether skill revision helps different model sizes and families.
Implementations
The skill format and retrieval strategy are fixed across models. Each rollout permits 50 interaction steps, followed by at most two revision rounds after failure. Provider endpoints and locally served open models use the settings recorded in the appendix.
4.2 Main Results
| Method | Chrome | GIMP | LibreOffice | Multi-Apps | OS | Thunderbird | VLC | VS Code | Overall | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Calc | Impress | Writer | |||||||||
| General Closed-Weights Models | |||||||||||
| Claude-Sonnet-4.5 Anthropic (2026) | 56.4 | 57.7 | 66.0 | 57.5 | 65.2 | 47.0 | 70.8 | 66.7 | 52.9 | 69.6 | 58.1 |
| Qwen3-VL-Flash Bai et al. (2025) | 56.4 | 42.3 | 23.0 | 51.0 | 56.5 | 22.0 | 54.2 | 66.7 | 34.4 | 69.6 | 41.6 |
| Specialized Open-Weights Models | |||||||||||
| UI-TARS-72B-DPO Qin et al. (2025) | 33.2 | 61.5 | 12.8 | 25.4 | 43.5 | 6.7 | 33.3 | 33.3 | 23.5 | 47.8 | 25.9 |
| Computer-Use-Preview | 36.9 | 34.6 | 14.9 | 29.7 | 26.1 | 15.8 | 70.8 | 66.7 | 11.8 | 69.6 | 31.2 |
| OpenCUA-32B Wang et al. (2025a) | 45.6 | 65.4 | 12.8 | 42.5 | 39.1 | 14.6 | 60.9 | 46.7 | 23.5 | 65.2 | 35.1 |
| EvoCUA-8B Xue et al. (2026) | 54.3 | 84.6 | 29.8 | 38.2 | 52.2 | 25.5 | 75.0 | 73.3 | 44.8 | 65.2 | 46.1 |
| EvoCUA-32B | 65.1 | 76.9 | 55.3 | 59.5 | 69.6 | 27.9 | 78.3 | 80.0 | 49.4 | 87.0 | 56.7 |
| GUI-Owl-1.5-32B Ye et al. (2025b) | 66.6 | 76.9 | 48.9 | 63.8 | 60.9 | 27.3 | 76.2 | 86.7 | 37.4 | 87.0 | 55.4 |
| Skill-based Works | |||||||||||
| MMSkills Zhang et al. (2026b) | 39.9 | 42.3 | 8.5 | 23.4 | 17.4 | 13.4 | 25.0 | 29.4 | 47.8 | 25.4 | |
| Our Method | |||||||||||
| GUI-Owl-1.5-8B | 35.6 | 80.8 | 29.8 | 57.5 | 69.6 | 21.5 | 70.8 | 73.3 | 43.8 | 81.8 | 46.7 |
| +EvoSkill-GUI | 41.3 | 88.5 | 37.8 | 65.8 | 69.6 | 30.5 | 66.7 | 93.3 | 86.5 | 87.0 | 54.8 |
| Improvement | +5.7 | +7.7 | +8.0 | +8.3 | +0.0 | +9.0 | -4.1 | +20.0 | +42.7 | +5.2 | +8.1 |
| Qwen3-VL-8B-Instruct | 28.1 | 26.9 | 19.1 | 31.8 | 47.8 | 7.7 | 16.7 | 60.0 | 35.3 | 26.1 | 23.8 |
| +EvoSkill-GUI | 43.4 | 46.2 | 29.8 | 34.4 | 52.2 | 15.2 | 41.7 | 46.7 | 50.6 | 47.8 | 34.3 |
| Improvement | +15.3 | +19.3 | +10.7 | +2.6 | +4.4 | +7.5 | +25.0 | -13.3 | +15.3 | +21.7 | +10.5 |
EvoSkill-GUI improves multiple base models on mobile GUI tasks.
On MobileWorld, Claude-Sonnet-4.6 improves from 57.1% to 67.6%, Qwen3.6-Plus from 53.3% to 69.5%, Qwen3.6-35B-A3B from 32.4% to 44.8%, and MAI-UI-8B from 29.5% to 37.1%. AndroidWorld gains are 2.6 and 6.0 percentage points for the two reported seeds.
EvoSkill-GUI extends to desktop computer-use tasks.
On OSWorld, GUI-Owl improves from 46.7% to 54.8% and Qwen3-VL-8B-Instruct from 23.8% to 34.3%. Some application-level results regress despite the overall gains.
4.3 Ablation Study
Ablations examine skill structure, immediate revision, critic isolation, retrieval, and accessibility observations.
| Skill Organization | Success Rate (%) |
|---|---|
| Single-file Skill | 66.67 |
| Skill Package (Ours) | 69.52 |
| Improvement | +2.85 |
Structured skill packages outperform monolithic skill files.
Separating skill components raises MobileWorld success from 66.67% for a single file to 69.52% for the structured package.
| Strategy | Success Rate (%) |
|---|---|
| EvoSkill-GUI (Ours) | 69.52 |
| w/o information isolation | 60.95 |
| w/o instant revision | 62.86 |
Instant in-rollout revision prevents cascading failures.
Removing within-rollout revision lowers success from 69.52% to 62.86%, supporting correction of mismatches before a full attempt fails.
Metadata-based retrieval outperforms full-text matching.
In twelve selected successful reuse cases, metadata retrieval finds the intended skill in all twelve at threshold 0.6; full-text matching finds one. This small selected-case analysis is supplemented by held-out retrieval evaluation.
Information isolation improves reflection quality.
Removing critic information isolation reduces success from 69.52% to 60.95%. Separate-session diagnosis helps avoid inheriting the executor’s mistaken assumptions.
| Task name | Metadata Score | Full-text Score | Meta@0.6 | Full-text@0.6 |
|---|---|---|---|---|
| Mastodon New Post | 0.8036 | 0.3431 | ✓ | ✗ |
| Mattermost Create Channel | 1.0000 | 0.4769 | ✓ | ✗ |
| TaoDian Item Checkout | 1.0000 | 0.6182 | ✓ | ✓ |
| Mastodon Reply (Moussaka) | 1.0000 | 0.3833 | ✓ | ✗ |
| Change Wallpaper (Sunflower) | 0.8333 | 0.4184 | ✓ | ✗ |
| Mattermost Send File | 0.7003 | 0.3414 | ✓ | ✗ |
| Mattermost Reply Result | 0.7833 | 0.3521 | ✓ | ✗ |
| Check Cart Price | 0.9500 | 0.4231 | ✓ | ✗ |
| Mastodon Unfollow | 0.8167 | 0.3876 | ✓ | ✗ |
| Mastodon Save Photos | 0.8500 | 0.3654 | ✓ | ✗ |
| Google Maps Alibaba South | 0.9167 | 0.4512 | ✓ | ✗ |
| Mastodon Conditional Favorite | 0.8500 | 0.2987 | ✓ | ✗ |
| Average / Hit Rate | 0.8753 | 0.4050 | 12/12 | 1/12 |
Each skill-package component contributes to performance.
Removing planning documents reduces success to 65.71%; removing failure reflection reduces it to 66.67%. The components provide complementary guidance.
Accessibility-tree utilities benefit grounding across models.
Accessibility-tree inputs improve success by 1.90–3.80 percentage points across the compared backbones, complementing visual grounding.
A moderate retrieval threshold balances skill reuse and selectivity.
Threshold 0.6 performs best in the reported sweep. A lower threshold admits weak matches; a higher threshold rejects useful experience.
| Retrieval Threshold | Success Rate (%) |
|---|---|
| 0.4 | 55.2 |
| 0.6 (Ours) | 69.5 |
| 0.8 | 66.7 |
4.4 Analysis
Skill evolution provides cumulative gains and largely saturates after three rounds.
Most improvement arrives within three execution rounds. In the extended Qwen3.6-Plus study, the first three rounds gain 12.4 points and two additional rounds add only 0.9.
| Skill Source | Tasks | Recovered | Recovery (%) |
|---|---|---|---|
| New skills | 72 | 8/24 | 33.3 |
| Reused skills | 160 | 19/65 | 29.2 |
| Overall | 232 | 27/89 | 30.3 |
| Retrieval Threshold | Success Rate (%) |
|---|---|
| 0.4 | 55.2 |
| 0.6 (Ours) | 69.5 |
| 0.8 | 66.7 |
| Failure Type | Target File | Recovered | Rate |
|---|---|---|---|
| Plan-level error | plan.md | 16/23 | 69.6 |
| Grounding error | backup.md | 6/12 | 50.0 |
| Missing contingency | recover.md | 1/4 | 25.0 |
Skill evolution outperforms repeated sampling under a comparable budget.
Under a comparable token budget, three evolution rounds reach 69.5% versus 62.8% for three independent baseline attempts, using 94.75 million versus 103.00 million tokens.
Evolved skill libraries remain reusable and effective on related tasks.
A library built on 116 AndroidWorld tasks benefits 116 related variants: success rises from 68.1% to 70.7% in phase one and from 55.2% to 61.2% in phase two. These results concern related parameterized tasks rather than arbitrary new applications.
Skill libraries accumulate reusable experience over task streams.
The sequential task stream accumulates more reusable packages, with the reported library reaching 98 skills. Reuse statistics depend on whether the paper measures a phase, stream, or selected subset.
Different failure types require different skill edits.
Plan edits repair 16 of 23 analyzed planning failures; localization edits repair 6 of 12 grounding failures, and recovery edits repair 1 of 4 missing-contingency failures. Repairability differs by error type.
5 Conclusion
Structured, editable skills can improve GUI execution without changing model weights. The gains come from targeted revision and reuse under the tested inference budgets.
Limitations
Incorrect perception or diagnosis can produce harmful edits, and no formal verifier currently rejects regressions. Benchmark tasks do not cover all interface changes or personal configurations. Failed attempts incur extra critic and revision calls; benefits depend on later reuse.
Ethics Statement
The authors discuss authorization, sandboxing, auditability, and review of reusable skills for deployment beyond benchmarks. They report no collection of personal user data in these experiments.
Appendix A Appendix
A.1 Complete Implementation Details
The appendix records provider-based and local serving configurations, shared evaluation settings, and research-use conditions.
Model serving.
The reported local configuration uses vLLM, bfloat16, a 65,536-token context, and at most five images per prompt. Exact serving parameters remain in the original implementation description.
Evaluation.
Runs permit 50 steps with a three-second per-step wait, and up to three execution iterations including the initial attempt.
Environment.
Environment settings include parallel benchmark containers, model-specific screenshot resizing, and normalized action coordinates. These choices affect runtime and reproducibility.
Excluded tasks.
Twelve MobileWorld Google-proxy tasks are excluded for evaluation stability.
A.2 Held-out skill transfer and retrieval quality
Evolved skills transfer to held-out tasks.
A library constructed on 87 MobileWorld tasks is tested on 30 held-out tasks. EvoSkill-GUI solves 22/30 versus 12/30 for pass@3. Five additional successes use retrieved skills and five use newly generated packages.
| Retrieval result | Tasks | Ours S/F | pass@3 S/F |
|---|---|---|---|
| Retrieved existing skill | 16 | 12/4 | 7/9 |
| No retrieval | 14 | 10/4 | 5/9 |
| Total | 30 | 22/8 | 12/18 |
Metadata-based retrieval remains precise on held-out tasks.
Of sixteen held-out tasks triggering retrieval, fifteen receive an appropriate skill. The remaining mismatch illustrates possible negative transfer.
A.3 Same-backbone self-diagnosis
Same-backbone critique preserves self-contained evolution.
Using one backbone for execution and critique avoids attributing improvement to a stronger supervisor, but critique still consumes inference tokens.
Information isolation separates critique from execution.
Critic sessions see the instruction and observed trajectory, excluding the skill package, executor CoT, and ground truth.
Same-backbone diagnosis is meaningful but imperfect.
Manual review finds 115 correct judgments among 143, or 80.4%. At least four false success judgments prematurely stop revision, demonstrating a concrete weakness of self-diagnosis.
| Judgment | Count | Rate (%) |
|---|---|---|
| Correct diagnosis | 115/143 | 80.4 |
| Incorrect diagnosis | 28/143 | 19.6 |
A.4 Analysis of negative OSWorld results
Two OSWorld application/model entries show negative changes despite overall improvement.
Rare negative cases are sensitive to run-level variation.
Only three tasks account for the negative changes, and reruns vary. Small application-level differences therefore provide weak evidence about systematic regressions.
Skill guidance can overcomplicate direct operations.
For short direct tasks, an imperfect skill can add unnecessary steps or reinforce an incorrect procedure. More guidance is not always helpful.
A.5 Token cost under repeated sampling.
The corrected accounting reports 94.75 million tokens for EvoSkill-GUI versus 103.00 million for pass@3. An earlier 168-million figure double-counted intermediate cumulative totals.
| Category | Parameter | Value |
| Execution | Temperature | 0.0 |
| Execution | Max output tokens | 8192 |
| Execution | History screenshots | 3 |
| Execution | Coordinate scale factor | 1000 |
| Execution | Max skill-tool calls/ step | 4 |
| Skill generation | Temperature | 0.3 |
| Skill generation | Max output tokens | 4096 |
| Critic | Temperature | 0.0 |
| Critic | Max output tokens | 4096 |
| Critic | Max screenshots | 8 |
| Critic | Screenshot max dimension | 1280 |
| Retrieval | BM25 | 1.5 |
| Retrieval | BM25 | 0.75 |
| Retrieval | Keyword threshold | 0.6 |
| Retrieval | Divergence threshold | 0.65 |
| Retrieval | Divergence weight | 0.3 |
| Retrieval | Domain bonus | 0.15 |
| Retrieval | Keyword bonus | 0.2 |
| Revision | Revise order | P B C |
| Revision | Failure examples | 3 at most |
| Method | Total (M) | Avg./ Task (M) | Avg./ Round (M) | Acc. (%) |
|---|---|---|---|---|
| Baseline pass@1 | 33.20 | 0.284 | 0.284 | 53.3 |
| Baseline pass@3 | 103.00 | 0.880 | 0.293 | 62.8 |
| Single-file skill pass@3 | 84.54 | 0.723 | 0.372 | 66.7 |
| EvoSkill-GUI (3 rounds) | 94.75 | 0.810 | 0.439 | 69.5 |
Accessibility trees add moderate runtime overhead but little token usage overhead.
Accessibility trees add collection and processing time while token usage remains broadly similar. Their cost is mainly runtime overhead in the reported examples.
| Task | Token cost(K) | Time(s) |
|---|---|---|
| AcceptMeetingTask | 67.2 / 71.1 | 437 / 288 |
| MastodonNewPostTask | 79.8 / 79.4 | 476 / 358 |
| SetAlarmTask | 189.1 / 186.0 | 526 / 405 |
| ScheduleLunchViaSmsTask | 800.0 / 820.1 | 1277 / 1125 |
A.6 Case Studies
Two case studies trace an observed failure through critic diagnosis, file edits, and later execution.
A.6.1 Case 1: Verification Checkpoint Insertion
In an email task, locating a resume file was mistaken for verifying and attaching it. The revised plan adds explicit document and attachment checks, and the next rollout succeeds.
| Round | Score | Steps | Failure Mode |
|---|---|---|---|
| 1 | 0.0 | 14 | Skipped file verification |
| 2 | 1.0 | 14 | Success |
A.6.2 Case 2: Completeness Enforcement for List Traversal
In a cart task, incomplete scrolling produces the wrong top-three price total. A revised traversal plan alone fails again; recording the concrete failed shortcut and enforcing item-count completion leads to success on the third attempt.
| Round | Score | Steps | Answer | GT |
|---|---|---|---|---|
| 1 | 0.0 | 25 | 2657 | 13186 |
| 2 | 0.0 | 28 | 2657 | 13186 |
| 3 | 1.0 | 11 | 13186 | 13186 |
A.6.3 Summary
The examples show how failure evidence changes specific planning, localization, and recovery files instead of merely prompting an independent retry.