Overview
Language-generated partner strategies, executable policy code and population-play training for zero-shot coordination. All original figures, tables, equations, algorithm, prompt boxes and code-diff examples are retained. Source code.
At a glance
PopGen first generates varied behavior descriptions, then writes and refines executable partner policies. PopPlay trains a cooperator against those frozen partners. Across four simulated environments it obtains the highest average reward with held-out scripted partners; gains over the strongest baseline have overlapping error bars in two environments. The experiments support strategy coverage and role adaptation, while human cooperation and judge-independent semantic validity remain untested.
1 Introduction
Different trajectories can implement the same strategy, so trajectory diversity need not cover incompatible coordination conventions. PopGen instead describes strategies in language, explores many candidates, selects plausible distinct descriptions, and converts them into code through rollout-guided refinement. The key hypothesis is that exposure to complementary roles and conventions helps a learned cooperator adapt to unseen partners. Semantic distance is itself an embedding-based proxy, and generated descriptions must actually be realized by the policies.
2 Related Work
The work combines population play with language-based behavior specification and executable policies. It evaluates both the diversity of descriptions and adherence of resulting agents, because either stage can fail.
Generating diverse populations for ZSC.
Prior populations pool self-play seeds/checkpoints, optimize trajectory/action diversity, or model human play. PopGen targets distinctions expressible in text rather than relying only on low-level trajectory distance. It complements, rather than proves invalid, earlier behavioral-diversity measures.
LLM-generated agents.
An LLM can choose actions online, generate a reward for an RL policy, or write explicit decision logic. Online calls are costly; a reward may be optimized without the intended behavior emerging. Code makes rules explicit and runs without per-step LLM calls. The controlled comparison with SemDiv-style reward generation uses identical descriptions to isolate this modeling choice.
3 Problem Formulation
The formal setting is cooperative sequential decision-making with shared reward, potentially partial observations and unknown partner conventions.
Multi-agent interaction.
A Dec-POMDP specifies agents, states, per-agent observations/actions, observation and transition models, shared reward and discount. Each policy acts from its observation history; coordination concerns joint behavior rather than independent task rewards.
Training against populations of partners.
Experiments use two exchangeable cooperative agents. First construct K partner policies, then optimize one cooperator’s expected shared return against sampled partners. PopGen associates each partner with a textual behavior. The displayed objective averages uniformly for exposition, while implemented PopPlay uses prioritized sampling of hard partners.
Equation 1
| (1) |
4 PopGen : Generating Diverse Agent Populations
Stage 1 produces descriptions that balance semantic variety with plausibility. Stage 2 turns each description into an executable policy and improves it using trajectory-based adherence feedback.
4.1 Stage 1: Generating Semantically Diverse Behaviors
Embed each behavior description as a unit vector and average pairwise distances √(2(1−cosine similarity)), the Euclidean distance on normalized embeddings. This measures linguistic/semantic spread under the selected embedding model, not directly the diversity of executed policies. Adherence evaluation supplies the separate grounding check.
Equation 2
| (2) |
The tension between quality and diversity of agent populations.
Quality sums three LLM-scored 0–5 rubrics: environment relevance, realism and specificity. PopGen oversamples descriptions with Verbalized Sampling, asking for candidate probabilities, then merges/deduplicates and selects a smaller high-quality set. Separating exploration from selection aims to prevent novelty through nonsense. The scores and verbalized probabilities are model judgments, not calibrated human ratings or behavior probabilities.
4.2 Stage 2: Instantiating Behaviors as Agents
The generator writes a Python policy that maps observations to actions. This avoids training a separate neural partner for every description and makes population rollouts inexpensive after generation. The neural cooperator is still trained with RL.
Measuring adherence with an evaluator.
A VLM receives sampled rollout states/images and returns a binary adherence verdict with rationale. Average accepted rollouts across multiple initial states to estimate adherence. It judges conformity to the target behavior, not necessarily high task reward, and its prompt allows justified good-faith attempts when the environment or teammate prevents full demonstration.
Equation 3
| (3) |
Iterative refinement loop.
Generate a candidate, run it with the designated scripted partner, score adherence and ask a separate advisor to diagnose failures. Regenerate with the feedback until a threshold is met or the attempt budget ends; otherwise return the highest-scoring attempt. Best-of-five results combine iterative repair and candidate selection, so their gain is not solely an isolated effect of the advisor.
4.3 Population Play with PopGen Partners
PopPlay freezes the generated partner code and trains only the recurrent PPO cooperator. It adopts MEP’s rank-based prioritization: partners yielding lower recent team return are sampled more often. This avoids spending all training on easy partners, though the training distribution then differs from the uniform objective stated earlier.
5 Experiments
Experiments test held-out coordination, growth of description diversity, quality-versus-diversity trade-offs, and the effect of code versus reward-based partner implementation.
5.1 Experimental Setup
Four environments cover discrete and continuous control, different observation scopes and complementary coordination conventions.
Environments.
Point-Mass Rendezvous requires agreement on one of four landmarks; Level-Based Foraging requires joint collection; Predator-Prey trades solo rabbit captures against coordinated stag captures; Overcooked requires a division of sequential cooking subtasks. All are simulated cooperative-agent tasks, with no physical-robot or human-partner evaluation.
LLM models.
The experiments use Gemini-3-Flash for generation and adherence classification, Gemini-3-Pro for feedback and quality judging, and Gemini-Embedding-001 for description embeddings. Results depend on these models and prompts; using related model families for generation and judgment can introduce shared biases.
Cooperator training.
Self-Play uses independently initialized MAPPO partners; FCP adds early/middle/final checkpoints; MEP rewards population entropy; CoMeDi seeks competent but incompatible conventions and uses an extra behavior-cloning objective. Most cooperators share PPO architecture and training-frame budget, while CoMeDi’s objective differs. FCP also has 12 partners versus four for the other main populations, so population size is not identical across all baselines.
ZSC evaluation.
Each environment has four fixed held-out scripted partner behaviors, designed in advance without LLMs and withheld from generation prompts. This separates explicit train/test behaviors but covers only a finite chosen synthetic test set; worst-case results below mean the worst of these four partners.
5.2 Training Cooperators with PopGen Populations
Evaluate team reward against unseen partners, then inspect Overcooked subtask choices to test whether the same cooperator changes roles with the partner.
Generalization to novel partners.
Across five runs, PopPlay’s held-out mean rewards are 1.70±0.56 in PMR, 0.87±0.07 in LBF, 1.59±0.08 in PP and 120.88±59.25 in Overcooked. It has the highest mean in all four; PMR and Overcooked overlap CoMeDi’s uncertainty. Its mean per-run worst-partner rewards are 1.41, 0.76, 1.33 and 31.2, with substantial variation, especially in PMR and Overcooked. This is empirical robustness on the designated test partners, not a bound over arbitrary teammates.
PopPlay adapts its role to the partner.
The same PopPlay cooperator plates/serves when the partner fills pots, and fills/cooks when the partner serves. Its averaged subtask shares span all displayed roles, unlike onion-specialized self-play populations whose cooperators fail to fetch onions when needed. CoMeDi is stronger with two serving-focused Overcooked partners, so PopPlay’s role coverage does not mean per-partner dominance. Figure 3 preserves both rollout panels and all task-share/worst-case tables.
5.3 Generating Diverse Populations for Cooperator Training
Population-size experiments link semantic spread to downstream reward; separate prompting comparisons assess whether that spread can be increased without sacrificing plausible descriptions.
Larger populations for more diversity, and more robustness.
Overcooked uses n=1,2,3,4,6,8 distinct behaviors while keeping eight training slots through replication and a fixed frame budget. Five independently generated populations/cooperators are run per size. Diversity rises from zero toward about 0.51 at n=4 and then levels off. One or two distinct partners leave worst-partner reward near zero; three or more improve it, but worst-case mean is not monotonic beyond four. The roughly 34% average-return increase at the fourth behavior and later plateau support coverage in this setting, without proving a universal four-partner optimum.
Preserving training agents’ quality while increasing diversity.
For sets of 30 descriptions, PopGen reports diversity 0.728 and quality 12.74/15, versus Verbalized Sampling at 0.726 and 12.03. Direct independent prompting scores higher quality (14.45) but lower diversity (0.446); SemDiv trades some diversity for quality. These figures describe the main reported comparison, while the six-environment oversampling ablation reports different averages. The claimed trade-off depends on embedding distances and the model judge’s rubric.
5.4 Modeling Agent Behaviors as Policy Code for Population Play
Keeping descriptions fixed, code-based partners yield higher downstream cooperator mean reward than reward-generated neural partners in all four environments. Table 12 shows best-attempt test adherence 66.3±7.2% for Policy Code versus 43.4±6.6% for Reward Function. The gap is largest in LBF (61.0% versus 12.5%) and Overcooked (63.3% versus 40.7%). This isolates implementation more directly than the full pipeline comparison, but does not establish that all reward-learning methods cannot realize these behaviors.
6 Conclusion
The results favor behavior coverage that is meaningful at the strategy level and faithfully executable. Oversampling/selection broadens descriptions, refinement improves code adherence, and population play exposes the cooperator to more complementary conventions.
Limitations.
All evaluation teammates are synthetic; human–AI coordination requires a human study. LLM population generation varies across runs, and the judge/embedding metrics are imperfect proxies. High mean reward with broad uncertainty and a finite worst-partner set should not be read as a universal cooperation guarantee.
Future work.
Future directions include generating new partners online to address the cooperator’s weaknesses, and improving the advisor’s context and code-generation reliability. The reported partners are generated before training and then frozen; online adaptation is not evaluated.
Acknowledgments
The authors acknowledge Antoine Bosselut, Seoyeon Choi and Google’s support.
AI Use Statement
The authors disclose generative-AI assistance with hypotheses, methodology, implementation, analysis, figures and writing, and state that they reviewed the resulting work. This disclosure is separate from the LLMs that are components of PopGen itself.
Reproducibility Statement
The paper supplies environment descriptions, held-out behaviors, baseline settings, detailed result tables and advisor/evaluator/quality-judge prompts. The code link and supplementary implementation support reproduction; the complete original prompt boxes are retained below.
Appendix A PopGen Details
Appendix A specifies environment mechanics and the adherence-driven refinement algorithm.
A.1 Environments
Two cooperative agents are trained in every core environment. LBF and Overcooked have six discrete actions; PMR and PP use two-dimensional continuous actions. LBF/PP are partially observable, while Overcooked/PMR use global observations. PP additionally contains five fixed-policy prey agents.
| Env | Num. Agents | Action Space | Observation | Objective |
| LBF | 2 (ego: 2) | Discrete (6) | Partial | Collect all food items |
| Overcooked | 2 (ego: 2) | Discrete (6) | Global | Serve as many soups as possible |
| PMR | 2 (ego: 2) | Continuous (2D) | Global | Meet at the same landmark |
| PP | 7 (ego: 2, adv.: 5) | Continuous (2D) | Partial | Capture prey |
Level-Based Foraging (LBF) ( Albrecht & Ramamoorthy, 2015 ) .
LBF rewards collected food level divided by the total food level in the field. Food can require combined agent capacity, so navigation and synchronized loading matter.
Overcooked ( Carroll et al., 2019 ) .
The large_room Overcooked layout rewards intermediate subtasks as well as delivery. Any chef can fetch onions, fill/start pots, plate and serve. Team return therefore includes shaping progress and is not simply a count of soups delivered.
Point-Mass Rendezvous (PMR) ( Lowe et al., 2017 ) .
PMR places four equally valid landmarks on a circle. The team receives reward 2 only when both agents converge to the same landmark, making incompatible conventions easy to diagnose.
Predator-Prey (PP) ( Lowe et al., 2017 ) .
PP uses two trained predators, three rabbits and two stags. A solo rabbit capture gives 0.5 reward; stag capture requires coordinated predators and gives 2. Prey policies are fixed by the environment.
A.2 Iterative Refinement for Adherence
The original pseudocode retains early acceptance at the adherence threshold, advisor feedback after failed attempts, and best-candidate fallback after the iteration budget.
Evaluating adherence to behavior.
Adherence evaluation uses a fixed partner type per environment: nearest-food loading in LBF, a coordinating scripted predator in PP, the Overcooked oracle planner, and a randomly selected landmark in PMR. Multiple initializations test robustness, but adherence with these partners does not guarantee the generated behavior under every teammate.
Feedback loop.
At each attempt, the generator receives the target behavior, environment and prior feedback. The VLM labels M rollouts; the advisor converts their rationales plus the current code into revision guidance. Accept a candidate once its adherence reaches α; otherwise retain the best attempt. The reported refinement ablation uses up to five attempts.
Advisor.
The advisor diagnoses recurring coordination failures and recommends actionable code edits. Its prompt and the contrasting code diffs are retained, allowing inspection of what the feedback actually asks the generator to change.
Appendix B Cooperator Training
All baselines construct partners first and train a separate cooperator afterward. The appendix distinguishes population-generation objectives from cooperator-training objectives.
B.1 Baselines
The shared population-play recipe does not imply identical populations or training losses. Baselines differ in partner construction, checkpoint count, prioritization and, for CoMeDi, the cooperator’s auxiliary objective.
Self-Play (SP).
Self-Play trains K independent MAPPO policies against copies of themselves and keeps final checkpoints. Diversity comes only from initialization and rollout noise, so multiple seeds can settle on the same convention.
Equation 4
| (4) |
Fictitious Co-Play (FCP).
FCP retains three evenly spaced early/middle/final checkpoints from each of four self-play runs, giving 12 partners. This increases skill variation but need not add new semantic roles when the original runs share conventions.
Equation 5
| (5) |
Maximum Entropy Population-based Training (MEP).
MEP augments team return with entropy of the population’s mean action policy, combining individual stochasticity and inter-policy disagreement. Its entropy coefficient is 0.001 for LBF/PMR and 0.005 for PP/Overcooked. Cooperator training ranks partners by recent episodic return and samples low-return partners more often; PopPlay uses the same prioritization with β=1.
Equations 6–8
| (6) |
| (7) |
| (8) |
CoMeDi.
CoMeDi adds partners sequentially, rewarding self-play competence while penalizing cross-play return against the closest existing convention. Mixed-play switches partway through a rollout from cross-play to self-play and scores the later return, discouraging deliberate incompetence as a shortcut to low cross-play. Coefficients are 0.25 for cross-play and 1.0 for mixed-play.
Equations 9–10
| (9) |
| (10) |
The CoMeDi cooperator.
CoMeDi’s cooperator adds behavior cloning from partner self-play demonstrations to its objective. This differs from the other PPO cooperators and inherits the convention coverage of its fixed population.
Equation 11
| (11) |
B.2 Held-out Evaluation Partners
Each environment has four held-out scripts: different food targets/loading timing in LBF; nearest, teammate-biased, rabbit-only or stag-only pursuit in PP; filling versus serving roles in Overcooked; and nearest, farthest, teammate-biased or regroup-then-diverge landmark behavior in PMR. The complete descriptions remain in Table 2.
| Env. | Name | Description |
| LBF | Closest-Food Greedy | Moves to the nearest food and loads it when possible. |
| Team-Center Food | Targets food near the center of the two agents. | |
| Teammate-Biased Food | Targets the food cheapest for the teammate to reach. | |
| Synchronized Loading | Waits at its food until the teammate is adjacent. | |
| PP | Closest-Prey Greedy | Chases the nearest prey, whether it is a rabbit or a stag. |
| Teammate-Biased Prey | Targets the prey the teammate is also pressuring. | |
| Rabbit Only | Always pursues the nearest rabbit, never a stag. | |
| Stag Only | Pursues stags only, so it scores only if the partner joins. | |
| Overcooked | Fill only | Only fetches onions and loads the pot. |
| Fetch & Serve | Only fetches a dish and delivers soup. | |
| Fill & Start | Loads onions and starts the pot, but never serves. | |
| Wait & Serve | Waits at the pot until the soup is cooked, then serves. | |
| PMR | Closest-Landmark Greedy | Moves toward the currently nearest landmark. |
| Farthest-Landmark Commitment | Always moves toward the currently farthest landmark. | |
| Teammate-Biased Landmark | Selects the landmark closest to the teammate. | |
| Regroup then Diverge | Approaches the teammate, then heads to a far landmark. |
B.3 Hyperparameters
Cooperators use recurrent GRU policies, discount 0.99, learning rate 10⁻⁴, PPO clipping 0.2, GAE λ=0.9 and prioritization β=1. Frame budgets are 4 million for LBF and Overcooked, 30 million for PMR and 80 million for PP. Partners remain frozen; baseline population generation uses MAPPO. Most main populations have four partners, versus FCP’s twelve.
B.4 Results Table
The full train/evaluation table shows PopPlay’s highest held-out mean in each environment, with a large Overcooked train–test drop from 211.66 to 120.88. Some methods score higher on evaluation than on training partners, so this difference is not always an OOD penalty. PopPlay has the top or tied top displayed mean for 14 of 16 partners; CoMeDi wins the two serving-focused Overcooked cases. The paper’s winner count assigns ties to PopPlay, while bold also marks overlapping error bars. Table 4 uses standard deviation; Table 5 uses standard error.
| Method | Partners | PMR | LBF | PP | Overcooked |
| Self-Play (SP) | Eval | ||||
| Train | |||||
| FCP | Eval | ||||
| Train | |||||
| MEP | Eval | ||||
| Train | |||||
| CoMeDi | Eval | ||||
| Train | |||||
| PopPlay (ours) | Eval | ||||
| Train |
| Held-out partner | PopPlay (ours) | SP | FCP | MEP | CoMeDi |
| Point-Mass Rendezvous (PMR) | |||||
| Closest-Landmark Greedy | |||||
| Farthest-Landmark Commitment | |||||
| Teammate-Biased Landmark | |||||
| Regroup then Diverge | |||||
| Level-Based Foraging (LBF) | |||||
| Closest-Food Greedy | |||||
| Team-Center Food | |||||
| Teammate-Biased Food | |||||
| Synchronized Loading | |||||
| Predator-Prey (PP) | |||||
| Closest-Prey Greedy | |||||
| Teammate-Biased Prey | |||||
| Rabbit Only | |||||
| Stag Only | |||||
| Overcooked | |||||
| Fill only | |||||
| Fetch & Serve | |||||
| Fill & Start | |||||
| Wait & Serve | |||||
| Best cooperator | /16 | /16 | /16 | /16 | /16 |
B.5 Mode Collapse in Self-Play Populations
Five self-play populations of four policies cover only 2.8 of four PMR landmarks on average by dominant convention; all four policies in one run select West. Overcooked self-play actions average 44% onion fetching and 36% pot loading, with no soup serving in the displayed runs. Intermediate FCP checkpoints add skill levels without necessarily adding missing roles. These diagnostics concern the tested training setup rather than all self-play algorithms.
| Landmark (%) | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | |||||||||||||||
| 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | |
| East | 0 | 0 | 0 | 100 | 35 | 100 | 0 | 0 | 0 | 0 | 0 | 0 | 85 | 0 | 0 | 100 | 0 | 0 | 0 | 0 |
| North | 100 | 15 | 0 | 0 | 0 | 0 | 75 | 0 | 0 | 0 | 100 | 0 | 15 | 0 | 30 | 0 | 0 | 0 | 0 | 0 |
| West | 0 | 0 | 75 | 0 | 0 | 0 | 0 | 0 | 100 | 0 | 0 | 0 | 0 | 0 | 70 | 0 | 100 | 100 | 70 | 100 |
| South | 0 | 85 | 25 | 0 | 65 | 0 | 25 | 100 | 0 | 100 | 0 | 100 | 0 | 100 | 0 | 0 | 0 | 0 | 30 | 0 |
| Stage (%) | Run 1 | Run 2 | Run 3 | Run 4 | Run 5 | ||||||||||||||||
| 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | Mean | |
| Fetch onion | 37 | 40 | 51 | 38 | 58 | 51 | 40 | 54 | 41 | 49 | 35 | 46 | 43 | 42 | 57 | 34 | 51 | 47 | 35 | 39 | 44 |
| Onionpot | 31 | 36 | 38 | 35 | 38 | 36 | 35 | 39 | 33 | 28 | 33 | 45 | 33 | 35 | 34 | 34 | 38 | 42 | 35 | 36 | 36 |
| Start pot | 9 | 12 | 0 | 12 | 0 | 0 | 12 | 0 | 8 | 8 | 11 | 0 | 11 | 10 | 0 | 10 | 0 | 0 | 12 | 12 | 6 |
| Fetch dish | 11 | 12 | 11 | 8 | 4 | 13 | 13 | 7 | 11 | 8 | 11 | 10 | 11 | 13 | 9 | 10 | 10 | 11 | 10 | 12 | 10 |
| Scoop soup | 11 | 0 | 0 | 8 | 0 | 0 | 0 | 0 | 8 | 8 | 11 | 0 | 2 | 0 | 0 | 10 | 0 | 0 | 10 | 1 | 3 |
| Serve | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
B.6 Task Event Counts in overcooked
Task-event counts over 50 episodes show PopPlay filling/cooking for serving partners and plating/serving for filling partners. CoMeDi loads pots even more often for serving partners, consistent with its higher reward in those cases. The caption mentions partner counts, but the displayed Table 8 contains cooperator event counts keyed by partner; it should not be interpreted as a separate measurement of the partner’s actions.
| Cooperator | ||||||
| Method | Partner | Onion pot | Start pot | Fetch dish | Scoop soup | Serve soup |
| PopPlay | Fill & Start | |||||
| Fill only | ||||||
| Fetch & Serve | ||||||
| Wait & Serve | ||||||
| Self-Play | Fill & Start | |||||
| Fill only | ||||||
| Fetch & Serve | ||||||
| Wait & Serve | ||||||
| FCP | Fill & Start | |||||
| Fill only | ||||||
| Fetch & Serve | ||||||
| Wait & Serve | ||||||
| MEP | Fill & Start | |||||
| Fill only | ||||||
| Fetch & Serve | ||||||
| Wait & Serve | ||||||
| CoMeDi | Fill & Start | |||||
| Fill only | ||||||
| Fetch & Serve | ||||||
| Wait & Serve | ||||||
Appendix C Diverse Behavior Generation
The description-generation study additionally includes Football and SMACv2. Their long or slow rollouts prevented inclusion in the main cooperator-training evaluation, so these results broaden text-generation analysis rather than proving coordination gains in six environments.
Football ( Bettini et al., 2024b ) .
Football descriptions vary positioning, offense and defense around the ball, goals and teammates. No main zero-shot cooperator reward result is supplied for Football.
SMACv2 ( Ellis et al., 2023 ) .
SMACv2 uses two allied Marines against four enemy Marines, encouraging varied target selection, movement, spacing and retreat. It is an additional behavior-generation domain, not one of the four main cooperation benchmarks.
Diverse text generation methods.
Direct prompting samples independently; One-Pass generates the set at once; Sequence uses batches; Multi-Turn conditions on prior descriptions; SimpleStrat samples explicit diversity axes. Verbalized Sampling requests candidate probabilities; its multi-turn version adds history. PopGen’s VS-OS oversamples and makes a separate merge/selection call. Table 9 retains the formal sampling definitions and call counts; default batches contain five candidates.
| Method | LLM Calls | Candidates | Turns | Prompt Example | Definition |
| 1. Instance-level Prompt | |||||
| Direct (DIR) | 1 | 1 | “Think step-by-step, generate a football player behavior” | ||
| 2. List-level Prompt | |||||
| One-Pass (OP) | 1 | “Generate 30 football player behaviors” | |||
| Sequence (SEQ) | 1 | “Generate 5 football player behaviors” | |||
| Multi-Turn (MT) | 1 | Turn 1: “Generate a football player behavior” Turn 2+: “Generate another football player behavior” | |||
| SimpleStrat (SS) (Wong et al., 2024) | 1 | “Generate a football player behavior following these characteristics: [strata]” | , | ||
| 3. Distribution-level Prompt | |||||
| Verbalized-Sampling (VS) (Zhang et al., 2025) | 1 | “Think step-by-step, generate 5 football player behaviors with probabilities” | |||
| VS-MT (Zhang et al., 2025) | Turn 1: “Generate 5 player behaviors with probabilities” Turn 2+: “Generate 5 more with probabilities” | ||||
| VS-OS (ours) | 1 | (VS) “Generate player behaviors with probs” (Merge) “Combine into best behaviors” | |||
C.1 Oversampling to Preserve Quality in Behavior Generation
Oversampling is evaluated as a way to explore broadly before selecting plausible distinct descriptions. More candidates need not always improve the final diversity.
| Diversity () | Quality () | ||||||||||||||
| Type | Ratio | football | lbf | pmr | pp | smacv2 | overcooked | avg | football | lbf | pmr | pp | smacv2 | overcooked | avg |
| Baseline (VS) | 1 | 0.730 | 0.726 | 0.636 | 0.691 | 0.756 | 0.612 | 0.692 | 12.77 | 11.16 | 13.52 | 12.20 | 12.00 | 13.32 | 12.50 |
| One Pass | 2 | 0.737 | 0.710 | 0.645 | 0.725 | 0.752 | 0.634 | 0.701 | 12.47 | 12.24 | 13.78 | 11.90 | 12.44 | 13.30 | 12.69 |
| 4 | 0.729 | 0.684 | 0.644 | 0.718 | 0.747 | 0.684 | 0.701 | 12.50 | 12.18 | 13.84 | 13.04 | 12.57 | 13.20 | 12.89 | |
| Independent | 2 | 0.737 | 0.654 | 0.637 | 0.735 | 0.738 | 0.616 | 0.686 | 12.44 | 11.97 | 13.84 | 12.93 | 12.20 | 13.26 | 12.77 |
| 4 | 0.726 | 0.672 | 0.643 | 0.721 | 0.749 | 0.614 | 0.688 | 13.01 | 12.57 | 13.88 | 12.77 | 12.11 | 13.10 | 12.91 | |
| Sequential | 2 | 0.733 | 0.703 | 0.649 | 0.737 | 0.740 | 0.613 | 0.696 | 13.01 | 11.97 | 13.78 | 13.14 | 12.87 | 13.20 | 13.00 |
| 4 | 0.738 | 0.685 | 0.640 | 0.719 | 0.694 | 0.594 | 0.678 | 12.84 | 12.36 | 13.74 | 12.66 | 12.80 | 13.22 | 12.94 | |
Oversampling ablation.
Across the six-environment table, sequential 2× oversampling gives quality 13.00 and diversity 0.696, close to the maximum diversity 0.701. It becomes the default. Sequential 4× lowers diversity to 0.678 and quality to 12.94, consistent with selection pruning unusual behaviors. Other variants likewise improve quality over baseline VS (12.50), but these averages are not the main four-environment Figure 5 values.
Diversity analysis and scaling of VS-OS.
Minimum embedding distance/coverage are 0.48/0.44 for VS, 0.35/0.38 for multi-turn VS, and 0.47/0.45 for VS-OS; lexical dissimilarity is similar. Scaling Flash to Pro gives multi-turn VS +1.00 quality but −0.06 diversity, versus VS-OS +0.57 and +0.04. Oversampling adds generation latency. These diagnostics support broader coverage under the chosen metrics, without proving that each extra semantic distinction becomes a distinct executable strategy.
Transfer of findings from other domains.
SimpleStrat reports quality 13.07 and diversity 0.532, below the displayed Pareto frontier. Verbalized-probability prompts transfer better to this task in the tested settings; multi-turn history tends to improve quality while narrowing variety. The comparison is empirical and model/domain dependent.
Appendix D Modeling Behaviors in Agents with LLMs
The adherence study separates development initializations used in refinement from ten unseen test initializations. This tests initialization generalization under the fixed evaluation partner and judge, not transfer to arbitrary partners or human judgments.
D.1 Behavior Modeling Methods
Compare online LLM actions, LLM-generated rewards followed by RL, and executable policy code. Each starts from a textual target behavior, with different generation and rollout costs.
| Name | Abbr. | Description |
| LLM-Embodied Agent | LLM-A | Uses an LLM directly as the decision-making agent, conditioned on textualized observations and a behavior specification: |
| Reward Function | RF | An LLM generates a reward function , whose optimization via reinforcement learning yields a neural network policy exhibiting the target behavior. |
| Policy Code | PC | An LLM synthesizes an executable policy that directly maps observations to actions, serving as the policy. |
Performance.
Use the explicit four-environment Table 12 values: best Policy Code adherence is 80.3% on development and 66.3% on test, versus Reward Function at 51.4%/43.4% and online LLM at 50.9%/49.2%. Online LLM is strongest on PP test adherence (93.3%), so code does not win every environment. Appendix prose instead says 81.3%/63.8% and cites SMACv2/Football figures absent from that table; those inconsistent claims are not used as verified results.
| Method | Generation latency (s) | Rollout latency (s) |
| LLM-A | — | |
| RF | ||
| PC |
Latency.
Reported average code synthesis takes 8.1±0.2 seconds and a rollout 9.8±1.9 seconds; reward-policy generation/training takes 2878.6±239.5 seconds and a rollout 11.7±0.6 seconds. Online LLM actions require no artifact-generation step but 110.1±1.5 seconds per rollout. These separate costs exclude a complete accounting of every refinement/judge/advisor call for an entire population; SMACv2 is excluded because engine overhead dominates.
D.2 Impact of the Refinement Loop on PopGen Agents’ Adherence
The refinement analysis compares the first candidate with the best of up to five attempts, then illustrates how an advisor changes code repairs.
Refinement loop to improve adherence.
Development adherence increases from 57.2% to 80.3% (+23.1 points); unseen-initialization adherence rises from 56.7% to 66.3% (+9.6 points). PP improves only on development (95%→98%) and stays 74% on test. The smaller test gain shows incomplete transfer from selection/refinement, and these averages remain below perfect adherence.
Feedback examples.
The advisor example fixes reversed movement directions, scores food using teammate motion, chooses an unoccupied support cell and holds a target for three steps. The no-advisor example adds loading without adjacency checks, chases the teammate instead of a support cell, and computes a heading it never uses. Both original truncated diffs remain, including indentation, colors and line-change counts. This is a qualitative comparison of one revision, not a quantified advisor-only ablation.
Original source text
@@ movement direction mapping
- if abs(dr) >= abs(dc) and abs(dr) > 0:
- return torch.tensor(1) if dr > 0 else torch.tensor(2)
+ action = 2 if dr > 0 else 1 # SOUTH if dr>0, NORTH if dr<0
+ return torch.tensor(action)
@@ target food scoring via teammate motion vector
- closest_dist = float('inf')
- for f in foods:
- dist = abs(f['r'] - team_pos[0]) + abs(f['c'] - team_pos[1])
+ def food_score(food):
+ f_dr = food[0] - team_row
+ f_dc = food[1] - team_col
+ return (abs(f_dr - team_dr) + abs(f_dc - team_dc), abs(f_dr), abs(f_dc))
+ best_food = sorted(foods, key=food_score)[0] if foods else None
@@ support-cell selection (closest cardinal cell, not teammate-occupied)
+ for d_row, d_col in [(-1,0),(1,0),(0,-1),(0,1)]:
+ cand_r, cand_c = target_row + d_row, target_col + d_col
+ if not (cand_r == team_row and cand_c == team_col):
+ valid_positions.append((cand_r, cand_c))
+ best_pos = min(valid_positions,
+ key=lambda p: abs(p[0]-self_row)+abs(p[1]-self_col))
@@ target commitment window
+ if not hasattr(policy, 'target_countdown'):
+ policy.target_countdown = 3
+ if policy.target_countdown == 0:
+ policy.target_food = best_food
+ policy.target_countdown = 3Original source text
@@ opportunistic LOAD without teammate adjacency check + if abs(target_r - self_r) <= 1 and abs(target_c - self_c) <= 1: + if food0_l > 0 and (self_l + team_l >= food0_l): + return torch.tensor(5) @@ fallback: move toward teammate rather than support cell + dr_to_t = team_r - self_r + dc_to_t = team_c - self_c + if dr_to_t != 0: + return torch.tensor(2 if dr_to_t > 0 else 1) \ + if abs(dr_to_t) > abs(dc_to_t) \ + else torch.tensor(3 if dc_to_t > 0 else 4) @@ heading estimation computed but never used in food selection + dr = policy.team_pos[-1][0] - policy.team_pos[-2][0] + dc = policy.team_pos[-1][1] - policy.team_pos[-2][1] + new_dir = 1 if dr > 0 else -1 # stored but unused
Appendix E Prompts
All advisor, adherence-evaluator and quality-judge system/user prompts are retained as original native renderings. Expand their source-text disclosures for the embedded copyable text.
E.1 Advisor
The advisor receives the target behavior, environment, attempted code and rollout evaluations, then diagnoses important failure patterns and recommends concrete edits. It does not itself supply a new policy in the displayed prompt.
Original source text
You are an advisor for a code generation pipeline.
Your task is to diagnose why a generated code fails to represent its
target behavior in rollouts and provide actionable feedback to improve
the next code attempt.
You will be given:
- The intended behavior
- The environment description
- The code attempt (type: {generation_type})
- Evaluation results from multiple rollout executions.
Focus on the most important failure patterns and give specific,
actionable suggestions to fix the code.Original source text
Code generation type: {generation_type}
Inputs
--------------
## Behavior
{behavior}
## Environment Description
{environment_description}
## Code Attempt
```
{code_attempt}
```
## Rollout Evaluation Results
{metrics_str}
--------------
Analyze the results and provide feedback for the next code attempt.E.2 Adherence Evaluator
The adherence evaluator prioritizes serialized states over images, focuses on the ego agent and requested behavior, and returns a boolean plus justification. It may accept a clearly explained good-faith effort when constraints prevent full demonstration. This permissive clause is part of what the reported adherence metric measures, rather than an objective task-success criterion.
Original source text
You are an expert evaluator of whether an agent's actions match a given
behavior description.
You will be given:
- a behavior description
- agent rollouts (trajectory states and possibly image sequences)
Your task is to determine whether the ego agent's actions adhere to the
behavior description.
# Guidelines:
- Evaluate alignment with the described behavior only.
- Do not consider any other possible goals.
- Focus only on the ego agent.
- Consider interactions with other agents only when relevant to behavior
following or when they affect the ego agent's ability to demonstrate
the behavior.
- Use states as the primary source of truth and images as supporting evidence.
- When serialized states are present, interpret them literally and use them
as the main basis for factual claims.
In rare cases, the agent may be unable to fully demonstrate the expected
behavior due to environment limitations or teammate actions. If the ego
agent still makes a clear, reasonable, good-faith effort to follow the
behavior, you may mark it as adherent. You must explicitly explain the
constraint and justify the decision in the reasoning field.
# Environment Description
{environment_description}
# Output Format
Return a JSON object with exactly these fields:
- "evaluation": a boolean indicating whether the rollout demonstrates
the behavior
- "reasoning": a detailed explanation of why the rollout does or does
not demonstrate the behavior, including the single most decisive reason
for the verdict and specific actions or states when relevantOriginal source text
Behavior:
{behavior}
Rollout (showing {N} serialized sampled states and {M} sampled images
in chronological order; step numbers are meaningful):
------------------------------------------------------------
[image_1] ... [image_M] # interleaved chronologically
[state_1]
[state_2]
...
[state_N]
------------------------------------------------------------E.3 Behavior Quality Judge
Quality is scored separately for relevance, realism and specificity, each 0–5. The detailed rubric permits unusual but conceivable styles while penalizing impossible actions or vague instructions. Scores sum to 15; they reflect the model judge’s interpretation of plausible human play, without human calibration in the reported study.
Original source text
System Prompt You are an expert in evaluating the quality of agent behavior descriptions. You will be provided with a behavior description and an environment description. Your task is to evaluate the quality of the behavior across three rubrics: 1. **Relevance** (0-5): Is the behavior relevant to the environment? - 0: Completely irrelevant, behavior has nothing to do with the environment or its objectives - 1: Mostly irrelevant, only tangentially related to the environment - 2: Somewhat relevant, addresses the environment but misses key aspects - 3: Moderately relevant, relates to the environment but could apply to many other contexts - 4: Highly relevant, clearly designed for this environment with appropriate context - 5: Perfectly relevant, deeply tied to the environment’s unique characteristics and objectives 2. **Realism** (0-5): Could a real player plausibly play this way, given the environment’s constraints? - 0: Violates hard environment constraints (e.g., impossible actions, broken physics) - 1: Technically allowed but contradicts the environment so fundamentally that no human would ever play this way - 2: Extremely unlikely — requires bizarre intent or misunderstanding, but not strictly impossible - 3: Unusual but conceivable — a small fraction of players (even 0.01%) might genuinely adopt this style - 4: Plausible — an uncommon but recognizable play style that some players would intentionally choose - 5: Natural — a common or intuitive way to play that many players would independently arrive at 3. **Specificity** (0-5): Is the behavior specific enough to be actionable? - 0: Extremely vague, no actionable guidance (e.g., "play well") - 1: Very generic, could mean almost anything (e.g., "be strategic") - 2: Somewhat vague, general direction but lacking concrete details - 3: Moderately specific, provides guidance but leaves room for interpretation - 4: Specific, clear behavior and conditions with minor ambiguities - 5: Highly specific, precise behavior, triggers, and success criteria that leave no ambiguity Provide detailed reasoning for each score.
Original source text
User Prompt
Environment Description:
{environment_description}
Behavior:
{behavior}
Please evaluate this behavior’s quality across the three rubrics (relevance, realism, specificity).