Overview
A centralized language-model critic reads joint trajectories during training. Each deployed agent subsequently acts from local observations without calling the critic.
At a glance
LLM-MCA uses a pretrained language model to turn shared team rewards and joint trajectories into individualized numerical feedback. LLM-TACA additionally supplies temporary task assignments to the agents during training, then withdraws those hints before decentralized execution.
Using Gemma-7B and Double DQN policies, the methods obtain strong returns in a climbing matrix game, Level-Based Foraging, Robotic Warehouse, and a new collision-sensitive Spaceworld grid environment. The paper retains natural-language explanations and releases annotated trajectories for future offline learning. These are small discrete simulations; the results do not establish hardware safety or validate the explanations as faithful causal attributions.
1. Introduction
In centralized training with decentralized execution, a critic sees the team’s joint observations, actions, and rewards, while each deployed policy sees local observations. A shared reward alone leaves uncertainty about which agents helped or hindered the outcome. This structural credit-assignment problem differs from assigning delayed credit among one agent’s earlier actions.
The authors cast cooperation as an agreement problem: agents should allocate the right number of participants to each subtask, avoiding both insufficient cooperation and redundant effort. A language-model critic receives explicit rules, objectives, examples, and definitions, then suggests per-agent rewards and explains its choices. The proposed advantage includes this extra semantic task knowledge and pretrained model capability; the comparison does not isolate credit decomposition from those priors.
The contributions are a credit-only critic, a task-and-credit variant, annotated offline trajectories, and Spaceworld. The batch-training diagram shows how retrospective trajectory feedback updates policies without retaining an LLM at deployment.
2. Related Works
The relevant prior work covers numerical credit assignment and language models as planners, reward generators, and critics.
2.1. Credit Assignment
Difference rewards and counterfactual methods estimate an agent’s contribution to a shared outcome. Value-factorization methods, including VDN, QMIX, and QTRAN, constrain how individual utilities combine into a team value; those assumptions can limit representation. MAVEN and Q-DPP address exploration or distinguishable behavior.
The evaluated baselines are LICA, which uses a hypernetwork critic without QMIX’s monotonicity constraint; MAPPO, which uses a centralized value function; and QMIX. These provide numerical comparisons, without establishing that all previous attribution methods are uninterpretable.
2.2. Large-Language Models in Reinforcement Learning
Prior work uses language models for task priors, hierarchical planning, reward shaping, reward-code generation, and dialogue-based coordination. Language-model critics for temporal credit assignment are particularly close. Here the language model trains ordinary numerical neural policies through parsed rewards rather than serving as the deployed linguistic agent.
3. Our Method
Both variants replace the learned centralized reward critic with a prompted language model. The task-and-credit variant adds a training-time communication channel to individual policies.
3.1. Problem Formulation
The setting is a cooperative decentralized partially observable decision process. Each agent maps its local observation to an action; the team receives one scalar reward. Equation 1 states the intended shared-return objective, and the retained problem statement asks how to assign an individual credit at each step.
A task needing more agents than are participating is under-collaborative; one assigned more participants than necessary is over-collaborative. The proposed critic identifies these patterns, suggests better joint behavior, and attributes useful changes to specific agents. Agreement is a heuristic for coordinating effort, not a proof that the learned credit maximizes the original reward or that every efficient task admits this simple participant-count description.
Equation 1
| (1) |
Problem 1 (Credit Assignment).
How can we determine the appropriate credit to assign to each policy at each timestep given the shared reward ?
3.2. Large Language Models for Multi-agent Credit Assignment (LLM-MCA)
The base prompt has four parts: environment rules and objectives, the input-data format, definitions and examples of coordination failures, and the critic’s requested role/output format. The complete Spaceworld prompt is preserved. It asks the model to interpret two robots, mirror segments, goals, collision hazards, and the shared efficiency reward.
During training the critic receives observations, actions, and rewards with interaction history. A regular-expression parser extracts individual numerical credits and applies a normalization step. The paper does not fully specify that normalization or prove that the resulting rewards preserve the team-optimal policy. The intended surrogate sums individual credits over time; Equation 2 retains the source’s compact vector-parser notation inside that sum.
Instead of querying at every environment step, the critic can retrospectively annotate a trajectory or batch when policies are updated. This supplies denser feedback for intermediate progress and amortizes language-model calls. The text claims lower compute and storage burden, but does not report a detailed timing or memory comparison. Natural-language rationales make the assigned feedback inspectable; their correctness and causal fidelity are not separately measured.
Definition Prompt 1.
Given a set of observations and actions as performed by the agents in the environment, there will be times when the agents will individually accomplish goals, but occasionally, we will need them to collaborate. When two agents correctly agree to collaborate on a task that requires two agents, then they have found a valid solution to the “Agreement Problem”.
Equation 2
| (2) |
3.3. Large Language Models for Task and Credit Assignment (LLM-TACA)
LLM-TACA makes the critic’s implied task plan explicit. In addition to credits, it can return short integer-array goals directly to a policy, or leave the hint absent. This is intended to reduce volatility when an agent’s emerging plan differs from the critic’s preferred assignment.
Policies reserve zero-padded hint inputs. During training, hints are increasingly replaced with zeros and dropout on associated inputs increases, preparing the agents to operate without them. The exact dropout/withdrawal schedule is not supplied. The retained example includes the complete feedback, Python-like arrays, task hints, and explanation; it is an illustrative model response rather than a formal attribution guarantee.
Equation 3
| (3) |
4. Results
The reported runs use the open Gemma-7B model and Double DQN policies on an NVIDIA 3080 Ti. No systematic language-model comparison is performed, so claims that other models or future reasoning improvements would transfer remain hypotheses.
The fully observed climbing game repeats a common-payoff matrix for 25 steps with constant observations. Level-Based Foraging requires agents’ combined levels to meet each food item’s requirement. Robotic Warehouse uses local 3×3 views and sparse delivery rewards, making navigation, congestion, and coordination difficult.
Spaceworld is a discrete mirror-transport task inspired by telescope servicing. Either agent may move either mirror. Attempting to enter the same cell destroys both robots and ends the episode with zero reward; slower completion loses reward, normalized to 0–10. This creates a safety–efficiency tradeoff within the simulator, without proving collision avoidance outside the evaluated scenarios.
4.1. Baselines
LICA, MAPPO, and QMIX compare implicit hypernetwork credit assignment, centralized policy-gradient training, and monotone value mixing respectively. Figures 5–6 and Table 1 report five independent training runs with 95% confidence intervals.
The complete table lists seven foraging and three warehouse configurations, plus the matrix game and Spaceworld: twelve tasks in total. Figure 6’s caption instead says twelve scenarios across the two standard benchmarks. Preserve its wording, but use the table’s explicit task counts when describing the evaluation.
4.2. Results Analysis
In Spaceworld both proposed methods score 10.0±0.0, versus 6.0±2.5 for MAPPO, 4.4±4.6 for QMIX, and 2.0±2.2 for LICA. This is strong performance on the tested randomized episodes, not a probabilistic safety guarantee.
Foraging means are 0.93 for LLM-MCA and 0.98 for LLM-TACA, versus 0.85 for MAPPO and 0.54 for QMIX. LLM-MCA does not win every configuration: in partially observed 10×10 foraging it scores 0.85 versus QMIX’s 0.89. Warehouse mean returns are 13.34 and 24.12 versus MAPPO’s 1.83; the gains are large, but not uniformly an order of magnitude for every method/configuration.
The matrix-game means are 195 for LLM-MCA and 210 for LLM-TACA. LICA’s 169 is poor relative to these methods but does not mean zero reward, despite the prose’s “completely fails” wording. The displayed payoff maximum 11 over 25 steps gives a theoretical 275, above every reported mean. Full table values and intervals are retained; broad claims of universal dominance are stronger than the measured evidence.
4.3. Offline Dataset
Training produces trajectories annotated with per-agent credits and task assignments. The project page provides dataset instructions. These are generated supervisory labels, not independently validated ground-truth contributions, and the paper does not demonstrate an offline-learning result using them.
5. Conclusions and Future Work
The experiments support language-model-generated rewards and temporary task hints as useful training signals for the selected cooperative simulations. The critic also produces explanations that users can inspect, although explanation fidelity is untested.
Language-model latency remains a limitation despite batching. Closed-model fees are a potential cost, while the reported experiments use an open model. Non-cooperative tasks, faster real-time evaluation, and real-robot deployment are future work. Missing prompt-normalization details, hint schedules, and comprehensive training hyperparameters limit reproducibility from the paper alone.
6. Appendix
The supplement records benchmark naming conventions and the complete numerical comparison.
6.1. Implementation Details
Foraging names encode grid size, player count, food count, optional sight range, and cooperative mode; the historical “-c” suffix became “-coop” in version 3. Warehouse names encode grid size and player count. The complete table preserves all means, 95% intervals, bold best values, and starred second-best entries.
These are collective environment returns across five runs. The caption’s wording about “no common reward or reward scalarization” is preserved, although the method explicitly supplies a shared reward to the critic. The appendix gives no further numerical parser-normalization formula, withdrawal/dropout schedule, or full optimizer/training-budget specification.
| Environments\Algorithms | LLM-MCA (Ours) | LLM-TACA (Ours) | LICA (Zhou et al., 2020) | MAPPO (Yu et al., 2022) | QMIX (Rashid et al., 2020) |
|---|---|---|---|---|---|
| Climbing Matrix Game | * | ||||
| Spaceworld-10x10 | * | ||||
| Level Based Foraging | |||||
| 8x8-2p-2f-c | * | ||||
| 8x8-2p-2f-2s-c | * | ||||
| 10x10-3p-3f | * | ||||
| 10x10-3p-3f-2s | * | ||||
| 15x15-3p-5f | * | ||||
| 15x15-4p-3f | * | ||||
| 15x15-4p-5f | * | ||||
| Average | * | ||||
| Robotic Warehouse | |||||
| Tiny 4p | * | ||||
| Small 4p | * | ||||
| Tiny 2p | * | ||||
| Average | * |