Literature
Display
← All papers

Leveraging Large Language Models for Effective and Explainable Multi-Agent Credit Assignment

Kartik Nagpal · Dayi Dong · Jean-Baptiste Bouvier · Negar Mehr

· AAMAS 2025 ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

A centralized language-model critic reads joint trajectories during training. Each deployed agent subsequently acts from local observations without calling the critic.

diagram
Figure 1. Overall architecture diagram for our LLM-MCA and LLM-TACA methods. Our centralized training architecture utilizes a centralized LLM-critic instantiated with our base prompt (environment description, our definitions, and task query). At each timestep, we update our LLM-critic with the global reward and latest observations from the environment. We then update our agents’ policies with the individualized feedback from our critic.

At a glance

LLM-MCA uses a pretrained language model to turn shared team rewards and joint trajectories into individualized numerical feedback. LLM-TACA additionally supplies temporary task assignments to the agents during training, then withdraws those hints before decentralized execution.

Using Gemma-7B and Double DQN policies, the methods obtain strong returns in a climbing matrix game, Level-Based Foraging, Robotic Warehouse, and a new collision-sensitive Spaceworld grid environment. The paper retains natural-language explanations and releases annotated trajectories for future offline learning. These are small discrete simulations; the results do not establish hardware safety or validate the explanations as faithful causal attributions.

1. Introduction

In centralized training with decentralized execution, a critic sees the team’s joint observations, actions, and rewards, while each deployed policy sees local observations. A shared reward alone leaves uncertainty about which agents helped or hindered the outcome. This structural credit-assignment problem differs from assigning delayed credit among one agent’s earlier actions.

The authors cast cooperation as an agreement problem: agents should allocate the right number of participants to each subtask, avoiding both insufficient cooperation and redundant effort. A language-model critic receives explicit rules, objectives, examples, and definitions, then suggests per-agent rewards and explains its choices. The proposed advantage includes this extra semantic task knowledge and pretrained model capability; the comparison does not isolate credit decomposition from those priors.

The contributions are a credit-only critic, a task-and-credit variant, annotated offline trajectories, and Spaceworld. The batch-training diagram shows how retrospective trajectory feedback updates policies without retaining an LLM at deployment.

diagram
Figure 2. Diagram for our batch-training process with our LLM-MCA method. Our centralized training process allows us to provide entire batches of trajectories at once to our centralized LLM-critic. Our LLM-MCA critic then generates individualized feedback for each agent, which we use to update their policies. After training, we no longer need our LLM-critic, and directly deploy our trained, decentralized agent policies.

2. Related Works

The relevant prior work covers numerical credit assignment and language models as planners, reward generators, and critics.

2.1. Credit Assignment

Difference rewards and counterfactual methods estimate an agent’s contribution to a shared outcome. Value-factorization methods, including VDN, QMIX, and QTRAN, constrain how individual utilities combine into a team value; those assumptions can limit representation. MAVEN and Q-DPP address exploration or distinguishable behavior.

The evaluated baselines are LICA, which uses a hypernetwork critic without QMIX’s monotonicity constraint; MAPPO, which uses a centralized value function; and QMIX. These provide numerical comparisons, without establishing that all previous attribution methods are uninterpretable.

2.2. Large-Language Models in Reinforcement Learning

Prior work uses language models for task priors, hierarchical planning, reward shaping, reward-code generation, and dialogue-based coordination. Language-model critics for temporal credit assignment are particularly close. Here the language model trains ordinary numerical neural policies through parsed rewards rather than serving as the deployed linguistic agent.

3. Our Method

Both variants replace the learned centralized reward critic with a prompted language model. The task-and-credit variant adds a training-time communication channel to individual policies.

3.1. Problem Formulation

The setting is a cooperative decentralized partially observable decision process. Each agent maps its local observation to an action; the team receives one scalar reward. Equation 1 states the intended shared-return objective, and the retained problem statement asks how to assign an individual credit at each step.

A task needing more agents than are participating is under-collaborative; one assigned more participants than necessary is over-collaborative. The proposed critic identifies these patterns, suggests better joint behavior, and attributes useful changes to specific agents. Agreement is a heuristic for coordinating effort, not a proof that the learned credit maximizes the original reward or that every efficient task admits this simple participant-count description.

Equation 1
(1) max𝜃​𝔼s0∼ρak∼πθ​(ok)​(∑k= 0R⁡(ok,ak)),\underset{\theta}{\max}\underset{\begin{subarray}{c}s_{0}\,\sim\,\rho\\ a_{k}\,\sim\,\pi_{\theta}(o_{k})\end{subarray}}{\mathbb{E}}\left(\sum_{k\,=\,0}R\big(o_{k},a_{k}\big)\right),
Problem 1 (Credit Assignment).

How can we determine the appropriate credit ckic_{k}^{i} to assign to each policy πi\pi^{i} at each timestep kk given the shared reward rkr_{k}?

3.2. Large Language Models for Multi-agent Credit Assignment (LLM-MCA)

The base prompt has four parts: environment rules and objectives, the input-data format, definitions and examples of coordination failures, and the critic’s requested role/output format. The complete Spaceworld prompt is preserved. It asks the model to interpret two robots, mirror segments, goals, collision hazards, and the shared efficiency reward.

During training the critic receives observations, actions, and rewards with interaction history. A regular-expression parser extracts individual numerical credits and applies a normalization step. The paper does not fully specify that normalization or prove that the resulting rewards preserve the team-optimal policy. The intended surrogate sums individual credits over time; Equation 2 retains the source’s compact vector-parser notation inside that sum.

Instead of querying at every environment step, the critic can retrospectively annotate a trajectory or batch when policies are updated. This supplies denser feedback for intermediate progress and amortizes language-model calls. The text claims lower compute and storage burden, but does not report a detailed timing or memory comparison. Natural-language rationales make the assigned feedback inspectable; their correctness and causal fidelity are not separately measured.

Definition Prompt 1.

Given a set of observations and actions as performed by the agents in the environment, there will be times when the agents will individually accomplish goals, but occasionally, we will need them to collaborate. When two agents correctly agree to collaborate on a task that requires two agents, then they have found a valid solution to the “Agreement Problem”.

Sample prompt
Figure 3. Example prompt for LLM-MCA in the “Spaceworld” benchmark. Our base prompt pbase:=(penv,pdesc,pdefn,ptask)p_{\text{base}}:=(p_{\text{env}},p_{\text{desc}},p_{\text{defn}},p_{\text{task}}) is divided into (1) a description of the scenario’s rules and objectives, (2) a description of the kinds of inputs it will now receive from that environment, (3) our agreement problem definitions with examples, and (4) a description of its role as a credit assignment agent along with the formatting requirements of its output.
Equation 2
(2) max𝜃​𝔼s0∼ρak∼πθ​(ok)​(∑k= 0∑i= 1NFM​C​A​(CL​L​M​(pk,R⁡(ok,ak),ok+1,ak))),\underset{\theta}{\max}\underset{\begin{subarray}{c}s_{0}\,\sim\,\rho\\ a_{k}\,\sim\,\pi_{\theta}(o_{k})\end{subarray}}{\mathbb{E}}\left(\sum_{k\,=\,0}\sum_{i\,=\,1}^{N}F_{MCA}\big(C_{LLM}\big(p_{k},R(o_{k},a_{k}),o_{k+1},a_{k}\big)\big)\right),

3.3. Large Language Models for Task and Credit Assignment (LLM-TACA)

LLM-TACA makes the critic’s implied task plan explicit. In addition to credits, it can return short integer-array goals directly to a policy, or leave the hint absent. This is intended to reduce volatility when an agent’s emerging plan differs from the critic’s preferred assignment.

Policies reserve zero-padded hint inputs. During training, hints are increasingly replaced with zeros and dropout on associated inputs increases, preparing the agents to operate without them. The exact dropout/withdrawal schedule is not supplied. The retained example includes the complete feedback, Python-like arrays, task hints, and explanation; it is an illustrative model response rather than a formal attribution guarantee.

Equation 3
(3) FT​A​C​A​(CL​L​M)​(pkT,R⁡(ok,ak),ok+1,ak)=(ck1,tk1,…,ckN,tkN).F_{TACA}(C_{LLM})(p^{T}_{k},R(o_{k},a_{k}),o_{k+1},a_{k})=(c_{k}^{1},t_{k}^{1},\ldots,c_{k}^{N},t_{k}^{N}).
Example output from our LLM-TACA method in the level-based foraging benchmark
Figure 4. Example output from our LLM-TACA method in the level-based foraging benchmark. Our LLM-critic provides individualized credit assignments for the previous timesteps, task assignments, and explanations for its decisions.

4. Results

The reported runs use the open Gemma-7B model and Double DQN policies on an NVIDIA 3080 Ti. No systematic language-model comparison is performed, so claims that other models or future reasoning improvements would transfer remain hypotheses.

The fully observed climbing game repeats a common-payoff matrix for 25 steps with constant observations. Level-Based Foraging requires agents’ combined levels to meet each food item’s requirement. Robotic Warehouse uses local 3×3 views and sparse delivery rewards, making navigation, congestion, and coordination difficult.

Spaceworld is a discrete mirror-transport task inspired by telescope servicing. Either agent may move either mirror. Attempting to enter the same cell destroys both robots and ends the episode with zero reward; slower completion loses reward, normalized to 0–10. This creates a safety–efficiency tradeoff within the simulator, without proving collision avoidance outside the evaluated scenarios.

Comparison of our methods with baselines on common benchmarks.
Figure 5. Comparison of our methods with baselines on benchmarks. All values reported are averaged across five separate trainings and the 95% confidence interval is illustrated by the error bars. Spaceworld is our custom benchmark where agents must avoid collision while transporting respective parts to their destinations.".

4.1. Baselines

LICA, MAPPO, and QMIX compare implicit hypernetwork credit assignment, centralized policy-gradient training, and monotone value mixing respectively. Figures 5–6 and Table 1 report five independent training runs with 95% confidence intervals.

The complete table lists seven foraging and three warehouse configurations, plus the matrix game and Spaceworld: twelve tasks in total. Figure 6’s caption instead says twelve scenarios across the two standard benchmarks. Preserve its wording, but use the table’s explicit task counts when describing the evaluation.

Comparison of our methods with baselines on common benchmarks.
Figure 6. Comparison of our methods with baselines on common MARL benchmarks. All values reported are averaged across five separate trainings and the 95% confidence interval is illustrated by the error bars. We compared our methods on 12 scenarios across two common MARL benchmarks. Level-Based Foraging (LBF) is a common benchmark where agents must coordinate to pick apples based on their levels and the level of the apple. Robotic Warehouse (RWARE) is a partially-observable benchmark where robots must move certain packages to the delivery zone. LBF scenario names follow the format: "{GRID SIZE}-{PLAYER COUNT}p-{FOOD COUNT}f-{SIGHT COUNT}{-c IF COOPERATIVE MODE}". RWARE environment names follow the format "{GRID SIZE} {PLAYER COUNT}p".

4.2. Results Analysis

In Spaceworld both proposed methods score 10.0±0.0, versus 6.0±2.5 for MAPPO, 4.4±4.6 for QMIX, and 2.0±2.2 for LICA. This is strong performance on the tested randomized episodes, not a probabilistic safety guarantee.

Foraging means are 0.93 for LLM-MCA and 0.98 for LLM-TACA, versus 0.85 for MAPPO and 0.54 for QMIX. LLM-MCA does not win every configuration: in partially observed 10×10 foraging it scores 0.85 versus QMIX’s 0.89. Warehouse mean returns are 13.34 and 24.12 versus MAPPO’s 1.83; the gains are large, but not uniformly an order of magnitude for every method/configuration.

The matrix-game means are 195 for LLM-MCA and 210 for LLM-TACA. LICA’s 169 is poor relative to these methods but does not mean zero reward, despite the prose’s “completely fails” wording. The displayed payoff maximum 11 over 25 steps gives a theoretical 275, above every reported mean. Full table values and intervals are retained; broad claims of universal dominance are stronger than the measured evidence.

4.3. Offline Dataset

Training produces trajectories annotated with per-agent credits and task assignments. The project page provides dataset instructions. These are generated supervisory labels, not independently validated ground-truth contributions, and the paper does not demonstrate an offline-learning result using them.

5. Conclusions and Future Work

The experiments support language-model-generated rewards and temporary task hints as useful training signals for the selected cooperative simulations. The critic also produces explanations that users can inspect, although explanation fidelity is untested.

Language-model latency remains a limitation despite batching. Closed-model fees are a potential cost, while the reported experiments use an open model. Non-cooperative tasks, faster real-time evaluation, and real-robot deployment are future work. Missing prompt-normalization details, hint schedules, and comprehensive training hyperparameters limit reproducibility from the paper alone.

6. Appendix

The supplement records benchmark naming conventions and the complete numerical comparison.

6.1. Implementation Details

Foraging names encode grid size, player count, food count, optional sight range, and cooperative mode; the historical “-c” suffix became “-coop” in version 3. Warehouse names encode grid size and player count. The complete table preserves all means, 95% intervals, bold best values, and starred second-best entries.

These are collective environment returns across five runs. The caption’s wording about “no common reward or reward scalarization” is preserved, although the method explicitly supplies a shared reward to the critic. The appendix gives no further numerical parser-normalization formula, withdrawal/dropout schedule, or full optimizer/training-budget specification.

Table 1. Comparison of our method with previous solutions on common benchmarks. All values reported are the means and 95% confidence interval for the collective return of the agents by each algorithm over 5 separate training runs with no common reward or reward scalarization techniques. The bold score denotes the highest value of a row and an asterisk*denotes the second highest.
Environments\Algorithms LLM-MCA (Ours) LLM-TACA (Ours) LICA (Zhou et al., 2020) MAPPO (Yu et al., 2022) QMIX (Rashid et al., 2020)
Climbing Matrix Game 195.0±33.5195.0\pm 33.5* 210.0±10.0\mathbf{210.0\pm 10.0} 169.0±13.4169.0\pm 13.4 175.0±0.0175.0\pm 0.0 175.0±54.8175.0\pm 54.8
Spaceworld-10x10 10.0±0.0\mathbf{10.0\pm 0.0} 10.0±0.0\mathbf{10.0\pm 0.0} 2.0±2.22.0\pm 2.2 6.0±2.56.0\pm 2.5* 4.4±4.64.4\pm 4.6
Level Based Foraging
8x8-2p-2f-c 1.00±0.00\mathbf{1.00\pm 0.00} 1.00±0.00\mathbf{1.00\pm 0.00} 0.00±0.000.00\pm 0.00 1.00±0.00\mathbf{1.00\pm 0.00} 0.81±0.070.81\pm 0.07*
8x8-2p-2f-2s-c 0.90±0.020.90\pm 0.02* 1.00±0.00\mathbf{1.00\pm 0.00} 0.01±0.000.01\pm 0.00 0.85±0.050.85\pm 0.05 1.00±0.00\mathbf{1.00\pm 0.00}
10x10-3p-3f 1.00±0.00\mathbf{1.00\pm 0.00} 1.00±0.00\mathbf{1.00\pm 0.00} 0.09±0.010.09\pm 0.01 0.98±0.020.98\pm 0.02* 0.76±0.080.76\pm 0.08
10x10-3p-3f-2s 0.85±0.040.85\pm 0.04 0.99±0.01\mathbf{0.99\pm 0.01} 0.11±0.010.11\pm 0.01 0.72±0.030.72\pm 0.03 0.89±0.010.89\pm 0.01*
15x15-3p-5f 0.88±0.060.88\pm 0.06* 0.96±0.04\mathbf{0.96\pm 0.04} 0.14±0.010.14\pm 0.01 0.75±0.040.75\pm 0.04 0.07±0.040.07\pm 0.04
15x15-4p-3f 0.92±0.080.92\pm 0.08* 1.00±0.00\mathbf{1.00\pm 0.00} 0.07±0.010.07\pm 0.01 0.91±0.050.91\pm 0.05 0.12±0.060.12\pm 0.06
15x15-4p-5f 0.93±0.020.93\pm 0.02* 0.94±0.02\mathbf{0.94\pm 0.02} 0.18±0.010.18\pm 0.01 0.74±0.030.74\pm 0.03 0.24±0.070.24\pm 0.07
Average 0.93±0.030.93\pm 0.03* 0.98±0.01\mathbf{0.98\pm 0.01} 0.09±0.010.09\pm 0.01 0.85±0.070.85\pm 0.07 0.54±0.050.54\pm 0.05
Robotic Warehouse
Tiny 4p 16.44±4.1616.44\pm 4.16* 33.65±2.44\mathbf{33.65\pm 2.44} 0.14±0.020.14\pm 0.02 2.05±0.122.05\pm 0.12 0.33±0.180.33\pm 0.18
Small 4p 13.47±3.1513.47\pm 3.15* 21.59±1.95\mathbf{21.59\pm 1.95} 0.03±0.010.03\pm 0.01 2.01±0.182.01\pm 0.18 0.12±0.110.12\pm 0.11
Tiny 2p 10.12±4.4310.12\pm 4.43* 17.13±2.27\mathbf{17.13\pm 2.27} 0.06±0.020.06\pm 0.02 1.43±0.261.43\pm 0.26 0.14±0.190.14\pm 0.19
Average 13.34±3.9113.34\pm 3.91* 24.12±2.22\mathbf{24.12\pm 2.22} 0.08±0.020.08\pm 0.02 1.83±0.191.83\pm 0.19 0.20±0.160.20\pm 0.16
Concept demonstration

Open image ↗