Overview
CRAFT uses language models to propose a curriculum and executable team rewards, and vision–language feedback to refine rewards after observing multi-robot policy rollouts.
At a glance
Coaching Reinforcement learning Autonomously using Foundation models for coordination Tasks combines staged task decomposition, generated reward code, policy training, and rollout-based feedback. Experiments cover two quadruped navigation tasks and simulated bimanual lifting. A navigation policy transfers to two physical quadrupeds with unchanged network weights but robot-specific command scaling and motion-capture observations. The approach improves the harder simulated tasks, while stochastic training, selected-trial reporting, supplied helper functions, and imperfect rollout evaluation limit the scope of the results.
1 Introduction
Coordination is difficult when agents must learn interacting behaviors from sparse feedback. CRAFT structures this learning as a sequence of subskills, using foundation models as a training-time coach. The resulting execution policy is decentralized and does not query the coach for every control step. Decentralization here does not mean agents lack information about one another: observations include other-agent poses or end-effector state.
2 Related Works
Curriculum methods stage progressively harder behaviors, while reward-generation methods produce executable objectives and revise them from feedback. CRAFT combines these ideas for multi-agent robotics, extending the motivation of CurricuLLM with a rollout-informed refinement loop. Eureka supplies the comparison for reward search without staged curricula. Human-provided environment descriptions, state variables, and helper functions still ground the automated components.
3 Problem Formulation
The target is a decentralized partially observable decision process with a shared team reward. Centralized training learns a shared policy whose execution inputs are agent-local observation vectors, including specified information about teammates. A curriculum changes the reward across stages while preserving the environment. The goal is a sequence of objectives that prepares the final policy for the target behavior, not a sequence of separately executed hand-designed task controllers.
4 Method
The four modules propose subtasks, generate reward code, judge trained policies, and advise/refine rewards after failure. Policies are trained between model queries, so this is a training workflow rather than a foundation model directly commanding the robots online.
4.1 Curriculum Generation Module
The language model generates several candidate curricula and refines them into a final sequence. Subtasks use supplied state-variable concepts, such as gripper-to-handle distance, so rewards and evaluators can refer to measurable quantities. The example prompts cap the curriculum at five tasks and forbid changing initial conditions, robot counts, objects, or termination rules. Candidate refinement is model judgment, not a guaranteed optimizer of curriculum difficulty.
4.2 Reward Function Generation Module
The reward model receives environment/task descriptions, example reward code, helper functions, and earlier subtask rewards. It writes a team-level reward decomposed into named components whose curves can later be inspected. The example helper functions encode substantial task structure, including grasp state, handle distances, and pot tilt. Training uses MAPPO under centralized training and decentralized execution.
4.3 Policy Evaluation Module
A vision–language evaluator receives rollout snapshots, numerical state trajectories, and current/earlier task descriptions. It decides whether to advance and describes failures or forgetting. This is richer than visual-only judgment, and the authors later find that numerical trajectories dominate its decisions. The quantitative reliability check uses final tasks, so it does not establish equal accuracy on all generated intermediate subtasks.
4.4 Reward Refinement Module
After a failed judgment, an advice model examines the reward, failure explanation, and component-learning curves, then a language model rewrites the reward. Up to three refinement rounds are used in the main method. If no attempt passes, a model chooses the best available policy and the curriculum still advances; completion of every stage is therefore not guaranteed. The next stage warm-starts actor weights while resetting exploration parameters and the value network to reduce loss of plasticity.
5 Experiments
Evaluation compares staged coaching with fixed rewards, the final generated reward trained from scratch, and two language-model baselines. Hardware evidence is limited to the gate-navigation task.
5.1 Experiment Setup
All three tasks require coordination between two agents. Observations, success conditions, selected-trial reporting, and modified baseline budgets are essential context for the reported rates.
5.1.1 Simulation Environments
Gate requires both quadrupeds to pass a narrow opening; Seesaw requires one robot to reach a platform while the other balances the board. Their actions are planar and yaw velocities, with lower-level locomotion handled separately. Lift uses two arms with six-dimensional end-effector pose increments plus gripper control; success requires pot height above 0.1 metres and tilt within thirty degrees. Agents observe teammate pose information but not future actions or intentions. These are structured-state control tasks rather than learned raw-image perception.
5.1.2 Baselines & Ablation Study
Fixed environment and example rewards test unstaged learning. From Scratch uses CRAFT’s final reward without curriculum. Eureka searches rewards directly on the final task; CurricuLLM uses a curriculum without the refinement loop. Each method runs five trials, but the main success table reports only the best three, each evaluated over one hundred random initial conditions. Separate nonzero-success fractions and best-trial scores partially expose variability; the main table is not an all-five-trial mean.
| Gate | Seesaw | Lift | |
| Env Reward | |||
| Example Reward | |||
| From Scratch | |||
| Eureka [17] | |||
| CurricuLLM [21] | |||
| CRAFT (Ours) |
5.2 Simulation Results
For the selected top three trials, CRAFT reports 78.33±9.88% Gate, 59.33±10.62% Seesaw, and 94.67±4.99% Lift. Eureka is higher on Gate at 83.33±3.57%, while much lower on Seesaw and Lift. Training CRAFT’s final reward from scratch gives 15.00%, 24.67%, and 8.0%, supporting the value of staging in these comparisons. Successful Lift rollouts look more like stable handle grasps than the twisting/incidental contacts found in Eureka examples, but bimanual hardware feasibility is not tested.
Across all five trials, CRAFT has nonzero final success in four Gate, three Seesaw, and four Lift trials, versus two, two, and one for CurricuLLM. Best CRAFT policies reach 92%, 72%, and 100%; these are maxima rather than expected run quality. “Curriculum success” means any nonzero final success, not reliable completion of every generated stage.
The evaluator confusion matrix has thirty correct judgments out of thirty-five: seven true positives, twenty-three true negatives, five false negatives, and no false positives. Overall accuracy is 85.71%, but only seven of twelve actually successful rollouts are accepted. The small final-task sample cannot establish zero false-positive risk generally. Rejected successes can trigger extra training, and later policy selection remains another model judgment rather than a proof that useful policies will always be recovered.
| CRAFT | CurricuLLM [21] | ||
| Gate | Curriculum Success Ratio | ||
| Maximum Success Rate | |||
| Seesaw | Curriculum Success Ratio | ||
| Maximum Success Rate | |||
| Lift | Curriculum Success Ratio | ||
| Maximum Success Rate |
| VLM Decision | |||
| Positive | Negative | ||
| Actual | Positive | (TP) | (FN) |
| Negative | (FP) | (TN) | |
5.3 Hardware Validation
A policy trained with a Go2 simulation is deployed on a Go2 and a Go1, with shared unchanged policy weights and motion-capture poses for both robots. Gate success is 65% over twenty trials, or thirteen successes. Robot-specific velocity scaling compensates for different low-level controllers and the Go1’s weak response to small commands. The reported zero-shot transfer therefore means no policy-weight finetuning, while deployment still includes sensing infrastructure and command calibration.
6 Conclusions and Limitations
The combination of generated curricula and rollout-based reward refinement learns useful coordination in the tested tasks. Limitations include substantial variation across model/training trials, reliance on engineered state/helper interfaces, and an evaluator that depends strongly on numerical trajectories and can miss visual coordination failures. Repeated simulation attempts and selecting successful policies mitigate some failures but do not establish reliable unattended training or broad hardware transfer. Video reasoning and more structured visual prompts are proposed future work.
Acknowledgments
The authors acknowledge Yuman Gao, Koushil Sreenath, and the Hybrid Robotics Lab for support with hardware experiments.
Appendix A Details on Experiments
The appendix details model versions, policy resets, baseline adaptations, observation spaces, rollout failures, and hardware command post-processing.
A.1 Implementation Details of CRAFT
Language modules use GPT-4o-2024-08-06 and vision–language modules use o4-mini-2025-04-16. MAPPO is implemented through OpenRL for quadrupeds and skrl for bimanual manipulation. Between subtasks, actor weights are retained, policy-standard-deviation exploration parameters return to initialization, and the critic is randomly reset. This preserves useful behavior while allowing exploration and new reward-value estimates; it is more specific than simply copying every network state unchanged.
A.2 Implementation Details of Baselines
Baselines share the MAPPO implementation and relevant language-model interfaces. CurricuLLM removes the feedback/refinement loop from the same prompt structure. Eureka receives CRAFT’s descriptions and helper functions in place of automatically extracted environment source, and uses three refinement iterations with three reward samples each, rather than the original five-by-sixteen setting. Its scalar selection metric is task success. These adaptations improve budget comparability but mean the experiment does not reproduce Eureka’s original search configuration.
A.3 Simulation Environment
Gate observations include agent identity, both robot poses, and gate position; Seesaw includes identity and both poses. Lift observes each arm’s end-effector/gripper and relevant handle relationships in the ego base frame, including teammate information. Quadrupeds output three velocity commands, while arms output pose increments and gripper action. Pot position is randomized at episode reset. These explicit teammate/state channels qualify the decentralized and partially observable formulation.
A.4 Additional Analysis on Experiments
Fixed rewards can stall at partial behaviors such as reaching or grasping without lifting. The complete five-trial Lift panels retain CRAFT’s failed zero-percent trial alongside successful 80–100% trials, and Eureka’s 8–52% trials with less natural contact strategies. A separate Gate experiment applies feedback refinement without curriculum and reaches roughly sixty percent mean success after five rounds. Per-iteration performance drops at one intermediate round before recovering, so refinement is not monotonically beneficial and the cumulative best curve should not be mistaken for each new policy’s performance.
(a) Eureka
(b) CRAFT
A.5 Hardware Experiments
Both real robots use identical actor parameters trained on the simulated Go2. A fixed robot-specific transformation scales velocity commands, including a larger small-command scaling for Go1, before low-level control. The trajectory figure shows one robot waiting while the other passes the gate. This illustrates the learned coordination but does not add hardware evidence for Seesaw or bimanual Lift.
Appendix B Example Prompts
All thirteen prompt listings and the original reward-code listing are preserved verbatim as semantic preformatted text, including placeholders, indentation, source inconsistencies, and full example trajectories. Their content documents the engineered constraints supplied to the models.
B.1 Curriculum Generation Module
Candidate and refinement prompts request task names, descriptions, and reasons, restrict descriptions to provided variables, and cap curricula at five stages. They forbid changing initial placements, counts, or termination conditions. The candidate prompt describes a centralized policy with access to all agents, whereas the method implements centralized training with decentralized execution; that original prompt wording is retained rather than silently rewritten.
You are a curriculum generator trying to generate a curriculum to solve multi-agent reinforcement learning tasks as effectively as possible. Your goal is to write a list of subtasks and corresponding reward function that will help agents to efficiently coordinate for target task. Some helpful tips for generating a curriculum: (1) Do not explore the world by doing random actions and do not try moving to manually defined position. (2) Try to make the curriculum simple. Do not generate more than 5 tasks. (3) The last task of your curriculum should align with the final goal of the environment. (4) In curriculum, you cannot change the environment or the terminating condition. Focus on the task description. (5) You are learning a centralized policy that have access to every agents. You will be given several variables you can use to describe each subtask in curriculum. Each task should be described using the given variables and do not introduce new variables. Your output should be Task 1 Name: Description: Reason: Task 2 Name: Description: Reason: ... Fill out the names, descriptions, and reasons in the format. <<Environment_Description>> Variables you can use for curriculum descriptions are <<State_Variables>> These are the rules that you have to consider when writing a curriculum (1) You cannot change start position of robots and objects. (2) You cannot change the number of robots and objects. (3) You should consider all robots in the environment.
You are a curriculum generator trying to generate a curriculum to solve multi-agent reinforcement learning tasks as effectively as possible. Your goal is to write a list of subtasks and corresponding reward function that will help agents to efficiently coordinate for target task. You will be given three candidates of curriculums that were generated by other generators. Pick the best version and refine it if needed. Note that the maximum possible curriculum is 5. You will be given several variables you can use to describe each subtask in curriculum. Each task should be described using the given variables and do not introduce new variables. Your output should be Task 1 Name: Description: Reason: Task 2 Name: Description: Reason: ... Fill out the names, descriptions, and reasons in the format. <<Environment_Description>> Variables you can use for curriculum descriptions are <<State_Variables>> These are the rules that you have to consider when writing a curriculum (1) You cannot change start position of robots and objects. (2) You cannot change the number of robots and objects. (3) You should consider all robots in the environment. Three candidates of curriculums are: Candidate 1: <<Candidate_1>> Candidate 2: <<Candidate_2>> Candidate 3: <<Candidate_3>>
B.2 Reward Function Generation Module
The reward prompt supplies past tasks/code, current objectives, example rewards, and allowed helper functions. It asks for named components, nonnegative normalized total reward in zero to one, and no new library imports or environment changes. Its use of “integer” return values conflicts with continuous normalized reward examples; the exact original wording remains visible.
You should write reward function for given task using useful variables from the environment. Your reward function is part of the curriculum learning which consists of learning sequence of different tasks. You will be given description of past tasks, reward code for past tasks, and current task description. Your gpt_reward(self) function's output should consist of three items (1) the total reward, an integer (2) a dictionary of each individual reward component where keys are names of reward component and items are each rewards (3) the maximum possible reward that is used to normalize the reward, an integer Some helpful tips for writing a reward function code: (1) Use numpy or scipy functions to write the reward function. numpy, scipy is already imported as np and scipy. Do not import additional library. (2) You may put higher weight on the reward for current task, but also include rewards from past tasks to avoid forgetting. (3) Total reward must stay within [0, 1]. Always, scale the reward to have maximum 1. (4) Only use provided variables. (5) Do not change other predefined parts in the code. <<Environment_Description>> You can use help functions to calculate reward. <<Helper_Functions>> Example reward function for the task is <<Example_Reward>> You should re-implement reward function to assign a proper reward function for given task. You can change the reward components that aligns well with the task. Do not include any new inputs in the functions. Note that you are designing a reward function for team, not a single agent. Generate a reward function code and command for Task Name: <<Task_Name>> Description: <<Task_Description>> Reason: <<Task_Reason>> Previous learned task was Task Name: <<Task_Name>> Description: <<Task_Description>> Reason: <<Task_Reason>> Code: ```python <<Task_Code>> ```
B.3 Policy Evaluation Module
The evaluator prompt asks for Success/Failure and reasons, explicitly checking current completion and forgetting. It supplies chronological images plus full state sequences and describes 0.03 as a scale intuition rather than a hard threshold. Its decision is therefore conditioned on privileged structured information as well as visuals.
You are a multi-agent reinforcement learning engineer asked to evaluate a rollout from a task in a curriculum. You will be provided with: - Snapshot images from the rollout - Trajectory data (state sequences) - Previous and current task descriptions Your job is to determine: 1. Whether the agents achieved the current goal. 2. Whether they have forgotten earlier subtasks. Your output should be Decision: [Success or Failure] Reason: - [reason 1] - [reason 2] ... Fill out the [] brackets in the given format. (Don't keep the [] brackets) Current task is: <<Current task>> Former tasks that RL succeeded to learn is: <<Former tasks>> Given images are snapshots in chronological order of the rollout. Trajectory data: <<Trajectory>> Note that the distance and orientation is considered as ``very close'' if the value is smaller than 0.03 (This is NOT a threshold value, this is given to have a sense of how close they are) Note that the trajectory data is sequential data for a whole episode. Make your decision on both the trajectory data and the image.
B.4 Reward Refinement Module
Advice is limited to three suggestions and the available helpers, with negative rewards forbidden. The refinement prompt must rewrite code within the supplied structure and consider earlier tasks. It says the output has “two items” but enumerates three; the source inconsistency is preserved. A downstream code interface must resolve such ambiguities, which the prompt text itself does not formally validate.
You are a multi agent reinforcement learning engineer trying to refine a reward function for a given task. You will be given: the reward function that was tried and failed, the experimental results, a description of why it failed, and the reward-component curve images. First, analyze the reason why the reward function failed to learn the task. Then, provide advice on how to improve or modify the reward function to achieve the task. These are some tips in giving advice: (1) Do not give more than three advice. (2) You cannot add a penalty in the reward function, e.g. no reward can be a negative value. These are the only functions that are allowed to used. <<Helper_Functions>> Information that cannot be obtained from the above functions cannot be used in the reward function. Current task is: <<Task_Name>> <<Task_Description>> <<Task_Reason>> Reward function that was tried: <<Reward_Function>> Failure reason: <<Failure_Reason>> Do not give an advice on the curriculum of the reward. Just consider how to improve the reward function given the current task. Do not provide the whole code of the reward function, just give advice on how to improve it.
You are a multi agent reinforcement learning engineer trying to refine a reward function for the given task.
The reward function is part of the curriculum learning which consists of learning sequence of different tasks.
You will be given description of past tasks, a reward function that was tried and failed, and an advice on how to improve it.
Follow the advice and provide a new reward function that should be able to achieve the given task.
Do not repeat the advice or the old reward function in your answer.
Your gpt_reward(self) function's output should consist of two items
(1) the total reward, an integer
(2) a dictionary of each individual reward component where keys are names of reward component and items are each rewards
(3) the maximum possible reward that is used to normalize the reward, an integer
Some helpful tips for writing a reward function code:
(1) Use numpy or scipy functions to write the reward function. numpy, scipy is already imported as np and scipy. Do not import additional library.
(2) You may put higher weight on the reward for current task, but also include rewards from past tasks to avoid forgetting.
(3) Total reward must stay within [0, 1]. Always, scale the reward to have maximum 1.
(4) Only use provided variables.
(5) Do not change other predefined parts in the code.
You can use help functions to calculate reward.
<<Helper_Functions>>
Current task is:
<<Current_Task>>
Former tasks that RL succeeded to learn is:
<<Former_Tasks>>
Reward function that was tried:
<<Reward_Function>>
Advice on how to improve the reward function:
<<Advice>>
Do not deviate from the reward structure given in the example.
Note that the advice might not acount for what information it can use in the reward function. So not all the advice might be applicable.
B.5 Task-Specific Substitutions for Two Arm Lift
Lift-specific substitutions provide the pot/handle geometry, height/tilt criterion, distances, orientation errors, grasp indicators, and helper functions. The full example reward combines reaching, alignment, lifting, grasping, and success, normalized by 7.5. Its helper success check concerns height, while a separate tilt coefficient gates terminal/lift reward. The listing preserves every operation and indentation level, rather than replacing executable details with a prose sketch.
A large pot with two handles is placed on a table top. The pot is rectangular shaped with U-shaped handles on the opposite side of the pot rim. The grip of the handle is level, parallel to the tabletop. Two robot arms are placed on opposite ends of the table. The two robot arms must each grab a handle and lift the pot together, above a certain height (0.1m), while keeping the pot level (no more tilted than 30 degrees). The pot location is randomized at the beginning of each episode.
- Euclidean distance from each robot's end-effector and its target handle - Orientation difference between each robot's end-effector and its target handle - Gripper state of each robot (whether it grasped the handle or not) - Pot elevation above the table - Pot tilt as cosine of the angle between its normal and the vertical
self._get_check_grasps(): returns two boolean value for each arm whether or not it's grasping the handle or not. self._get_pot_elevation(): returns the distance (in meters) from pot's bottom to the table top self._get_tilt_degree(): returns the cosine value of the tilted degree of the pot. The tilted degree is the difference between the z axis and pot's normal vector self._get_gripper_to_handle_distance(): returns two float value of distance from each robot's end-effector to each handle self._get_gripper_to_handle_rotation(): returns two rotation difference (calculated with R1.T @ R2) between each robot's end-effector to each handle with shape: (3,3) each self._check_success(): returns True if the pot is higher than 0.1
def gpt_reward(self):
self.gpt_rew = 0
self.gpt_rew_dict = {}
max_reward = 7.5
# reward components
reach_reward = 0.0
align_reward = 0.0
lift_reward = 0.0
grasp_reward = 0.0
success_reward = 0.0
# check if pot is tilted less than 30 degrees
cos_z = self._get_tilt_degree()
cos_30 = np.cos(np.pi / 6)
direction_coef = 1 if cos_z >= cos_30 else 0
# check for goal completion: cube is higher than the table top above a margin
if self._check_success():
success_reward += max_reward * direction_coef
else:
# Reaching reward - max 0.2
_g0h_dist, _g1h_dist = self._get_gripper_to_handle_distance()
reach_reward += 0.1 * (1 - np.tanh(10.0 * _g0h_dist))
reach_reward += 0.1 * (1 - np.tanh(10.0 * _g1h_dist))
# aligning reward - max 0.3
_g0h_rot, _g1h_rot = self._get_gripper_to_handle_rotation()
_g0h_rot_diff = np.trace(np.eye(3) - _g0h_rot)
_g1h_rot_diff = np.trace(np.eye(3) - _g1h_rot)
align_reward += 0.15 * (1 - np.tanh(10.0 * _g0h_rot_diff))
align_reward += 0.15 * (1 - np.tanh(10.0 * _g1h_rot_diff))
# lifting reward - max 5.0
elevation = self._get_pot_elevation()
r_lift = min(max(elevation, 0), 0.2)
lift_reward += 25.0 * direction_coef * r_lift
# Grasping reward - max 2.0
g0_grasp, g1_grasp = self._get_check_grasps()
if g0_grasp:
grasp_reward += 1.0
if g1_grasp:
grasp_reward += 1.0
# Add and normalize the reward
self.gpt_rew = reach_reward + align_reward + lift_reward + grasp_reward + success_reward
self.gpt_rew *= 1.0 / max_reward
self.gpt_rew_dict = {
'reach_reward': reach_reward,
'align_reward': align_reward,
'lift_reward': lift_reward,
'grasp_reward': grasp_reward,
'success_reward': success_reward,
}
return self.gpt_rew, self.gpt_rew_dict, max_reward
Appendix C Examples in VLM-guided Reward Refinement
The final appendix shows the actual seesaw evaluation and reward-advice exchange, including rollout snapshots, numeric trajectories, failed reward code, component curves, and complete model responses. These examples demonstrate the workflow rather than independently validating every generated statement.
C.1 Example Results of Policy Evaluation
The evaluation example asks whether one robot traverses the seesaw while the other stabilizes its start, after an alignment stage. It provides both robots’ full position sequences and an instruction not to be too strict because this is intermediate. The retained response judges success based on sequential entry and stability. This illustrates how prompt framing and quantitative trajectories can influence advancement.





Current task is: Task Name: 2_Seesaw Entry and Stability Task Description: Train both agents to coordinate so that Agent 0 enters and traverses the seesaw toward the end at position [x=7, y=0], ensuring gradual and controlled movement to maintain balance. Meanwhile, Agent 1 enters the seesaw afterward, stopping and stabilizing at the start position [x=4, y=0, z=1.0]. Reason: Establishing sequential entry and motion ensures Agent 0 can proceed unimpeded while Agent 1 stabilizes the seesaw. Effective coordination during this phase prevents excessive tilting, allowing for smooth advancement toward the goal. Former tasks that RL succeeded to learn is: Task Name: 1_Initial Alignment Task Description: Ensure Agent 0 (Blue) aligns itself with the start of the seesaw at position [x=4, y=0, z=1.0] while Agent 1 (Green) maintains its initial position, preparing to follow Agent 0. Both agents should aim to keep y position near 0 to stay within the seesaw's width. Reason: This task sets the foundational entry sequence where Agent 0 leads the way, ensuring it is ready to proceed onto the seesaw without obstruction from Agent 1. This preparation is crucial for establishing entry control and ensuring proper seesaw engagement. Given images are snapshots in chronological order of the rollout. Agent 0 is colored with blue and Agent 1 is colored with green. Trajectory data: seesaw_center: [5.500 0.000 1.000] seesaw_start: [4.000 0.000] seesaw_end: [7.000 0.000] target_pos: [7.700 0.000 1.500] agent_0_pos: [[0.920 -0.730 0.420] [1.380 -0.550 0.270] [2.130 -0.250 0.280] [2.910 -0.070 0.290] [3.640 -0.010 0.330] [4.360 0.030 0.520] [5.210 0.140 0.740] [5.910 0.320 0.920] [6.250 0.240 0.960] [6.910 0.210 1.150] [7.280 0.200 1.110] [7.250 0.080 0.970] [7.200 -0.030 1.010] [7.270 -0.020 1.240] [7.850 0.110 1.300] [7.950 -0.080 1.290] [7.980 -0.250 1.280] [8.070 -0.370 1.280] [8.160 -0.450 1.220] [8.160 -0.390 1.220] [8.090 -0.370 1.220]] agent_1_pos: [[0.920 0.830 0.420] [1.120 0.850 0.270] [1.500 0.600 0.280] [2.190 0.400 0.290] [2.890 0.370 0.280] [3.590 0.390 0.350] [4.140 0.390 0.450] [4.210 0.370 0.430] [4.150 0.180 0.420] [4.210 0.160 0.450] [4.320 0.200 0.630] [4.380 0.150 0.740] [4.380 0.260 0.610] [4.380 0.120 0.460] [4.460 -0.050 0.510] [4.510 0.070 0.570] [4.400 0.040 0.530] [4.440 0.070 0.550] [4.340 -0.010 0.510] [4.300 0.010 0.470] [4.220 -0.110 0.420]] Note that this is not the final goal in the curriculum, don't be too strict on the decision.
Decision: Success Reason: - Agent 0 led the sequence, entered the seesaw first, and traversed to and beyond the target end (around x = 7.2-8.2) with smooth, controlled height (z) changes, indicating balance was maintained. - Agent 1 only stepped onto the seesaw after Agent 0 had committed, then remained stably near the start region (x ~ 4.2, y ~ 0, small z-oscillations), satisfying the stabilization requirement. - The initial-alignment behaviour is intact (Agent 1 kept close to y = 0 while Agent 0 aligned to the seesaw before entry), showing no forgetting of the previous subtask.
C.2 Example Results of Reward Advice
The failed ascension reward contains many components and negative fall/collision terms despite the general nonnegative-reward instruction. The advice proposes strengthening forward/height incentives, replacing negative penalties with survival bonuses, and concentrating the second robot’s balance signal. Some suggestions require checking that the existing helpers expose the needed quantities, as the refinement prompt itself cautions. The complete original code, curves, failure explanation, and advice remain available for assessing this gap.








Current task is:
Name: Target Platform Ascension Task
Description: Ensure Agent 0 continues confidently toward the end of the seesaw and reaches the target platform [x coordinate > 7, height > 1.3], while Agent 1 maintains stability at the seesaw's starting point. Agent 1 should ensure it provides enough counterbalance to prevent the seesaw from destabilizing.
Reason: This task focuses on culmination and assessment of learned coordination strategies, where Agent 0 must finalize its movement to reach the target while relying on Agent 1 to maintain seesaw balance. Successfully reaching the platform signifies the completion of the objective and tests the agents' ability to coordinate effectively under the primary task constraints.
Reward function that was tried:
def gpt_reward(self, state, action):
reward = torch.zeros((self.num_envs, self.num_agents), device=self.env.device)
rew_dict = {}
max_reward = 23.0
# Reward agent 0 for progressing towards the target platform - max 3.0
agent_0_progress_reward = 3.0 * self._progress_to_target(state, action)
# Only reward agent 0 when progressing towards the target
agent_0_progress_reward[:, 1] = 0.0
agent_0_progress_reward = self._check_reward_shape(agent_0_progress_reward)
reward += agent_0_progress_reward
rew_dict['agent_0_progress'] = torch.mean(agent_0_progress_reward[:, 0])
# Reward agent 1 for holding position at the seesaw start - max 3.0
agent_1_stability_reward = 3.0 * self._progress_to_seesaw_start(state, action)
# Only reward agent 1 for maintaining stability
agent_1_stability_reward[:, 0] = 0.0
agent_1_stability_reward = self._check_reward_shape(agent_1_stability_reward)
reward += agent_1_stability_reward
rew_dict['agent_1_stability'] = torch.mean(agent_1_stability_reward[:, 1])
# Reward for maintaining y position close to 0 for both agents - max 2.0
y_alignment_reward = 2.0 * self._y_alignment(state, action)
y_alignment_reward = self._check_reward_shape(y_alignment_reward)
reward += y_alignment_reward
rew_dict['y_alignment'] = torch.mean(y_alignment_reward)
# Reward for height progression for Agent 0, aiming to reach above 1.3m - max 2.0
height_reward = 2.0 * self._normalized_height(state, action)
height_reward[:, 1] = 0.0 # Only agent 0 should get height reward
height_reward = self._check_reward_shape(height_reward)
reward += height_reward
rew_dict['height_reward'] = torch.mean(height_reward[:, 0])
# Penalty for falling off the seesaw - min -2.0
fall_penalty = -2.0 * self._fall(state, action).float()
fall_penalty = self._check_reward_shape(fall_penalty)
reward += fall_penalty
rew_dict['fall_penalty'] = torch.mean(fall_penalty)
# Penalty for colliding with walls or structures - min -1.0
collision_penalty = -1.0 * self._wall_collision(state, action).float()
collision_penalty = self._check_reward_shape(collision_penalty)
reward += collision_penalty
rew_dict['collision_penalty'] = torch.mean(collision_penalty)
# Reward for maintaining an appropriate distance between agents - max 1.0
agent_distance = self._agent_distance(state, action)
distance_reward = 1.0 * torch.clamp(torch.square(agent_distance), max=1.0)
distance_reward = self._check_reward_shape(distance_reward)
reward += distance_reward
rew_dict['distance_reward'] = torch.mean(distance_reward)
# Reward for success when Agent 0 reaches the platform - max 10.0
success = self._success(state, action)
success_reward = 10.0 * success.float()
success_reward = self._check_reward_shape(success_reward)
reward += success_reward
rew_dict['success'] = torch.mean(success_reward)
# Normalize the reward
reward = self._check_reward_shape(reward)
reward /= max_reward
return reward, rew_dict, max_reward
Failure reason:
- Agent 0 never reached the target platform region defined by x>7 and height z>1.3; its maximum z was only ~1.1 near x~6.86 before falling back down.
- Agent 1 did not hold the seesaw at the start point (x~4); it remained around x=0.8~0.9 throughout, indicating it forgot the earlier alignment and stability subtasks.
Do not give an advice on the curriculum of the reward. Just consider how to improve the reward function given the current task.
Do not provide the whold code of the reward function, just give advices on how to improve it.
Here are three focused suggestions for reshaping your reward so that Agent 0 is driven all the way up onto the platform and Agent 1 stays firmly at the seesaw pivot: 1. Re-weight and de-crowd your signal for Agent 0's forward/vertical progress - Right now ``agent_0_progress'' (*3) and ``height_reward'' (*2) get drowned in the many small terms and then all get divided by 23. Even at full scale they only amount to ~0.1 per step. Boost those two terms by an order of magnitude (e.g. *20-30 for progress, *10-15 for height) and then remove or dramatically shrink the less critical terms (y_alignment, distance_reward) so that the agent sees a strong, unambiguous gradient pulling it to x>7 and height>1.3. 2. Turn your fall-and-collision ``penalties'' into positive stay-alive/balance bonuses - You've defined negative fall/collision terms, but the spec forbids negative rewards anyway, and they barely ever fire (1e-9 scale). Instead give a small constant ``survival'' bonus every step that you remain upright and collision-free (e.g. +1 per step of life/collision-free). This keeps all components >0 and provides a steady shaping signal without explicit negatives. 3. Concentrate Agent 1's reward on seesaw balance, not just position drift - '_progress_to_seesaw_start' only measures how close Agent 1 got initially, but says nothing about ongoing balance. Repurpose your y_alignment term (or distance_reward) so that it's applied only to Agent 1 and heavily weighted: reward 1 - tilt (i.e. how flat the seesaw is) or reward the inverse of Agent 1's deviation from the pivot. This gives Agent 1 a clear continuous incentive to lock itself at the fulcrum and keep the seesaw horizontal while Agent 0 climbs.