Literature
Display
← All papers

CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks

Seoyeon Choi · Kanghyun Ryu · Jonghoon Ock · Negar Mehr

Preprint posted · arXiv (preprint) ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

CRAFT uses language models to propose a curriculum and executable team rewards, and vision–language feedback to refine rewards after observing multi-robot policy rollouts.

At a glance

Coaching Reinforcement learning Autonomously using Foundation models for coordination Tasks combines staged task decomposition, generated reward code, policy training, and rollout-based feedback. Experiments cover two quadruped navigation tasks and simulated bimanual lifting. A navigation policy transfers to two physical quadrupeds with unchanged network weights but robot-specific command scaling and motion-capture observations. The approach improves the harder simulated tasks, while stochastic training, selected-trial reporting, supplied helper functions, and imperfect rollout evaluation limit the scope of the results.

1 Introduction

Coordination is difficult when agents must learn interacting behaviors from sparse feedback. CRAFT structures this learning as a sequence of subskills, using foundation models as a training-time coach. The resulting execution policy is decentralized and does not query the coach for every control step. Decentralization here does not mean agents lack information about one another: observations include other-agent poses or end-effector state.

Refer to caption
Refer to caption
Figure 1: (Left) Overview of CRAFT. A curriculum LLM decomposes the target multi-agent task into a sequence of subtasks that progressively build towards coordination. For each subtask ℳk\mathcal{M}_{k}, a reward LLM generates a reward RkjR_{k}^{j}, MARL trains a decentralized policy πkj\pi_{k}^{j}, and a VLM evaluates rollout images and state trajectories. If the policy fails, an advice VLM generates advice on refining the reward from reward component learning curves. Otherwise, the curriculum proceeds to the next subtask. (Right) Demonstration of the Quadruped Gate task, where two quadrupeds coordinate to pass through a narrow gate without collision.

2 Related Works

Curriculum methods stage progressively harder behaviors, while reward-generation methods produce executable objectives and revise them from feedback. CRAFT combines these ideas for multi-agent robotics, extending the motivation of CurricuLLM with a rollout-informed refinement loop. Eureka supplies the comparison for reward search without staged curricula. Human-provided environment descriptions, state variables, and helper functions still ground the automated components.

3 Problem Formulation

The target is a decentralized partially observable decision process with a shared team reward. Centralized training learns a shared policy whose execution inputs are agent-local observation vectors, including specified information about teammates. A curriculum changes the reward across stages while preserving the environment. The goal is a sequence of objectives that prepares the final policy for the target behavior, not a sequence of separately executed hand-designed task controllers.

4 Method

The four modules propose subtasks, generate reward code, judge trained policies, and advise/refine rewards after failure. Policies are trained between model queries, so this is a training workflow rather than a foundation model directly commanding the robots online.

Refer to caption
Figure 2: Example of curriculum refinement for task lift and balance the pot. Three different candidate curricula 𝒞1\mathcal{C}^{1} to 𝒞3\mathcal{C}^{3}, generated by the curriculum module, are re-provided to the LLM for refinement. In 𝒞1\mathcal{C}^{1}, Task 1 focuses only on minimizing distance, while Task 1 in 𝒞3\mathcal{C}^{3} is defined as minimizing distance and matching orientation. In contrast, Task 3 and Task 4 in 𝒞1\mathcal{C}^{1} break down the lifting into two stages of first lifting halfway and then to a full height, whereas 𝒞3\mathcal{C}^{3} represents lifting as a single task. The curriculum LLM merges these candidates into a final curriculum 𝒞\mathcal{C} by selecting the stronger tasks definitions from each candidate.

4.1 Curriculum Generation Module

The language model generates several candidate curricula and refines them into a final sequence. Subtasks use supplied state-variable concepts, such as gripper-to-handle distance, so rewards and evaluators can refer to measurable quantities. The example prompts cap the curriculum at five tasks and forbid changing initial conditions, robot counts, objects, or termination rules. Candidate refinement is model judgment, not a guaranteed optimizer of curriculum difficulty.

4.2 Reward Function Generation Module

The reward model receives environment/task descriptions, example reward code, helper functions, and earlier subtask rewards. It writes a team-level reward decomposed into named components whose curves can later be inspected. The example helper functions encode substantial task structure, including grasp state, handle distances, and pot tilt. Training uses MAPPO under centralized training and decentralized execution.

4.3 Policy Evaluation Module

A vision–language evaluator receives rollout snapshots, numerical state trajectories, and current/earlier task descriptions. It decides whether to advance and describes failures or forgetting. This is richer than visual-only judgment, and the authors later find that numerical trajectories dominate its decisions. The quantitative reliability check uses final tasks, so it does not establish equal accuracy on all generated intermediate subtasks.

Refer to caption
Figure 3: Example of reward refinement of subtask Coordinate Preliminary Lift. Through the first reward-refinement loop, Rk=31R^{1}_{k=3} was produced and the evaluation VLM marked the policy as a failure since the pot never reached the required elevation of 0.05 m. The reward component learning curves were then passed to the advice VLM, which identified that lift_reward was too weak compared to balance_reward. It recommended removing the square on elevation, increasing the lift weight, and decreasing the balance weight. The revised reward Rk=32R^{2}_{k=3} reflects these changes: the square on elevation was removed, the lift weight increased from 80 to 200, and the balance weight decreased from 2 to 1. With this reward, the policy successfully achieved the 0.05 m elevation and satisfied the subtask.

4.4 Reward Refinement Module

After a failed judgment, an advice model examines the reward, failure explanation, and component-learning curves, then a language model rewrites the reward. Up to three refinement rounds are used in the main method. If no attempt passes, a model chooses the best available policy and the curriculum still advances; completion of every stage is therefore not guaranteed. The next stage warm-starts actor weights while resetting exploration parameters and the value network to reduce loss of plasticity.

5 Experiments

Evaluation compares staged coaching with fixed rewards, the final generated reward trained from scratch, and two language-model baselines. Hardware evidence is limited to the gate-navigation task.

5.1 Experiment Setup

All three tasks require coordination between two agents. Observations, success conditions, selected-trial reporting, and modified baseline budgets are essential context for the reported rates.

5.1.1 Simulation Environments

Gate requires both quadrupeds to pass a narrow opening; Seesaw requires one robot to reach a platform while the other balances the board. Their actions are planar and yaw velocities, with lower-level locomotion handled separately. Lift uses two arms with six-dimensional end-effector pose increments plus gripper control; success requires pot height above 0.1 metres and tilt within thirty degrees. Agents observe teammate pose information but not future actions or intentions. These are structured-state control tasks rather than learned raw-image perception.

Refer to caption
Refer to caption
Refer to caption
Figure 4: Illustrative snapshot showing successful execution of multi-agent coordination tasks by policies trained by CRAFT.

5.1.2 Baselines & Ablation Study

Fixed environment and example rewards test unstaged learning. From Scratch uses CRAFT’s final reward without curriculum. Eureka searches rewards directly on the final task; CurricuLLM uses a curriculum without the refinement loop. Each method runs five trials, but the main success table reports only the best three, each evaluated over one hundred random initial conditions. Separate nonzero-success fractions and best-trial scores partially expose variability; the main table is not an all-five-trial mean.

Gate Seesaw Lift
Env Reward 0.60±1.200.60\pm 1.20 0.00±0.000.00\pm 0.00 1.33±1.891.33\pm 1.89
Example Reward 0.00±0.000.00\pm 0.00 4.19±5.084.19\pm 5.08 9.33±3.779.33\pm 3.77
From Scratch 15.00±21.2115.00\pm 21.21 24.67±6.6024.67\pm 6.60 8.0±11.318.0\pm 11.31
Eureka [17] 83.33±3.57\mathbf{83.33\pm 3.57} 15.00±11.3415.00\pm 11.34 41.33±12.3641.33\pm 12.36
CurricuLLM [21] 27.75±35.5927.75\pm 35.59 21.82±24.1321.82\pm 24.13 25.33±35.8325.33\pm 35.83
CRAFT (Ours) 78.33±9.88\mathbf{78.33\pm 9.88} 59.33±10.62\mathbf{59.33\pm 10.62} 94.67±4.99\mathbf{94.67\pm 4.99}

Table 1: Success Rate(%) of the top-three out of 55 trials from each environment. Each environment is evaluated by 100 random initial conditions. CRAFT achieves the highest success rates in Seesaw and Lift, and remains competitive with the strongest baseline in Gate.
Refer to caption
(a) Eureka
Refer to caption
(b) CRAFT
Figure 5: Highest success rate Lift policies from Eureka and CRAFT. Eureka shows unnatural joint configurations, while CRAFT demonstrates more physically feasible motions.

5.2 Simulation Results

For the selected top three trials, CRAFT reports 78.33±9.88% Gate, 59.33±10.62% Seesaw, and 94.67±4.99% Lift. Eureka is higher on Gate at 83.33±3.57%, while much lower on Seesaw and Lift. Training CRAFT’s final reward from scratch gives 15.00%, 24.67%, and 8.0%, supporting the value of staging in these comparisons. Successful Lift rollouts look more like stable handle grasps than the twisting/incidental contacts found in Eureka examples, but bimanual hardware feasibility is not tested.

Across all five trials, CRAFT has nonzero final success in four Gate, three Seesaw, and four Lift trials, versus two, two, and one for CurricuLLM. Best CRAFT policies reach 92%, 72%, and 100%; these are maxima rather than expected run quality. “Curriculum success” means any nonzero final success, not reliable completion of every generated stage.

The evaluator confusion matrix has thirty correct judgments out of thirty-five: seven true positives, twenty-three true negatives, five false negatives, and no false positives. Overall accuracy is 85.71%, but only seven of twelve actually successful rollouts are accepted. The small final-task sample cannot establish zero false-positive risk generally. Rejected successes can trigger extra training, and later policy selection remains another model judgment rather than a proof that useful policies will always be recovered.

CRAFT CurricuLLM [21]
Gate Curriculum Success Ratio 𝟒/𝟓\mathbf{4/5} 2/52/5
Maximum Success Rate 𝟗𝟐%\mathbf{92\%} 78%78\%
Seesaw Curriculum Success Ratio 𝟑/𝟓\mathbf{3/5} 2/52/5
Maximum Success Rate 𝟕𝟐%\mathbf{72\%} 55%55\%
Lift Curriculum Success Ratio 𝟒/𝟓\mathbf{4/5} 1/51/5
Maximum Success Rate 𝟏𝟎𝟎%\mathbf{100\%} 76%76\%

Table 2: Curriculum success ratio measures the fraction of curricula with nonzero success across 55 trials, while maximum success rate reports the best policy over 55 trials. CRAFT produces more reliable curricula than CurricuLLM, and its higher maximum success rate shows the benefit of reward refinement for generating better policies.
VLM Decision
Positive Negative
Actual Positive 7/357/35 (TP) 5/355/35 (FN)
Negative 0/350/35 (FP) 23/3523/35 (TN)

Table 3: Confusion matrix for VLM policy evaluation on final task rollouts, aggregated across environments. TP, FN, FP, and TN indicate true positives, false negatives, false positives, and true negatives. The VLM achieves 85.71%85.71\% accuracy, with all errors corresponding to false negatives.

5.3 Hardware Validation

A policy trained with a Go2 simulation is deployed on a Go2 and a Go1, with shared unchanged policy weights and motion-capture poses for both robots. Gate success is 65% over twenty trials, or thirteen successes. Robot-specific velocity scaling compensates for different low-level controllers and the Go1’s weak response to small commands. The reported zero-shot transfer therefore means no policy-weight finetuning, while deployment still includes sensing infrastructure and command calibration.

6 Conclusions and Limitations

The combination of generated curricula and rollout-based reward refinement learns useful coordination in the tested tasks. Limitations include substantial variation across model/training trials, reliance on engineered state/helper interfaces, and an evaluator that depends strongly on numerical trajectories and can miss visual coordination failures. Repeated simulation attempts and selecting successful policies mitigate some failures but do not establish reliable unattended training or broad hardware transfer. Video reasoning and more structured visual prompts are proposed future work.

Acknowledgments

The authors acknowledge Yuman Gao, Koushil Sreenath, and the Hybrid Robotics Lab for support with hardware experiments.

Appendix A Details on Experiments

The appendix details model versions, policy resets, baseline adaptations, observation spaces, rollout failures, and hardware command post-processing.

A.1 Implementation Details of CRAFT

Language modules use GPT-4o-2024-08-06 and vision–language modules use o4-mini-2025-04-16. MAPPO is implemented through OpenRL for quadrupeds and skrl for bimanual manipulation. Between subtasks, actor weights are retained, policy-standard-deviation exploration parameters return to initialization, and the critic is randomly reset. This preserves useful behavior while allowing exploration and new reward-value estimates; it is more specific than simply copying every network state unchanged.

A.2 Implementation Details of Baselines

Baselines share the MAPPO implementation and relevant language-model interfaces. CurricuLLM removes the feedback/refinement loop from the same prompt structure. Eureka receives CRAFT’s descriptions and helper functions in place of automatically extracted environment source, and uses three refinement iterations with three reward samples each, rather than the original five-by-sixteen setting. Its scalar selection metric is task success. These adaptations improve budget comparability but mean the experiment does not reproduce Eureka’s original search configuration.

A.3 Simulation Environment

Gate observations include agent identity, both robot poses, and gate position; Seesaw includes identity and both poses. Lift observes each arm’s end-effector/gripper and relevant handle relationships in the ego base frame, including teammate information. Quadrupeds output three velocity commands, while arms output pose increments and gripper action. Pot position is randomized at episode reset. These explicit teammate/state channels qualify the decentralized and partially observable formulation.

A.4 Additional Analysis on Experiments

Fixed rewards can stall at partial behaviors such as reaching or grasping without lifting. The complete five-trial Lift panels retain CRAFT’s failed zero-percent trial alongside successful 80–100% trials, and Eureka’s 8–52% trials with less natural contact strategies. A separate Gate experiment applies feedback refinement without curriculum and reaches roughly sixty percent mean success after five rounds. Per-iteration performance drops at one intermediate round before recovering, so refinement is not monotonically beneficial and the cumulative best curve should not be mistaken for each new policy’s performance.

Refer to caption
Refer to caption
Refer to caption
Figure 6: Illustrative snapshot of policies trained with Env Reward without curriculum. The policy shows suboptimal behaviors, such as only one agent passing the gate, agent failing to balance the seesaw, or only managing to grasp the pot, rather than lifting it.
Refer to caption
(a) SR 8% (F)
Refer to caption
(b) SR 12% (F)
Refer to caption
(c) SR 24% (F)
Refer to caption
(d) SR 48% (S)
Refer to caption
(e) SR 52% (S)

(a) Eureka

Refer to caption
(f) SR 0% (F)
Refer to caption
(g) SR 80% (S)
Refer to caption
(h) SR 88% (S)
Refer to caption
(i) SR 96% (S)
Refer to caption
(j) SR 100% (S)

(b) CRAFT

Figure 7: Illustrative snapshots from the Lift task across 55 trials for each method. Each snapshot is labeled with the policy success rate (SR) and whether the shown rollout succeeds (S) or fails (F). Eureka often achieves partial success but exhibits less physically feasible motions, whereas CRAFT produces more consistent successful rollouts. For Eureka, snapshots are taken from the first rollout of each trained policy. For CRAFT, snapshots are taken from the rollouts used by the policy evaluation module.
Original figure
(a) Success rate at each refinement iteration.
Original figure
(b) Best success rate up to each iteration.
Figure 8: Effect of VLM-guided reward refinement in the Gate environment without staged curriculum training. We report the mean and standard deviation over five runs. The per-iteration success rate measures the policy trained with the reward from each refinement iteration, while the cumulative best success rate reports the best performance achieved up to that iteration. Although performance drops at sample 3, it recovers at sample 4, showing that the refinement loop can overcome occasional unsuccessful updates.

A.5 Hardware Experiments

Both real robots use identical actor parameters trained on the simulated Go2. A fixed robot-specific transformation scales velocity commands, including a larger small-command scaling for Go1, before low-level control. The trajectory figure shows one robot waiting while the other passes the gate. This illustrates the learned coordination but does not add hardware evidence for Seesaw or bimanual Lift.

Original figure
Original figure
Original figure
Figure 9: Trajectory of quadrupeds in our hardware experiments for Quadruped Gate policy trained with CRAFT. Two quadrupeds exhibit coordination behaviors in real world and manage to pass the gate sequentially without collision.

Appendix B Example Prompts

All thirteen prompt listings and the original reward-code listing are preserved verbatim as semantic preformatted text, including placeholders, indentation, source inconsistencies, and full example trajectories. Their content documents the engineered constraints supplied to the models.

B.1 Curriculum Generation Module

Candidate and refinement prompts request task names, descriptions, and reasons, restrict descriptions to provided variables, and cap curricula at five stages. They forbid changing initial placements, counts, or termination conditions. The candidate prompt describes a centralized policy with access to all agents, whereas the method implements centralized training with decentralized execution; that original prompt wording is retained rather than silently rewritten.

Prompt 1: LLM prompt for generating curriculum candidates.
You are a curriculum generator trying to generate a curriculum to solve multi-agent reinforcement learning tasks as effectively as possible. Your goal is to write a list of subtasks and corresponding reward function that will help agents to efficiently coordinate for target task.

Some helpful tips for generating a curriculum:
(1) Do not explore the world by doing random actions and do not try moving to manually defined position.
(2) Try to make the curriculum simple. Do not generate more than 5 tasks.
(3) The last task of your curriculum should align with the final goal of the environment.
(4) In curriculum, you cannot change the environment or the terminating condition. Focus on the task description.
(5) You are learning a centralized policy that have access to every agents.

You will be given several variables you can use to describe each subtask in curriculum.
Each task should be described using the given variables and do not introduce new variables.

Your output should be
Task 1
Name:
Description:
Reason:

Task 2
Name:
Description:
Reason:

...

Fill out the names, descriptions, and reasons in the format.

<<Environment_Description>>

Variables you can use for curriculum descriptions are
<<State_Variables>>

These are the rules that you have to consider when writing a curriculum
(1) You cannot change start position of robots and objects.
(2) You cannot change the number of robots and objects.
(3) You should consider all robots in the environment.
Prompt 2: LLM prompt for refining curriculum candidates.
You are a curriculum generator trying to generate a curriculum to solve multi-agent reinforcement learning tasks as effectively as possible. Your goal is to write a list of subtasks and corresponding reward function that will help agents to efficiently coordinate for target task.

You will be given three candidates of curriculums that were generated by other generators.
Pick the best version and refine it if needed.
Note that the maximum possible curriculum is 5.

You will be given several variables you can use to describe each subtask in curriculum.
Each task should be described using the given variables and do not introduce new variables.

Your output should be
Task 1
Name:
Description:
Reason:

Task 2
Name:
Description:
Reason:

...

Fill out the names, descriptions, and reasons in the format.

<<Environment_Description>>

Variables you can use for curriculum descriptions are
<<State_Variables>>

These are the rules that you have to consider when writing a curriculum
(1) You cannot change start position of robots and objects.
(2) You cannot change the number of robots and objects.
(3) You should consider all robots in the environment.

Three candidates of curriculums are:

Candidate 1:
<<Candidate_1>>

Candidate 2:
<<Candidate_2>>

Candidate 3:
<<Candidate_3>>

B.2 Reward Function Generation Module

The reward prompt supplies past tasks/code, current objectives, example rewards, and allowed helper functions. It asks for named components, nonnegative normalized total reward in zero to one, and no new library imports or environment changes. Its use of “integer” return values conflicts with continuous normalized reward examples; the exact original wording remains visible.

Prompt 3: LLM prompt for generating base reward function.
You should write reward function for given task using useful variables from the environment.
Your reward function is part of the curriculum learning which consists of learning sequence of different tasks.
You will be given description of past tasks, reward code for past tasks, and current task description.

Your gpt_reward(self) function's output should consist of three items
(1) the total reward, an integer
(2) a dictionary of each individual reward component where keys are names of reward component and items are each rewards
(3) the maximum possible reward that is used to normalize the reward, an integer

Some helpful tips for writing a reward function code:
(1) Use numpy or scipy functions to write the reward function. numpy, scipy is already imported as np and scipy. Do not import additional library.
(2) You may put higher weight on the reward for current task, but also include rewards from past tasks to avoid forgetting.
(3) Total reward must stay within [0, 1]. Always, scale the reward to have maximum 1.
(4) Only use provided variables.
(5) Do not change other predefined parts in the code.

<<Environment_Description>>

You can use help functions to calculate reward.
<<Helper_Functions>>

Example reward function for the task is
<<Example_Reward>>

You should re-implement reward function to assign a proper reward function for given task.
You can change the reward components that aligns well with the task.
Do not include any new inputs in the functions.

Note that you are designing a reward function for team, not a single agent.

Generate a reward function code and command for
Task Name: <<Task_Name>>
Description: <<Task_Description>>
Reason: <<Task_Reason>>

Previous learned task was
Task Name: <<Task_Name>>
Description: <<Task_Description>>
Reason: <<Task_Reason>>
Code:
```python
<<Task_Code>>
```

B.3 Policy Evaluation Module

The evaluator prompt asks for Success/Failure and reasons, explicitly checking current completion and forgetting. It supplies chronological images plus full state sequences and describes 0.03 as a scale intuition rather than a hard threshold. Its decision is therefore conditioned on privileged structured information as well as visuals.

Prompt 4: VLM prompt for policy evaluation.
You are a multi-agent reinforcement learning engineer asked to evaluate a rollout from a task in a curriculum.

You will be provided with:
- Snapshot images from the rollout
- Trajectory data (state sequences)
- Previous and current task descriptions

Your job is to determine:
1. Whether the agents achieved the current goal.
2. Whether they have forgotten earlier subtasks.

Your output should be
Decision: [Success or Failure]

Reason:
- [reason 1]
- [reason 2]
...

Fill out the [] brackets in the given format. (Don't keep the [] brackets)

Current task is:
<<Current task>>
Former tasks that RL succeeded to learn is:
<<Former tasks>>

Given images are snapshots in chronological order of the rollout.

Trajectory data:
<<Trajectory>>

Note that the distance and orientation is considered as ``very close'' if the value is smaller than 0.03 (This is NOT a threshold value, this is given to have a sense of how close they are)
Note that the trajectory data is sequential data for a whole episode.

Make your decision on both the trajectory data and the image.

B.4 Reward Refinement Module

Advice is limited to three suggestions and the available helpers, with negative rewards forbidden. The refinement prompt must rewrite code within the supplied structure and consider earlier tasks. It says the output has “two items” but enumerates three; the source inconsistency is preserved. A downstream code interface must resolve such ambiguities, which the prompt text itself does not formally validate.

Prompt 5: VLM prompt for generating reward refinement advice.
You are a multi agent reinforcement learning engineer trying to refine a reward function for a given task.
You will be given: the reward function that was tried and failed, the experimental results, a description of why it failed, and the reward-component curve images.
First, analyze the reason why the reward function failed to learn the task.
Then, provide advice on how to improve or modify the reward function to achieve the task.

These are some tips in giving advice:
(1) Do not give more than three advice.
(2) You cannot add a penalty in the reward function, e.g. no reward can be a negative value.

These are the only functions that are allowed to used.
<<Helper_Functions>>
Information that cannot be obtained from the above functions cannot be used in the reward function.

Current task is:
<<Task_Name>>
<<Task_Description>>
<<Task_Reason>>

Reward function that was tried:
<<Reward_Function>>

Failure reason:
<<Failure_Reason>>

Do not give an advice on the curriculum of the reward. Just consider how to improve the reward function given the current task.
Do not provide the whole code of the reward function, just give advice on how to improve it.
Prompt 6: LLM prompt for refining a reward function using advice.
You are a multi agent reinforcement learning engineer trying to refine a reward function for the given task.
The reward function is part of the curriculum learning which consists of learning sequence of different tasks.
You will be given description of past tasks, a reward function that was tried and failed, and an advice on how to improve it.
Follow the advice and provide a new reward function that should be able to achieve the given task.
Do not repeat the advice or the old reward function in your answer.

Your gpt_reward(self) function's output should consist of two items
    (1) the total reward, an integer
    (2) a dictionary of each individual reward component where keys are names of reward component and items are each rewards
    (3) the maximum possible reward that is used to normalize the reward, an integer

Some helpful tips for writing a reward function code:
(1) Use numpy or scipy functions to write the reward function. numpy, scipy is already imported as np and scipy. Do not import additional library.
(2) You may put higher weight on the reward for current task, but also include rewards from past tasks to avoid forgetting.
(3) Total reward must stay within [0, 1]. Always, scale the reward to have maximum 1.
(4) Only use provided variables.
(5) Do not change other predefined parts in the code.

You can use help functions to calculate reward.
<<Helper_Functions>>

Current task is:
<<Current_Task>>
Former tasks that RL succeeded to learn is:
<<Former_Tasks>>

Reward function that was tried:
<<Reward_Function>>

Advice on how to improve the reward function:
<<Advice>>

Do not deviate from the reward structure given in the example.
Note that the advice might not acount for what information it can use in the reward function. So not all the advice might be applicable.

B.5 Task-Specific Substitutions for Two Arm Lift

Lift-specific substitutions provide the pot/handle geometry, height/tilt criterion, distances, orientation errors, grasp indicators, and helper functions. The full example reward combines reaching, alignment, lifting, grasping, and success, normalized by 7.5. Its helper success check concerns height, while a separate tilt coefficient gates terminal/lift reward. The listing preserves every operation and indentation level, rather than replacing executable details with a prose sketch.

Prompt 7: <<Environment_Description>>for Two Arm Lift.
A large pot with two handles is placed on a table top. The pot is rectangular shaped with U-shaped handles on the opposite side of the pot rim. The grip of the handle is level, parallel to the tabletop. Two robot arms are placed on opposite ends of the table. The two robot arms must each grab a handle and lift the pot together, above a certain height (0.1m), while keeping the pot level (no more tilted than 30 degrees). The pot location is randomized at the beginning of each episode.
Prompt 8: <<State_Variables>>for Two Arm Lift.
- Euclidean distance from each robot's end-effector and its target handle
- Orientation difference between each robot's end-effector and its target handle
- Gripper state of each robot (whether it grasped the handle or not)
- Pot elevation above the table
- Pot tilt as cosine of the angle between its normal and the vertical
Prompt 9: <<Helper_Functions>>for Two Arm Lift.
self._get_check_grasps(): returns two boolean value for each arm whether or not it's grasping the handle or not.
self._get_pot_elevation(): returns the distance (in meters) from pot's bottom to the table top
self._get_tilt_degree(): returns the cosine value of the tilted degree of the pot. The tilted degree is the difference between the z axis and pot's normal vector
self._get_gripper_to_handle_distance(): returns two float value of distance from each robot's end-effector to each handle
self._get_gripper_to_handle_rotation(): returns two rotation difference (calculated with R1.T @ R2) between each robot's end-effector to each handle with shape: (3,3) each
self._check_success(): returns True if the pot is higher than 0.1
Code 1: <<Example_Reward>>for Two Arm Lift.
def gpt_reward(self):

    self.gpt_rew = 0
    self.gpt_rew_dict = {}
    max_reward = 7.5

    # reward components
    reach_reward = 0.0
    align_reward = 0.0
    lift_reward = 0.0
    grasp_reward = 0.0
    success_reward = 0.0

    # check if pot is tilted less than 30 degrees
    cos_z = self._get_tilt_degree()
    cos_30 = np.cos(np.pi / 6)
    direction_coef = 1 if cos_z >= cos_30 else 0

    # check for goal completion: cube is higher than the table top above a margin
    if self._check_success():
        success_reward += max_reward * direction_coef

    else:

        # Reaching reward - max 0.2
        _g0h_dist, _g1h_dist = self._get_gripper_to_handle_distance()
        reach_reward += 0.1 * (1 - np.tanh(10.0 * _g0h_dist))
        reach_reward += 0.1 * (1 - np.tanh(10.0 * _g1h_dist))

        # aligning reward - max 0.3
        _g0h_rot, _g1h_rot = self._get_gripper_to_handle_rotation()
        _g0h_rot_diff = np.trace(np.eye(3) - _g0h_rot)
        _g1h_rot_diff = np.trace(np.eye(3) - _g1h_rot)
        align_reward += 0.15 * (1 - np.tanh(10.0 * _g0h_rot_diff))
        align_reward += 0.15 * (1 - np.tanh(10.0 * _g1h_rot_diff))

        # lifting reward - max 5.0
        elevation = self._get_pot_elevation()
        r_lift = min(max(elevation, 0), 0.2)
        lift_reward += 25.0 * direction_coef * r_lift

        # Grasping reward - max 2.0
        g0_grasp, g1_grasp = self._get_check_grasps()
        if g0_grasp:
            grasp_reward += 1.0
        if g1_grasp:
            grasp_reward += 1.0

    # Add and normalize the reward
    self.gpt_rew = reach_reward + align_reward + lift_reward + grasp_reward + success_reward
    self.gpt_rew *= 1.0 / max_reward

    self.gpt_rew_dict = {
        'reach_reward': reach_reward,
        'align_reward': align_reward,
        'lift_reward': lift_reward,
        'grasp_reward': grasp_reward,
        'success_reward': success_reward,
    }

    return self.gpt_rew, self.gpt_rew_dict, max_reward

Appendix C Examples in VLM-guided Reward Refinement

The final appendix shows the actual seesaw evaluation and reward-advice exchange, including rollout snapshots, numeric trajectories, failed reward code, component curves, and complete model responses. These examples demonstrate the workflow rather than independently validating every generated statement.

C.1 Example Results of Policy Evaluation

The evaluation example asks whether one robot traverses the seesaw while the other stabilizes its start, after an alignment stage. It provides both robots’ full position sequences and an instruction not to be too strict because this is intermediate. The retained response judges success based on sequential entry and stability. This illustrates how prompt framing and quantitative trajectories can influence advancement.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 10: Rollout snapshots provided to the policy evaluation VLM.
Prompt 10: Example VLM prompt for evaluating ‘seesaw entry and stability task’ in Quadruped Seesaw.
Current task is:
Task Name: 2_Seesaw Entry and Stability Task
Description: Train both agents to coordinate so that Agent 0 enters and traverses the seesaw toward the end at position [x=7, y=0], ensuring gradual and controlled movement to maintain balance. Meanwhile, Agent 1 enters the seesaw afterward, stopping and stabilizing at the start position [x=4, y=0, z=1.0].
Reason: Establishing sequential entry and motion ensures Agent 0 can proceed unimpeded while Agent 1 stabilizes the seesaw. Effective coordination during this phase prevents excessive tilting, allowing for smooth advancement toward the goal.
Former tasks that RL succeeded to learn is:

Task Name: 1_Initial Alignment Task
Description: Ensure Agent 0 (Blue) aligns itself with the start of the seesaw at position [x=4, y=0, z=1.0] while Agent 1 (Green) maintains its initial position, preparing to follow Agent 0. Both agents should aim to keep y position near 0 to stay within the seesaw's width.
Reason: This task sets the foundational entry sequence where Agent 0 leads the way, ensuring it is ready to proceed onto the seesaw without obstruction from Agent 1. This preparation is crucial for establishing entry control and ensuring proper seesaw engagement.

Given images are snapshots in chronological order of the rollout.
Agent 0 is colored with blue and Agent 1 is colored with green.

Trajectory data:
seesaw_center:
[5.500 0.000 1.000]
seesaw_start:
[4.000 0.000]
seesaw_end:
[7.000 0.000]
target_pos:
[7.700 0.000 1.500]
agent_0_pos:
[[0.920 -0.730 0.420]
 [1.380 -0.550 0.270]
 [2.130 -0.250 0.280]
 [2.910 -0.070 0.290]
 [3.640 -0.010 0.330]
 [4.360 0.030 0.520]
 [5.210 0.140 0.740]
 [5.910 0.320 0.920]
 [6.250 0.240 0.960]
 [6.910 0.210 1.150]
 [7.280 0.200 1.110]
 [7.250 0.080 0.970]
 [7.200 -0.030 1.010]
 [7.270 -0.020 1.240]
 [7.850 0.110 1.300]
 [7.950 -0.080 1.290]
 [7.980 -0.250 1.280]
 [8.070 -0.370 1.280]
 [8.160 -0.450 1.220]
 [8.160 -0.390 1.220]
 [8.090 -0.370 1.220]]
agent_1_pos:
[[0.920 0.830 0.420]
 [1.120 0.850 0.270]
 [1.500 0.600 0.280]
 [2.190 0.400 0.290]
 [2.890 0.370 0.280]
 [3.590 0.390 0.350]
 [4.140 0.390 0.450]
 [4.210 0.370 0.430]
 [4.150 0.180 0.420]
 [4.210 0.160 0.450]
 [4.320 0.200 0.630]
 [4.380 0.150 0.740]
 [4.380 0.260 0.610]
 [4.380 0.120 0.460]
 [4.460 -0.050 0.510]
 [4.510 0.070 0.570]
 [4.400 0.040 0.530]
 [4.440 0.070 0.550]
 [4.340 -0.010 0.510]
 [4.300 0.010 0.470]
 [4.220 -0.110 0.420]]

Note that this is not the final goal in the curriculum, don't be too strict on the decision.
Prompt 11: Example VLM answer for evaluating ‘seesaw entry and stability task’ in Quadruped Seesaw.
Decision: Success

Reason:
- Agent 0 led the sequence, entered the seesaw first, and traversed to and beyond the target end (around x = 7.2-8.2) with smooth, controlled height (z) changes, indicating balance was maintained.
- Agent 1 only stepped onto the seesaw after Agent 0 had committed, then remained stably near the start region (x ~ 4.2, y ~ 0, small z-oscillations), satisfying the stabilization requirement.
- The initial-alignment behaviour is intact (Agent 1 kept close to y = 0 while Agent 0 aligned to the seesaw before entry), showing no forgetting of the previous subtask.

C.2 Example Results of Reward Advice

The failed ascension reward contains many components and negative fall/collision terms despite the general nonnegative-reward instruction. The advice proposes strengthening forward/height incentives, replacing negative penalties with survival bonuses, and concentrating the second robot’s balance signal. Some suggestions require checking that the existing helpers expose the needed quantities, as the refinement prompt itself cautions. The complete original code, curves, failure explanation, and advice remain available for assessing this gap.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Refer to caption
Figure 11: Reward component learning curves provided to the reward advice VLM.
Prompt 12: Example VLM prompt for evaluating ‘target platform ascension task’ in Quadruped Seesaw.
Current task is:
Name: Target Platform Ascension Task
Description: Ensure Agent 0 continues confidently toward the end of the seesaw and reaches the target platform [x coordinate > 7, height > 1.3], while Agent 1 maintains stability at the seesaw's starting point. Agent 1 should ensure it provides enough counterbalance to prevent the seesaw from destabilizing.
Reason: This task focuses on culmination and assessment of learned coordination strategies, where Agent 0 must finalize its movement to reach the target while relying on Agent 1 to maintain seesaw balance. Successfully reaching the platform signifies the completion of the objective and tests the agents' ability to coordinate effectively under the primary task constraints.

Reward function that was tried:
def gpt_reward(self, state, action):

    reward = torch.zeros((self.num_envs, self.num_agents), device=self.env.device)
    rew_dict = {}
    max_reward = 23.0

    # Reward agent 0 for progressing towards the target platform - max 3.0
    agent_0_progress_reward = 3.0 * self._progress_to_target(state, action)
    # Only reward agent 0 when progressing towards the target
    agent_0_progress_reward[:, 1] = 0.0
    agent_0_progress_reward = self._check_reward_shape(agent_0_progress_reward)
    reward += agent_0_progress_reward
    rew_dict['agent_0_progress'] = torch.mean(agent_0_progress_reward[:, 0])

    # Reward agent 1 for holding position at the seesaw start - max 3.0
    agent_1_stability_reward = 3.0 * self._progress_to_seesaw_start(state, action)
    # Only reward agent 1 for maintaining stability
    agent_1_stability_reward[:, 0] = 0.0
    agent_1_stability_reward = self._check_reward_shape(agent_1_stability_reward)
    reward += agent_1_stability_reward
    rew_dict['agent_1_stability'] = torch.mean(agent_1_stability_reward[:, 1])

    # Reward for maintaining y position close to 0 for both agents - max 2.0
    y_alignment_reward = 2.0 * self._y_alignment(state, action)
    y_alignment_reward = self._check_reward_shape(y_alignment_reward)
    reward += y_alignment_reward
    rew_dict['y_alignment'] = torch.mean(y_alignment_reward)

    # Reward for height progression for Agent 0, aiming to reach above 1.3m - max 2.0
    height_reward = 2.0 * self._normalized_height(state, action)
    height_reward[:, 1] = 0.0  # Only agent 0 should get height reward
    height_reward = self._check_reward_shape(height_reward)
    reward += height_reward
    rew_dict['height_reward'] = torch.mean(height_reward[:, 0])

    # Penalty for falling off the seesaw - min -2.0
    fall_penalty = -2.0 * self._fall(state, action).float()
    fall_penalty = self._check_reward_shape(fall_penalty)
    reward += fall_penalty
    rew_dict['fall_penalty'] = torch.mean(fall_penalty)

    # Penalty for colliding with walls or structures - min -1.0
    collision_penalty = -1.0 * self._wall_collision(state, action).float()
    collision_penalty = self._check_reward_shape(collision_penalty)
    reward += collision_penalty
    rew_dict['collision_penalty'] = torch.mean(collision_penalty)

    # Reward for maintaining an appropriate distance between agents - max 1.0
    agent_distance = self._agent_distance(state, action)
    distance_reward = 1.0 * torch.clamp(torch.square(agent_distance), max=1.0)
    distance_reward = self._check_reward_shape(distance_reward)
    reward += distance_reward
    rew_dict['distance_reward'] = torch.mean(distance_reward)

    # Reward for success when Agent 0 reaches the platform - max 10.0
    success = self._success(state, action)
    success_reward = 10.0 * success.float()
    success_reward = self._check_reward_shape(success_reward)
    reward += success_reward
    rew_dict['success'] = torch.mean(success_reward)

    # Normalize the reward
    reward = self._check_reward_shape(reward)
    reward /= max_reward

    return reward, rew_dict, max_reward

Failure reason:
- Agent 0 never reached the target platform region defined by x>7 and height z>1.3; its maximum z was only ~1.1 near x~6.86 before falling back down.
- Agent 1 did not hold the seesaw at the start point (x~4); it remained around x=0.8~0.9 throughout, indicating it forgot the earlier alignment and stability subtasks.

Do not give an advice on the curriculum of the reward. Just consider how to improve the reward function given the current task.
Do not provide the whold code of the reward function, just give advices on how to improve it.
Prompt 13: Example VLM advice for improving reward function for ‘target platform ascension task’ in Quadruped Seesaw.
Here are three focused suggestions for reshaping your reward so that Agent 0 is driven all the way up onto the platform and Agent 1 stays firmly at the seesaw pivot:

1.  Re-weight and de-crowd your signal for Agent 0's forward/vertical progress
   - Right now ``agent_0_progress'' (*3) and ``height_reward'' (*2) get drowned in the many small terms and then all get divided by 23.  Even at full scale they only amount to ~0.1 per step.  Boost those two terms by an order of magnitude (e.g. *20-30 for progress, *10-15 for height) and then remove or dramatically shrink the less critical terms (y_alignment, distance_reward) so that the agent sees a strong, unambiguous gradient pulling it to x>7 and height>1.3.

2.  Turn your fall-and-collision ``penalties'' into positive stay-alive/balance bonuses
   - You've defined negative fall/collision terms, but the spec forbids negative rewards anyway, and they barely ever fire (1e-9 scale).  Instead give a small constant ``survival'' bonus every step that you remain upright and collision-free (e.g. +1 per step of life/collision-free).  This keeps all components >0 and provides a steady shaping signal without explicit negatives.

3.  Concentrate Agent 1's reward on seesaw balance, not just position drift
   - '_progress_to_seesaw_start' only measures how close Agent 1 got initially, but says nothing about ongoing balance. Repurpose your y_alignment term (or distance_reward) so that it's applied only to Agent 1 and heavily weighted: reward 1 - tilt (i.e. how flat the seesaw is) or reward the inverse of Agent 1's deviation from the pivot. This gives Agent 1 a clear continuous incentive to lock itself at the fulcrum and keep the seesaw horizontal while Agent 0 climbs.
Concept demonstration

Open image ↗