At a glance
CurricuLLM uses language models to propose a sequence of easier-to-harder training tasks, generate reward/goal-distribution code for each task, and select among trained policy candidates using rollout statistics. Policies are fine-tuned sequentially through the curriculum. Results cover simulated manipulation, maze navigation, and humanoid locomotion, plus a limited humanoid walking demonstration. The approach reduces task-specific curriculum design while still relying on supplied environment variables, training infrastructure, and human-designed constraints.
I INTRODUCTION
Direct reinforcement learning on a difficult target can waste experience before useful behavior appears. Manually chosen intermediate tasks can help, but require expertise. CurricuLLM separates language-level curriculum design from executable task-code generation and policy evaluation. Its contribution is an automated sequence of reward functions and goal ranges within an existing simulator, rather than automatic construction of the entire robot learning system.
II Related Works
II-A Curriculum Learning
Earlier automatic curricula vary initial states, goals, opponents, or environments. This work focuses on task-level progression expressed through changing rewards and goal distributions. It differs from domain-randomization and terrain-generation approaches, while sharing their use of language models to propose training conditions.
II-B Large Language Model for Robotics
Language models have been used for discrete task planning, action-code composition, reward design, and direct evaluation. CurricuLLM uses their code-generation ability to train continuous control policies instead of assuming a fixed skill library. Rather than repeatedly optimizing a reward against a predefined task score, it samples several candidates per stage and asks a model to judge their rollout summaries against the subtask description. This changes the evaluator interface; it does not establish that the model supplies an unbiased or optimal ranking.
III Problem Formulation
Each subtask is a goal-conditioned Markov decision process with state space, goal space, action space, dynamics, reward, and goal distribution. State/action/goal spaces and dynamics are fixed across stages. A curriculum is an ordered list of reward-and-goal-distribution pairs, implemented as task code. Its objective is the target task’s expected discounted return after sequential policy training. This formal objective defines what a good curriculum should achieve; the proposed language-model search is a heuristic rather than an optimizer with a guarantee.
IV Method
Three modules generate subtask descriptions, sample executable task codes, and select the most aligned trained policy. Each candidate at a stage starts from the preceding selected policy. The winner and its task code become the starting point and history for the next stage.
IV-A Generating Sequence of Language Description
The curriculum model receives the robot, available state variables, initial conditions, and target task. Prompts require a short sequence, use of known variables, and the original target as the last task. A humanoid example progresses from stability or basic movement toward broader velocity and heading commands. Grounding descriptions in simulator variables makes coding more concrete, but does not guarantee that the proposed difficulty ordering is correct.
IV-B Task Code Generation
For each subtask, the model samples up to five reward/goal-code candidates in the reported experiments. Normalized observations help it choose reward scales. Previous task descriptions and selected code remain in context, and instructions emphasize the current objective while retaining earlier rewards to reduce forgetting. Each candidate requires policy fine-tuning. Therefore, candidate count and stage count contribute to total training cost beyond any single learning curve.
IV-C Large Language Model for Policy Evaluation
The evaluator receives rollout statistics, the current description, and task history, then chooses a policy to carry forward. It operates between training runs rather than providing rewards at every control step. The humanoid prompt also imposes an explicit minimum episode length of 100, so selection includes a human-specified health criterion even though no dedicated numerical subtask score is written. Algorithm 1 retains the complete sampling, training, selection, and history-update loop.
Input Language description of target task , Language description of environment
Hyperparameter Number of task code samples , Total number of training timesteps for each subtask
Output Policy
V Experiments
Evaluation covers Fetch manipulation, ant navigation, and humanoid locomotion, followed by a hardware demonstration. Different tasks use different reinforcement-learning algorithms and permitted curriculum changes.
V-A Gymnasium Environments
Fetch-Slide and Fetch-Push use a seven-degree-of-freedom arm; AntMaze-UMaze navigates a U-shaped maze. CurricuLLM changes only rewards for Fetch, but can change goal-distance limits for AntMaze. Comparisons use SAC, hindsight experience replay [52], and a one-shot language-model reward. Experiments use five seeds per method and ten for the one-shot baseline, with GPT-4-turbo for language-model stages. The strongest advantage is in AntMaze, where the plotted curriculum policy reaches substantially higher success than the alternatives. Results are mixed on Fetch: HER reaches a higher final success rate on Fetch-Push, while Fetch-Slide has low and variable success across methods. The authors explicitly report high sensitivity to curriculum quality. The reward-code example shows a richer AntMaze reward that encourages motion and angular stability, compared with a sparse one-shot reward, but that example is not a complete causal ablation of every design choice.
V-B Berkeley Humanoid
V-B 1 Simulation Results
The Berkeley Humanoid simulator has 45 observed dimensions and twelve desired-joint-position actions. The target ranges are ±2 m/s in both planar velocity components and a full heading range. CurricuLLM adjusts rewards and command ranges; all methods use PPO [54]. Human-designed motion penalties and accumulated-tracking-error termination are added to the generated reward to discourage undesirable motion and standing-only solutions. CurricuLLM reports 0.46±0.38 m/s tracking error and 8.79±4.72% early failure, versus 0.41±0.10 and 7.31±3.12% for the designed human reward. These are comparable rather than superior results. The one-shot reward has much higher error and variability. Adding an alive bonus to the human reward reduces early failure to 0.51% but produces standing behavior with 1.37 m/s tracking error. Because failure means termination before two seconds, low failure alone does not imply task success.
| Tracking Error (m/s) | Failure (%) | Episode Length (steps) | |
| CurricuLLM | |||
| LLM-zeroshot | |||
| Human reward | |||
| Human reward with live bonus |
V-B 2 Hardware Validation
Hardware validation restricts commanded planar speeds to ±1 m/s because of hardware limitations. The retained plot contains both forward and lateral velocity traces, although its caption names only the forward component. The demonstration shows qualitative tracking with visible deviations and transients; it does not establish the simulated ±2 m/s running capability on hardware or provide repeated-trial success statistics.
VI CONCLUSIONS
Language-generated curricula can help a pretrained sequence of policies acquire difficult tasks, with the clearest reported benefit in maze navigation. Remaining limitations include variable curriculum quality, multiple candidate-training costs, dependence on supplied simulator interfaces and additional human reward/termination design, heuristic rollout evaluation, and limited hardware validation. The authors propose adapting training duration and stopping from learning-progress feedback in future work.
VII APPENDIX
VII-A Prompts
All three complete prompts are retained. Curriculum generation permits at most five tasks and requires the original target last. Code generation restricts the model to supplied normalized observations and PyTorch operations, specifies command-range formatting, and tells it to retain earlier skills. Policy evaluation asks for a selected agent and explanation while rejecting episodes shorter than 100. The generated curriculum and code therefore depend on explicit human interface and selection rules.
You should write reward function for given task using useful variables from the environment
Your reward function is part of the curriculum learning consist of learning sequence of different tasks.
You will be given description of past tasks, reward code for past tasks, and current task descriptiion.
Your reward_curriculum() function's return should be torch.Tensor of shape (number of environments)
The code output of reward_curriculum() should be formatted as a python code string: "```python ... ```".
Your have to specify your command range.
The code output should be formatted as "```command ... ```"
Some helpful tips for writing the reward function code:
(1) Use torch functions to write the reward function. Do not import additional library.
(2) If you want to get a size of value in 2d tensor, use torch.sum(torch.square(variable), dim=1)
(3) If you want to get a size of value in 1d tensor, use torch.square(variable)
(4) You may normalize each reward arguments
(5) You should use a weighting parameter outside of the transformation function
(6) You may put higher weight on the reward for current task. However, you should also give reward for past tasks to avoid forgetting.
(7) Focus on stability and learning healthy in early stages. However, focus on command tracking in later stages.
(8) Most importantly, you must only use given variables
(9) Do not change other predefined parts in the code.
The hub is a bipedal robot that can walk and run in a fashin similar to that of humans or animals.
The original task in the environment is for hub to walk or run by following random command.
Speed of the hub robot will be specified in a command consist of [linear velocity x, linear velocity y, angular velocity z, heading angle]
The command range in original task should follow this: "linear velocity x: [-2, 2] , linear velocity y: [-2, 2], heading angle is [-pi, pi]"
observation from normalized_obs() includes
base_lin_vel: Linear velocity of base in xyz direction, 2 dimension with shape (number of envs, 3)
base_ang_vel: angular velocity of base in xyz direction, 2 dimension with shape (number of envs, 3)
nonflat_base_orientation: Size of deviation of base orientation from default standing position, 1 dimension with shape (number of envs)
base_height_diff: height difference between measured base height and default stainding height, 1 dimension with shape (number of envs)
torques: Torques applied to the joint, 2 dimension with shape (number of envs, number of joints)
hip_pos_deviation: Relative position of hip joint from default standing position, 2 dimension with shape (number of envs, number of hip joints)
knee_pos_deviation: Relative position of knee joint from default standing position, 2 dimension with shape (number of envs, number of knee joints)
joint_vel: Velocity of each joint, 2 dimension with shape (number of envs, number of joints)
joint_acc: Acceleration of each joint, 2 dimension with shape (number of envs, number of joints)
is_alive: 1 if the environment is healthy, 0 if the environment is terminated due to failure, 1 dimension with shape (number of envs)
commands: Commands that robot base should follow.
Consist of [linear velocity x, linear velocity y, angular velocity z].
2 dimension with shape (number ov envs, 3)
Note that default values are initialized to make the bipedal robot stand.
Your output for the environment code should follow this format
```python
def reward_curriculum(env: ManagerBasedRLEnv, asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")) -> torch.Tensor:
"""Compute the reward for the current curriculum step."""
command, base_lin_vel, base_ang_vel, hip_pos_deviation, knee_pos_deviation, \
joint_vel, joint_acc, torque, \
nonflat_base_orientation, base_height_diff = normalized_obs(env, asset_cfg)
is_alive = (~env.termination_manager.terminated).float()
n_envs = env.scene.num_envs
# Implement your reward function here
reward = torch.zeros(n_envs, device=env.device)
return reward
```
You should re-implement reward_curriculum function to assign a proper reward function for given task.
Note that you should try to follow command linear velocity and angular velocity z.
Especially, third component of base angular velocity should accurately follow angular velocity z in command.
Your output for the command value should follow this format.
You should change the range of lin_vel_x, lin_vel_y, and heading for your specific task.
You cannot change the range of ang_vel_z.
```command
ranges=mdp.UniformVelocityCommandCfg.Ranges(lin_vel_x=(-2.0, 2.0), lin_vel_y=(-2.0, 2.0), ang_vel_z=(-1.0, 1.0), heading=(-math.pi, math.pi)),
```
Generate a reward function code and command for
Task Name: <<Task_Name>>
Description: <<Task_Description>>
Reason: <<Task_Reason>>
Previous learnd task was
Task Name: <<Task_Name>>
Description: <<Task_Description>>
Reason: <<Task_Reason>>
Code:
```python
<<Task_Code>>
```
VII-B CurricuLLM Examples
The complete example curriculum, both task-code listings, and the full evaluation input/output are preserved. Four stages progress from ±0.5 m/s commands to ±1 and then ±2; the last two stages share the full command range, so this example is not a strictly increasing sequence by that measure. Code 1 and Code 2 retain the exact velocity, orientation, height, torque, and survival terms, their weights, and command settings. The example evaluator selects Agent 1 from four candidates and rejects Agent 3 for a very short episode. The supplied averages and explanation illustrate the selection process but do not establish that Agent 1 is optimal under a specified quantitative tracking objective.
def reward_curriculum(env: ManagerBasedRLEnv, asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")) -> torch.Tensor:
"""Compute the reward for the Basic Locomotion task."""
command, base_lin_vel, base_ang_vel, hip_pos_deviation, knee_pos_deviation, \
joint_vel, joint_acc, torques, \
nonflat_base_orientation, base_height_diff = normalized_obs(env, asset_cfg)
is_alive = (~env.termination_manager.terminated).float()
n_envs = env.scene.num_envs
# Locomotion Rewards
reward_lin_vel_x = -torch.abs(command[:, 0] - base_lin_vel[:, 0])
reward_lin_vel_y = -torch.abs(command[:, 1] - base_lin_vel[:, 1])
reward_ang_vel_z = -torch.abs(command[:, 2] - base_ang_vel[:, 2])
# Stability and Efficiency Rewards
reward_base_orientation = -torch.abs(nonflat_base_orientation)
reward_base_height_diff = -torch.abs(base_height_diff)
reward_torques = -torch.sum(torch.abs(torques), dim=1)
# Weighting the rewards
weight_lin_vel_x = 0.2
weight_lin_vel_y = 0.2
weight_ang_vel_z = 0.4
weight_base_orientation = 0.1
weight_base_height_diff = 0.05
weight_torques = 0.05
weight_alive = 1.0
# Combining the rewards
reward = weight_lin_vel_x * reward_lin_vel_x + weight_lin_vel_y * reward_lin_vel_y + \
weight_ang_vel_z * reward_ang_vel_z + weight_base_orientation * reward_base_orientation + \
weight_base_height_diff * reward_base_height_diff + weight_torques * reward_torques + \
weight_alive * is_alive
return reward
ranges=mdp.UniformVelocityCommandCfg.Ranges(lin_vel_x=(-0.5, 0.5), lin_vel_y=(-0.5, 0.5), ang_vel_z=(-1.0, 1.0), heading=(-math.pi/4, math.pi/4)),
def reward_curriculum(env: ManagerBasedRLEnv, asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")) -> torch.Tensor:
"""Compute the reward for the Original task."""
command, base_lin_vel, base_ang_vel, hip_pos_deviation, knee_pos_deviation, \
joint_vel, joint_acc, torques, \
nonflat_base_orientation, base_height_diff = normalized_obs(env, asset_cfg)
is_alive = (~env.termination_manager.terminated).float()
n_envs = env.scene.num_envs
# Original Task Rewards (evaluating adherence to any given command accurately and efficiently)
reward_lin_vel_x = -torch.abs(command[:, 0] - base_lin_vel[:, 0])
reward_lin_vel_y = -torch.abs(command[:, 1] - base_lin_vel[:, 1])
reward_ang_vel_z = -torch.abs(command[:, 2] - base_ang_vel[:, 2]) * 1.5 # emphasize on angular velocity
# Stability and Agility Rewards
reward_base_orientation = -torch.abs(nonflat_base_orientation) * 0.5
reward_base_height_diff = -torch.abs(base_height_diff) * 0.5
reward_torques = -torch.sum(torch.abs(torques), dim=1) * 0.1 # adjusted to account for necessary torques during dynamic movements
# Combining the rewards with adjusted weights to reflect the integration of all learned skills
# Maintained proportion but slightly adjusted for overall performance and efficiency
weight_lin_vel_x = 0.2
weight_lin_vel_y = 0.2
weight_ang_vel_z = 0.4 # Further emphasized for the original task
weight_base_orientation = 0.05
weight_base_height_diff = 0.05
weight_torques = 0.05 # Slight adjustment for embracing dynamic actions
weight_alive = 1.5 # Balance between surviving and task accomplishment
# Final Reward Combination reflecting the task's goal
reward = weight_lin_vel_x * reward_lin_vel_x + weight_lin_vel_y * reward_lin_vel_y + \
weight_ang_vel_z * reward_ang_vel_z + weight_base_orientation * reward_base_orientation + \
weight_base_height_diff * reward_base_height_diff + weight_torques * reward_torques + \
weight_alive * is_alive
return reward
ranges=mdp.UniformVelocityCommandCfg.Ranges(lin_vel_x=(-2.0, 2.0), lin_vel_y=(-2.0, 2.0), ang_vel_z=(-1.0, 1.0), heading=(-math.pi, math.pi)),