Literature
Display
← All papers

CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills Using Large Language Models

Kanghyun Ryu · Qiayuan Liao · Zhongyu Li · Payam Delgosha · Koushil Sreenath · Negar Mehr

· ICRA 2025 ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

CurricuLLM uses language models to propose a sequence of easier-to-harder training tasks, generate reward/goal-distribution code for each task, and select among trained policy candidates using rollout statistics. Policies are fine-tuned sequentially through the curriculum. Results cover simulated manipulation, maze navigation, and humanoid locomotion, plus a limited humanoid walking demonstration. The approach reduces task-specific curriculum design while still relying on supplied environment variables, training infrastructure, and human-designed constraints.

I INTRODUCTION

Direct reinforcement learning on a difficult target can waste experience before useful behavior appears. Manually chosen intermediate tasks can help, but require expertise. CurricuLLM separates language-level curriculum design from executable task-code generation and policy evaluation. Its contribution is an automated sequence of reward functions and goal ranges within an existing simulator, rather than automatic construction of the entire robot learning system.

Original figure
Fig. 1: CurricuLLM takes natural language description of environments, robots, and target task that we wish the robot to learn, and then generates a sequence of subtasks. In each subtasks, it samples different task codes and evaluates the resulting trained policy to find the policy which is best aligned within the current subtask. These iterations are repeated throughout the curriculum subtasks to sequentially train a policy that reaches complex target task.

II Related Works

II-A Curriculum Learning

Earlier automatic curricula vary initial states, goals, opponents, or environments. This work focuses on task-level progression expressed through changing rewards and goal distributions. It differs from domain-randomization and terrain-generation approaches, while sharing their use of language models to propose training conditions.

Original figure
Fig. 2: Curriculum generation LLM receives the natural language form of a curriculum prompt as well as the environment description to generate a sequence of subtasks. Our prompt includes instruction for tje curriculum designer, rules for how to describe the subtasks, and other tips on describing the curriculum. Environment description consists of the robot and its state variable description, the target task, and the initial state description.

II-B Large Language Model for Robotics

Language models have been used for discrete task planning, action-code composition, reward design, and direct evaluation. CurricuLLM uses their code-generation ability to train continuous control policies instead of assuming a fixed skill library. Rather than repeatedly optimizing a reward against a predefined task score, it samples several candidates per stage and asks a model to judge their rollout summaries against the subtask description. This changes the evaluator interface; it does not establish that the model supplies an unbiased or optimal ranking.

III Problem Formulation

Each subtask is a goal-conditioned Markov decision process with state space, goal space, action space, dynamics, reward, and goal distribution. State/action/goal spaces and dynamics are fixed across stages. A curriculum is an ordered list of reward-and-goal-distribution pairs, implemented as task code. Its objective is the target task’s expected discounted return after sequential policy training. This formal objective defines what a good curriculum should achieve; the proposed language-model search is a heuristic rather than an optimizer with a guarantee.

IV Method

Three modules generate subtask descriptions, sample executable task codes, and select the most aligned trained policy. Each candidate at a stage starts from the preceding selected policy. The winner and its task code become the starting point and history for the next stage.

IV-A Generating Sequence of Language Description

The curriculum model receives the robot, available state variables, initial conditions, and target task. Prompts require a short sequence, use of known variables, and the original target as the last task. A humanoid example progresses from stability or basic movement toward broader velocity and heading commands. Grounding descriptions in simulator variables makes coding more concrete, but does not guarantee that the proposed difficulty ordering is correct.

Original figure
Fig. 3: Our task code generation and evaluation framework in each subtask. Task code generation LLM takes the environment and target task description, current and past task information, and the reward function used for previous task. Then, KK task code candidates for current subtask is sampled and used for fine-tuning policies from previous subtask. Then, evaluation LLM receives the statistics of trajectory rollout from trained policy and find a policy that best aligns with current subtask description.

IV-B Task Code Generation

For each subtask, the model samples up to five reward/goal-code candidates in the reported experiments. Normalized observations help it choose reward scales. Previous task descriptions and selected code remain in context, and instructions emphasize the current objective while retaining earlier rewards to reduce forgetting. Each candidate requires policy fine-tuning. Therefore, candidate count and stage count contribute to total training cost beyond any single learning curve.

IV-C Large Language Model for Policy Evaluation

The evaluator receives rollout statistics, the current description, and task history, then chooses a policy to carry forward. It operates between training runs rather than providing rewards at every control step. The humanoid prompt also imposes an explicit minimum episode length of 100, so selection includes a human-specified health criterion even though no dedicated numerical subtask score is written. Algorithm 1 retains the complete sampling, training, selection, and history-update loop.

Algorithm 1 Overall algorithm of CurricuLLM

Input Language description of target task lTl_{T}, Language description of environment lEl_{E}
   Hyperparameter Number of task code samples KK, Total number of training timesteps for each subtask MM

1: # Generate curriculum from LLM
2: [l1,l2,…,lN]∼L​L​MC​u​r​r​i​c​u​l​u​m​(lT,lE)[l_{1},l_{2},\dots,l_{N}]\sim LLM_{Curriculum}(l_{T},l_{E})
3: π∼random policy\pi\sim\text{random policy}  # Initialize with random policy
4: h=[]h=[\>]
5: for lnl_{n} for n=1:Nn=1:N do
6:   for k=1:Kk=1:K do
7:    # Sample task code from LLM
8:    (rnk,ρg,nk)∼L​L​MT​a​s​k​C​o​d​e​G​e​n​e​r​a​t​i​o​n​(ln,lE,h)(r_{n}^{k},\rho_{g,n}^{k})\sim LLM_{TaskCodeGeneration}(l_{n},l_{E},h)
9:    πnk∼train​(π,rnk,ρg,nk,M)\pi_{n}^{k}\sim\text{train}(\pi,r_{n}^{k},\rho_{g,n}^{k},M)
10:   end for
11:   # Roll out trajectories from policy candidates
12:   τn1,τn2,…,τnK∼πn1,πn2,…,πnK\tau_{n}^{1},\tau_{n}^{2},\dots,\tau_{n}^{K}\sim\pi_{n}^{1},\pi_{n}^{2},\dots,\pi_{n}^{K}
13:   # Evaluate the resulting trajectories
14:   (rn∗,ρg,n∗),πn∗∼L​L​ME​v​a​l​u​a​t​i​o​n​(ln,τn1,τn2,…,τnK)(r_{n}^{*},\rho_{g,n}^{*}),\pi_{n}^{*}\sim LLM_{Evaluation}(l_{n},\tau_{n}^{1},\tau_{n}^{2},\dots,\tau_{n}^{K})
15:   # Update the policy and training history
16:   π←πn∗\pi\leftarrow\pi_{n}^{*}
17:   h.a​p​p​e​n​d​(ln,rn∗,ρg,n∗)h.append(l_{n},r_{n}^{*},\rho_{g,n}^{*})  
18: end for

Output Policy π\pi

V Experiments

Evaluation covers Fetch manipulation, ant navigation, and humanoid locomotion, followed by a hardware demonstration. Different tasks use different reinforcement-learning algorithms and permitted curriculum changes.

Refer to caption
Refer to caption
Refer to caption
Refer to caption
Fig. 4: Snapshot of Environments: From left to right, Fetch-Slide, Fetch-Push, AntMaze-UMaze, and Berkeley Humanoid.
Original figure
((a)) Fetch-Slide environment
Original figure
((b)) Fetch-Push environment
Original figure
((c)) AntMaze-UMaze environment
Fig. 5: Success rate of tasks in Gynmasium-Robotics environments.

V-A Gymnasium Environments

Fetch-Slide and Fetch-Push use a seven-degree-of-freedom arm; AntMaze-UMaze navigates a U-shaped maze. CurricuLLM changes only rewards for Fetch, but can change goal-distance limits for AntMaze. Comparisons use SAC, hindsight experience replay [52], and a one-shot language-model reward. Experiments use five seeds per method and ten for the one-shot baseline, with GPT-4-turbo for language-model stages. The strongest advantage is in AntMaze, where the plotted curriculum policy reaches substantially higher success than the alternatives. Results are mixed on Fetch: HER reaches a higher final success rate on Fetch-Push, while Fetch-Slide has low and variable success across methods. The authors explicitly report high sensitivity to curriculum quality. The reward-code example shows a richer AntMaze reward that encourages motion and angular stability, compared with a sparse one-shot reward, but that example is not a complete causal ablation of every design choice.

Original figure
Fig. 6: Example reward codes generated from CurricuLLM and reward code generated in a zero-shot manner from LLM for AntMaze environment. The reward code designed from CurricuLLM encompasses diverse behaviors that motivates learning the target task while directly querying a reward code for the target task leads to sparse-reward, uninformative reward function.

V-B Berkeley Humanoid

V-B 1 Simulation Results

The Berkeley Humanoid simulator has 45 observed dimensions and twelve desired-joint-position actions. The target ranges are ±2 m/s in both planar velocity components and a full heading range. CurricuLLM adjusts rewards and command ranges; all methods use PPO [54]. Human-designed motion penalties and accumulated-tracking-error termination are added to the generated reward to discourage undesirable motion and standing-only solutions. CurricuLLM reports 0.46±0.38 m/s tracking error and 8.79±4.72% early failure, versus 0.41±0.10 and 7.31±3.12% for the designed human reward. These are comparable rather than superior results. The one-shot reward has much higher error and variability. Adding an alive bonus to the human reward reduces early failure to 0.51% but produces standing behavior with 1.37 m/s tracking error. Because failure means termination before two seconds, low failure alone does not imply task success.

TABLE I: Results of Berkeley Humanoid Simulation
Tracking Error (m/s) Failure (%) Episode Length (steps)
CurricuLLM 0.46±0.38\mathbf{0.46\pm 0.38} 8.79±4.72\mathbf{8.79\pm 4.72} 315.44±315.44\pm 73.9573.95
LLM-zeroshot 1.13±0.571.13\pm 0.57 28.57±45.1928.57\pm 45.19 146.66±146.66\pm 114.12114.12
Human reward 0.41±0.10\mathbf{0.41\pm 0.10} 7.31±3.12\mathbf{7.31\pm 3.12} 397.67±397.67\pm 42.0442.04
Human reward with live bonus 1.37±0.661.37\pm 0.66 0.51±0.230.51\pm 0.23 167.57±167.57\pm 98.2298.22

V-B 2 Hardware Validation

Hardware validation restricts commanded planar speeds to ±1 m/s because of hardware limitations. The retained plot contains both forward and lateral velocity traces, although its caption names only the forward component. The demonstration shows qualitative tracking with visible deviations and transients; it does not establish the simulated ±2 m/s running capability on hardware or provide repeated-trial success statistics.

Refer to caption
Fig. 7: Snapshot of hardware experiment.
Original figure
Fig. 8: Odometry data and command data of base linear velocity vxv_{x} in hardware experiments. The policy trained with CurricuLLM successfully follows the command in real-world.

VI CONCLUSIONS

Language-generated curricula can help a pretrained sequence of policies acquire difficult tasks, with the clearest reported benefit in maze navigation. Remaining limitations include variable curriculum quality, multiple candidate-training costs, dependence on supplied simulator interfaces and additional human reward/termination design, heuristic rollout evaluation, and limited hardware validation. The authors propose adapting training duration and stopping from learning-progress feedback in future work.

VII APPENDIX

VII-A Prompts

All three complete prompts are retained. Curriculum generation permits at most five tasks and requires the original target last. Code generation restricts the model to supplied normalized observations and PyTorch operations, specifies command-range formatting, and tells it to retain earlier skills. Policy evaluation asks for a selected agent and explanation while rejecting episodes shorter than 100. The generated curriculum and code therefore depend on explicit human interface and selection rules.

Prompt 1: Curriculum generation prompt for Berkeley Humanoid.
You are a curriculum generator trying to generate a curriculum to solve reinforcement learning tasks as effectively as possible.
Your goal is to write a list of tasks for the agent that will help the agent learn the main task in the environment described in the text.
Some helpful tips for generating a curriculum:
(1) Do not explore or learn dynamics by doing random actions. Each task should have a specific goal to achieve.
(2) Curriculum should be made as short as possible. Be concise.
(3) Never generate more than 5 tasks
(4) The last task of your curriculum should align with the original task description
(5) Focus on stability and learning healthy in early stages. However, focus on command tracking in later stages.
You will be given several variables you can use to describe each task in curriculum.
Each task should be described using the given variables and do not introduce new variables.
You should also describe the command that will be given to the robot.
You will be given several variables you can use to describe each task in curriculum.
Each task should be described using the given variables and do not introduce new variables.
Your output should be
Task 1
Name: []
Description: []
Reason: []
Task 2
Name: []
Description: []
Reason: []
Task n
Name: Original task
Description: []
Reason: []
Fill out the inside of brackets in the format.
Remove the brackets after filling out them.
The hub is a bipedal robot that can walk and run in a fashin similar to that of humans or animals.
The original task in the environment is for hub to walk or run by following random command.
Speed of the hub robot will be specified in a command consist of [linear velocity x, linear velocity y, heading angle]
The command range in original task should follow this: "linear velocity x: [-2, 2] , linear velocity y: [-2, 2], heading angle [-pi, pi]"
In each subtask, you can train the agent for randomized command with specific command range.
You have to specify the range of command value in the task description.
Note that the default initial values are initialized to make hub stand.
The variables you can use to describe the curriculum includes
base_lin_vel: Linear velocity of base in xyz direction
base_ang_vel: angular velocity of base in xyz direction,
nonflat_base_orientation: Size of deviation of base orientation from default standing position,
base_height_diff: height difference between measured base height and default stainding height,
torques: Torques applied to the joint,
hip_pos_deviation: Relative position of hip joint from default standing position,
knee_pos_deviation: Relative position of knee joint from default standing position,
joint_vel: Velocity of each joint,
joint_acc: Acceleration of each joint,
is_alive: 1 if the environment is healthy, 0 if the environment is terminated due to failure,
commands: Range of commands that robot base should follow.
Consist of [linear velocity x range [], linear velocity y range [], heading angle range []].
Prompt 2: Task code generation prompt for Berkeley Humanoid.
You should write reward function for given task using useful variables from the environment
Your reward function is part of the curriculum learning consist of learning sequence of different tasks.
You will be given description of past tasks, reward code for past tasks, and current task descriptiion.

Your reward_curriculum() function's return should be torch.Tensor of shape (number of environments)
The code output of reward_curriculum() should be formatted as a python code string: "```python ... ```".
Your have to specify your command range.
The code output should be formatted as "```command ... ```"

Some helpful tips for writing the reward function code:
(1) Use torch functions to write the reward function. Do not import additional library.
(2) If you want to get a size of value in 2d tensor, use torch.sum(torch.square(variable), dim=1)
(3) If you want to get a size of value in 1d tensor, use torch.square(variable)
(4) You may normalize each reward arguments
(5) You should use a weighting parameter outside of the transformation function
(6) You may put higher weight on the reward for current task. However, you should also give reward for past tasks to avoid forgetting.
(7) Focus on stability and learning healthy in early stages. However, focus on command tracking in later stages.
(8) Most importantly, you must only use given variables
(9) Do not change other predefined parts in the code.

The hub is a bipedal robot that can walk and run in a fashin similar to that of humans or animals.
The original task in the environment is for hub to walk or run by following random command.
Speed of the hub robot will be specified in a command consist of [linear velocity x, linear velocity y, angular velocity z, heading angle]
The command range in original task should follow this: "linear velocity x: [-2, 2] , linear velocity y: [-2, 2], heading angle is [-pi, pi]"

observation from normalized_obs() includes
base_lin_vel: Linear velocity of base in xyz direction, 2 dimension with shape (number of envs, 3)
base_ang_vel: angular velocity of base in xyz direction, 2 dimension with shape (number of envs, 3)
nonflat_base_orientation: Size of deviation of base orientation from default standing position, 1 dimension with shape (number of envs)
base_height_diff: height difference between measured base height and default stainding height, 1 dimension with shape (number of envs)
torques: Torques applied to the joint, 2 dimension with shape (number of envs, number of joints)
hip_pos_deviation: Relative position of hip joint from default standing position, 2 dimension with shape (number of envs, number of hip joints)
knee_pos_deviation: Relative position of knee joint from default standing position, 2 dimension with shape (number of envs, number of knee joints)
joint_vel: Velocity of each joint, 2 dimension with shape (number of envs, number of joints)
joint_acc: Acceleration of each joint, 2 dimension with shape (number of envs, number of joints)
is_alive: 1 if the environment is healthy, 0 if the environment is terminated due to failure, 1 dimension with shape (number of envs)

commands: Commands that robot base should follow.
          Consist of [linear velocity x, linear velocity y, angular velocity z].
          2 dimension with shape (number ov envs, 3)

Note that default values are initialized to make the bipedal robot stand.

Your output for the environment code should follow this format
```python
def reward_curriculum(env: ManagerBasedRLEnv, asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")) -> torch.Tensor:
    """Compute the reward for the current curriculum step."""
    command, base_lin_vel, base_ang_vel, hip_pos_deviation, knee_pos_deviation, \
    joint_vel, joint_acc, torque, \
    nonflat_base_orientation, base_height_diff = normalized_obs(env, asset_cfg)
    is_alive = (~env.termination_manager.terminated).float()

    n_envs = env.scene.num_envs

    # Implement your reward function here
    reward = torch.zeros(n_envs, device=env.device)
    return reward
```

You should re-implement reward_curriculum function to assign a proper reward function for given task.
Note that you should try to follow command linear velocity and angular velocity z.
Especially, third component of base angular velocity should accurately follow angular velocity z in command.


Your output for the command value should follow this format.
You should change the range of lin_vel_x, lin_vel_y, and heading for your specific task.
You cannot change the range of ang_vel_z.
```command
ranges=mdp.UniformVelocityCommandCfg.Ranges(lin_vel_x=(-2.0, 2.0), lin_vel_y=(-2.0, 2.0), ang_vel_z=(-1.0, 1.0), heading=(-math.pi, math.pi)),
```

Generate a reward function code and command for
Task Name: <<Task_Name>>
Description: <<Task_Description>>
Reason: <<Task_Reason>>

Previous learnd task was
Task Name: <<Task_Name>>
Description: <<Task_Description>>
Reason: <<Task_Reason>>
Code:
```python
<<Task_Code>>
```
Prompt 3: Policy evaluation prompt for Berkeley Humanoid
You are a robotics engineer trying to evaluate the robot trajectory is following the given task.
You will be given a language Description of a task, and trajectory statistics of robots.
Based on the trajectory rollout information, choose the best agent for following the task Description.
Your task is part of curriculum learning consist of sequence of tasks.
While you should focus on evaluating the agent based on current task, also consider their performance on previous tasks.
Your priority is selecting healthy policy.
If episode length is shorter than 100, do not select it.
Follow the output format.
Decision: Agent [number]
Reason:
Find the best agent for following task description in below
Task Name: <<Task_Name>>
Description: <<Task_Description>>
Reason: <<Task_Reason>>
Previous tasks in curriculum are
Task Name: <<Task_Name>>
Description: <<Task_Description>>
Reason: <<Task_Reason>>

VII-B CurricuLLM Examples

The complete example curriculum, both task-code listings, and the full evaluation input/output are preserved. Four stages progress from ±0.5 m/s commands to ±1 and then ±2; the last two stages share the full command range, so this example is not a strictly increasing sequence by that measure. Code 1 and Code 2 retain the exact velocity, orientation, height, torque, and survival terms, their weights, and command settings. The example evaluator selects Agent 1 from four candidates and rejects Agent 3 for a very short episode. The supplied averages and explanation illustrate the selection process but do not establish that Agent 1 is optimal under a specified quantitative tracking objective.

Output 1: Example curriculum for Berkeley Humanoid
Task 1
Name: Basic Locomotion
Description: In this task, commands will have a limited range to encourage initial movements without requiring precise control over speed or direction. The commands range should be linear velocity x: [-0.5, 0.5], linear velocity y: [-0.5, 0.5], heading angle: [-pi/4, pi/4].
Reason: To introduce the hub to basic movements and to ensure it learns to balance and take steps forward, backward, and to the sides within a controlled environment. This will also help stabilize its base and familiarize it with simple locomotion commands.
Task 2
Name: Advanced Locomotion
Description: Commands will be moderately challenging with increased range to enhance the hub’s ability to move and change directions. The commands range should be linear velocity x: [-1, 1], linear velocity y: [-1, 1], heading angle: [-pi/2, pi/2].
Reason: To improve the hub’s response to a wider range of motions and directions, teaching it to adapt its balance and coordination for faster and more varied movements. This serves as a bridge to mastering complex locomotion required in the original task.
Task 3
Name: Full Speed and Agility Training
Description: Commands will cover the full range specified for the original task to maximize the hub’s movement capabilities. The commands range should be linear velocity x: [-2, 2], linear velocity y: [-2, 2], heading angle: [-pi, pi].
Reason: To ensure the hub is capable of handling the full spectrum of speed and directional changes, preparing it for any possible command within the original task’s requirements. This task aims at fine-tuning the hub’s motor control, agility, and responsiveness.
Task 4
Name: Original task
Description: Hub to walk or run by following random commands within the specified range: linear velocity x: [-2, 2], linear velocity y: [-2, 2], heading angle: [-pi, pi].
Reason: To assess the hub’s ability to follow any given command accurately and efficiently, combining all learned skills from previous tasks. This final task evaluates how well the hub can integrate its training to achieve the goal of dynamic, responsive locomotion.
Code 1: Example task code for Berkeley Humanoid. Task code for Basic Locomotion task.
def reward_curriculum(env: ManagerBasedRLEnv, asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")) -> torch.Tensor:
    """Compute the reward for the Basic Locomotion task."""
    command, base_lin_vel, base_ang_vel, hip_pos_deviation, knee_pos_deviation, \
    joint_vel, joint_acc, torques, \
    nonflat_base_orientation, base_height_diff = normalized_obs(env, asset_cfg)
    is_alive = (~env.termination_manager.terminated).float()

    n_envs = env.scene.num_envs

    # Locomotion Rewards
    reward_lin_vel_x = -torch.abs(command[:, 0] - base_lin_vel[:, 0])
    reward_lin_vel_y = -torch.abs(command[:, 1] - base_lin_vel[:, 1])
    reward_ang_vel_z = -torch.abs(command[:, 2] - base_ang_vel[:, 2])

    # Stability and Efficiency Rewards
    reward_base_orientation = -torch.abs(nonflat_base_orientation)
    reward_base_height_diff = -torch.abs(base_height_diff)
    reward_torques = -torch.sum(torch.abs(torques), dim=1)

    # Weighting the rewards
    weight_lin_vel_x = 0.2
    weight_lin_vel_y = 0.2
    weight_ang_vel_z = 0.4
    weight_base_orientation = 0.1
    weight_base_height_diff = 0.05
    weight_torques = 0.05
    weight_alive = 1.0

    # Combining the rewards
    reward = weight_lin_vel_x * reward_lin_vel_x + weight_lin_vel_y * reward_lin_vel_y + \
             weight_ang_vel_z * reward_ang_vel_z + weight_base_orientation * reward_base_orientation + \
             weight_base_height_diff * reward_base_height_diff + weight_torques * reward_torques + \
             weight_alive * is_alive

    return reward

ranges=mdp.UniformVelocityCommandCfg.Ranges(lin_vel_x=(-0.5, 0.5), lin_vel_y=(-0.5, 0.5), ang_vel_z=(-1.0, 1.0), heading=(-math.pi/4, math.pi/4)),
Code 2: Example task code for Berkeley Humanoid. Task code for final task which is following command within 2m/s.
def reward_curriculum(env: ManagerBasedRLEnv, asset_cfg: SceneEntityCfg = SceneEntityCfg("robot")) -> torch.Tensor:
    """Compute the reward for the Original task."""
    command, base_lin_vel, base_ang_vel, hip_pos_deviation, knee_pos_deviation, \
    joint_vel, joint_acc, torques, \
    nonflat_base_orientation, base_height_diff = normalized_obs(env, asset_cfg)
    is_alive = (~env.termination_manager.terminated).float()

    n_envs = env.scene.num_envs

    # Original Task Rewards (evaluating adherence to any given command accurately and efficiently)
    reward_lin_vel_x = -torch.abs(command[:, 0] - base_lin_vel[:, 0])
    reward_lin_vel_y = -torch.abs(command[:, 1] - base_lin_vel[:, 1])
    reward_ang_vel_z = -torch.abs(command[:, 2] - base_ang_vel[:, 2]) * 1.5  # emphasize on angular velocity

    # Stability and Agility Rewards
    reward_base_orientation = -torch.abs(nonflat_base_orientation) * 0.5
    reward_base_height_diff = -torch.abs(base_height_diff) * 0.5
    reward_torques = -torch.sum(torch.abs(torques), dim=1) * 0.1  # adjusted to account for necessary torques during dynamic movements

    # Combining the rewards with adjusted weights to reflect the integration of all learned skills
    # Maintained proportion but slightly adjusted for overall performance and efficiency
    weight_lin_vel_x = 0.2
    weight_lin_vel_y = 0.2
    weight_ang_vel_z = 0.4  # Further emphasized for the original task
    weight_base_orientation = 0.05
    weight_base_height_diff = 0.05
    weight_torques = 0.05  # Slight adjustment for embracing dynamic actions
    weight_alive = 1.5  # Balance between surviving and task accomplishment

    # Final Reward Combination reflecting the task's goal
    reward = weight_lin_vel_x * reward_lin_vel_x + weight_lin_vel_y * reward_lin_vel_y + \
             weight_ang_vel_z * reward_ang_vel_z + weight_base_orientation * reward_base_orientation + \
             weight_base_height_diff * reward_base_height_diff + weight_torques * reward_torques + \
             weight_alive * is_alive

    return reward

ranges=mdp.UniformVelocityCommandCfg.Ranges(lin_vel_x=(-2.0, 2.0), lin_vel_y=(-2.0, 2.0), ang_vel_z=(-1.0, 1.0), heading=(-math.pi, math.pi)),
Output 2: Example evaluation result for Basic Locomotion task in Berkeley Humanoid. Average value from trajectory rollout is provided.
Find the best agent for following task description in below
Task Name: Basic Locomotion
Description: In this task, commands will have a limited range to encourage initial movements without requiring precise control over speed or direction. The commands range should be linear velocity x: [-0.5, 0.5], linear velocity y: [-0.5, 0.5], heading angle: [-pi/4, pi/4].
Reason: To introduce the hub to basic movements and to ensure it learns to balance and take steps forward, backward, and to the sides within a controlled environment. This will also help stabilize its base and familiarize it with simple locomotion commands.
Agent 0:
base_lin_vel: [-0.031 0.023 -0.013]
base_ang_vel: [0.004 0.033 0.003]
projected_gravity: [ 0.444 -0.197 -0.87 ]
velocity_command: [ 0.461 -0.165 0.109]
hip_pos: [-0.366 -0.175 -0.362 0.096 -0.38 -0.71 ]
kfe_pos: [0.077 0.378]
ffe_pos: [-0.144 -0.174]
faa_pos: [-0.058 0.004]
joint_vel: [-0.016 0.023 0.085 -0.037 -0.086 -0.076 0.061 0.052 0.078 0.101
0.351 -0.23 ]
last_actions: [-9.600e-02 -7.650e-01 -7.860e-01 1.520e-01 -3.640e-01 -1.033e+00
-2.810e-01 1.000e-03 6.983e+00 1.152e+00 1.985e+00 -1.913e+00]
episode_length: 1000.0
Agent 1:
base_lin_vel: [-0.099 -0.166 0.012]
base_ang_vel: [0.007 0.003 0.127]
projected_gravity: [-0.042 -0.058 -0.995]
velocity_command: [-0.113 -0.489 0.113]
hip_pos: [ 0.46 -0.245 0.427 -0.27 0.027 -0.129]
kfe_pos: [0.019 0.237]
ffe_pos: [-0.044 -0.117]
faa_pos: [-0.075 -0.123]
joint_vel: [ 0.038 0.008 0.093 0.032 -0.037 -0.02 0.057 0.074 0.012 0.039
0.343 0.193]
last_actions: [ 1.104 -0.784 0.728 -0.352 -0.11 -0.41 -0.821 -0.711 -3.689 0.257
0.051 0.38 ]
episode_length: 1000.0
Agent 2:
base_lin_vel: [ 0.079 -0.289 0.003]
base_ang_vel: [-0.007 0.035 -0.143]
projected_gravity: [ 0.051 0.021 -0.994]
velocity_command: [ 0.172 -0.428 -0.176]
hip_pos: [ 0.309 0.16 0.036 -0.004 0.029 -0.101]
kfe_pos: [0.051 0.252]
ffe_pos: [-0.098 -0.052]
faa_pos: [-0.043 -0.1 ]
joint_vel: [ 0.024 0.001 0.063 0.032 -0.02 -0.035 0.052 0.058 0.03 0.111
0.232 0.26 ]
last_actions: [ 0.352 0.266 0.466 0.102 0.028 -0.264 -0.262 -0.835 2.257 -3.157
0.269 -0.447]
episode_length: 1000.0
Agent 3:
base_lin_vel: [ 0.023 -0.131 -0.383]
base_ang_vel: [2.006 0.577 0.047]
projected_gravity: [ 0.023 -0.063 -0.995]
velocity_command: [0.002 0.017 0.005]
hip_pos: [-0.073 -0.068 0.025 0.19 -0.169 -0.13 ]
kfe_pos: [0.28 0.258]
ffe_pos: [ 0.02 -0.032]
faa_pos: [-0.034 0.052]
joint_vel: [-2.201 -2.652 0.502 6.555 -4.148 -2.601 6.557 5.742 0.477 -0.715
-1.158 0.67 ]
last_actions: [-0.181 -2.088 -0.139 3.195 -1.09 -0.826 1.922 1.813 1.214 -1.612
-0.212 -0.147]
episode_length: 4.354
Decision: Agent 1
Reason: Among the agents, Agent 1 demonstrates the most relevant and balanced performance for the "Basic Locomotion" task. The task specifies a need for movements within a certain range of linear velocities and heading angles. Agent 1’s trajectory data shows velocity commands (-0.113, -0.489, 0.113) and base linear velocities (-0.099, -0.166, 0.012) that are reasonably within or close to the specified command ranges for the task (linear velocity x and y: [-0.5, 0.5], heading angle: [-pi/4, pi/4]).
The actions of Agent 1 also indicate movements that are modest and controlled compared to the other agents, which is suitable for basic locomotion learning and aligns with the aim of familiarizing the hub with simple motions. Agent 1’s episode length, like all but Agent 3’s, meets the criterion of being longer than 100 which reflects a healthy policy that can sustain activity over an acceptable duration.
Agent 0 and Agent 2 also show potential, but their velocity commands and base linear velocities are less aligned with the task’s specified ranges than those of Agent 1. Agent 3 is disqualified for having an episode length (4.354) far below the acceptable threshold of 100, indicating an unhealthy or unstable policy for this task despite its possibly relevant actions and velocities, which cannot be reliably evaluated given the short episode length.
In consideration of the constrained environment meant to introduce basic movements, Agent 1’s performance data suggests a balance between obeying the velocity command constraints and exhibiting a level of control conducive to learning basic movements.
Concept demonstration

Open image ↗