Overview
CMPO trains an action-chunk policy through a world model that predicts the whole future latent chunk in one call. At deployment it executes only the first action and replans from the new observation. This edition retains all seven figure groups, three numbered tables, the original algorithm, eight numbered equations, unnumbered derivations and all appendices. The central result is improved learning with a shorter computational graph; direct multi-step prediction still has model error, and the hardware evidence is a small Go2 standing experiment.
At a glance
A chunk actor and chunk dynamics model replace recurrent imagined rollouts with one differentiable prediction of future latents. The actor optimizes predicted rewards plus a terminal critic while avoiding repeated model-policy calls and their backpropagation-through-time chain. On 19 HumanoidBench tasks the reported average return and sample efficiency exceed the chosen baselines. Walk/run timing is roughly halved for training and about 30 times faster per inference than TD-MPC2. A simulation-trained Go2 policy maintains a bipedal posture for 20 seconds in 7/10 physical trials, versus 4/10 for SAC.
1 Introduction
Learned dynamics can improve data efficiency, but recurrent rollouts introduce sequential computation, accumulated state-prediction error and long products of Jacobians. Action chunking has already aided imitation and value learning; CMPO applies it jointly to the actor and world model for gradient-based online policy optimization. The model evaluates a proposed action sequence during training, while deployment uses a direct actor. Chunk generation therefore does not imply open-loop execution of the entire chunk.
2 Related Work
The paper separates three overlapping uses of world models: generate synthetic training transitions, plan actions at decision time, or differentiate imagined returns to update a policy. CMPO belongs primarily to the third family, related to stochastic value gradients, MAAC and Dreamer. It inherits most components of TD-MPC2 but replaces one-step dynamics and policy outputs with chunks. Decoupled Q-Chunking motivates first-action execution, while MAC explores chunks in offline model-based learning. Optional test-time planning is a separate choice.
3 Chunk-wise Model-based Policy Optimization
The method changes the temporal structure of actor optimization while retaining learned state encoding, reward prediction and one-step action-value learning. The following sections distinguish the motivation, concrete objectives and deployment behavior.
3.1 Background
In an infinite-horizon discounted Markov decision process, a learned latent dynamics and reward model provide an imagined finite-horizon return terminated by a critic. Equation 1 gives the conventional H-step expansion. Such an estimate can support planning, value targets or policy training; CMPO uses a shorter expansion specifically as an actor-training objective.
Equation 1
| (1) |
3.2 Motivation
A recurrent model needs H sequential state predictions; the next prediction depends on the preceding predicted latent. Under a uniform one-step error bound and Lipschitz dynamics, the appendix bounds rollout error by a geometric sum. This is an upper bound under stated assumptions, not proof that every rollout actually grows exponentially. Recurrent differentiation also introduces Jacobian products that can amplify or suppress gradients and require a stored temporal graph. Equation 3 presents the state-mediated recurrence but omits an explicit direct policy-parameter derivative; the original expression is retained. CMPO removes this recurrent temporal depth, rather than guaranteeing an unbiased model or stable optimization for arbitrary horizons.
Equations 2–4
| (2) |
| (3) |
| (4) |
3.3 Collaborative Chunk-wise Modeling
The actor predicts H actions from the current latent, and the dynamics model predicts H future latents from that latent plus the complete action chunk. A single model invocation supplies the actor objective: H−1 predicted reward steps plus a terminal single-action critic and entropy regularization. Using the final predicted action in that terminal critic avoids querying the actor again. State encoding, latent consistency with a stop-gradient target, two-hot reward prediction and one-step critic learning otherwise follow TD-MPC2. The model and critic losses weight later terms by powers of a decay parameter; the target critic uses an exponential moving average. Only the first action is executed before replanning. Original notation has unresolved inconsistencies: the unnumbered expansion uses an action index beyond the stated chunk, whereas Equation 8 uses its final action; Equation 7’s printed target omits the discount factor. These formulas remain unchanged, and this summary follows the stated first-action/Equation 8 mechanism without treating the printed equations as a complete implementation specification.
Equations 5–8
| (5) | ||||||||
| (6) |
| (7) |
| (8) |
4 Experiments
Experiments test average performance and interaction efficiency on high-dimensional humanoid tasks, computational cost, transfer to physical Go2 standing, and sensitivity to gradient structure, prediction horizon and optional planning. The reported benchmark is state-based continuous control rather than a visual manipulation-policy evaluation.
4.1 High-dimensional Humanoid Continuous Control
Nineteen HumanoidBench tasks use a 61-dimensional action space. Curves show means over three runs, with shaded 95% confidence intervals, up to one million environment steps. CMPO has the strongest average return and performs well on many tasks, but the complete plots show that it is not uniformly best. SAC, PPO, DreamerV3 and TD-MPC2 comparison curves are taken from the original benchmark; the MAAC-style comparator is implemented on TD-MPC2’s architecture. Separate timing on one RTX 4090, without Torch Compile, measures 200,000 training steps: walk takes 1.74 versus 3.57 hours, and run 1.72 versus 3.54 hours, for CMPO versus TD-MPC2. One inference takes 1.4 versus 40.8 ms and 1.3 versus 39.5 ms. These figures support the reported roughly twofold training and 30-fold inference gains for those settings, not every task or hardware platform.
4.2 Quadruped Locomotion & Sim-to-Real Deployment
CMPO and SAC share a simulation-training and hardware-deployment pipeline for hind-leg standing locomotion on a Unitree Go2. CMPO discovers standing earlier and obtains higher simulated linear-tracking reward. Physical success means retaining the bipedal posture for 20 seconds: 7/10 CMPO trials versus 4/10 SAC trials. This small experiment does not establish broad locomotion robustness or a statistically precise success-rate advantage. TD-MPC2 is omitted from the hardware comparison because its iterative planner does not fit the same low-latency TorchScript feed-forward interface. The main text describes tracking gated by base height, whereas the explicit appendix formulas gate it by upright orientation; the original reward table is preserved and this discrepancy is recorded.
4.3 Discussion
MAAC shows larger gradient excursions than CMPO in the reported three-seed traces and horizon comparison. The first plot uses separate vertical scales, so raw curve heights are not a shared numerical scale. The horizon plot’s error bars are means of each run’s minima and maxima, not confidence intervals. Direct chunk prediction reduces recurrent error accumulation but still becomes less accurate at greater horizons: H=2 and H=4 work best in the shown performance study, H=5 degrades, and the main comparisons use H=3. Keeping a chunk actor but reverting to a one-step recurrent model yields walk/run/stair/slide returns of 22.2/9.6/37.1/36.1 versus 954.1/629.2/60.9/265.8 for CMPO. Adding the TD-MPC2 MPPI planner gives 931.1/864.5/71.9/391.2: it helps three of these four tasks while slightly lowering walk return. This is optional added test-time computation, and the proposed explanation of better exploration is an interpretation rather than a separately isolated causal test.
5 Conclusion
Joint chunk prediction makes the world model a differentiable training evaluator rather than a recursively queried surrogate environment. The reported gains combine more effective actor updates and avoiding test-time planning. Evidence includes 19 simulation tasks, controlled structural comparisons and a small physical standing study. The paper does not prove universal model-gradient stability or eliminate the difficulty of predicting distant futures.
Limitation
Most network design is inherited from TD-MPC2, leaving chunk-specific architectures unexplored. First-action execution maintains feedback but discards the remaining proposed actions, which may waste useful computation in slower-changing environments. Adaptive execution horizons are a proposed extension, not an evaluated capability. Larger prediction horizons already degrade performance, and broader physical tasks remain outside the demonstrated scope.
Acknowledgments
The authors acknowledge Shanghai Municipal Science and Technology Major Project support and National Natural Science Foundation of China grants. The funding identifiers remain available in the original paper.
Appendix A Proof
The proposition compares deterministic true and learned dynamics under the same initial latent and the same open-loop action sequence. It assumes a uniform one-step model error bound over every visited state-action pair and Lipschitz continuity of the true dynamics in the latent state. Adding and subtracting the true transition at the imagined state yields e(next) ≤ epsilon + L times e(current). Repeated substitution gives the geometric-sum bound; for L=1 it reduces to k times epsilon, and otherwise to epsilon times (L to the kth power minus one)/(L minus one). It does not establish the same bound when the two rollouts choose different feedback actions or prove a corresponding uniform error bound for the chunk model.
Equations
Appendix B Implementation Details
Implementation details provide the original collection/update algorithm and explain the limited architectural changes from TD-MPC2. The appendix does not tabulate a complete set of optimizer, network-width and training hyperparameters; inherited settings are described by reference.
B.1 Pseudocode for CMPO
The original algorithm initializes the actor, chunk dynamics, encoder, reward model and critics. Each environment step samples a full action sequence, executes its first action and saves the transition. Updates sample contiguous replay chunks, optimize model/encoder/reward prediction, then the critic and actor, and finally average target-critic parameters. The native algorithm retains its Require block, nested loops, comments, equations and all 12 numbered steps. Its original variable-name and spelling inconsistencies are preserved rather than silently rewritten.
B.2 Network Settings
The policy output grows from action dimension to action dimension times horizon. Dynamics input grows from latent plus one action to latent plus H actions, and dynamics output from one latent to H latents. Both remain based on TD-MPC2 multilayer perceptrons. These changes remove recurrent calls while increasing input/output sizes; a single call does not mean computational cost is independent of horizon. Most other architecture and hyperparameter choices are inherited, limiting how independently the paper specifies a reproducible configuration.
Appendix C Experimental Details
The experimental appendix enumerates benchmark tasks, distinguishes reused baseline results from the controlled MAAC implementation, specifies Go2 actuation and rewards, and supplies the complete per-task learning curves.
C.1 Simulation Benchmark Overview
The Unitree H1hand simulation benchmark covers locomotion, terrain traversal, balance and scene/object interaction, all with 61 action dimensions. The fourteen original visualization panels cover balance, basketball, bookshelf, cube, door, hurdle, maze, pole, run/stand/walk, sit, slide, spoon, stair and window; difficulty variants and the grouped locomotion panel account for the 19 tasks. Detailed task reward definitions are delegated to HumanoidBench.
C.2 Simulation Baseline Specification
SAC, PPO, DreamerV3 and TD-MPC2 benchmark results are reused from the original HumanoidBench report, not all newly rerun under this paper’s implementation. The MAAC-style policy update is instead implemented on the same TD-MPC2 architecture as CMPO. This makes the recurrent-versus-chunk comparison more controlled, while the broad benchmark comparisons still combine results from different training implementations.
C.3 Real World Experiment Setup
Policies are trained in Isaac Gym and transferred to a physical Go2. The actor outputs residual joint-position commands for all 12 actuators, tracked by a proportional-derivative controller. The task combines hind-leg posture with commanded linear/angular velocity tracking and penalties for aggressive or wasteful motion. Physical evidence is the reported 20-second posture trial, not a measured benchmark across varied terrains or disturbances.
C.3.1 Reward Design
Table 3 retains all eleven reward terms and weights: linear/angular tracking, acceleration, joint power, height, collision, action rate, action smoothness, orientation, reference pose and hip regularization. Tracking uses width 0.25 and the upright gate −projected-gravity-x > 0.9; desired height is 0.55 m. Per-leg hip/thigh/calf reference angles are 0, 1.8 and −1.1 radians. The table uses body-frame axes appropriate to the standing posture, including negative base-z velocity for the forward command and base-x angular velocity for yaw. This explicit orientation gate differs from the height-gating description in the main text. The complete formulas are preserved so those conventions and the configured reward scales remain inspectable.
| Reward term | Definition | Weight |
|---|---|---|
| Linear velocity tracking | ||
| Angular velocity tracking | ||
| DOF acceleration | ||
| Joint power | ||
| Base height | ||
| Collision | ||
| Action rate | ||
| Action smoothness | ||
| Upright orientation | ||
| Upright reference pose | ||
| Hip regularization |
C.4 Complete Simulation Results
Figure 7 preserves all 19 task-return panels plus their aggregate, including simpler/harder balance, bookshelf and sitting variants. All curves share the one-million-step interaction budget and show three-run means with 95% intervals. The aggregate improvement coexists with task-dependent weaknesses: CMPO is not dominant on every scene interaction, and tasks such as cube remain difficult. Returns have different scales across tasks, so the plotted average should not be read as a uniform success-rate gain.