Overview
This edition follows the 34-page arXiv v1 dated 19 October 2026, retaining all 19 numbered figures, seven numbered tables, five algorithms, 18 displayed equation blocks and six appendices. Five search-tree figures are restored from native PDF art so node images, edges and small labels remain available at original detail. Search supplies training data; deployment still samples and scores grasp/place candidates and uses low-level motion planning. Read the three-trial hardware results as preliminary evidence.
At a glance
DISCO alternates physics-simulator tree search with policy/value distillation to extend single-step manipulation data into longer-horizon behavior. A learned policy proposes continuous grasp/place actions; progressive-widening MCTS explores feasible futures, and a learned value ranks candidates. The final model acts from segmented point clouds without simulator-based search at test time. Held-out shelf/stacking success reaches 80.0%/66.7%; physical shelf packing and table bussing succeed in two and one of three trials, respectively.
1 Introduction
An immediately feasible placement can block later objects. DISCO uses simulation search to expose those dependencies, then trains a reactive proposal-and-ranking model on the resulting trees. This adapts the policy/value/search loop familiar from board games to continuous manipulation and point-cloud observations. One model pair is trained per task on randomized instances; it is not a single generalist controller. Initial data contain local grasp/place successes, not complete long-horizon demonstrations.
2 Related Work
The work connects progressive-widening search, plan distillation and 3D policies. It uses a physics simulator rather than learning a transition model, and does not require a symbolic task representation. Compared with Neural Tree Expansion, which retains online search, the final policy/value pair executes without test-time tree construction. Its proposal architecture modifies 3D FlowMatch Actor.
3 Problem Statement and Assumptions
Training assumes object poses, a simulator, task goal/progress functions and short-horizon demonstration pairs. At test time, networks receive segmented point clouds and propose continuous end-effector grasp/place poses; cuRobo converts those into executable trajectories. A valid training plan must satisfy simulated transitions and the terminal goal. Progress reward is the difference in a task-specific scalar, not a learned reward or a test-time ranking signal.
4 DISCO: Distilled Search over Continuous Actions
Initialize the policy from single-object data, generate trees on simulated task instances, and periodically retrain both policy and value. Improved models then guide the next planning sweep. The final closed-loop controller observes again after executing each selected action.
4.1 Monte-Carlo Tree Search over Continuous Actions
Each tree vertex stores a simulator state, visits and search value; edges store continuous actions. Policy samples are motion-planned and simulated, and feasible successors become children. Widening allows additional children while their count is below kNᵅ, with k=2 and α=0.3. This bounds early branching while permitting later alternatives. High visits and high values correlate imperfectly; the appendix’s measured rank correlation is only 0.15.
4.2 Value-Guided Rollout
Rollouts continue the same search tree: sample batches, add feasible children and choose using value plus a UCB exploration term. Initially an unvisited child uses its incoming progress reward; later rounds use the learned success prior. Visited children instead use averaged backed-up root-to-leaf progress returns. Descents stop at a goal, horizon or dead end. Several descents run concurrently, so a GPU batch can contain actions from different starting vertices; every simulator environment resets to its own state.
4.3 Iterative Distillation
Value targets are successful visited leaves divided by all visited leaves below each non-root node. Weighted binary cross-entropy uses log(1+leaf count), normalized within the minibatch. These targets reflect search continuations rather than direct policy rollouts. Policy distillation uses every feasible expanded edge, including unsuccessful partial paths, with flow matching; the original single-step dataset is not mixed back in. Ranking is delegated to the value model. Two planning/distillation rounds update both networks.
Equation 1
| (1) |
5 Experiments
Experiments separate training-time planning effectiveness, component ablations, scaling with object count, held-out direct rollout and hardware transfer.
5.1 Datasets and Tasks
Shelf packing and stacking use 50 randomized training instances and 15 held-out instances per task in ManiSkill3, with ACRONYM meshes. Constrained packing compares online planners; physical tests use a Franka Panda with a Robotiq gripper. A single RTX 4090 supports training/evaluation. Shelf search takes 47.7 hours, generating 263,650 vertices with 11,015 expanded; trees store simulator states rather than point clouds, at a reported 95 MB per tree.
5.2 Evaluation
Beam search and MPPI replace the training-time plan generator, then distill controllers; RL-PD learns an actor/critic; ablations remove learned proposals or progressive widening. Except the explicitly online planners, evaluation uses candidate sampling/value ranking without simulator search. DISCO’s held-out shelf/stack success is 80.0%/66.7%, with reward 90.9/94.7; captions report 32 environment seeds and 95% intervals, not 32 independent training runs. Training-task success reaches 100%/98%, which must not be confused with held-out performance. Constrained packing reaches 85.9%, slightly below SPOT’s 86.7% under the adapted protocol. Hardware succeeds 2/3 on shelf packing and 1/3 on table bussing, versus 0/3 for the initial policy on both. Distillation improves the policy alone from 13.3% to 46.7%, and the final value-guided system to 80.0%.

6 Conclusion
Simulation search can supply long-horizon supervision that a segmented-point-cloud controller later uses without online tree search. The evidence supports this loop in the evaluated pick/place tasks; it does not show universal reasoning, reliable hardware performance across many trials, or removal of low-level planning.
7 Limitations
Search depends on an initial policy that proposes enough feasible actions and on sufficiently faithful simulation. Only end-effector pick/place actions are tested; joint-space and dexterous control remain future work. Misaligned progress shaping can bias search even though value targets use terminal goal labels. Foundation-model integration is also untested. Segmentation, geometric execution interfaces, task-specific training and small hardware samples further bound deployment claims.
Acknowledgments
The authors credit Toyota Research Institute, ONR MURI and Korean IITP/MSIT support, plus feedback and camera-calibration assistance. David Held’s concurrent Amazon Scholar appointment is disclosed; the work is attributed to CMU rather than Amazon.
Part Appendix
The six appendices retain baseline implementation, algorithms and architectures, augmentation/data generation, task definitions, quantitative/qualitative analyses and mathematical arguments.
Appendix A Baseline Details
Training-time planning baselines share the task, action interface, reward and budget wherever specified; the separate constrained-packing comparison retains online planners.
A.1 Beam Search
Beam search keeps 16 vertices, expands 16 proposed candidates per beam element and ranks successors by accumulated progress, breaking ties randomly. It has no UCB exploration, progressive widening or learned-value ranking during search. It stops at a goal, dead end, object-count horizon or exhausted budget. Its trees then train policy/value networks for direct evaluation.
Equation
A.2 MPPI
MPPI samples 32 policy-proposed action sequences with a horizon equal to object count. It exponentially weights cumulative progress returns using λ=10, updates the policy from weighted samples and replans after the selected action. It does not use a learned value during planning or retain MCTS backups. The final plans supply the distillation/evaluation pipeline.
Equations
A.3 RL-PD
RL-PD uses the same local demonstrations, observations, composite actions, progress rewards and horizon, combining prior data with online simulation transitions. Soft actor-critic targets include entropy and goal-conditioned termination. Table 3 retains the reported discount 0.99, entropy weight 0.1, batch 64, ten updates per environment iteration, critic/actor learning rates 0.1/0.01 and million-transition buffer. These unusually large listed learning rates are source values, not silently corrected.
Equation
| Hyperparameter | Value |
|---|---|
| Prior data | Single-step demonstrations used by DISCO |
| Discount factor | |
| Entropy temperature | |
| Batch size | |
| Gradient updates per environment iteration | |
| Critic learning rate | |
| Actor learning rate | |
| Replay buffer capacity |
A.4 DISCO without Policy Prior
Removing the learned proposal does not remove geometric knowledge. Shelf proposals use stored object-frame grasps, pose transforms, inverse-kinematics filtering and bounded placement samples. Stacking chooses base/moved objects, one of six downward faces and a support offset, then checks pick/place feasibility. The same MCTS, progress reward, widening and backups remain, with k=2, α=0.3 and C=1.5. This compares learned versus structured heuristic proposals, not completely uninformed random actions.
A.5 Test-Time Planning Baselines
The constrained-packing evaluation repeats five released scenes with 100 seeds each, for 500 trials; these are not 500 distinct scene geometries. DISCO executes directly, whereas SPOT, Points2Plans and relational baselines plan online. The Points2Plans execution interface is modified to use Jacobian-based velocity control toward DISCO-predicted grasps rather than its original MoveIt inverse kinematics. Results therefore describe the adapted common-interface experiment.
Appendix B Method Details
The detailed algorithms retain how actions become feasible edges, how tree statistics update and how the learned proposal/value pair operates.
B.1 Algorithm Details
UCBSelect descends by stored values/visits. Expand samples candidates, motion-plans and simulates them, discards infeasible transitions and scores new children at the parent observation. Rollout repeatedly expands and selects within the same graph; Backpropagate updates visit counts and running mean returns along parent links and records goal leaves. Algorithms 2–5 preserve exact operations. Deployment retains proposal/value scoring and physical execution, with no graph, UCB bonus or simulator call.
B.2 MCTS Hyperparameters
Total simulator budgets are 270,000 calls for packing and 27,000 for stacking. Expansion caps at 32 candidates, with 32 GPU environments handling concurrent descents; the task horizon is the 3–7 object count. Widening uses k=2, α=0.3 and UCB C=1.5. The appendix specifies distillation after a complete training-set planning sweep, twice in total. This clarifies the main text’s more generic “every few MCTS iterations” wording.
| Hyperparameter | Value |
|---|---|
| Simulation budget | 270,000 (shelf packing), 27,000 (object stacking) |
| Progressive-widening coefficient | 2.0 |
| Progressive-widening exponent | 0.3 |
| Exploration constant | 1.5 |
| Expansion batch size | 32 |
| Maximum task horizon | number of objects (3–7 per task) |
| Number of distillation rounds | 2 |
| Number of training seeds/tasks | 50 |
| Number of held-out seeds/tasks | 15 |
B.3 Policy Architecture
Separate rectified-flow grasp and grasp-conditioned place actors consume 4,096-point clouds. Two EdgeConv layers use an xyz k-nearest-neighbor graph with k=16 to produce width-120 tokens; a transformer with rotary 3D positions and adaptive normalization predicts flow velocities. Euler integration starts from Gaussian pose noise. Each output is position, quaternion and fixed grasp/place gripper state. Target-object clouds retain shelf/packed-object context; stacking adds a target mask, while packing uses xyz channels. The controller invokes the policy for remaining segmented objects.
B.4 Value Function Architecture
The value model sees the full scene plus two grasp/place pose tokens, embedding points with a per-point MLP rather than EdgeConv. Four shared attention layers, eight heads and a learned action-conditioned value query produce a sigmoid success score. Targets are search-subtree leaf fractions, with logarithmic evidence weighting. Figure 10 shows mostly zero/one targets and greater weighted influence for intermediate outcomes. A low training loss does not establish calibrated success probabilities on unseen real scenes.
B.5 Sim-to-Real Augmentation
Networks use normalized xyz point clouds, without RGB, simulator-state or proprioceptive inputs; segmented objects are assumed in both domains. Training adds Gaussian noise with sampled scale up to 1 cm, removes local neighborhoods and substitutes bounding-box points. Separate object yaw and shelf 3D rotations transform matching grasp/place labels. Centroid subtraction and maximum-radius scaling are inverted for deployment actions. These augmentations reduce some depth/segmentation/pose sensitivity but do not remove the segmentation assumption.
B.6 Pre-training Dataset for Initial Policy
Task-specific geometric samplers generate the initial local data: cached grasps and shelf placements, or base/moved-object face-and-offset stacks. Exclude objects already packed/stacked, filter with inverse kinematics, execute in simulation and retain successful transitions. The resulting supervision contains individual manipulation steps rather than demonstrations solving the full long-horizon task.
| Model | Metric | Training Steps | Final |
|---|---|---|---|
| Grasp policy | () | 82.8k | 0.0457 |
| Grasp policy | () | 82.8k | 0.714 |
| Place policy | () | 58.9k | 0.0530 |
| Place policy | () | 58.9k | 0.384 |
| Q function | () | 30k | 0.00681 |
| Q function | () | 30k | 0.0160 |
B.7 Policy-value training metrics.
The grasp/place models train for 82.8k/58.9k steps, with reported final translation errors 0.0457/0.0530 and rotation errors 0.714/0.384. The value model trains 30k steps, reaching training MSE 0.00681 and MAE 0.0160 on subtree targets. The paper interprets pose errors as compatible with multimodal proposals; these diagnostics alone do not demonstrate multimodality or deployment accuracy. Search samples 32 candidates and filters infeasible executions.
Appendix C Task Setup
Task progress and success are hand-defined from simulated or measured arrangements. Their precise geometric criteria matter for interpreting both reward and completion.
C.1 Shelf Packing.
Randomize shelf geometry/pose and collision-free table objects. Packed progress is the volume-weighted fraction of objects whose centers lie inside the shelf cuboid; success requires all centers inside. This is a center-based inclusion test, not a requirement that every point of every object lies within the shelf.
C.2 Object Stacking.
Remove the shelf but retain corresponding object layouts. A placement must settle with contact, horizontal proximity and the moved object above its selected base, while base drift stays within tolerance. Progress is stacked-chain length divided by object count, and success reaches the required tower height.
C.3 Constrained Packing
Objects must finish on the shelf surface without stacking or disturbing previously placed objects. The evaluation explicitly differs from Points2Plans’ original relational-predicate metric: placing behind an existing object is allowed if the final arrangement is valid. Preserve that protocol difference when comparing the 85.9% result with published planners.
C.4 Real-World Shelf Packing and Table Bussing
Hardware shelf packing uses the simulated center-based reward/success definition. Table bussing counts objects whose centroids are inside the tray and whose global roll and pitch stay below 30°. Progress is the fraction satisfying that upright-tray condition; success requires every object. Three trials per task provide limited evidence about reliability.
Appendix D Additional Quantitative Analysis
The appendix separates changes in proposal quality, value ranking, search statistics and the supervision produced by widening.
D.1 How do distillation iterations improve performance?
Policy-only shelf success improves 13.3→20.0→46.7% over two rounds. Holding policy P1 fixed and replacing Q1 with Q2 improves guided success 40.0→66.7%; P2+Q2 reaches 80.0%. Thus both proposal and ranking changes contribute, rather than attributing the entire gain to the policy alone.
D.2 Distillation Ablations
Training on all 8,655 feasible edges gives 52.8% subsequent-search success versus 32.3% from 1,868 successful-plan edges, with wide reported spreads. This changes both selection and dataset size, so it is not an equal-data isolation. Initial-data ablation uses 444, 888, 1,332 and 1,776 demonstrations; reward after 25 search iterations largely saturates at the last two sizes.
| Policy trained on | Success (%) () | Dataset size |
|---|---|---|
| Only successful plans | 1,868 | |
| All feasible edges (Ours) | 8,655 |
| Sub-task data | #Demos | Reward () |
|---|---|---|
| 25% of | 444 | |
| 50% of | 888 | |
| 75% of | 1,332 | |
| 100% of (Ours) | 1,776 |
D.3 Does MCTS converge?
Figure 13 plots selected root-action values in one training environment and their distance to their own final estimates. Frequently useful actions receive further updates; poor ones can be abandoned. Falling distance to the finite-budget endpoint indicates stabilization in this run, not a convergence proof or an error bound against true action values.
D.4 Do better nodes get visited more?
Across 50 training trees, root-action value rises sharply over the first few visits, then plateaus. Spearman correlation is positive but modest at 0.15. Visit count therefore weakly ranks already-promising actions, motivating a learned value rather than equating visits with quality.
D.5 Why Progressive Widening Matters for Distillation
The repeated results emphasize widening’s effect on supervision as well as search coverage. New children expose alternative futures for value learning. A fixed first-visit set of 16 children produces narrower supervision and zero held-out packing success here, despite substantial training plan counts.
| Shelf Packing (Sim) Metrics | Object Stacking Metrics | |||||
|---|---|---|---|---|---|---|
| Algorithm | Reward () | Success (%) () | Train Plans () | Reward () | Success (%) () | Train Plans () |
| Beam Search | ||||||
| MPPI | ||||||
| RL-PD | ||||||
| DISCO w/o Policy | ||||||
| DISCO w/o PW | ||||||
| DISCO (Ours) | ||||||
D.6 Interpretation of Baseline Behavior
Baseline rewards can indicate useful partial completion while full success degrades with object count. The authors interpret policy proposals as reducing search width, value estimates as improving continuation choices, and widening as preserving exploration. Those are empirical interpretations of the component comparisons, not separate quantified complexity bounds.
D.7 Search as Training-Time Supervision
Deployment computation does not include MCTS, so the improvement is carried by trained proposal/value networks. Successful-plan count alone is insufficient: RL-PD reports more stacking training plans than DISCO in the table but much lower held-out stacking success. Search quality, state coverage and ranking supervision all matter.
Appendix E Additional Qualitative Analysis
The original tree visualizations retain node states, edges, IDs and values for selected decisions, later discoveries and alternative solutions.
E.1 Analysis of key decision states
The example favors stable placements of the light-blue object on flat supports, with values 0.944/0.917, over the unstable node-29 placement at 0.25. It illustrates a useful ranking in one explored state rather than a calibrated general stability predictor.
E.2 Analysis of funnel states
A selected funnel state requires the pink mug to be placed before the blue object because it supplies later support. The higher-valued branch illustrates an ordering dependency absent from isolated single-step demonstrations.
E.3 Progressive widening can find better actions later
Progressive widening later adds node 1010, whose yellow-object placement leaves better support for future stacking than previously preferred node 140. The example shows why revisiting a state can discover a useful new action instead of fixing its first proposal set.
E.4 Multi-modality
Figures 18–19 retain the full set of illustrated candidate placements and multiple successful stacking paths. They show diversity in these selected trees, rather than measuring complete coverage of the continuous action space.
Appendix F Mathematical Details
The mathematics formalizes a deterministic finite-horizon simulator problem, success-based scoring and progress-backed search values. The explanatory guarantees depend on exact values and matched continuation distributions; learned scores and correlated search leaves need not satisfy those idealizations.
F.1 Markovian Formulation
Each task fixes an initial state and goal; object poses and composite grasp/place actions define a deterministic simulator transition that includes motion planning. Stop on success, horizon or no feasible action. A deterministic observation function yields segmented point clouds, and stochasticity enters through policy action sampling. The factorized plan distribution is a model assumption; partial point-cloud observations are not proven sufficient Markov state for arbitrary physical manipulation.
Equations 2–7
| (2) |
| (3) |
| (4) |
| (5) |
| (6) |
| (7) |
F.2 Success Probability Objective
The objective maximizes expected terminal success across tasks. Tree targets estimate success among search-generated continuations, not necessarily the raw policy’s action value. With exact matching policy values, best-of-B sampling is at least as good as one sample in expectation over candidate sets; it is not a guarantee for every sampled batch or an imperfect learned ranking. The binomial-style variance motivation for leaf-count weighting is approximate because MCTS continuations are adaptively sampled and can be correlated.
Equations 8–11
| (8) | ||||
| (9) |
| (10) |
| (11) |
F.3 Policy Distillation Target
At each visited state, fitting all feasible edges approximates the proposal distribution restricted to executable actions. Search also shifts training toward deeper useful states absent from local demonstrations. Successful-plan-only supervision is sharper but discards feasible partial paths and provides far fewer examples. The intended division is diverse feasible proposals plus a success-oriented ranking function.
Equation 12
| (12) |
F.4 Value Prior and Progress Backups
An unvisited child receives the learned success prior; its first backup replaces that with observed root-to-leaf progress. Under the stated zero-initial-progress and success-iff-progress-one assumptions, terminal success probability is bounded by expected progress for the same continuation distribution. This does not guarantee that an imperfect learned prior is conservative, or that its value is ordered correctly against different siblings’ sampled returns. Goal-only labels train the value model, and deployment ranks with that model rather than the shaping reward.
Equation 13
| (13) |
