Literature
Display
← All papers

Long-Horizon Multi-Object Rearrangement via Distilled Tree Search over Continuous Actions

Kallol Saha · Yining Hong · Amber Li · Yejin Choi · Maxim Likhachev · David Held

Preprint posted · CoRL 2026 (accepted) ↗

Robotics14 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the 34-page arXiv v1 dated 19 October 2026, retaining all 19 numbered figures, seven numbered tables, five algorithms, 18 displayed equation blocks and six appendices. Five search-tree figures are restored from native PDF art so node images, edges and small labels remain available at original detail. Search supplies training data; deployment still samples and scores grasp/place candidates and uses low-level motion planning. Read the three-trial hardware results as preliminary evidence.

Refer to caption
Figure 1: DISCO: Distilled Tree Search over Continuous Actions. DISCO addresses long-horizon multi-object rearrangement by using progressive-widening Monte Carlo Tree Search over continuous actions. Using an initial behavior-cloned policy to propose short-horizon actions, DISCO expands search trees in parallel GPU simulation to discover long-horizon plans. The search trees are then distilled into a policy and a value function, enabling direct rollout on held-out tasks, such as real-world shelf packing. MCTS and simulation are used only during training. At test time, the distilled model acts directly from point clouds.

At a glance

DISCO alternates physics-simulator tree search with policy/value distillation to extend single-step manipulation data into longer-horizon behavior. A learned policy proposes continuous grasp/place actions; progressive-widening MCTS explores feasible futures, and a learned value ranks candidates. The final model acts from segmented point clouds without simulator-based search at test time. Held-out shelf/stacking success reaches 80.0%/66.7%; physical shelf packing and table bussing succeed in two and one of three trials, respectively.

1 Introduction

An immediately feasible placement can block later objects. DISCO uses simulation search to expose those dependencies, then trains a reactive proposal-and-ranking model on the resulting trees. This adapts the policy/value/search loop familiar from board games to continuous manipulation and point-cloud observations. One model pair is trained per task on randomized instances; it is not a single generalist controller. Initial data contain local grasp/place successes, not complete long-horizon demonstrations.

2 Related Work

The work connects progressive-widening search, plan distillation and 3D policies. It uses a physics simulator rather than learning a transition model, and does not require a symbolic task representation. Compared with Neural Tree Expansion, which retains online search, the final policy/value pair executes without test-time tree construction. Its proposal architecture modifies 3D FlowMatch Actor.

3 Problem Statement and Assumptions

Training assumes object poses, a simulator, task goal/progress functions and short-horizon demonstration pairs. At test time, networks receive segmented point clouds and propose continuous end-effector grasp/place poses; cuRobo converts those into executable trajectories. A valid training plan must satisfy simulated transitions and the terminal goal. Progress reward is the difference in a task-specific scalar, not a learned reward or a test-time ranking signal.

4 DISCO: Distilled Search over Continuous Actions

Initialize the policy from single-object data, generate trees on simulated task instances, and periodically retrain both policy and value. Improved models then guide the next planning sweep. The final closed-loop controller observes again after executing each selected action.

4.1 Monte-Carlo Tree Search over Continuous Actions

Each tree vertex stores a simulator state, visits and search value; edges store continuous actions. Policy samples are motion-planned and simulated, and feasible successors become children. Widening allows additional children while their count is below kNᵅ, with k=2 and α=0.3. This bounds early branching while permitting later alternatives. High visits and high values correlate imperfectly; the appendix’s measured rank correlation is only 0.15.

Refer to caption
Figure 2: MCTS with DISCO. We embed a policy πθ\pi_{\theta} and a value function QϕQ_{\phi} into the MCTS loop. The policy proposes continuous actions for tree expansion, while the value function guides rollout action selection toward terminal states. Periodically, both models are updated from newly generated search trees, enabling direct policy-value deployment at test time.
Algorithm 1 DISCO
Input: πθ,Qϕ,fsim,r,𝒢,(k,α)\pi_{\theta},Q_{\phi},f_{\mathrm{sim}},r,\mathcal{G},(k,\alpha)
Output: Updated πθ\pi_{\theta}, updated QϕQ_{\phi}
1 Initialize G=⟨V,E⟩G=\langle V,E\rangle with root 𝐯0\mathbf{v}_{0}, Vsucc←∅V_{\mathrm{succ}}\leftarrow\emptyset
2 while search is running do
    3 𝐯←\mathbf{v}\leftarrow UCBSelect(G)(G)
    4 if |children⁡(𝐯)|<k​N​(𝐯)α|\mathrm{children}(\mathbf{v})|<kN(\mathbf{v})^{\alpha} then
       5 G←G\leftarrow Expand(G,𝐯,πθ,fsim)(G,\mathbf{v},\pi_{\theta},f_{\mathrm{sim}})
    6 end if
    7 R←R\leftarrow Rollout(𝐯,πθ,Qϕ,r,𝒢)(\mathbf{v},\pi_{\theta},Q_{\phi},r,\mathcal{G})
    8 G←G\leftarrow Backpropagate(G,𝐯,R)(G,\mathbf{v},R)
    9 Vsucc←Vsucc∪{𝐯∈V∣𝒢⁡(s𝐯)=1}V_{\mathrm{succ}}\leftarrow V_{\mathrm{succ}}\cup\{\mathbf{v}\in V\mid\mathcal{G}(s_{\mathbf{v}})=1\}
    10 if distillation interval reached then
       11 (πθ,Qϕ)←(\pi_{\theta},Q_{\phi})\leftarrow Distill(G,Vsucc)(G,V_{\mathrm{succ}})
       12 Reset GG to root 𝐯0\mathbf{v}_{0}, Vsucc←∅V_{\mathrm{succ}}\leftarrow\emptyset
    13 end if
14 end while
15 return πθ,Qϕ\pi_{\theta},Q_{\phi}

4.2 Value-Guided Rollout

Rollouts continue the same search tree: sample batches, add feasible children and choose using value plus a UCB exploration term. Initially an unvisited child uses its incoming progress reward; later rounds use the learned success prior. Visited children instead use averaged backed-up root-to-leaf progress returns. Descents stop at a goal, horizon or dead end. Several descents run concurrently, so a GPU batch can contain actions from different starting vertices; every simulator environment resets to its own state.

4.3 Iterative Distillation

Value targets are successful visited leaves divided by all visited leaves below each non-root node. Weighted binary cross-entropy uses log(1+leaf count), normalized within the minibatch. These targets reflect search continuations rather than direct policy rollouts. Policy distillation uses every feasible expanded edge, including unsuccessful partial paths, with flow matching; the original single-step dataset is not mixed back in. Ranking is delegated to the value model. Two planning/distillation rounds update both networks.

Equation 1
yi=∑𝐯ℓ∈leaves⁡(𝐯i)𝟙𝒢⁡(sℓ)|leaves⁡(𝐯i)|,ℒV​(ϕ)=∑i∈ℬwi​BCE​((Qϕ​(𝐨pi,𝐚i)),yi)∑i∈ℬwi.\begin{aligned} y_{i}&=\frac{\sum_{\mathbf{v}_{\ell}\in\mathrm{leaves}(\mathbf{v}_{i})}\mathbbm{1}_{\mathcal{G}(s_{\ell})}}{|\mathrm{leaves}(\mathbf{v}_{i})|},\end{aligned}\qquad\begin{aligned} \mathcal{L}_{V}(\phi)&=\frac{\sum_{i\in\mathcal{B}}w_{i}\,\mathrm{BCE}\left((Q_{\phi}(\mathbf{o}_{p_{i}},\mathbf{a}_{i})),y_{i}\right)}{\sum_{i\in\mathcal{B}}w_{i}}.\end{aligned} (1)

5 Experiments

Experiments separate training-time planning effectiveness, component ablations, scaling with object count, held-out direct rollout and hardware transfer.

Refer to caption
Figure 3: Task Setup: Representative states from the tasks used in our evaluation. In Shelf Packing and Object Stacking, the robot must fit objects into a shelf and construct stable stacks, respectively. Constrained Packing requires objects to be placed inside a bounded region without collisions. We also evaluate on a real-world version of Shelf Packing along with a Table Bussing task where dinnerware must be placed upright in a tray.

5.1 Datasets and Tasks

Shelf packing and stacking use 50 randomized training instances and 15 held-out instances per task in ManiSkill3, with ACRONYM meshes. Constrained packing compares online planners; physical tests use a Franka Panda with a Robotiq gripper. A single RTX 4090 supports training/evaluation. Shelf search takes 47.7 hours, generating 263,650 vertices with 11,015 expanded; trees store simulator states rather than point clouds, at a reported 95 MB per tree.

5.2 Evaluation

Beam search and MPPI replace the training-time plan generator, then distill controllers; RL-PD learns an actor/critic; ablations remove learned proposals or progressive widening. Except the explicitly online planners, evaluation uses candidate sampling/value ranking without simulator search. DISCO’s held-out shelf/stack success is 80.0%/66.7%, with reward 90.9/94.7; captions report 32 environment seeds and 95% intervals, not 32 independent training runs. Training-task success reaches 100%/98%, which must not be confused with held-out performance. Constrained packing reaches 85.9%, slightly below SPOT’s 86.7% under the adapted protocol. Hardware succeeds 2/3 on shelf packing and 1/3 on table bussing, versus 0/3 for the initial policy on both. Distillation improves the policy alone from 13.3% to 46.7%, and the final value-guided system to 80.0%.

Refer to caption
Figure 4: Simulation Results. DISCO improves reward and success with increasing simulation budget on shelf packing and object stacking, and remains more robust as task complexity increases. The simulation budget counts simulator calls made by search during offline training, not at test time.
Shelf Packing (Sim) Metrics Object Stacking Metrics
Algorithm Reward (↑\uparrow) Success (%) (↑\uparrow) Train Plans (↑\uparrow) Reward (↑\uparrow) Success (%) (↑\uparrow) Train Plans (↑\uparrow)
Beam Search 24.3±8.024.3\scriptstyle{\pm 8.0} 6.7±5.16.7\scriptstyle{\pm 5.1} 4.26±1.144.26\scriptstyle{\pm 1.14} 79.9±4.279.9\scriptstyle{\pm 4.2} 33.3±13.633.3\scriptstyle{\pm 13.6} 5.84±1.005.84\scriptstyle{\pm 1.00}
MPPI 36.7±7.836.7\scriptstyle{\pm 7.8} 13.3±7.213.3\scriptstyle{\pm 7.2} 12.20±3.2512.20\scriptstyle{\pm 3.25} 60.7±6.260.7\scriptstyle{\pm 6.2} 13.3±12.213.3\scriptstyle{\pm 12.2} 3.32±0.503.32\scriptstyle{\pm 0.50}
RL-PD 71.0±4.871.0\scriptstyle{\pm 4.8} 20.0±13.720.0\scriptstyle{\pm 13.7} 138.52±40.81138.52\scriptstyle{\pm 40.81} 64.5±5.764.5\scriptstyle{\pm 5.7} 20.0±8.520.0\scriptstyle{\pm 8.5} 46.46±10.13\mathbf{46.46\scriptstyle{\pm 10.13}}
DISCO w/o Policy 65.9±6.265.9\scriptstyle{\pm 6.2} 6.7±2.36.7\scriptstyle{\pm 2.3} 7.56±1.597.56\scriptstyle{\pm 1.59} 75.2±5.075.2\scriptstyle{\pm 5.0} 33.3±8.933.3\scriptstyle{\pm 8.9} 3.02±0.563.02\scriptstyle{\pm 0.56}
DISCO w/o PW 39.6±7.239.6\scriptstyle{\pm 7.2} 0.0±0.00.0\scriptstyle{\pm 0.0} 133.14±31.88133.14\scriptstyle{\pm 31.88} 64.8±7.564.8\scriptstyle{\pm 7.5} 26.7±10.826.7\scriptstyle{\pm 10.8} 22.98±3.7922.98\scriptstyle{\pm 3.79}
DISCO (Ours) 90.9±6.7\mathbf{90.9\scriptstyle{\pm 6.7}} 80.0±9.5\mathbf{80.0\scriptstyle{\pm 9.5}} 463.16±79.76\mathbf{463.16\scriptstyle{\pm 79.76}} 94.7±2.0\mathbf{94.7\scriptstyle{\pm 2.0}} 66.7±5.5\mathbf{66.7\scriptstyle{\pm 5.5}} 34.12±5.7834.12\scriptstyle{\pm 5.78}
Table 1: Simulation Metrics. We report average reward and success rate for DISCO against baselines on a held-out test set for shelf packing and object stacking. We also report number of plans found by each method during training. Results are averaged over 32 environment seeds and show a 95% confidence interval.
Refer to caption
Figure 5: Constrained Packing Metrics. We rollout DISCO zero-shot against test-time abstraction-based planning methods.
Shelf Packing (Real) Table Bussing (Real)
Algorithm Reward (↑\uparrow) Success (↑\uparrow) Reward (↑\uparrow) Success (↑\uparrow)
3DFA 33.3±19.233.3\scriptstyle{\pm 19.2} 0/30/3 27.8±5.627.8\scriptstyle{\pm 5.6} 0/30/3
DISCO (Ours) 88.9±11.1\mathbf{88.9\scriptstyle{\pm 11.1}} 𝟐/𝟑\mathbf{2/3} 83.3±9.6\mathbf{83.3\scriptstyle{\pm 9.6}} 𝟏/𝟑\mathbf{1/3}
Table 2: Real World Metrics. We report average reward and success rate across 3 trials of the shelf packing and table bussing task for our method compared to 3DFA.
Refer to caption
Figure 6: Real World Rollout of DISCO on the table bussing task. In the point cloud, we see that DISCO’s value function gives high scores for cup placements that leave space for the plates to be placed in the future.
Refer to caption
Figure 7: Iterative Distillation. Held-out shelf packing success of the policy PkP_{k} after the kk-th distillation round, executed alone (solid) or with test-time value guidance from QkQ_{k} (hatched). Both the policy and the value function improve across rounds.
Original S5.F8.fig2 native PDF artwork with complete original panels, labels and layout.
Figure 8: Long-Horizon Dependency. An object stacking search tree in which only placing the pink mug first leads to success, since it supports the blue object placed later. MCTS assigns this action a higher value (Q=0.99Q=0.99) than its alternatives (Q=0.75Q=0.75).

6 Conclusion

Simulation search can supply long-horizon supervision that a segmented-point-cloud controller later uses without online tree search. The evidence supports this loop in the evaluated pick/place tasks; it does not show universal reasoning, reliable hardware performance across many trials, or removal of low-level planning.

7 Limitations

Search depends on an initial policy that proposes enough feasible actions and on sufficiently faithful simulation. Only end-effector pick/place actions are tested; joint-space and dexterous control remain future work. Misaligned progress shaping can bias search even though value targets use terminal goal labels. Foundation-model integration is also untested. Segmentation, geometric execution interfaces, task-specific training and small hardware samples further bound deployment claims.

Acknowledgments

The authors credit Toyota Research Institute, ONR MURI and Korean IITP/MSIT support, plus feedback and camera-calibration assistance. David Held’s concurrent Amazon Scholar appointment is disclosed; the work is attributed to CMU rather than Amazon.

Part Appendix

The six appendices retain baseline implementation, algorithms and architectures, augmentation/data generation, task definitions, quantitative/qualitative analyses and mathematical arguments.

Appendix A Baseline Details

Training-time planning baselines share the task, action interface, reward and budget wherever specified; the separate constrained-packing comparison retains online planners.

A.1 Beam Search

Beam search keeps 16 vertices, expands 16 proposed candidates per beam element and ranks successors by accumulated progress, breaking ties randomly. It has no UCB exploration, progressive widening or learned-value ranking during search. It stops at a goal, dead end, object-count horizon or exhausted budget. Its trees then train policy/value networks for direct evaluation.

Equation
R(ρ0:k)=∑t=0k−1r(st,𝐚t,st+1).R(\rho_{0:k})=\sum_{t=0}^{k-1}r(s_{t},\mathbf{a}_{t},s_{t+1}).

A.2 MPPI

MPPI samples 32 policy-proposed action sequences with a horizon equal to object count. It exponentially weights cumulative progress returns using λ=10, updates the policy from weighted samples and replans after the selected action. It does not use a learned value during planning or retain MCTS backups. The final plans supply the distillation/evaluation pipeline.

Equations
ρk:k+H(b)=[sk,𝐚k(b),sk+1(b),…,𝐚k+H−1(b),sk+H(b)],\rho^{(b)}_{k:k+H}=[s_{k},\mathbf{a}^{(b)}_{k},s^{(b)}_{k+1},\ldots,\mathbf{a}^{(b)}_{k+H-1},s^{(b)}_{k+H}],
R(ρk:k+H(b))=∑t=kk+H−1r(st(b),𝐚t(b),st+1(b)).R(\rho^{(b)}_{k:k+H})=\sum_{t=k}^{k+H-1}r(s^{(b)}_{t},\mathbf{a}^{(b)}_{t},s^{(b)}_{t+1}).
w(b)=exp(λR(ρk:k+H(b)))∑j=1Bexp(λR(ρk:k+H(j))).w^{(b)}=\frac{\exp(\lambda R(\rho^{(b)}_{k:k+H}))}{\sum_{j=1}^{B}\exp(\lambda R(\rho^{(j)}_{k:k+H}))}.

A.3 RL-PD

RL-PD uses the same local demonstrations, observations, composite actions, progress rewards and horizon, combining prior data with online simulation transitions. Soft actor-critic targets include entropy and goal-conditioned termination. Table 3 retains the reported discount 0.99, entropy weight 0.1, batch 64, ten updates per environment iteration, critic/actor learning rates 0.1/0.01 and million-transition buffer. These unusually large listed learning rates are source values, not silently corrected.

Equation
yk=rk+γ⁡(1−𝒢⁡(sk+1))​[Qϕ​(𝐨k+1,𝐚k+1)−α​log⁡πθ​(𝐚k+1|𝐨k+1)],y_{k}=r_{k}+\gamma\left(1-\mathcal{G}(s_{k+1})\right)\left[Q_{\phi}(\mathbf{o}_{k+1},\mathbf{a}_{k+1})-\alpha\log\pi_{\theta}(\mathbf{a}_{k+1}|\mathbf{o}_{k+1})\right],
Hyperparameter Value
Prior data Single-step demonstrations used by DISCO
Discount factor γ\gamma 0.990.99
Entropy temperature α\alpha 0.10.1
Batch size 6464
Gradient updates per environment iteration 1010
Critic learning rate 0.10.1
Actor learning rate 0.010.01
Replay buffer capacity 10610^{6}
Table 3: RL-PD Hyperparameters. We report the actor-critic hyperparameters used for the RL-PD baseline. RL-PD uses the same progress reward, action representation, and task horizon as DISCO, but replaces search-generated supervision with off-policy reinforcement learning from prior and online data.

A.4 DISCO without Policy Prior

Removing the learned proposal does not remove geometric knowledge. Shelf proposals use stored object-frame grasps, pose transforms, inverse-kinematics filtering and bounded placement samples. Stacking chooses base/moved objects, one of six downward faces and a support offset, then checks pick/place feasibility. The same MCTS, progress reward, widening and backups remain, with k=2, α=0.3 and C=1.5. This compares learned versus structured heuristic proposals, not completely uninformed random actions.

A.5 Test-Time Planning Baselines

The constrained-packing evaluation repeats five released scenes with 100 seeds each, for 500 trials; these are not 500 distinct scene geometries. DISCO executes directly, whereas SPOT, Points2Plans and relational baselines plan online. The Points2Plans execution interface is modified to use Jacobian-based velocity control toward DISCO-predicted grasps rather than its original MoveIt inverse kinematics. Results therefore describe the adapted common-interface experiment.

Appendix B Method Details

The detailed algorithms retain how actions become feasible edges, how tree statistics update and how the learned proposal/value pair operates.

B.1 Algorithm Details

UCBSelect descends by stored values/visits. Expand samples candidates, motion-plans and simulates them, discards infeasible transitions and scores new children at the parent observation. Rollout repeatedly expands and selects within the same graph; Backpropagate updates visit counts and running mean returns along parent links and records goal leaves. Algorithms 2–5 preserve exact operations. Deployment retains proposal/value scoring and physical execution, with no graph, UCB bonus or simulator call.

Algorithm 2 Expand
Input: G=⟨V,E⟩G=\langle V,E\rangle, vertex 𝐯\mathbf{v}, policy πθ\pi_{\theta}, simulator fsimf_{\mathrm{sim}}, batch size BB
Output: Updated graph GG
1 Construct 𝐨𝐯\mathbf{o}_{\mathbf{v}} from 𝐬𝐯\mathbf{s}_{\mathbf{v}}
// batched expansion
2 Sample {𝐚(b)}b=1B∼πθ​(𝐚|𝐨𝐯)\{\mathbf{a}^{(b)}\}_{b=1}^{B}\sim\pi_{\theta}(\mathbf{a}|\mathbf{o}_{\mathbf{v}})
3 {τ(b)}b=1B←MotionPlan​(𝐬𝐯,{𝐚(b)}b=1B)\{\tau^{(b)}\}_{b=1}^{B}\leftarrow\textsc{MotionPlan}(\mathbf{s}_{\mathbf{v}},\{\mathbf{a}^{(b)}\}_{b=1}^{B})
4 {𝐬(b)}b=1B←fsim​(𝐬𝐯,{τ(b)}b=1B)\{\mathbf{s}^{(b)}\}_{b=1}^{B}\leftarrow f_{\mathrm{sim}}(\mathbf{s}_{\mathbf{v}},\{\tau^{(b)}\}_{b=1}^{B})
5 for b=1,…,Bb=1,\ldots,B do
    6 if τ(b)\tau^{(b)} is feasible then
       7 Create child 𝐯(b)\mathbf{v}^{(b)} with state 𝐬(b)\mathbf{s}^{(b)}
       8 Initialize Q⁡(𝐯)←Qϕ​(𝐨𝐯,𝐚(b))Q(\mathbf{v})\leftarrow Q_{\phi}(\mathbf{o}_{\mathbf{v}},\mathbf{a}^{(b)}) e(b)←⟨𝐯,𝐚(b),τ(b),𝐯(b)⟩e^{(b)}\leftarrow\langle\mathbf{v},\mathbf{a}^{(b)},\tau^{(b)},\mathbf{v}^{(b)}\rangle
       9 G←⟨V∪{𝐯(b)},E∪{e(b)}⟩G\leftarrow\langle V\cup\{\mathbf{v}^{(b)}\},E\cup\{e^{(b)}\}\rangle
    10 end if
11 end for
12 return GG
Algorithm 3 Rollout
Input: G=⟨V,E⟩G=\langle V,E\rangle, vertex 𝐯\mathbf{v}, policy πθ\pi_{\theta}, value function QϕQ_{\phi}, reward rr, goal 𝒢\mathcal{G}, horizon HH, batch size BB
Output: return RR, terminal vertex 𝐯T\mathbf{v}_{T}, updated graph GG
1 𝐯cur←𝐯\mathbf{v}_{\mathrm{cur}}\leftarrow\mathbf{v}
2 R←0R\leftarrow 0
3 for h=1,…,Hh=1,\ldots,H do
    4 if 𝒢⁡(𝐬𝐯cur)=1\mathcal{G}(\mathbf{s}_{\mathbf{v}_{\mathrm{cur}}})=1 then
       5 break
    6 end if
    7 G←Expand​(G,𝐯cur,πθ,fsim,B)G\leftarrow\textsc{Expand}(G,\mathbf{v}_{\mathrm{cur}},\pi_{\theta},f_{\mathrm{sim}},B)
    8 Construct 𝐨cur\mathbf{o}_{\mathrm{cur}} from 𝐬𝐯cur\mathbf{s}_{\mathbf{v}_{\mathrm{cur}}}, get actions {𝐚(b)}b=1B\{\mathbf{a}^{(b)}\}_{b=1}^{B} from children⁡(𝐯cur)\mathrm{children}(\mathbf{v}_{\mathrm{cur}})
    9 𝐯c∗←arg⁡max​UCB​(𝐯,C)\mathbf{v}_{c}^{*}\leftarrow\arg\max\textsc{UCB}(\mathbf{v},C)
    10 ri←r⁡(𝐬𝐯cur,𝐚∗,𝐬𝐯c∗)r_{i}\leftarrow r(\mathbf{s}_{\mathbf{v}_{\mathrm{cur}}},\mathbf{a}^{*},\mathbf{s}_{\mathbf{v}_{c}^{*}})
    11 R←R+riR\leftarrow R+r_{i}
    12 𝐯cur←𝐯c∗\mathbf{v}_{\mathrm{cur}}\leftarrow\mathbf{v}_{c}^{*}
13 end for
14 𝐯T←𝐯cur\mathbf{v}_{T}\leftarrow\mathbf{v}_{\mathrm{cur}}
15 return R,𝐯T,GR,\mathbf{v}_{T},G
Algorithm 4 UCBSelect
Input: Search graph G=⟨V,E⟩G=\langle V,E\rangle, root vertex 𝐯0\mathbf{v}_{0}, exploration constant CC
Output: Leaf vertex 𝐯\mathbf{v}
1 𝐯←𝐯0\mathbf{v}\leftarrow\mathbf{v}_{0}
2 while children⁡(𝐯)≠∅\mathrm{children}(\mathbf{v})\neq\emptyset do
    3 𝐯c∗←arg⁡max​UCB​(𝐯,C)\mathbf{v}_{c}^{*}\leftarrow\arg\max\textsc{UCB}(\mathbf{v},C)
    4 𝐯←𝐯c∗\mathbf{v}\leftarrow\mathbf{v}_{c}^{*}
5 end while
6 return 𝐯\mathbf{v}
Algorithm 5 Backpropagate
Input: GG, terminal vertex 𝐯T\mathbf{v}_{T}, return RR
Output: updated GG
1 𝐯←𝐯T\mathbf{v}\leftarrow\mathbf{v}_{T}
2 while 𝐯≠∅\mathbf{v}\neq\emptyset do
    3 N⁡(𝐯)←N⁡(𝐯)+1N(\mathbf{v})\leftarrow N(\mathbf{v})+1
    4 Q⁡(𝐯)←Q⁡(𝐯)+R−Q⁡(𝐯)N⁡(𝐯)Q(\mathbf{v})\leftarrow Q(\mathbf{v})+\frac{R-Q(\mathbf{v})}{N(\mathbf{v})}
    5 if 𝒢⁡(𝐬𝐯)=1\mathcal{G}(\mathbf{s}_{\mathbf{v}})=1 then
       6 Vsucc←Vsucc∪{𝐯}V_{\mathrm{succ}}\leftarrow V_{\mathrm{succ}}\cup\{\mathbf{v}\}
    7 end if
    8 𝐯←parent⁡(𝐯)\mathbf{v}\leftarrow\mathrm{parent}(\mathbf{v})
9 end while
10 return GG

B.2 MCTS Hyperparameters

Total simulator budgets are 270,000 calls for packing and 27,000 for stacking. Expansion caps at 32 candidates, with 32 GPU environments handling concurrent descents; the task horizon is the 3–7 object count. Widening uses k=2, α=0.3 and UCB C=1.5. The appendix specifies distillation after a complete training-set planning sweep, twice in total. This clarifies the main text’s more generic “every few MCTS iterations” wording.

Hyperparameter Value
Simulation budget β\beta 270,000 (shelf packing), 27,000 (object stacking)
Progressive-widening coefficient kk 2.0
Progressive-widening exponent α\alpha 0.3
Exploration constant CC 1.5
Expansion batch size BB 32
Maximum task horizon KK number of objects (3–7 per task)
Number of distillation rounds 2
Number of training seeds/tasks 50
Number of held-out seeds/tasks 15
Table 4: MCTS and distillation hyperparameters.

B.3 Policy Architecture

Separate rectified-flow grasp and grasp-conditioned place actors consume 4,096-point clouds. Two EdgeConv layers use an xyz k-nearest-neighbor graph with k=16 to produce width-120 tokens; a transformer with rotary 3D positions and adaptive normalization predicts flow velocities. Euler integration starts from Gaussian pose noise. Each output is position, quaternion and fixed grasp/place gripper state. Target-object clouds retain shelf/packed-object context; stacking adds a target mask, while packing uses xyz channels. The controller invokes the policy for remaining segmented objects.

Original figure
Figure 9: Policy and Value Function Architectures. Top: the policy πθ\pi_{\theta} encodes the target-object point cloud with EdgeConv layers and denoises a pose token with a flow-matching transformer to predict a grasp or place pose. Bottom: the value function QϕQ_{\phi} embeds the full-scene point cloud with a per-point MLP, and an action-conditioned query attends to the scene tokens to predict the success probability Qϕ​(𝐨,𝐚)∈[0,1]Q_{\phi}(\mathbf{o},\mathbf{a})\in[0,1].

B.4 Value Function Architecture

The value model sees the full scene plus two grasp/place pose tokens, embedding points with a per-point MLP rather than EdgeConv. Four shared attention layers, eight heads and a learned action-conditioned value query produce a sigmoid success score. Targets are search-subtree leaf fractions, with logarithmic evidence weighting. Figure 10 shows mostly zero/one targets and greater weighted influence for intermediate outcomes. A low training loss does not establish calibrated success probabilities on unseen real scenes.

Refer to caption
Figure 10: Value Function Targets. We visualize the distribution of value-function targets and the effect of evidence weighting. Left: histogram of targets v=nsucc/nleavesv=n_{\mathrm{succ}}/n_{\mathrm{leaves}} over 11,015 eligible nodes from 50 training trees, showing that the distribution is dominated by saturated targets at 0 and 1. Right: raw target-bucket mass compared to the effective training mass under the weighted BCE loss with w=log⁡(1+nleaves)w=\log(1+n_{\mathrm{leaves}}), which increases the contribution of intermediate targets backed by more rollout evidence.

B.5 Sim-to-Real Augmentation

Networks use normalized xyz point clouds, without RGB, simulator-state or proprioceptive inputs; segmented objects are assumed in both domains. Training adds Gaussian noise with sampled scale up to 1 cm, removes local neighborhoods and substitutes bounding-box points. Separate object yaw and shelf 3D rotations transform matching grasp/place labels. Centroid subtraction and maximum-radius scaling are inverted for deployment actions. These augmentations reduce some depth/segmentation/pose sensitivity but do not remove the segmentation assumption.

B.6 Pre-training Dataset for Initial Policy

Task-specific geometric samplers generate the initial local data: cached grasps and shelf placements, or base/moved-object face-and-offset stacks. Exclude objects already packed/stacked, filter with inverse kinematics, execute in simulation and retain successful transitions. The resulting supervision contains individual manipulation steps rather than demonstrations solving the full long-horizon task.

Model Metric Training Steps Final
Grasp policy ℒpos\mathcal{L}_{\mathrm{pos}} (↓\downarrow) 82.8k 0.0457
Grasp policy ℒrot\mathcal{L}_{\mathrm{rot}} (↓\downarrow) 82.8k 0.714
Place policy ℒpos\mathcal{L}_{\mathrm{pos}} (↓\downarrow) 58.9k 0.0530
Place policy ℒrot\mathcal{L}_{\mathrm{rot}} (↓\downarrow) 58.9k 0.384
Q function ℒMSEQ\mathcal{L}_{\mathrm{MSE}}^{Q} (↓\downarrow) 30k 0.00681
Q function ℒMAEQ\mathcal{L}_{\mathrm{MAE}}^{Q} (↓\downarrow) 30k 0.0160
Table 5: Policy and Value Training Metrics. We report final training losses for the grasp policy, place policy, and Q function. The grasp policy is trained for 82.8k steps and the place policy is trained for 58.9k steps using the flow-matching policy objective, where ℒpos\mathcal{L}_{\mathrm{pos}} denotes translation error and ℒrot\mathcal{L}_{\mathrm{rot}} denotes rotation error. The Q function is trained for 30k steps, where ℒMSEQ\mathcal{L}_{\mathrm{MSE}}^{Q} and ℒMAEQ\mathcal{L}_{\mathrm{MAE}}^{Q} denote mean squared error and mean absolute error on subtree-success targets.

B.7 Policy-value training metrics.

The grasp/place models train for 82.8k/58.9k steps, with reported final translation errors 0.0457/0.0530 and rotation errors 0.714/0.384. The value model trains 30k steps, reaching training MSE 0.00681 and MAE 0.0160 on subtree targets. The paper interprets pose errors as compatible with multimodal proposals; these diagnostics alone do not demonstrate multimodality or deployment accuracy. Search samples 32 candidates and filters infeasible executions.

Appendix C Task Setup

Task progress and success are hand-defined from simulated or measured arrangements. Their precise geometric criteria matter for interpreting both reward and completion.

C.1 Shelf Packing.

Randomize shelf geometry/pose and collision-free table objects. Packed progress is the volume-weighted fraction of objects whose centers lie inside the shelf cuboid; success requires all centers inside. This is a center-based inclusion test, not a requirement that every point of every object lies within the shelf.

Refer to caption
Figure 11: Shelf Packing Environments. Visualization of all 50 train environments and 15 test environments for the shelf packing task setup.

C.2 Object Stacking.

Remove the shelf but retain corresponding object layouts. A placement must settle with contact, horizontal proximity and the moved object above its selected base, while base drift stays within tolerance. Progress is stacked-chain length divided by object count, and success reaches the required tower height.

Refer to caption
Figure 12: Object Stacking Environments. Visualization of all 50 train environments and 15 test environments for the object stacking task setup.

C.3 Constrained Packing

Objects must finish on the shelf surface without stacking or disturbing previously placed objects. The evaluation explicitly differs from Points2Plans’ original relational-predicate metric: placing behind an existing object is allowed if the final arrangement is valid. Preserve that protocol difference when comparing the 85.9% result with published planners.

C.4 Real-World Shelf Packing and Table Bussing

Hardware shelf packing uses the simulated center-based reward/success definition. Table bussing counts objects whose centroids are inside the tray and whose global roll and pitch stay below 30°. Progress is the fraction satisfying that upright-tray condition; success requires every object. Three trials per task provide limited evidence about reliability.

Appendix D Additional Quantitative Analysis

The appendix separates changes in proposal quality, value ranking, search statistics and the supervision produced by widening.

D.1 How do distillation iterations improve performance?

Policy-only shelf success improves 13.3→20.0→46.7% over two rounds. Holding policy P1 fixed and replacing Q1 with Q2 improves guided success 40.0→66.7%; P2+Q2 reaches 80.0%. Thus both proposal and ranking changes contribute, rather than attributing the entire gain to the policy alone.

D.2 Distillation Ablations

Training on all 8,655 feasible edges gives 52.8% subsequent-search success versus 32.3% from 1,868 successful-plan edges, with wide reported spreads. This changes both selection and dataset size, so it is not an equal-data isolation. Initial-data ablation uses 444, 888, 1,332 and 1,776 demonstrations; reward after 25 search iterations largely saturates at the last two sizes.

Policy trained on Success (%) (↑\uparrow) Dataset size
Only successful plans 32.3±24.932.3\scriptstyle{\pm 24.9} 1,868
All feasible edges (Ours) 52.8±38.9\mathbf{52.8\scriptstyle{\pm 38.9}} 8,655
Sub-task data #Demos Reward (↑\uparrow)
25% of 𝒟\mathcal{D} 444 28.4±7.528.4\scriptstyle{\pm 7.5}
50% of 𝒟\mathcal{D} 888 32.5±6.832.5\scriptstyle{\pm 6.8}
75% of 𝒟\mathcal{D} 1,332 41.0±8.041.0\scriptstyle{\pm 8.0}
100% of 𝒟\mathcal{D} (Ours) 1,776 41.9±8.2\mathbf{41.9\scriptstyle{\pm 8.2}}
Table 6: Distillation Ablations. Left: training the policy on all feasible edges gives higher success in subsequent search than training on only successful plans. Right: pre-training on fractions of the sub-task dataset 𝒟\mathcal{D}. Reward after 25 search iterations saturates beyond 1,332 demonstrations.

D.3 Does MCTS converge?

Figure 13 plots selected root-action values in one training environment and their distance to their own final estimates. Frequently useful actions receive further updates; poor ones can be abandoned. Falling distance to the finite-budget endpoint indicates stabilization in this run, not a convergence proof or an error bound against true action values.

Refer to caption
Figure 13: Convergence of QQ values. We visualize the convergence of root-action value estimates during MCTS in a representative training environment. Left: QQ trajectories for representative depth-1 actions, where markers indicate updates on iterations when the corresponding action is selected. Right: mean absolute gap to the final estimate, |Q⁡(t)−Qfinal|\lvert Q(t)-Q_{\mathrm{final}}\rvert, averaged over all visited root actions. Root-action values fan out from noisy early estimates and converge steadily as search budget increases.

D.4 Do better nodes get visited more?

Across 50 training trees, root-action value rises sharply over the first few visits, then plateaus. Spearman correlation is positive but modest at 0.15. Visit count therefore weakly ranks already-promising actions, motivating a learned value rather than equating visits with quality.

Refer to caption
Figure 14: Visit count compared to QQ-values. We plot the MCTS value estimate of each visited root action against its visit count across the training trees. Gray points show individual depth-1 nodes, and the blue curve shows the binned mean with standard error. Spearman ρ\rho measures rank correlation between visit count and QQ without assuming a linear relationship. The positive but modest correlation (ρ=+0.15\rho=+0.15) reflects an early rise followed by a plateau. Mean QQ rises sharply over the first few visits and then saturates, indicating that MCTS quickly abandons poor actions and allocates additional search budget primarily to actions already confirmed to be promising.

D.5 Why Progressive Widening Matters for Distillation

The repeated results emphasize widening’s effect on supervision as well as search coverage. New children expose alternative futures for value learning. A fixed first-visit set of 16 children produces narrower supervision and zero held-out packing success here, despite substantial training plan counts.

Refer to caption
Figure 15: Simulation Results. DISCO improves reward and success with increasing simulation budget on shelf packing and object stacking, and remains more robust as task complexity increases.
Shelf Packing (Sim) Metrics Object Stacking Metrics
Algorithm Reward (↑\uparrow) Success (%) (↑\uparrow) Train Plans (↑\uparrow) Reward (↑\uparrow) Success (%) (↑\uparrow) Train Plans (↑\uparrow)
Beam Search 24.3±8.024.3\scriptstyle{\pm 8.0} 6.7±5.16.7\scriptstyle{\pm 5.1} 4.26±1.144.26\scriptstyle{\pm 1.14} 79.9±4.279.9\scriptstyle{\pm 4.2} 33.3±13.633.3\scriptstyle{\pm 13.6} 5.84±1.005.84\scriptstyle{\pm 1.00}
MPPI 36.7±7.836.7\scriptstyle{\pm 7.8} 13.3±7.213.3\scriptstyle{\pm 7.2} 12.20±3.2512.20\scriptstyle{\pm 3.25} 60.7±6.260.7\scriptstyle{\pm 6.2} 13.3±12.213.3\scriptstyle{\pm 12.2} 3.32±0.503.32\scriptstyle{\pm 0.50}
RL-PD 71.0±4.871.0\scriptstyle{\pm 4.8} 20.0±13.720.0\scriptstyle{\pm 13.7} 138.52±40.81138.52\scriptstyle{\pm 40.81} 64.5±5.764.5\scriptstyle{\pm 5.7} 20.0±8.520.0\scriptstyle{\pm 8.5} 46.46±10.13\mathbf{46.46\scriptstyle{\pm 10.13}}
DISCO w/o Policy 65.9±6.265.9\scriptstyle{\pm 6.2} 6.7±2.36.7\scriptstyle{\pm 2.3} 7.56±1.597.56\scriptstyle{\pm 1.59} 75.2±5.075.2\scriptstyle{\pm 5.0} 33.3±8.933.3\scriptstyle{\pm 8.9} 3.02±0.563.02\scriptstyle{\pm 0.56}
DISCO w/o PW 39.6±7.239.6\scriptstyle{\pm 7.2} 0.0±0.00.0\scriptstyle{\pm 0.0} 133.14±31.88133.14\scriptstyle{\pm 31.88} 64.8±7.564.8\scriptstyle{\pm 7.5} 26.7±10.826.7\scriptstyle{\pm 10.8} 22.98±3.7922.98\scriptstyle{\pm 3.79}
DISCO (Ours) 90.9±6.7\mathbf{90.9\scriptstyle{\pm 6.7}} 80.0±9.5\mathbf{80.0\scriptstyle{\pm 9.5}} 463.16±79.76\mathbf{463.16\scriptstyle{\pm 79.76}} 94.7±2.0\mathbf{94.7\scriptstyle{\pm 2.0}} 66.7±5.5\mathbf{66.7\scriptstyle{\pm 5.5}} 34.12±5.7834.12\scriptstyle{\pm 5.78}
Table 7: Simulation Metrics. We report average reward and success rate for DISCO against baselines on a held-out test set for shelf packing and object stacking. We also report number of plans found by each method during training. Results are averaged over 32 environment seeds and show a 95% confidence interval.

D.6 Interpretation of Baseline Behavior

Baseline rewards can indicate useful partial completion while full success degrades with object count. The authors interpret policy proposals as reducing search width, value estimates as improving continuation choices, and widening as preserving exploration. Those are empirical interpretations of the component comparisons, not separate quantified complexity bounds.

D.7 Search as Training-Time Supervision

Deployment computation does not include MCTS, so the improvement is carried by trained proposal/value networks. Successful-plan count alone is insufficient: RL-PD reports more stacking training plans than DISCO in the table but much lower held-out stacking success. Search quality, state coverage and ranking supervision all matter.

Appendix E Additional Qualitative Analysis

The original tree visualizations retain node states, edges, IDs and values for selected decisions, later discoveries and alternative solutions.

E.1 Analysis of key decision states

The example favors stable placements of the light-blue object on flat supports, with values 0.944/0.917, over the unstable node-29 placement at 0.25. It illustrates a useful ranking in one explored state rather than a calibrated general stability predictor.

Original A5.F16 native PDF artwork with complete original panels, labels and layout.
Figure 16: Decision Making. The graph above shows an object stacking task vertex with several children. In this graph we see that the higher Q-values of 0.944 and 0.917 are given most often to stable placements of the light blue object, by placing it on flat surfaces, rather than the unstable placement in node id=29 with a Q-value of 0.25

E.2 Analysis of funnel states

A selected funnel state requires the pink mug to be placed before the blue object because it supplies later support. The higher-valued branch illustrates an ordering dependency absent from isolated single-step demonstrations.

E.3 Progressive widening can find better actions later

Progressive widening later adds node 1010, whose yellow-object placement leaves better support for future stacking than previously preferred node 140. The example shows why revisiting a state can discover a useful new action instead of fixing its first proposal set.

Original A5.F17 native PDF artwork with complete original panels, labels and layout.
Figure 17: Mind Change. The object stacking graph above shows how the DISCO algorithm changes its mind after progressive widening adds a new action and a new vertex. Here, vertex id=140 was the preferred vertex with more objects getting stacked. Later during search, vertex id=1010 is added, offering a better placement for the yellow object, that allows more objects to be placed on top. Thus, progressive widening leads DISCO to find better actions later during search.

E.4 Multi-modality

Figures 18–19 retain the full set of illustrated candidate placements and multiple successful stacking paths. They show diversity in these selected trees, rather than measuring complete coverage of the continuous action space.

Original A5.F18 native PDF artwork with complete original panels, labels and layout.
Figure 18: Multi-Modal Proposals. The graph above shows multi-modal action proposals of object stacking originating from a single node.
Original A5.F19 native PDF artwork with complete original panels, labels and layout.
Figure 19: Multi-Modal Plans. The graph of above visualizes multi-modal plans found by DISCO. Here, we show a graph of the object stacking environment where DISCO finds many different ways of creating the stack of objects.

Appendix F Mathematical Details

The mathematics formalizes a deterministic finite-horizon simulator problem, success-based scoring and progress-backed search values. The explanatory guarantees depend on exact values and matched continuation distributions; learned scores and correlated search leaves need not satisfy those idealizations.

F.1 Markovian Formulation

Each task fixes an initial state and goal; object poses and composite grasp/place actions define a deterministic simulator transition that includes motion planning. Stop on success, horizon or no feasible action. A deterministic observation function yields segmented point clouds, and stochasticity enters through policy action sampling. The factorized plan distribution is a model assumption; partial point-cloud observations are not proven sufficient Markov state for arbitrary physical manipulation.

Equations 2–7
sk+1=fsim​(sk,𝐚k).s_{k+1}=f_{\mathrm{sim}}(s_{k},\mathbf{a}_{k}). (2)
p⁡(ρ∣s0)=∏k=0T−1p⁡(sk+1∣sk,𝐚k)​p​(𝐚k∣sk).p(\rho\mid s_{0})=\prod_{k=0}^{T-1}p(s_{k+1}\mid s_{k},\mathbf{a}_{k})\,p(\mathbf{a}_{k}\mid s_{k}). (3)
p⁡(𝐚k∣sk)=∫𝐨kp⁡(𝐚k∣𝐨k)​p​(𝐨k∣sk)​d​𝐨k.p(\mathbf{a}_{k}\mid s_{k})=\int_{\mathbf{o}_{k}}p(\mathbf{a}_{k}\mid\mathbf{o}_{k})\,p(\mathbf{o}_{k}\mid s_{k})\,d\mathbf{o}_{k}. (4)
𝐨k=Ω⁡(sk),sk+1=fsim​(sk,𝐚k).\mathbf{o}_{k}=\Omega(s_{k}),\qquad s_{k+1}=f_{\mathrm{sim}}(s_{k},\mathbf{a}_{k}). (5)
p⁡(ρ∣s0)=∏k=0T−1p⁡(sk+1∣sk,𝐚k)​p​(𝐚k∣sk),p⁡(𝐚k∣sk)≈πθ​(𝐚k∣𝐨k).\begin{gathered}p(\rho\mid s_{0})=\prod_{k=0}^{T-1}p(s_{k+1}\mid s_{k},\mathbf{a}_{k})\,p(\mathbf{a}_{k}\mid s_{k}),\\ p(\mathbf{a}_{k}\mid s_{k})\approx\pi_{\theta}(\mathbf{a}_{k}\mid\mathbf{o}_{k}).\end{gathered} (6)
pθ​(ρ∣s0)=∏k=0T−1πθ​(𝐚k∣𝐨k)p_{\theta}(\rho\mid s_{0})=\prod_{k=0}^{T-1}\pi_{\theta}(\mathbf{a}_{k}\mid\mathbf{o}_{k}) (7)

F.2 Success Probability Objective

The objective maximizes expected terminal success across tasks. Tree targets estimate success among search-generated continuations, not necessarily the raw policy’s action value. With exact matching policy values, best-of-B sampling is at least as good as one sample in expectation over candidate sets; it is not a guarantee for every sampled batch or an imperfect learned ranking. The binomial-style variance motivation for leaf-count weighting is approximate because MCTS continuations are adaptively sampled and can be correlated.

Equations 8–11
pθ​(𝒢⁡(sT)=1∣s0)\displaystyle p_{\theta}(\mathcal{G}(s_{T})=1\mid s_{0}) =∫ρp⁡(𝒢⁡(sT)=1∣ρ,s0)​pθ​(ρ∣s0)​𝑑ρ\displaystyle=\int_{\rho}p(\mathcal{G}(s_{T})=1\mid\rho,s_{0})\,p_{\theta}(\rho\mid s_{0})\,d\rho (8)
=𝔼ρ∼pθ​(ρ∣s0)​[𝟙𝒢⁡(sT)],\displaystyle=\mathbb{E}_{\rho\sim p_{\theta}(\rho\mid s_{0})}\left[\mathbbm{1}_{\mathcal{G}(s_{T})}\right],
θ∗=arg⁡maxθ​𝔼⟨s0,𝒢⟩∼𝒯​𝔼ρ∼pθ​(ρ∣s0)​[𝟙𝒢⁡(sT)].\theta^{*}=\arg\max_{\theta}\;\mathbb{E}_{\langle s_{0},\mathcal{G}\rangle\sim\mathcal{T}}\;\mathbb{E}_{\rho\sim p_{\theta}(\rho\mid s_{0})}\left[\mathbbm{1}_{\mathcal{G}(s_{T})}\right]. (9)
Qπ​(s,𝐚)=P⁡(𝒢⁡(sT)=1∣sk=s,𝐚k=𝐚,π),Q^{\pi}(s,\mathbf{a})=P\big(\mathcal{G}(s_{T})=1\mid s_{k}=s,\ \mathbf{a}_{k}=\mathbf{a},\ \pi\big), (10)
Qπθ​(s,𝐚∗)=maxb⁡Qπθ​(s,𝐚(b))≥𝔼𝐚∼πθ​[Qπθ​(s,𝐚)],Q^{\pi_{\theta}}(s,\mathbf{a}^{*})=\max_{b}Q^{\pi_{\theta}}(s,\mathbf{a}^{(b)})\;\geq\;\mathbb{E}_{\mathbf{a}\sim\pi_{\theta}}\big[Q^{\pi_{\theta}}(s,\mathbf{a})\big], (11)

F.3 Policy Distillation Target

At each visited state, fitting all feasible edges approximates the proposal distribution restricted to executable actions. Search also shifts training toward deeper useful states absent from local demonstrations. Successful-plan-only supervision is sharper but discards feasible partial paths and provides far fewer examples. The intended division is diverse feasible proposals plus a success-oriented ranking function.

Equation 12
πnew(𝐚∣𝐨)∝πθ(𝐚∣𝐨) 1[𝐚∈ℱ(s)].\pi_{\mathrm{new}}(\mathbf{a}\mid\mathbf{o})\;\propto\;\pi_{\theta}(\mathbf{a}\mid\mathbf{o})\,\mathbbm{1}\big[\mathbf{a}\in\mathcal{F}(s)\big]. (12)

F.4 Value Prior and Progress Backups

An unvisited child receives the learned success prior; its first backup replaces that with observed root-to-leaf progress. Under the stated zero-initial-progress and success-iff-progress-one assumptions, terminal success probability is bounded by expected progress for the same continuation distribution. This does not guarantee that an imperfect learned prior is conservative, or that its value is ordered correctly against different siblings’ sampled returns. Goal-only labels train the value model, and deployment ranks with that model rather than the shaping reward.

Equation 13
P⁡(𝒢⁡(sT)=1)=P⁡(R≥1)≤𝔼⁡[R].P\big(\mathcal{G}(s_{T})=1\big)=P(R\geq 1)\leq\mathbb{E}[R]. (13)
Concept demonstration

Open image ↗