Literature
Display
← All papers

Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs

Yining Hong · Huang Huang · Manling Li · Li Fei-Fei · Leonidas Guibas · Jiajun Wu · Yejin Choi

Preprint posted · arXiv (preprint) ↗

AI agents17 min text · CondensedOriginal paper ↗
Contents

Overview

The agent learns during a deployment episode: propose several actions, predict their usefulness, execute one, then revisit earlier decisions using actual outcomes. Its external evaluator stays frozen while the action generator and internal evaluator adapt.

Complete original PDF S0.F1
Figure 1: Conceptual overview of Reflective Test-Time Planning. The agent (a) receives a long-horizon task, (b) performs reflection-in-action by internally simulating and scoring candidate actions, (c) performs reflection-on-action by updating its beliefs and decision systems based on execution outcomes, and (d) conducts retrospective reflection to revise earlier decisions with hindsight.

At a glance

Reflective Test-Time Planning couples best-of-N action selection with parameter updates driven by verbal feedback and hindsight scores. On the authors’ household benchmark, mean category success rises from 10.4% without either reflection mechanism to 33.65%. Cupboard fitting and limited sim-to-real tests also improve, although fitting any compartment and reaching the correct compartment give different rankings.

1 Introduction

A physically successful action can still be strategically wrong: placing a small object may occupy the only space for a larger one. Immediate feedback diagnoses the local outcome; retrospective reflection revises earlier scores after downstream consequences become visible. The mechanism operationalizes reflective practice and double-loop learning by changing both decisions and the evaluator that guides them.

This edition covers all 33 pages of arXiv v3, dated 25 May 2026. It retains eight numbered figures, 18 numbered tables, 13 numbered equations, six additional displays including four training schemas, Algorithm 1 and all 46 references. The catalog’s original posting date remains distinct from the edition’s source version.

2 Related Works

Unlike Reflexion and Self-Refine, which primarily reuse critiques as context, this method turns hindsight feedback into deployment-time weight updates. Unlike fixed predictive scorers, it updates the internal evaluator after encountering failures. LoRA makes these updates smaller; spatial memory and 3D grounding provide the observations on which they depend.

The paper contrasts its single-episode adaptation with conventional planning and reinforcement-learning training loops. That comparison motivates this implementation; it does not establish that all planning or reinforcement-learning methods must keep their models fixed at deployment.

3 Reflective Test-Time Planning

Initialize a unified embodied model through supervised fine-tuning, then instantiate three copies: action policy π, internal evaluator Vᵢ and frozen external evaluator Vₑ. Action execution supplies new observations and a binary outcome signal. The learning signal is self-generated language grounded in that feedback, not an independent human reward oracle.

Complete original PDF S3.F2
Figure 2: Method overview. (a) Reflection-in-action: multiple candidate actions are generated and scored by an internal reflection LLM prior to execution. (b) Reflection-on-action: iteratively invoked when working memory hits K or at key milestones. Executed actions are critiqued by an external reflection LLM and stored in a working memory buffer; at milestones, hindsight re-evaluation assigns long-horizon credit. The resulting verbal reflections form self-supervised training data to update both the internal reflection LLM (supervised loss) and the action LLM (policy gradient) via test-time training, enabling agents to learn from execution experience during deployment.

3.1 Reflection-in-Action

Construct a prompt from the task, current observation, previous action and previous external reflection. Sample N candidates at elevated temperature; score each with an internal reflection and a number from 0 to 100; execute the maximum-score candidate. Equations 1–4 contrast greedy decoding with this selection process. The internal “simulation” is model reasoning, not a verified rollout of physical dynamics.

Algorithm 1 Reflective Test-Time Planning
Complete original PDF alg1
Original selectable source
0:  Task τ\tau, initial observation o1o_{1}; action LLM πθ\pi_{\theta}, internal reflection LLM VϕiV_{\phi_{i}}, external reflection LLM VϕeV_{\phi_{e}}; window size KK
1:  Initialize a0,fe0←Nonea_{0},f_{e}^{0}\leftarrow\text{None}; buffers 𝒲,𝒟train,𝒟retro←∅\mathcal{W},\mathcal{D}_{\text{train}},\mathcal{D}_{\text{retro}}\leftarrow\emptyset; temperature TT; max step 𝒯\mathcal{T}
2:  for t=1,…,𝒯t=1,\ldots,\mathcal{T} do
3:   // Reflection-in-Action
4:   Construct xactionx_{\text{action}} from (τ,ot,at−1,fet−1)(\tau,o_{t},a_{t-1},f_{e}^{t-1}) and sample {atk}k=1N∼πθ(⋅|xaction;T)\{a_{t}^{k}\}_{k=1}^{N}\sim\pi_{\theta}(\cdot|x_{\text{action}};T)
5:   for k=1,…,Nk=1,\ldots,N do
6:    Construct xinternalkx_{\text{internal}}^{k} and score fit,k,sit,k←Vϕi​(xinternalk)f_{i}^{t,k},s_{i}^{t,k}\leftarrow V_{\phi_{i}}(x_{\text{internal}}^{k})
7:   end for
8:   Select at∗=at(arg⁡maxk⁡sit,k)a_{t}^{*}=a_{t}^{(\arg\max_{k}s_{i}^{t,k})}; execute at∗a_{t}^{*} and observe (ot+1,et)(o_{t+1},e_{t}); break if task complete
9:   // Reflection-on-Action
10:   Construct xexternalx_{\text{external}} and generate feedback fet,set←Vϕe​(xexternal)f_{e}^{t},s_{e}^{t}\leftarrow V_{\phi_{e}}(x_{\text{external}})
11:   𝒲←𝒲∪{(ot,at∗,fet,set)}\mathcal{W}\leftarrow\mathcal{W}\cup\{(o_{t},a_{t}^{*},f_{e}^{t},s_{e}^{t})\}
12:   if |𝒲|=K|\mathcal{W}|=K or hit key milestone then
13:    // Retrospective Reflection: Re-evaluate with Hindsight
14:    for each (oj,aj,fj,sj)∈𝒲∪𝒟retro(o_{j},a_{j},f^{j},s^{j})\in\mathcal{W}\cup\mathcal{D}_{\text{retro}} do
15:     Construct xretrojx_{\text{retro}}^{j} from (τ,aj,𝒲,ot+1,fj,sj)(\tau,a_{j},\mathcal{W},o_{t+1},f^{j},s^{j})
16:     Generate frj,srj←Vϕe​(xretroj)f_{r}^{j},s_{r}^{j}\leftarrow V_{\phi_{e}}(x_{\text{retro}}^{j}) and set 𝒟retro​[aj]←(aj,frj,srj)\mathcal{D}_{\text{retro}}[a_{j}]\leftarrow(a_{j},f_{r}^{j},s_{r}^{j})
17:    end for
18:    𝒟train←𝒟retro\mathcal{D}_{\text{train}}\leftarrow\mathcal{D}_{\text{retro}}
19:    // Regularization: Prevent Catastrophic Forgetting
20:    for sampled unexplored action ala_{l} do
21:     Construct xinternallx_{\text{internal}}^{l}; get original output fil,sil←Vϕi​(xinternall)f_{i}^{l},s_{i}^{l}\leftarrow V_{\phi_{i}}(x_{\text{internal}}^{l}); add (al,fil,sil)(a_{l},f_{i}^{l},s_{i}^{l}) to 𝒟train\mathcal{D}_{\text{train}}
22:    end for
23:    // Test-Time Training
24:    for (a,f,sr)∈𝒟train(a,f,s_{r})\in\mathcal{D}_{\text{train}} do
25:     Construct xinternal,xactionx_{\text{internal}},x_{\text{action}} from aa; reward r=2​sr/100−1r=2s_{r}/100-1
26:     Update internal LLM: ℓϕ=−log⁡pϕi​(f|xinternal)\ell_{\phi}=-\log p_{\phi_{i}}(f|x_{\text{internal}}), ϕi←ϕi−ηϕ​∇ϕiℓϕ\phi_{i}\leftarrow\phi_{i}-\eta_{\phi}\nabla_{\phi_{i}}\ell_{\phi}
27:     Update action LLM: ℓθ=−r​log⁡pθ​(a|xaction)\ell_{\theta}=-r\log p_{\theta}(a|x_{\text{action}}), θ←θ−ηθ​∇θℓθ\theta\leftarrow\theta-\eta_{\theta}\nabla_{\theta}\ell_{\theta}
28:    end for
29:    Clear working memory: 𝒲←∅\mathcal{W}\leftarrow\emptyset
30:   end if
31:  end for
Equations 1–4
at=arg⁡maxa∈𝒜​pθ​(a|ot)a_{t}=\arg\max_{a\in\mathcal{A}}p_{\theta}(a|o_{t}) (1)
atk∼pθ(⋅|xaction;T)k=1,…,Na_{t}^{k}\sim p_{\theta}(\cdot|x_{\text{action}};T)\ \ \ k=1,...,N (2)
fit,k,sit,k=Vϕi​(xinternalk)f_{i}^{t,k},s_{i}^{t,k}=V_{\phi_{i}}(x_{\text{internal}}^{k}) (3)
at∗=at(arg⁡maxk∈[N]⁡sit,k)a_{t}^{*}=a_{t}^{(\arg\max_{k\in[N]}s_{i}^{t,k})} (4)

3.2 Reflection-on-Action

Execution can falsify a persuasive internal prediction. Post-action observations and success/failure feedback therefore support retrospective supervision for both the evaluator and policy. Only those two models update; the external teacher remains fixed.

3.2.1 Multi-Scale External Reflection

After each action, Vₑ describes what happened and assigns an immediate score. Accumulate recent transitions in a working buffer. At a milestone, such as leaving a room, or at K steps, re-evaluate earlier actions using the current window, latest observation and previous reflection. Store the newest hindsight assessment for each historical action; older actions may be subsampled as memory grows.

Equations 5–6
fet,set=Vϕe​(xexternal)f_{e}^{t},s_{e}^{t}=V_{\phi_{e}}(x_{\text{external}}) (5)
frj,srj=Vϕe​(xretroj)f_{r}^{j},s_{r}^{j}=V_{\phi_{e}}(x_{\text{retro}}^{j}) (6)

3.2.2 Test-Time Training Dataset Construction.

Training mixes hindsight-supervised action/reflection/score triples with randomly sampled unexplored actions labeled by the internal model’s current predictions. The second group anchors behavior outside the recent trajectory; the household implementation uses a 50:50 mixture. These are self-distillation targets, not newly observed outcomes for unexplored actions.

Equations 7–8
𝒟retro=(aj,frj,srj)\mathcal{D}_{\text{retro}}={(a_{j},f_{r}^{j},s_{r}^{j})} (7)
𝒟reg=(al,fil,sil)\mathcal{D}_{\text{reg}}={(a_{l},f_{i}^{l},s_{i}^{l})} (8)

3.2.3 Test-Time Training

Train Vᵢ by negative log-likelihood of retrospective reflection text. Train π with a REINFORCE-style token-sequence loss, converting score s to reward 2s/100−1. Positive rewards increase the sampled action’s likelihood; negative rewards decrease it. Algorithm 1 includes both retrospective and regularization examples in the updates, then clears the short working buffer while retaining retrospective memory.

The displayed evaluator loss predicts f rather than an explicitly separate scalar-score loss. Scoring and repeated updates depend on the teacher’s calibration; the paper gives no general convergence, unbiased-policy-gradient or safe-adaptation guarantee.

Equations 9–13
ℒinternal​(ϕi)=𝔼⁡(xinternal,f,s)∼𝒟train​[−log⁡pϕi​(f|x)]\mathcal{L}_{\text{internal}}(\phi_{i})=\mathbb{E}{(x_{\text{internal}},f,s)\sim\mathcal{D}_{\text{train}}}[-\log p_{\phi_{i}}(f|x)] (9)
ϕi(e+1)=ϕi(e)−ηϕ​∇ϕiℒinternal\phi_{i}^{(e+1)}=\phi_{i}^{(e)}-\eta_{\phi}\nabla_{\phi_{i}}\mathcal{L}_{\text{internal}} (10)
log⁡pθ​(a|xaction)=∑i=1|a|log⁡pθ​(ai|a<i,xaction)\log p_{\theta}(a|x_{\text{action}})=\sum_{i=1}^{|a|}\log p_{\theta}(a_{i}|a_{<i},x_{\text{action}}) (11)
ℓθ=−r⋅logpθ(a|xaction)\ell_{\theta}=-r\cdot\log p_{\theta}(a|x_{\text{action}}) (12)
θ(s+1)=θ(s)−ηθ​∇θ​∑(xaction,f,sr)∈𝒟trainℓθ​(xaction,f,sr)\theta^{(s+1)}=\theta^{(s)}-\eta_{\theta}\nabla_{\theta}\sum_{(x_{\text{action}},f,s_{r})\in\mathcal{D}_{\text{train}}}\ell_{\theta}(x_{\text{action}},f,s_{r}) (13)

4 Experiments on Long-Horizon Household Tasks

The household benchmark deliberately stresses mistakes whose consequences span rooms and actions. Its reported averages are unweighted means of four category percentages, not an explicitly reported overall task-weighted success rate.

4.1 Long-Horizon Household Task Construction

Build tasks on BEHAVIOR-1K scenes: Fitting tests geometry/capacity, Selection tests preferences and exploration, Preparation tests ordering constraints, and Hybrid combines them. GPT-5 receives complete scene graphs and proposes tasks, placements, trajectories and reflection scores across 3–7 rooms. Execute generated trajectories in OmniGibson and reject contradictory outcomes.

A shared LLaVA-3D model learns action and reflection formats before its three deployment copies are separated. Evaluation uses disjoint descriptions, scene configurations, placements and trajectories. Appendix P defines a 30-action budget and requires spatial, preference and ordering constraints as well as final object placement.

Table 1: Baseline comparisons and ablations of Reflective Test-Time Planning on long-horizon household tasks. RIA denotes reflection-in-action and ROA denotes reflection-on-action. Act. denotes action model loss; Int. denotes internal reflection model loss.
Complete original PDF S4.T1
Original selectable source
Baselines Reflective Test-Time Planning Ablations
Reflexion Self-Refine ReflectVLM PPO DreamerV3 3DLLM-Mem w/o RIA/ROA w/o RIA w/o ROA w/o Act. w/o Int. Ours
Fitting 8.51% 10.6% 2.12% 0% 4.26% 10.6% 0% 33.5% 6.38% 25.5% 12.8% 44.7%
Selection 8.82% 11.8% 5.88% 2.94% 11.8% 14.7% 17.6% 5.88% 11.8% 26.5% 8.82% 32.4%
Preparation 15.9% 12.7% 14.3% 7.94% 11.1% 9.52% 11.1% 3.17% 19.0% 20.6% 17.5% 31.7%
Hybrid 6.45% 9.68% 6.45% 3.23% 12.9% 9.68% 12.9% 3.23% 12.9% 16.1% 9.68% 25.8%
Average 9.92% 11.20% 7.19% 3.53% 10.02% 11.13% 10.40% 11.45% 12.52% 22.18% 12.20% 33.65%
#Data 1 Epoch 2 Epoch 3 Epoch w/o RIA/ROA
25% 15.6 20.3 22.1 1.3
50% 18.7 24.8 19.6 4.4
100% (Ours) 33.6 20.7 16.1 10.4
200% 24.5 19.6 9.1 13.4
300% 20.7 17.5 7.5 15.9
Table 2: SFT scaling analysis. Too little data fails to produce correct formats; too much overfits to SFT distribution, destabilizing TTT.
Method Fit Sel. Prep. Hyb. Avg. Overhead
TTT on Ext. Refl. 22.4 17.6 12.7 9.69 15.6 ∼\sim4×\times
TTT on Retro (Ours) 44.7 32.4 31.7 25.8 33.7 1×\times
Table 3: Retrospective vs. immediate external reflection as TTT signal. Retro-reflection achieves better credit assignment at lower overhead, as external reflection triggers at every step.

4.2 Experimental Results & Analysis

Table 1 reports 33.65% mean success versus 11.20% for Self-Refine, 11.13% for 3DLLM-Mem and 10.40% without both reflection modes. Full category scores are 44.7/32.4/31.7/25.8% for fitting/selection/preparation/hybrid. Removing candidate reflection gives 11.45%; removing online training gives 12.52%; removing action-model or internal-model loss gives 22.18% or 12.20%. Both components help overall, but interactions differ by category.

SFT scaling peaks at the reported 100%-data/one-epoch setting, 33.6%; more epochs or data can reduce adaptation quality. Hindsight-based training gives 33.7% versus 15.6% for immediate-feedback training at roughly one quarter of its overhead. These ablations support the complete recipe; they do not establish that initialization supplies only formatting or that all reasoning is acquired during testing.

Transfer to 41 HM3D preparation tasks reaches 19.5% versus 7.32% for the strongest listed baseline. This is no additional domain-specific offline fine-tuning, but deployment-time updates still occur. No uncertainty intervals or repeated-seed protocol accompany these main comparisons.

Results Analysis.

Household success averages 33.65% across the four categories, with 44.7% fitting success. Removing either reflection component reduces the overall score, but some category interactions are non-monotonic. Thus the ablations support the combined recipe without proving that both mechanisms are indispensable in every task. HM3D transfer reaches 19.5% on 41 preparation tasks versus 7.32% for the strongest listed baseline. Training uses synthetic BEHAVIOR scenes, while deployment still includes online updates; no repeated-seed uncertainty accompanies these comparisons.

Necessity and Reliability of SFT Initialization and External LLM.

SFT initialization helps the system produce usable outputs, and the tested data/epoch settings show a peak followed by degradation. Initialization-only success is 10.4%, versus about 33.7% for the complete reflective system. These comparisons do not isolate initialization as format-only teaching or establish that every reasoning ability is acquired at test time. The frozen external teacher receives observed outcomes and simulator feedback. Human ratings support useful reflections but omit sample sizes, agreement and uncertainty. Immediate-feedback training reaches 15.6% at approximately four times the hindsight-training overhead, indicating that signal placement and credit assignment matter in this setup.

Complete original PDF S4.F3
Figure 3: Cupboard Fitting results. Blue bars show correct-placement rate, pink bars show fit rate. RIA means Reflection-in-action; ROA means Reflection-on-action. “W/o external reflection”: we don’t use external reflection as the input to the action generation LLM. We implement two test-time training variants for ROA: 1) test-time training on all base weights; 2) test-time training on LoRA parameters only. Reflective Test-Time Planning significantly improves both success metrics.

5 Experiments on the Cupboard Fitting Task

Cupboard Fitting uses MuJoCo, a Franka arm, colored objects and compartments with geometric constraints. Fit rate counts objects in any compartment; correct rate counts designated compartments. Full LoRA adaptation reaches 60.2% fit and 25.3% correct, versus 44.5/17.6% without either mode. Base-weight adaptation gives 57.4/26.0%.

The full model has the best fit rate, but not the best correct rate: the LoRA variant without action-model loss reaches 58.8% fit and 30.8% correct. Thus “improves both metrics” is true against basic baselines, not dominance over every ablation. Figure 3’s complete chart, absent from the HTML graphics, is restored from the PDF.

On the real Franka, direct sim-to-real transfer gives 40.6% fit/13.4% correct versus 13.1/4.5% without reflection. With real-domain fine-tuning, results are 44.2/16.6% versus 20.7/7.2%. “Zero-shot” permits online adaptation; real-task counts, fine-tuning data volume and statistical uncertainty are not specified. Figure 5 illustrates local corrections and later reassignment of blame.

Complete original PDF S5.F4
Figure 4: The Cupboard Fitting Task.
Complete original PDF S5.F5
Figure 5: Qualitative Examples. Steps and reflections simplified for better presentations. Blue text: internal reflection. Orange text: external reflection. red text: retrospective reflection. (a) Long-Horizon Household example. We use retro & internal because the generated retro reflection is also used to train the internal model. (b) Real-robot Cupboard Fitting example. We put reflection scores inside brackets, omit detailed reflections and only present the scores for simplicity.
Setting Fit % Correct %
Zero-shot, w/o RIA & ROA 13.1 4.5
Fine-tuned, w/o RIA & ROA 20.7 7.2
Zero-shot, full 40.6 13.4
Fine-tuned, full 44.2 16.6
Table 4: Real-robot Cupboard Fitting on a Franka Panda arm. Zero-shot denotes direct sim-to-real transfer.
Method Preparation (%)
Reflexion 2.44
Self-Refine 4.88
ReflectVLM 0.00
PPO 0.00
DreamerV3 2.44
3DLLM-Mem 7.32
Ours 19.5
Table 5: Zero-shot generalization to HM3D.

Real-Robot Experiments.

On the real Franka, the full model reaches 40.6% fit and 13.4% correct placement without target-domain offline fine-tuning, versus 13.1% and 4.5% without reflection. After real-domain fine-tuning, the respective results are 44.2/16.6% and 20.7/7.2%. These object-level metrics support better performance in the tested setup, while missing trial counts and uncertainty limit stronger comparisons. Zero-shot transfer still permits online adaptation. Execution uses registered positions, structured grasp/place commands and bounds feedback rather than an end-to-end learned perception/control pipeline.

6 Conclusion

The evidence favors combining candidate selection, hindsight and online parameter updates on these constructed benchmarks. Remaining limits include imperfect external scoring, low absolute exact-placement rates, dependence on initialized formats and execution feedback, computational overhead and incomplete evaluation documentation. Tactile sensing is a proposed extension, not an evaluated capability.

Acknowledgments

Support includes ONR, Korea’s IITP/MSIT, Toyota Research Institute and Google’s academic compute program.

Appendix A Contribution Statement

Yining Hong is credited with the idea, implementation, data, experiments and writing; Huang Huang helped set up the physical robot; the remaining authors provided research guidance.

Appendix B Broader Impacts

The paper identifies possible benefits in failure recovery and risks from changing behavior during deployment, biased verbal assessments and reduced oversight. Interpretable reflections help inspection but do not themselves constrain adapted actions or prove safety.

Appendix C Test-Time Cost Analysis

The cost analyses distinguish extra reasoning and training time from environment interaction count. Their approximate ratios use different reference configurations, so they should not be combined without checking the denominator.

C.1 Computational Comparison

Average per-step wall time is reported as approximately three times the no-reflection baseline across the two benchmarks. Candidate generation/scoring and periodic updates create this cost. The paper does not provide absolute latency, hardware-normalized throughput or a complete end-to-end timing breakdown here.

Equation
Timefull≈3.0×Timeno-RIA/ROA.\text{Time}_{\text{full}}\approx 3.0\times\text{Time}_{\text{no-RIA/ROA}}.

C.2 Why the Overhead is Justified

The authors argue that fewer repeated mistakes can compensate for slower decisions and turn deployment time into learning. That is a task-dependent rationale; measured per-step overhead alone does not establish practical real-time control performance.

Deployment-Oriented Adaptation.

The additional computation occurs during deployment and can support adaptation to observed failures. This moves part of the learning cost from offline preparation to online operation. Whether that trade is practical depends on the task, hardware and allowable decision delay; the experiment does not establish general real-time control performance.

Reduction of Execution Waste.

Candidate reflection can reject poor actions before execution, while retrospective updates can reduce repeated failures. The reported comparisons show more successful task behavior despite slower decisions. They do not separately measure whether saved physical actions repay the full wall-clock, teacher-model and hardware cost in every deployment.

Conversion of Time into Learning.

Extra test-time computation updates the internal models instead of merely extending the action rollout. In Table 6, tripling the vanilla action budget gives 8.46% mean success versus 8.79% vanilla and 33.65% full. Its vanilla hybrid score differs from Table 1, so the two tables are not one identical baseline run. The result favors the tested reflective recipe under approximate budget matching; it does not prove that all alternative uses of test-time compute are inferior.

w/o RIA w/o ROA Ours (Full)
Vanilla Same Time Budget
Fitting 0.00% 0.00% 44.7%
Selection 17.6% 14.7% 32.4%
Preparation 11.1% 12.7% 31.7%
Hybrid 6.45% 6.45% 25.8%
Average 8.79% 8.46% 33.65%
Table 6: Performance comparison between (1) vanilla ablation without RIA or ROA, (2) vanilla baseline with matched time budget (3× steps), and (3) our full reflective model on Long-Horizon Household Tasks.

C.3 Compute-Matched Experiment

Tripling the vanilla action budget is an approximate time-matched comparison. Table 6 gives 8.79% vanilla, 8.46% expanded-budget and 33.65% full success. However, its vanilla hybrid score is 6.45%, whereas Table 1 lists 12.9%, yielding a different baseline average. Preserve both reported tables rather than treating them as one identical run.

Equation
Stepsbaselineexpanded≈3×Stepsbaseline.\text{Steps}_{\text{baseline}}^{\text{expanded}}\approx 3\times\text{Steps}_{\text{baseline}}.

Conclusion.

The paper reports roughly threefold per-step latency and better task performance than a vanilla policy with three times the action budget. This is an empirical comparison of the tested systems. The different nominal baselines and incomplete hardware-normalized timing prevent a general conclusion that the overhead is always practical or uniquely valuable.

Appendix D Efficiency Improvements via Parallelization

Cumulative optimizations add parallel scoring/KV caching, batched hindsight, early stopping, 4-bit QLoRA and multi-GPU execution. Table 7 reduces time from 1.00 to 0.45 relative to the full pipeline, while success falls 33.7→31.5%.

The concluding prose instead describes a change from 3× to 0.45×, mixing normalization conventions. If 0.45 is relative to the original full pipeline and that pipeline takes 3× vanilla time, the corresponding implied ratio is about 1.35× vanilla, not 0.45×. Hardware/resource changes also matter.

Configuration Success Overhead
Ours (full) 33.7% 1.00×\times
+Parallel Scoring + KV Cache 33.0% 0.85×\times
+Batched Retro-Reflection 33.5% 0.66×\times
+TTT Early-Stop 32.3% 0.60×\times
+QLoRA 32.0% 0.52×\times
+Multi-GPU Parallelism 31.5% 0.45×\times
Table 7: Cumulative efficiency improvements. Each row adds to the previous. The combined pipeline reduces overhead to 0.45×0.45\times with minimal performance degradation.

Appendix E Generalization to Habitat-Matterport 3D Scenes: More Details

HM3D tests transfer from synthetic household scenes to reconstructions of real interiors. It evaluates planning in those scenes, not physical deployment in all such homes.

E.1 Experimental Setup

Use HM3D preparation tasks with irregular layouts, unfamiliar appearances and longer navigation dependencies. Only this task category is evaluated in the reported domain-transfer table.

E.2 Task Construction and Adaptation

The 41 tasks require cross-room retrieval, ordered placement/assembly and scene-specific spatial reasoning. That small set supports the reported domain shift result; it does not cover all household categories.

E.3 Implementation Details

Keep LLaVA-3D-7B, point-cloud inputs, prompts and online-learning settings from the household experiment. No HM3D-specific offline fine-tuning is performed, but the reflective system adapts during each evaluation episode.

Appendix F Reliability of the External Reflection Model

A frozen teacher avoids simultaneous drift of all three models, but stability is not correctness. The paper checks external reflections through human judgments and replacement by human-written feedback.

F.1 Human Evaluation of V ϕ e Quality

Table 8 gives factual/causal/usefulness ratings of 98/95/99% in BEHAVIOR and 96/96/95% in HM3D. Annotator counts, sampled-output counts, agreement statistics and detailed rating procedures are not reported, limiting precision and reproducibility.

Setting Factual Causal Useful
In-domain (BEHAVIOR) 98 95 99
Out-of-domain (HM3D) 96 96 95
Table 8: Human evaluation of VϕeV_{\phi_{e}} reflection quality (%). Quality remains high under distribution shift.

F.2 Human Oracle Comparison

Replacing external reflections with human feedback raises HM3D success 19.5→21.9%, leaves Cleanup at 40.0%, and raises Rearrange 26.7→33.3%. These results are encouraging but do not establish statistical equivalence to human supervision without denominators and uncertainty estimates.

Method HM3D Cleanup Rearrange
Ours w/ VϕeV_{\phi_{e}} 19.5% 40.0% 26.7%
Ours w/ Human Oracle 21.9% 40.0% 33.3%
Table 9: VϕeV_{\phi_{e}} vs. human oracle under distribution shift. Near-identical performance confirms VϕeV_{\phi_{e}} provides supervision of equivalent quality to human judgment.

Appendix G SFT Initialization Analysis

Initialization quality changes the ability to parse actions and adapt. The benchmark-specific backbone and data settings must be distinguished from model-scale comparisons.

G.1 Zero-SFT Baselines

Without SFT, household performance is 0% with or without reflection; few-shot examples yield 1.3% and 2.1%. Initialized models reach 10.4% without reflection and 33.7% with it. This establishes a prerequisite in the tested setup, not a separation of learned formatting from learned task knowledge.

Method Accuracy (%)
Zero-SFT, w/o RIA & ROA 0.0
Zero-SFT, w/ RIA & ROA 0.0
Zero-SFT, Few-Shot In-Context, w/o RIA & ROA 1.3
Zero-SFT, Few-Shot In-Context, w/ RIA & ROA 2.1
Ours, w/o RIA & ROA 10.4
Ours (Full) 33.7
Table 10: Zero-SFT baselines confirm that test-time reflection cannot function without minimal SFT initialization.

G.2 Base Model Scale Analysis

Table 11’s cupboard model comparison gives post-SFT fit rates 48.5/60.2/52.7% for 2B/3B/7B models. Before SFT, format-parse fractions are 0.19/0.27/0.31, improving to 0.95/0.99/0.99; describing the initial fractions as near-zero is imprecise. The 2B and larger models also differ in family/version. Neither these observations nor semantic-similarity scores isolate overparameterization as the cause of the 7B result.

Model Sem. Sim. Human Eval Format Parse Fit %
Qwen2-VL 2B (no SFT) 0.59 76 0.19 4.7
Qwen2.5-VL 3B (no SFT) 0.73 89 0.27 8.0
Qwen2.5-VL 7B (no SFT) 0.75 91 0.31 11.9
Qwen2-VL 2B + SFT — — 0.95 48.5
Qwen2.5-VL 3B + SFT — — 0.99 60.2
Qwen2.5-VL 7B + SFT — — 0.99 52.7
Table 11: Base model scale analysis. Without SFT, 3B/7B models have sufficient semantic quality but near-zero format parse rates; SFT fixes format. Qwen2.5-VL 3B hits the optimal fit rate; 7B degrades due to over-parameterization.

Appendix H Retro-Reflection Quality Analysis

Hindsight score-revision direction agrees with the reported ground truth 99% in BEHAVIOR and 95% in HM3D. Table 12 repeats the high factual/causal/usefulness ratings, but does not report sample size or whether these are independent samples from Table 8.

Metric BEHAVIOR HM3D (OOD)
Revision Direction (%) 99 95
Human Eval — Factual 98 96
Human Eval — Causal 95 96
Human Eval — Useful 99 95
Table 12: Retro-reflection quality. Score revision directions align with ground-truth outcomes at ≥\geq95% even out-of-domain, and human evaluators rate reflections as highly factual, causally grounded, and useful.

Appendix I Search-Based Baseline Comparison

A language-guided MCTS baseline uses the action model for expansion and internal evaluator for values, with depth capped at three. It reaches 15.4% mean success at a reported 5–8 times full-method cost, versus 33.7% for the complete method. This comparison favors this implementation of adaptation over this bounded search; it does not rule out all search/update combinations.

Method Fit Sel. Prep. Hybrid Avg. Overhead
LG-MCTS 14.9 14.7 15.8 16.1 15.4 ∼\sim5–8×\times
Ours w/o TTT 6.4 11.8 19.0 12.9 12.5 0.5×\times
Ours 44.7 32.4 31.7 25.8 33.7 1×\times
Table 13: LG-MCTS is inferior to our full method at 5–8×\times cost. Search-based lookahead is ungrounded in physical uncertainties that can only be resolved through interaction; TTT, not search depth, is the key.

Appendix J Generalization to Human-Written Instructions

Human-written instructions give 32.3% mean success versus 33.7% for generated instructions. The text mentions ambiguous instructions and held-out task types, but the table aggregates four categories without separate ambiguity scores or evaluation counts.

Instruction Type Fit Sel. Prep. Hybrid Avg.
GPT-Generated (current) 44.7 32.4 31.7 25.8 33.7
Human-Written 42.5 33.6 25.7 27.4 32.3
Table 14: Performance generalizes to human-written instructions with marginal degradation.

Appendix K Commercial LLM Baselines

Under partial observations, GPT-5, Gemini 2.5 Pro and Qwen2.5 score 20.7/19.7/14.2% versus 33.7% for the trained reflective pipeline. GPT-5’s data-generation role instead sees complete scene graphs. This is not a matched-training or matched-test-compute model-capability comparison.

This appendix and Table 15 call the household system a 3B model, conflicting with the LLaVA-3D-7B architecture specified in Appendices E/P. The explicitly documented Qwen2.5-VL-3B backbone belongs to Cupboard Fitting. The discrepancy is unresolved.

Method Fit Sel. Prep. Hybrid Avg.
GPT-5 19.2 23.5 17.4 22.6 20.7
Gemini 2.5 Pro 14.9 20.5 20.6 22.6 19.7
Qwen2.5 12.8 14.6 14.3 15.1 14.2
Ours 44.7 32.4 31.7 25.8 33.7
Table 15: Commercial LLM zero-shot baselines. Our framework outperforms all commercial models despite using a 3B base model, demonstrating that test-time learning, not model scale, drives the gains. GPT-5 as a zero-shot baseline receives only partial observations; its data-generation role uses complete scene graphs.

Appendix L GPT-5 Data Quality Validation

Raw generated data scores 85% agreement with simulation and 80% with human judgment. Rejecting simulator-inconsistent cases yields 100% simulation consistency by construction and 98% human agreement. Table 17 gives naturalness/reflection/calibration/diversity ratings of 95/99/97/92. Sampling sizes and annotator details are absent, and filtering does not independently prove every language explanation correct.

Method Consistent w/ Sim. Consistent w/ Human
GPT Raw Scores 85% 80%
GPT + Sim. Validation 100% 98%
Table 16: GPT-5 data quality after physics-simulation validation. Simulator filtering raises human consistency from 80% to 98%.
Dimension Score
Task Naturalness 95
Reflection Quality 99
Score Calibration 97
Diversity 92
Table 17: Human evaluation of GPT-5-generated data quality.

Appendix M Real-Robot Experiment Details

The physical Franka uses a top-down offset RGB camera and registered object/compartment positions. High-level commands invoke grasp and place routines; binary feedback checks whether the object stays inside target bounds. This evaluates reflective high-level planning over a structured manipulation interface, not end-to-end learned localization and motor control.

Appendix N Hyperparameter Analyses for Cupboard Fitting Task

Figure 6 varies candidate count, temperature, LoRA capacity and action budget for cupboard fitting. It reports selected-point performance without error bars; causal explanations of small changes are hypotheses rather than isolated mechanisms.

Complete original PDF A14.F6
Figure 6: Hyperparameter ablation studies on Cupboard Fitting. Top Left: Performance vs. number of candidate actions. Peak performance (60.0%) occurs at N=6 candidates, demonstrating that internal reflection effectively identifies superior actions from diverse pools. Beyond N=6, performance plateaus as excessive candidates add computational cost without improving the best candidate quality. Top Right: Performance vs. sampling temperature. Optimal temperature range (T=1.25-1.5) balances candidate diversity with quality—temperatures below 0.5 produce overly similar candidates that limit reflection value, while temperatures above 1.75 generate incoherent actions that even accurate reflection cannot salvage. Bottom Left: Performance vs. LoRA configuration (rank, alpha). The optimal configuration (r=8, α\alpha=16) achieves 60.0% performance, balancing adaptation capacity with training stability. Smaller configurations like (4,4) underfit with insufficient capacity (52.5%), while larger configurations cause mode collapse during test-time training—(16,32) drops to 41.5% and (32,32) collapses to 34.8% as the model begins predicting identical outputs for all inputs, losing the ability to distinguish between different spatial configurations and task contexts. Bottom Right: Performance vs. action budget (maximum steps). Performance improves dramatically from 30 steps (51.5%) to 50 steps (60.0%), but slightly degrades to 59.4% at 100 steps, suggesting that excessive action budgets allow suboptimal exploration strategies that accumulate errors over longer horizons.

N.1 Test-Time Scaling: Number of Candidate Actions

Fit performance rises from 53.0% at one candidate to 60.0% at six, then reaches 58.8% at ten. The curve is not strictly monotonic: N=4 slightly exceeds N=5. Six is the best tested point, not a universal optimal search width.

N.2 Sampling Temperature Analysis

Temperature 1.25 gives 60.0%, versus 52.6% for greedy decoding, 59.8% at 1.5, 49.5% at 1.75 and 47.0% at 2.0. Moderate diversity helps this scorer; high-temperature degradation is consistent with poorer candidate quality, but candidate-quality measurements are not supplied.

N.3 LoRA Configuration Analysis

The tested optimum is rank 8/alpha 16, 60.0% fit. Rank 4/alpha 4 gives 52.5%; rank 16/alpha 32 gives 41.5%; rank 32/alpha 32 gives 34.8%. The authors attribute large-adapter failure to repetitive outputs under aggressive updates on about 10–15 hindsight examples. Both rank and alpha vary, and the general method includes regularization despite this paragraph saying test-time training lacks it.

N.4 Action Budget Analysis

Increasing the maximum budget from 30 to 50 actions raises fit rate 51.5→60.0%; 100 actions gives 59.4%. No uncertainty is provided for the 0.6-point decrease, so accumulated-error explanations should not be treated as demonstrated causal effects.

Appendix O Single-Step Action Generation vs. Receding Horizon Planning

Compare single-action selection with generating 5–10 actions, executing only the first, updating, and replanning. Table 18 reports single-step fit/correct of 60.2/25.3% versus 57.8/25.8% for receding horizon, at relative compute 1× versus 5×. The latter slightly improves correct placement despite lower fit.

The prose alternates between 60.0 and 60.2% and between four-times-more and 5× compute; the table is retained as printed. This limited comparison does not establish a fundamental incompatibility between online training and receding-horizon planning. Hindsight can teach long-term consequences without explicitly regenerating a full plan each step.

Table 18: Single-step action generation vs. receding horizon planning on Cupboard Fitting. Despite using 5× more computation to plan sequences at each step, receding horizon shows degraded performance, demonstrating that single-step action generation is more effective for our reflective learning framework.
Method Fit Correct Compute
Ours (Receding Horizon) 57.8% 25.8% 5.0×
Ours (Single-Step) 60.2% 25.3% 1.0×

Appendix P Experiments on Long-Horizon Household Tasks: More Details

The household appendix details generation, physics filtering, observations, training and evaluation. Several reporting gaps remain: absolute SFT dataset size, full evaluation counts, human-rating sample sizes and hardware timings are not supplied in these descriptions.

P.1 Task Generation & Validation Overview

Initialize scenes, place generated objects, execute trajectories and capture observations after each interaction. Compare execution outcomes with the generated task specification before retaining examples; this supplies supervised action/reflection tuples.

P.2 Why Build Upon BEHAVIOR?

BEHAVIOR provides scene assets, object affordances, simulation and interaction interfaces; the authors add tasks designed around failure recovery and delayed consequences. Its name denotes 1,000 activities, while this appendix’s claim of “1,000+ scenes” is not established by the paper’s experiment inventory.

P.3 Task Categories

Fitting introduces size, occlusion and blocked-capacity failures; Selection rewards exploring for better alternatives; Preparation enforces prerequisite order; Hybrid combines these dependencies. Figure 7 gives category proportions of 26.9/19.4/36.0/17.7%, which should not be confused with the equal-category averaging in Table 1.

Complete original PDF A16.F7
Figure 7: Distribution of task categories in the dataset.

P.4 GPT-5 Task Generation Prompting Strategy

Prompts include room connections, object boxes/affordances, explicit failure modes and 2–3 examples per category. Required outputs interleave angle-bracket actions, internal critiques, execution status, external critiques and hindsight updates. Low-score unexecuted actions train the internal evaluator, not the action policy.

Calibration bands are 0–20 physical failure/dead end, 21–40 strategically harmful, 41–60 acceptable/suboptimal, 61–80 good and 81–100 near-optimal. Prompt refinement adds geometry and diverse failure examples. The paper describes this recipe but does not print a complete reusable generation prompt.

P.5 Physical Validation in BEHAVIOR.

Reject impossible placements and any trajectory whose annotated success/failure disagrees with simulation. Use uniform object scaling. Three external 560×560 RGB-D views are transformed into robot-base-frame point clouds after interactions; retain the most recent room observation. Validation is relative to this simulator and labeling procedure.

P.6 Training Data Construction

Four exact input/output schemas are retained below. Action inputs contain task/rooms/observations and previous feedback; internal scoring adds a proposed action; external scoring adds executed action, outcome and before/after observations; hindsight adds the historical action, room history and last reflection. The printed schemas have minor brace inconsistencies, preserved without silent correction.

Equations
Input: {Task,All Rooms,Explored Rooms,\displaystyle\{\text{Task},\text{All Rooms},\text{Explored Rooms},
Current Room,Observations,\displaystyle\text{Current Room},\text{Observations},
Previous Action,Previous External Reflection}\displaystyle\text{Previous Action},\text{Previous External Reflection}\}
Output: Actiont\displaystyle\text{Action}_{t}
Input: {Task,All Rooms,Explored Rooms,\displaystyle\{\text{Task},\text{All Rooms},\text{Explored Rooms},
Current Room,Observations,\displaystyle\text{Current Room},\text{Observations},
Previous Action,Previous External Reflection}\displaystyle\text{Previous Action},\text{Previous External Reflection}\}
Potential Actiont}\displaystyle\text{Potential Action}_{t}\}
Output: {Internal Reflectiont,Scoret}\displaystyle\{\text{Internal Reflection}_{t},\text{Score}_{t}\}
Input: {Task,All Rooms,Explored Rooms,\displaystyle\{\text{Task},\text{All Rooms},\text{Explored Rooms},
Current Room,Observations (before and after),\displaystyle\text{Current Room},\text{Observations (before and after)},
Previous Action,Previous External Reflection}\displaystyle\text{Previous Action},\text{Previous External Reflection}\}
Executed Actiont,Execution Resultt}\displaystyle\text{Executed Action}_{t},\text{Execution Result}_{t}\}
Output: {External Reflectiont,Scoret}\displaystyle\{\text{External Reflection}_{t},\text{Score}_{t}\}
Input: {Task,Context,Current Observations,\displaystyle\{\text{Task},\text{Context},\text{Current Observations},
Actionj,Room Action History,\displaystyle\text{Action}_{j},\text{Room Action History},
Last Reflectionj}\displaystyle\text{Last Reflection}_{j}\}
Output: {Retro Reflectionj,Updated Scorej}\displaystyle\{\text{Retro Reflection}_{j},\text{Updated Score}_{j}\}

P.7 Model Architecture and Training

Household models use a 3D encoder, multimodal projector and Llama-based 7B backbone. Unified SFT uses AdamW, learning rate 2×10⁻⁵, per-GPU batch eight, four-step accumulation, three epochs and weight decay 0.01. Fused point-cloud memory follows 3DLLM-Mem. Three copies are then instantiated, with only the external evaluator frozen.

This three-epoch configuration is not reconciled with Table 2’s peak at 100% data/one epoch. Keep the reported configuration and sensitivity result distinct.

P.8 Test-Time Training Configuration

Household LoRA uses rank four, alpha eight, dropout 0.15 on query/value projections. Internal-reflection updates use rate 5×10⁻⁵, three epochs and gradient clipping 0.3; action updates use SGD at 10⁻³, three epochs and clipping 0.5. Mix equal hindsight and self-distillation examples. These settings differ from cupboard LoRA.

P.9 Evaluation Protocol and Baselines

Require all task constraints within 30 actions; trigger hindsight at room exits or K=5 steps. Main deployment uses model-generated reflections grounded in simulator outcomes; the human-oracle study is a separate intervention.

Baselines include contextual reflection, up to three Self-Refine rounds, learned value scoring, spatial memory, PPO and Dreamer-style world-model learning. Matching environment-interaction counts is not necessarily matching computation. The appendix labels a DreamerV3 citation with the RoboDreamer reference, a bibliographic mismatch retained in the original references.

Appendix Q Cupboard Fitting: More Details

Cupboard experiments isolate placement failure with images and structured high-level commands rather than household point-cloud navigation.

Q.1 Cupboard Setup

Each task has 6–8 compartments and 6–10 objects. Compartments have known bounding boxes; multiple objects can share space. The authors stipulate one valid complete designated arrangement. Reported rates are fractions of placed objects, not the fraction of episodes that realize this complete solution.

Q.2 Base Model and Supervised Fine-tuning.

Use Qwen2.5-VL-3B with unified action/internal/external SFT: global batch 128, AdamW, learning rate 10⁻⁵, weight decay 0.1 and three epochs. This is the explicit 3B configuration, distinct from the household 7B description.

Q.3 Test-Time Training Variants.

Base-weight adaptation freezes vision and updates nonvisual parameters with SGD for three epochs: 10⁻³ for actions, 5×10⁻⁵ for the internal evaluator, weight decay 10⁻⁴ and no momentum. LoRA targets linear layers outside vision, embeddings and output head, with rank eight, alpha 16 and dropout 0.1; learning rates are 10⁻² for actions and 0.2 for the evaluator. Reported trainable-parameter reduction exceeds 95%, but the paper gives no full memory/latency accounting.

Appendix R More Qualitative Examples

The full Figure 8 preserves the gift-box search and both cupboard sequences. Local feedback first revises failed placements; later hindsight downgrades a previously acceptable box or watermelon placement after better choices or blocked capacity become visible. These are selected, simplified traces, not quantitative evidence of how often each behavior occurs.

Complete original PDF A18.F8
Figure 8: Additional qualitative examples. Blue text: internal reflection for candidate selection. Orange text: external reflection after execution. Red text: retrospective reflection and model updates. Reflection scores shown in brackets. Steps and reflections simplified for presentation.
Concept demonstration

Open image ↗