Literature
Display
← All papers

Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning

Yoonwoo Kim · Raghav Arora · Roberto Martín-Martín · Peter Stone · Ben Abbatematteo · Yoonchang Sung

· ICRA 2026 ↗

Robotics10 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers arXiv v2 dated 6 March 2026, including the complete appendix and real-robot execution traces. CoCo-TAMP project. The six authors are affiliated with UT Austin and Nanyang Technological University; Ben Abbatematteo and Yoonchang Sung share advising credit.

At a glance

CoCo-TAMP uses language-model knowledge to initialize object-location beliefs and transfer evidence from observed objects to semantically similar or dissimilar objects. Hierarchical Bayes filters represent rooms and surfaces; a particle filter tracks continuous poses and distinguishes an occluded object from an absent one.

A conventional task-and-motion planner then selects manipulation and information-gathering actions. The paper reports 62.7% less cumulative planning/execution time in simulation and 72.6% in its real-robot example versus uniform priors without co-location. Known room/surface geometry, stationary objects and predefined actions are assumed.

I Introduction

Occlusion makes deterministic manipulation plans brittle: a failed observation can mean the object is elsewhere or merely hidden. CoCo-TAMP combines typical-placement priors with evidence that related objects often share a location, while disabling that relation for broadly distributed categories such as switches.

Following the LLM-Modulo view, the LLM supplies approximate semantic knowledge and model-based planning handles geometric feasibility. The paper demonstrates efficiency gains; it does not establish a new completeness guarantee for learned beliefs or optimal solution of the full POMDP.

II Problem Description

The semantic map of rooms and surfaces is known, for example from Hydra. Each task-relevant object has categorical room/surface beliefs and a continuous SE(3) pose belief. The specification supplies objects, predicates, initial/goal literals and parameterized actions.

SS-Replan supplies the belief-space replanning framework; PDDLStream connects symbolic actions with certified continuous samples. Besides pick, place and move, detect samples an object pose and a robot view. Its heuristic cost is one divided by the joint belief, favoring promising observations. This density-based proxy is not a calibrated detection probability or a general equivalence between minimum cost and maximum success.

Evaluation measures accumulated planning/execution time and replanning count after failures. Timing depends on implementation and hardware; replans provide a complementary measure of wasted observations.

Equation

k∈{1,…,K}k\in\{1,\dots,K\}

xr,tk∈ℛx_{r,t}^{k}\in\mathcal{R}

xs,tk∈𝒮x_{s,t}^{k}\in\mathcal{S}

xp,tk∈S​E​(3)x_{p,t}^{k}\in SE(3)

K<OK<O

(𝒪,𝒫,ℐ,𝒢,𝒜)(\mathcal{O},\mathcal{P},\mathcal{I},\mathcal{G},\mathcal{A})

cabel​(xr,tk,xs,tk,xp,0k)\frac{c_{a}}{\text{bel}(x_{r,t}^{k},x_{s,t}^{k},x_{p,0}^{k})}

ca=1c_{a}=1

III COCO-TAMP

First obtain semantic priors and uniform pose samples. Plan and execute; after an observation, estimate visibility, optionally transfer co-location evidence and update the hierarchical belief. Replan on execution failure until the goal is reached. Figure 1 retains the complete loop.

Refer to caption
Fig. 1: The initial beliefs about the semantic locations of objects, bel​(xr,0k)\text{bel}(x_{r,0}^{k}) and bel​(xs,0k)\text{bel}(x_{s,0}^{k}), are derived from LLMs, while the initial beliefs about their poses, bel​(xp,0k)\text{bel}(x_{p,0}^{k}), are uniformly distributed across all surfaces. The TAMP problem specification (𝒪,𝒫,ℐ,𝒢,𝒜)(\mathcal{O,P,I,G,A}), where the beliefs are incorporated into the cost of the observation action, is provided to a TAMP planner PDDLStream. The planner outputs a plan, which the robot executes. Upon executing the observation action, the co-location toggler determines whether to use the co-location model or not, based on the observed object. Then, the beliefs are updated using the proposed observation model, and the planner replans with the updated beliefs. Planning and execution are repeated until the goal state 𝒢\mathcal{G} is reached. Further implementation details, derivations, additional experimental results, and link to the code are available at our project page: https://coco-tamp.github.io.

III-A Initial belief generation with LLMs

Room and surface names become single-letter multiple-choice answers. The next-token log scores for those answers are normalized with softmax, giving the initial categorical beliefs. The toaster prompt is retained below; the surface query uses the same structure.

These probabilities express the model’s preference among the supplied options. Calibration, option-order robustness and coverage of atypical placements are not established by the normalization itself.

Predict the location of a toaster.
(A) Kitchen
(B) Bathroom
(C) Livingroom
(D) Garage
Return the letter that represents the location:

III-B Object state estimation

The state belief factors into pose conditioned on surface/room, surface conditioned on room, and room. Pose uses a particle filter; categorical levels use discrete Bayes updates. Visibility in [0,1] measures the fraction of initially uniform surface particles covered by the current view.

The factorization assumes conditional independence of observations and stationary objects. It provides a tractable approximation, not an unrestricted joint posterior over all object relationships.

Equation

bel​(xp,tk)=P⁡(xp,tk|xs,tk,xr,tk,zp,tk)\text{bel}(x_{p,t}^{k})=P(x_{p,t}^{k}|x_{s,t}^{k},x_{r,t}^{k},z_{p,t}^{k})

bel​(xs,tk)=P⁡(xs,tk|xr,tk,𝒵s,t,vs,t)\text{bel}(x_{s,t}^{k})=P(x_{s,t}^{k}|x_{r,t}^{k},\mathcal{Z}_{s,t},v_{s,t})

bel​(xr,tk)=P⁡(xr,tk|𝒵r,t,vr,t)\text{bel}(x_{r,t}^{k})=P(x_{r,t}^{k}|\mathcal{Z}_{r,t},v_{r,t})

vr,t,vs,t∈[0,1]v_{r,t},v_{s,t}\in[0,1]

III-B 1 Estimating object’s semantic location

With stationary objects, the previous posterior becomes the prediction. Multiply it by a visibility-aware target likelihood and independent cross-object likelihood factors, then normalize. Surface coverage is the seen-particle fraction; room coverage aggregates those counts across surfaces.

For a true location, detection likelihood is (1−pfn)v and a miss has likelihood 1−v+vpfn. At an incorrect location, detection has likelihood pfpv and a miss 1−vpfp. Both error rates are fixed to 0.01. Thus a small visible fraction provides weak negative evidence.

The cross-object factorization follows correlational object search. Shared occluders and scene structure can correlate observations beyond the assumed conditioning; the paper does not validate this independence or calibrate the error rates.

Equations 1–7
bel​(xr,tk)\displaystyle\text{bel}(x_{r,t}^{k}) =P⁡(xr,tk|𝒵r,t,vr,t),\displaystyle=P(x_{r,t}^{k}|\mathcal{Z}_{r,t},v_{r,t}), (1)
=η​P​(𝒵r,t|xr,tk,vr,t)​bel¯​(xr,tk),\displaystyle=\eta P(\mathcal{Z}_{r,t}|x_{r,t}^{k},v_{r,t})\overline{\text{bel}}(x_{r,t}^{k}), (2)
vs,t=ns,t(seen)ns,0,vr,t=∑s∈𝒮ns,t(seen)∑s∈𝒮ns,0.\displaystyle v_{s,t}=\frac{n_{s,t}^{\text{(seen)}}}{n_{s,0}},v_{r,t}=\frac{\sum_{s\in\mathcal{S}}n_{s,t}^{\text{(seen)}}}{\sum_{s\in\mathcal{S}}n_{s,0}}. (3)
P⁡(zti,ztj|xtj)=P⁡(zti|xtj)​P​(ztj|xtj).\displaystyle P(z_{t}^{i},z_{t}^{j}|x_{t}^{j})=P(z_{t}^{i}|x_{t}^{j})P(z_{t}^{j}|x_{t}^{j}). (4)
P⁡(𝒵r,t|xr,tk,vr,t)=P⁡(zr,tk|xr,tk,vr,t)\displaystyle P(\mathcal{Z}_{r,t}|x_{r,t}^{k},v_{r,t})=P(z_{r,t}^{k}|x_{r,t}^{k},v_{r,t}) (5)
×∏j=1,j≠kKP⁡(zr,tj|xr,tk).\displaystyle\times\prod_{j=1,j\neq k}^{K}P(z_{r,t}^{j}|x_{r,t}^{k}).
P⁡(zr,tk∣xr,tk,vr,t)={(1−pfn)​vr,t,Det.,(1−vr,t)+vr,t​pfn,No Det..\displaystyle P(z_{r,t}^{k}\mid x_{r,t}^{k},v_{r,t})=\begin{cases}(1-p_{\mathrm{fn}})\,v_{r,t},&\text{Det.},\\ (1-v_{r,t})+v_{r,t}\,p_{\mathrm{fn}},&\text{No Det.}.\end{cases} (6)
P⁡(zr,tk∣xr,tk,vr,t)={pfp​vr,t,Det.,1−vr,t​pfp,No Det.\displaystyle P(z_{r,t}^{k}\mid x_{r,t}^{k},v_{r,t})=\begin{cases}p_{\mathrm{fp}}\,v_{r,t},&\text{Det.},\\ 1-v_{r,t}p_{\mathrm{fp}},&\text{No Det}.\end{cases} (7)

zr,tk=xr,tkz_{r,t}^{k}=x_{r,t}^{k}

zr,tk≠xr,tkz_{r,t}^{k}\neq x_{r,t}^{k}

III-B 2 co-location model

Generate three sentences describing each object’s common uses, embed the concatenation and compute cosine similarity. For nonnegative similarity, interpolate between uniform room probability and the same-room delta; for negative similarity, interpolate toward the normalized different-room complement. The definitions and piecewise equation remain exact.

A detected banana can increase the apple belief nearby; a dissimilar object can decrease it. A separate GPT-4o Boolean query disables co-location for categories normally spread across a house. Semantic similarity is a heuristic relation, not a measured physical co-location frequency. The negative-similarity complement divides by R−1 and needs special handling when only one room is available.

Equations 8–9
P⁡(zr,tj|xr,tk)=∑xr,tj∈ℛP⁡(zr,tj|xr,tj)​P​(xr,tj|xr,tk).\displaystyle P(z_{r,t}^{j}|x_{r,t}^{k})=\sum_{x_{r,t}^{j}\in\mathcal{R}}P(z_{r,t}^{j}|x_{r,t}^{j})P(x_{r,t}^{j}|x_{r,t}^{k}). (8)
P⁡(xr,tj∣xr,tk)\displaystyle P(x_{r,t}^{j}\mid x_{r,t}^{k}) ={sim​(j,k)​δj​k+(1−sim​(j,k))​uj​k, sim​(j,k)≥0,abs​(sim​(j,k))​δ¯j​k+(1+sim​(j,k))​uj​k, sim​(j,k)≤0.\displaystyle=\begin{cases}\texttt{sim}(j,k)\,\delta_{jk}\\ +(1-\texttt{sim}(j,k))\,u_{jk},\texttt{ sim}(j,k)\!\geq\!0,\\ \texttt{abs}(\texttt{sim}(j,k))\bar{\delta}_{jk}\\ +(1+\texttt{sim}(j,k))\,u_{jk},\texttt{ sim}(j,k)\!\leq\!0.\end{cases} (9)

sim​(j,k)∈[−1,1]\texttt{sim}(j,k)\in[-1,1]

δj​k=δ⁡(xr,tj,xr,tk)\delta_{jk}=\delta(x_{r,t}^{j},x_{r,t}^{k})

δ¯j​k=1−δ⁡(xr,tj,xr,tk)R−1\bar{\delta}_{jk}=\frac{1-\delta(x_{r,t}^{j},x_{r,t}^{k})}{R-1}

uj​k=u⁡(xr,tj,xr,tk)u_{jk}=u(x_{r,t}^{j},x_{r,t}^{k})

III-B 3 Estimating object pose

A target detection weights particles by Gaussian distance to its observed pose. A miss retains occluded/out-of-view hypotheses and assigns zero weight to visible ones. If another object is detected, an exponential distance kernel interpolates between attraction and repulsion according to semantic similarity; λ controls the distance scale.

The exact Gaussian, missed-observation and similarity equations are preserved. The continuous miss rule hard-zeros visible particles despite the categorical model’s nonzero false-negative rate. Eq. (14) also includes the previous particle weight inside a quantity called a likelihood; how it combines with Eq. (12)’s prior should be distinguished from Algorithm 1’s direct weight assignment. No silently corrected formula is substituted.

Equations 10–15
bel​(xp,tk)\displaystyle\text{bel}(x_{p,t}^{k}) =P⁡(xp,tk|xs,tk,xr,tk​zp,tk),\displaystyle=P(x_{p,t}^{k}|x_{s,t}^{k},x_{r,t}^{k}z_{p,t}^{k}), (10)
=η​P​(zp,tk|xp,tk,xs,tk,xr,tk)​bel¯​(xp​(t)), or\displaystyle=\eta P(z_{p,t}^{k}|x_{p,t}^{k},x_{s,t}^{k},x_{r,t}^{k})\overline{\text{bel}}(x_{p}(t)),\text{ or} (11)
=η​P​(zp,tk¯|xp,tk,xs,tk,xr,tk)​bel¯​(xp​(t)).\displaystyle=\eta P(\overline{z_{p,t}^{k}}|x_{p,t}^{k},x_{s,t}^{k},x_{r,t}^{k})\overline{\text{bel}}(x_{p}(t)). (12)
P⁡(zp,tk|xp,t[i],k,xs,tk,xr,tk)=1η​exp⁡(−d22​σ2),\displaystyle P(z_{p,t}^{k}|x_{p,t}^{[i],k},x_{s,t}^{k},x_{r,t}^{k})=\frac{1}{\eta}\exp(-\frac{d^{2}}{2\sigma^{2}}), (13)
P⁡(zp,tk¯|xp,t[i],k,xs,tk,xr,tk)={0,Visible,1η​wt−1k,[i],Invisible,\displaystyle P(\overline{z_{p,t}^{k}}|x_{p,t}^{[i],k},x_{s,t}^{k},x_{r,t}^{k})=\begin{cases}0,&\text{Visible},\\ \frac{1}{\eta}w_{t-1}^{k,[i]},&\text{Invisible},\end{cases} (14)
wtk\displaystyle w_{t}^{k} =1+sim​(j,k)2e−d/λ+1−sim​(j,k)2(1−e−d/λ),\displaystyle=\frac{1+\texttt{sim}(j,k)}{2}e^{-d/\lambda}+\frac{1-\texttt{sim}(j,k)}{2}(1-e^{-d/\lambda}), (15)

IV Experiments

Experiments use Housekeep-derived geometric household simulations and Toyota’s HSR in a mock apartment. GPT-4o is selected after an MCQA comparison with smaller models and used for the remaining experiments. The embedding model, particle count, measurement σ and similarity-distance λ are not specified in this paper.

IV-A Large-scale simulations

Housekeep human placement annotations determine room–surface–object sampling, reducing manual placement choices favorable to the method. Simulations use simple geometric boxes because the planner consumes semantic labels rather than visual semantics; this does not test end-to-end perception robustness.

Refer to caption
Fig. 2: Example of a simulated household environment.

IV-A 1 Comparative analysis

Six variants separate initialization and updates: uniform/Bayesian baseline; uniform plus co-location; uniform plus direct LLM belief updates (LGBU); MCQA priors alone; MCQA plus co-location; and MCQA plus LGBU. LGBU queries current belief, inspected location, visibility, detection result and co-detected objects, then normalizes answer-token probabilities.

Table I preserves the exact configurations. Its first row repeats the surface-belief symbol, while the caption identifies both room and surface priors; this is retained as a source notation discrepancy.

Baseline Co-Model LGBU MCQA MCQA with Co-Model MCQA with LGBU
b​e​l​(xs,0k)bel(x_{s,0}^{k}), b​e​l​(xs,0k)bel(x_{s,0}^{k}) Uniform Uniform Uniform MCQA MCQA MCQA
co-location model No Yes No No Yes No
LLM generated update No No Yes No No Yes
TABLE I: Comparison method details.The terms b​e​l​(xs,0k)bel(x_{s,0}^{k}) and b​e​l​(xr,0k)bel(x_{r,0}^{k}) denote the initial belief over surfaces and rooms, respectively, for object kk.
4 rooms 8 surfs 4 rooms 16 surfs 6 rooms 12 surfs 6 rooms 24 surfs 8 rooms 16 surfs 8 rooms 32 surfs
Baseline 632.5±124.2632.5\pm 124.2 715.1±240.5715.1\pm 240.5 1205.3±2721205.3\pm 272 1994±602.71994\pm 602.7 1564.9±3621564.9\pm 362 3017.5±1066.13017.5\pm 1066.1
Co-Model 121.2±58.4121.2\pm 58.4 70.2±198.570.2\pm 198.5 185.1±145.6185.1\pm 145.6 355.3±398.7355.3\pm 398.7 398.8±272398.8\pm 272 00
MCQA 99.4±95.399.4\pm 95.3 169.9±306.4169.9\pm 306.4 248.2±262.2248.2\pm 262.2 1212.6±735.11212.6\pm 735.1 134±231134\pm 231 1925.3±1320.81925.3\pm 1320.8
MCQA with Co-Model 00 00 00 00 00 50.5±834.850.5\pm 834.8
LGBU 164.4±92.9164.4\pm 92.9 605.8±288.8605.8\pm 288.8 894.8±551.9894.8\pm 551.9 509.2±405509.2\pm 405 787.2±321.3787.2\pm 321.3 1066.9±1054.31066.9\pm 1054.3
MCQA with LGBU 0.2±48.80.2\pm 48.8 339.2±286.5339.2\pm 286.5 101.0±210.7101.0\pm 210.7 351.1±361.2351.1\pm 361.2 288.8±308.3288.8\pm 308.3 408.5±871.7408.5\pm 871.7
TABLE II: Pairwise comparison of cumulative planning and execution time (in seconds) relative to the best-performing method. Numbers represent the mean ±\pm 95% confidence interval calculated over 50 problems, and 0 indicates the best-performing method.
4 rooms 8 surfs 4 rooms 16 surfs 6 rooms 12 surfs 6 rooms 24 surfs 8 rooms 16 surfs 8 rooms 32 surfs
Baseline 9.5±3.09.5\pm 3.0 7.1±6.87.1\pm 6.8 17.2±5.417.2\pm 5.4 25.9±8.825.9\pm 8.8 19.8±6.419.8\pm 6.4 35.3±13.735.3\pm 13.7
Co-Model 3.0±1.93.0\pm 1.9 3.4±5.83.4\pm 5.8 5.5±3.15.5\pm 3.1 7.9±5.57.9\pm 5.5 10.6±6.110.6\pm 6.1 10.5±7.410.5\pm 7.4
MCQA 2.4±2.72.4\pm 2.7 00 2.9±4.92.9\pm 4.9 12.8±8.712.8\pm 8.7 00 13.9±14.113.9\pm 14.1
MCQA with Co-Model 0.6±1.40.6\pm 1.4 0.3±6.60.3\pm 6.6 00 00 1.4±4.11.4\pm 4.1 00
LGBU 4.1±2.94.1\pm 2.9 18.5±10.718.5\pm 10.7 23.9±14.623.9\pm 14.6 17.9±10.117.9\pm 10.1 21.3±8.321.3\pm 8.3 15.5±8.815.5\pm 8.8
MCQA with LGBU 11.2±9.411.2\pm 9.4 00 3.1±5.53.1\pm 5.5 10.2±7.810.2\pm 7.8 6.6±7.06.6\pm 7.0 6.1±8.26.1\pm 8.2
TABLE III: Pairwise comparison of number of replans relative to the best-performing method. Numbers represent the mean ±\pm 95% confidence interval calculated over 50 problems, and 0 indicates the best-performing method.

IV-A 2 Comparison results

There are six layouts: 4/6/8 rooms, each with either two or four surfaces per room. Fifty environments per layout vary semantic labels and placements, with obstacles producing occlusions. Tables II–III report paired excess time/replans relative to the best method in each layout, with 95% confidence intervals. Zero means best in that comparison, not zero runtime or replanning.

At six rooms/twelve surfaces, the baseline takes 1,205.3±272.0 extra seconds and 17.2±5.4 extra replans; MCQA plus co-location is the reference best. Co-location alone adds 185.1±145.6 seconds and 5.5±3.1 replans. MCQA plus co-location is best by mean time in five layouts; co-location alone wins the largest layout, and several replan comparisons favor MCQA or tie with LGBU variants. The broad claim of uniformly lowest means/variability exceeds the tables.

Direct LGBU often underperforms Bayesian variants, but it is faster than the uninformed baseline in every tabled layout, so the blanket statement that it is worse than baseline on both metrics is too strong. In five adversarial layouts with randomized placements, Bayesian methods finish, while the caption reports LGBU and MCQA+LGBU hitting the 100-replan cap in three of five trials. This small stress test supports a limited robustness claim; it does not establish universal recovery from misleading priors.

Refer to caption
Fig. 3: Average cumulative planning and execution time over 50 environments in household layout with 6 rooms and 12 surfaces with 95% confidence interval.
Refer to caption
Fig. 4: Pairwise comparison of number of replans required to complete the task compared to MCQA with co-location Model with 95% confidence interval.
Refer to caption
Fig. 5: Performance metrics over 50 environments in a household layout (6 rooms, 12 surfaces), comparing different LLMs for the MCQA setting. The plots show the number of replans with 95% confidence intervals. We see GPT-4o outperforms the other smaller models, and hence we have used GPT-4o in all other experiments in this paper.
Refer to caption
Fig. 6: Average number of replans over 5 environments in household layout with 6 rooms and 12 surfaces with 95% confidence interval.
Note: The LGBU and MCQA with LGBU experiments had a total limit of 100 replans, which was exceeded 3 out of 5 times this experiment was run.

IV-B Real-world experiments

HSR moves an apple from the kitchen table to the living-room coffee table in a two-room, three-surface setup. A banana supplies positive semantic evidence, a screwdriver dissimilar evidence, and cereal/cracker boxes hide the target.

Reported planning/execution times are 365 s for baseline, 112 s with co-location, and 100 s with MCQA plus co-location: the last is 72.6% below baseline. This is one described scenario without repeated-trial uncertainty or a broad real-world success-rate evaluation.

Refer to caption
Fig. 7: Environment for real-world experiments.

V Related Work

Belief-space TAMP handles uncertainty but remains computationally demanding; object-search systems exploit observation structure without generally supplying a complete manipulation task model. CoCo-TAMP improves semantic belief estimation inside an existing planner.

Other LLM/TAMP work infers constraints, proposes plans, reasons about failures or translates language into PDDL. Here the LLM supplies placement priors and object relations; geometric planning and observations constrain execution. That distinction does not make every object-search system a formal special case of this implementation.

VI Conclusion

The combination of semantic priors and observation-grounded Bayesian updates improves search efficiency in the reported household tasks. Transfer to factories/hospitals and unknown room/surface layouts is untested. Dynamic objects, calibrated semantic uncertainty and reliable handling of atypical arrangements remain important boundaries.

ACKNOWLEDGMENT

The work acknowledges UT Austin LARG support from NSF, ONR, ARO, DARPA, Lockheed Martin and Good Systems. Peter Stone’s compensated Sony AI chief-scientist role was reviewed under UT Austin’s research-objectivity policy.

APPENDIX

The complete appendix supplies all prompts, the detect action and two PDDL streams, the belief factorization and surface equations, the particle-filter algorithm, dataset construction and the real-robot traces. These executable/formal examples remain intact.

VI-A Co-location Model Prompt

The three-sentence toaster example illustrates the text embedded for semantic similarity. The exact prompt and GPT-4o response are retained; this is a qualitative example, not evidence that cosine similarity is a calibrated location probability.

Equation

Prompt: Explain the common use of a toaster in three sentences.
LLM Response (GPT4o): A toaster is commonly used to toast slices of bread, making them crispy and golden brown. It’s also useful for heating up and adding a slight crunch to bagels, English muffins, and some pastries. Additionally, many people use a toaster to quickly prepare breakfast items like Pop-Tarts or frozen waffles.

VI-B Co-Location Toggler Prompt

The toggler asks whether a category is commonly distributed throughout a household, as with switches or doorknobs. A True/False response determines whether to suppress co-location transfer. The exact system/user prompts remain below.

Equation

System Prompt: You will be given an object that commonly appears in a typical household environment. Using common sense, determine if the object tends to be distributed throughout a typical household, such as doorknobs and light switches.
User Prompt: The following is the object: [object]
LLM Response (GPT4o): True / False

VI-C LGBU Prompt

LGBU receives the current belief, inspected location, coverage, detection result and other detected objects. It selects a likely location from supplied choices; answer-token probabilities become the updated belief. The exact structured prompts are retained for comparison with the explicit Bayesian update.

Equation

System Prompt: You will receive:

  • •

    Current belief about where an object might be located.

  • •

    observation_location: the location that was just inspected.

  • •

    visibility: how much of the location was visible. 0 means not visible at all, 1 means fully visible.

  • •

    result: whether the object was found there.

  • •

    co_detected: other objects found at the same location.

Based on this information and common sense, predict where the object is most likely to be now. Choose the most likely location from the given options.
User Prompt:

  • •

    Current belief about [object_name]: [belief_text]

  • •

    observation_location: [obs_loc]

  • •

    visibility: [visibility]

  • •

    result: [result]

  • •

    co_detected: [co_detected_text]

Given this information, where is object_name most likely to be? [options_text]
Similar to the MCQA method, the belief is updated using next token prediction, where [option_text] is a multiple choice with a character paired with a surface or room.

VI-D Detect action

detect requires typed object/surface/room arguments, the current robot base, a sampled object pose and a certified visibility configuration. Its effects assert the sampled pose/support and increase total cost. sample-PoseB certifies a sampled surface pose; inverse-visibility samples base/head configurations and a head trajectory whose viewing cone covers that pose.

All three original PDDL blocks are preserved exactly, including indentation. The explanatory prose repeats ?pb where the Vis predicate uses ?bq; the actual listing has ?bq. View-cone certification is distinct from successful perception through an occluder, which the execution/belief loop must handle.

(:action detect
 :param (?o ?s ?r ?pb ?bq ?hq ?ht)
 :pre (and (IsItem ?o) (IsSurf ?s) (IsRoom ?r)
           (AtBConf ?bq) (PoseB ?o ?pb ?s)
           (Vis ?o ?pb ?bq ?hq ?ht))
 :eff (and (Supported ?o ?pb ?s) (AtPoseB ?o ?pb)
           (increase (total-cost)
           (DetectCost ?o ?s ?r ?pb))))
(:stream sample-PoseB
 :inp (?o ?s)
 :dom (and (IsItem ?o) (IsSurf ?s))
 :out (?pb)
 :cert (and (PoseB ?o ?pb ?s)))
(:stream inverse-visibility
:inp (?o ?pb ?s)
:dom (and (IsItem ?o) (IsSurf ?s) (PoseB ?o ?pb ?s))
:out (?hq ?ht ?bq)
:cert (and (Vis ?o ?pb ?bq ?hq ?ht)))

VI-E Object State Estimation Derivation

The derivation applies chain rules, conditional independence and a Markov simplification to separate pose, surface and room beliefs. It assumes that knowing an object’s room/surface removes extra semantic-observation information from pose estimation.

Equations (17)–(21) repeat the room variable in the pose-conditioning arguments where the preceding description requires room and surface. The final removal of history relies on the recursive belief retaining past evidence; it is not a generic consequence of state Markovity alone. The entire printed derivation is unchanged.

Equations 16–21
bel(xr,tk,xs,tk,xp,tk)=P(xr,tk,xs,tk,xp,tk|𝒵r,1:t,𝒵s,1:t,zp,1:tk,vr,1:t,vs,1:t)\displaystyle\text{bel}(x_{r,t}^{k},x_{s,t}^{k},x_{p,t}^{k})=P(x_{r,t}^{k},x_{s,t}^{k},x_{p,t}^{k}|\mathcal{Z}_{r,1:t},\mathcal{Z}_{s,1:t},z_{p,1:t}^{k},v_{r,1:t},v_{s,1:t}) (16)
=P(xp,tk|xr,tk,xr,tk,𝒵r,1:t,𝒵s,1:t,zp,1:tk,vr,1:t,vs,1:t)P(xr,tk,xs,tk|𝒵r,1:t,𝒵s,1:t,zp,1:tk,vr,1:t,vs,1:t)\displaystyle=P(x_{p,t}^{k}|x_{r,t}^{k},x_{r,t}^{k},\mathcal{Z}_{r,1:t},\mathcal{Z}_{s,1:t},z_{p,1:t}^{k},v_{r,1:t},v_{s,1:t})P(x_{r,t}^{k},x_{s,t}^{k}|\mathcal{Z}_{r,1:t},\mathcal{Z}_{s,1:t},z_{p,1:t}^{k},v_{r,1:t},v_{s,1:t}) (17)
=P(xp,tk|xr,tk,xr,tk,zp,1:tk)P(xr,tk,xs,tk|𝒵r,1:t,𝒵s,1:t,vr,1:t,vs,1:t)\displaystyle=P(x_{p,t}^{k}|x_{r,t}^{k},x_{r,t}^{k},z_{p,1:t}^{k})P(x_{r,t}^{k},x_{s,t}^{k}|\mathcal{Z}_{r,1:t},\mathcal{Z}_{s,1:t},v_{r,1:t},v_{s,1:t}) (18)
=P(xp,tk|xr,tk,xr,tk,zp,1:tk)P(xs,tk|xr,tk,𝒵r,1:t,𝒵s,1:t,vr,1:t,vs,1:t)P(xr,tk|𝒵r,1:t,𝒵s,1:t,vr,1:t,vs,1:t)\displaystyle=P(x_{p,t}^{k}|x_{r,t}^{k},x_{r,t}^{k},z_{p,1:t}^{k})P(x_{s,t}^{k}|x_{r,t}^{k},\mathcal{Z}_{r,1:t},\mathcal{Z}_{s,1:t},v_{r,1:t},v_{s,1:t})P(x_{r,t}^{k}|\mathcal{Z}_{r,1:t},\mathcal{Z}_{s,1:t},v_{r,1:t},v_{s,1:t}) (19)
=P(xp,tk|xr,tk,xr,tk,zp,1:tk)P(xs,tk|xr,tk,𝒵s,1:t,vs,1:t)P(xr,tk|𝒵r,1:t,vr,1:t)\displaystyle=P(x_{p,t}^{k}|x_{r,t}^{k},x_{r,t}^{k},z_{p,1:t}^{k})P(x_{s,t}^{k}|x_{r,t}^{k},\mathcal{Z}_{s,1:t},v_{s,1:t})P(x_{r,t}^{k}|\mathcal{Z}_{r,1:t},v_{r,1:t}) (20)
=P⁡(xp,tk|xr,tk,xr,tk,zp,tk)​P​(xs,tk|xr,tk,𝒵s,t,vs,t)​P​(xr,tk|𝒵r,t,vr,t)\displaystyle=P(x_{p,t}^{k}|x_{r,t}^{k},x_{r,t}^{k},z_{p,t}^{k})P(x_{s,t}^{k}|x_{r,t}^{k},\mathcal{Z}_{s,t},v_{s,t})P(x_{r,t}^{k}|\mathcal{Z}_{r,t},v_{r,t}) (21)

VI-F Belief over Surfaces

Surface updates mirror room updates, conditioned additionally on the room: a target detection/miss likelihood times cross-object evidence, followed by normalization. The surface relation sums over possible room–surface locations of the other object. All six equations and the detected/not-detected cases are retained.

Equations 22–27
bel​(xs,tk)\displaystyle\text{bel}(x_{s,t}^{k}) =P⁡(xs,tk|xr,tk,𝒵s,t,vs,t),\displaystyle=P(x_{s,t}^{k}|x_{r,t}^{k},\mathcal{Z}_{s,t},v_{s,t}), (22)
=η​P​(𝒵s,t|xs,tk,xr,tk,vs,t)​bel¯​(xs,tk).\displaystyle=\eta P(\mathcal{Z}_{s,t}|x_{s,t}^{k},x_{r,t}^{k},v_{s,t})\overline{\text{bel}}(x_{s,t}^{k}). (23)
P⁡(𝒵s,t∣xs,tk,xr,tk,vs,t)\displaystyle P(\mathcal{Z}_{s,t}\mid x_{s,t}^{k},x_{r,t}^{k},v_{s,t}) =P⁡(zs,tk∣xs,tk,xr,tk,vs,t)×∏j=1j≠kKP⁡(zs,tj∣xs,tk,xr,tk).\displaystyle=P(z_{s,t}^{k}\mid x_{s,t}^{k},x_{r,t}^{k},v_{s,t})\times\prod_{\begin{subarray}{c}j=1\\ j\neq k\end{subarray}}^{K}P\!\bigl(z_{s,t}^{j}\mid x_{s,t}^{k},x_{r,t}^{k}\bigr). (24)
P⁡(zs,tk|xs,tk,xr,tk,vs,t)={(1−pfn)​vs,t,Detected,(1−vs,t)+vs,t​pfn,Not detected.\displaystyle P(z_{s,t}^{k}|x_{s,t}^{k},x_{r,t}^{k},v_{s,t})=\begin{cases}(1-p_{\text{fn}})\,v_{s,t},&\text{Detected},\\ (1-v_{s,t})+v_{s,t}\,p_{\text{fn}},&\text{Not detected}.\end{cases} (25)
P⁡(zs,tk|xs,tk,xr,tk,vs,t)={pfp​vs,t,Detected,1−vs,t​pfp,Not detected.\displaystyle P(z_{s,t}^{k}|x_{s,t}^{k},x_{r,t}^{k},v_{s,t})=\begin{cases}p_{\text{fp}}\,v_{s,t},&\text{Detected},\\ 1-v_{s,t}p_{\text{fp}},&\text{Not detected}.\end{cases} (26)
P(zs,tj|xs,tk,xr,tk)=∑xr,tj∈ℛxs,tj∈𝒮P(zs,tj|xs,tj,xr,tj)×P(xs,tj,xr,tj|xs,tk,xr,tk).\displaystyle P(z_{s,t}^{j}|x_{s,t}^{k},x_{r,t}^{k})=\sum_{\begin{subarray}{c}x_{r,t}^{j}\in\mathcal{R}\\ x_{s,t}^{j}\in\mathcal{S}\end{subarray}}P(z_{s,t}^{j}|x_{s,t}^{j},x_{r,t}^{j})\times P(x_{s,t}^{j},x_{r,t}^{j}|x_{s,t}^{k},x_{r,t}^{k}). (27)

zs,tk=xs,tkz_{s,t}^{k}=x_{s,t}^{k}

zs,tk≠xs,tkz_{s,t}^{k}\neq x_{s,t}^{k}

VI-G Occlusion Aware Particle Filter Algorithm

Algorithm 1 keeps object poses stationary, weights by a target observation or an occlusion-aware miss, optionally multiplies in semantic-distance evidence from another detected object, then resamples N particles. Both the original PDF layout and copyable MathML version are retained. If all weights vanish, the displayed algorithm supplies no fallback.

Complete original Algorithm 1 with all line numbers, nesting and mathematical expressions.
Algorithm 1 Particle filter for object pose estimation
Copyable particle-filter algorithm
1: procedure Particle Filter(Xp,t−1k,zp,tk,zp,tk¯X_{p,t-1}^{k},z_{p,t}^{k},\overline{z_{p,t}^{k}})
2: Xp,tk←∅,X¯p,tk←∅X_{p,t}^{k}\leftarrow\emptyset,\overline{X}_{p,t}^{k}\leftarrow\emptyset
3: for i=1i=1 to NN do
4: xp,tk,[i]←xp,t−1k,[i]x_{p,t}^{k,[i]}\leftarrow x_{p,t-1}^{k,[i]} ⊳\triangleright Stationary objects
5: if object kk is detected then
6: wtk,[i]←P⁡(zp,tk|xp,tk,[i],xs,tk,xr,tk)w_{t}^{k,[i]}\leftarrow P(z_{p,t}^{k}|x_{p,t}^{k,[i]},x_{s,t}^{k},x_{r,t}^{k}) ⊳\triangleright Eq. 13
7: else if object j≠kj\neq k is detected then
8: wtk,[i]←P⁡(zp,tk¯|xp,tk,[i],xs,tk,xr,tk)⋅wtk,[i]w_{t}^{k,[i]}\leftarrow P(\overline{z_{p,t}^{k}}|x_{p,t}^{k,[i]},x_{s,t}^{k},x_{r,t}^{k})\cdot w_{t}^{k,[i]} ⊳\triangleright Eq. 14, 15
9: else
10: wtk,[i]←P⁡(zp,tk¯|xp,tk,[i],xs,tk,xr,tk)w_{t}^{k,[i]}\leftarrow P(\overline{z_{p,t}^{k}}|x_{p,t}^{k,[i]},x_{s,t}^{k},x_{r,t}^{k}) ⊳\triangleright Eq. 14
11: end if
12: X¯p,tk←X¯p,tk∪⟨xp,tk,[i],wtk,[i]⟩\overline{X}_{p,t}^{k}\leftarrow\overline{X}_{p,t}^{k}\cup\langle x_{p,t}^{k,[i]},w_{t}^{k,[i]}\rangle
13: end for
14: for i=1i=1 to NN do
15: draw jj with probability ∝wtk,[j]\propto w_{t}^{k,[j]}
16: Xp,tk←Xp,tk∪xp,tk,[j]X_{p,t}^{k}\leftarrow X_{p,t}^{k}\cup x_{p,t}^{k,[j]}
17: end for
18: end procedure

VI-H Simulation Environment Generation

Housekeep supplies 1,799 object categories, 585 surface types, 15 room types and 45,556 entries, with ten annotators ranking candidate placements. Correct placements score from N downward; incorrect placements from −N upward. Remove surfaces judged implausible by a majority, sum scores and normalize for sampling.

Rooms, surfaces and objects are sampled hierarchically to construct each environment. The exact multiport-hub ranking/scoring examples remain below. Mapping aggregate scores into [0,1] does not by itself specify normalization to a probability distribution; those details and annotation disagreement limit interpretation as calibrated placement likelihoods.

Object: multiport hub | Room: kitchen
Correct (ranked): carpet, fridge, table, counter, sink
Incorrect (ranked): chest, cooktop, microwave, dishwasher, stove
Implausible: oven, shelf, top cabinet, chair, bottom cabinet
Object: multiport hub | Room: kitchen
Correct: carpet=15, fridge=14, table=13, counter=12, sink=11
Incorrect: chest=-15, cooktop=-14, microwave=-13, dishwasher=-12, stove=-11

VII Real World Experiment Analysis

The baseline inspects coffee table, kitchen table, bench, coffee table and kitchen table, requiring four replans. Co-location alone needs three inspections/two replans: after seeing the banana behind the first failed kitchen view, it retries that table from another view. MCQA plus co-location starts at the kitchen table and needs two inspections/one replan.

The initial/goal literals and all three exact execution traces are retained. Failed kitchen observations arise from the cracker/cereal occluders; other failures inspect the wrong surface. The demonstration separates prior-guided first choice from co-location-guided persistence after occlusion.

Initial literal:
(IsItem ?apple),(IsItem ?banana),(IsItem ?cracker_box),
(IsItem ?cereal_box),(IsItem ?screwdriver),
(IsSurf ?coffee_table),(IsSurf ?bench),(IsSurf ?table),
(IsRoom ?living_room),(IsRoom ?kitchen),
(HandEmpty ?arm), (AtBConf ?start)
Goal literal:
(At ?apple ?coffee_table)
Baseline:
detect ?apple ?coffee_table ?living_room (1)
detect ?apple ?table ?kitchen            (2)
detect ?apple ?bench ?living_room        (3)
detect ?apple ?coffee_table ?living_room (4)
detect ?apple ?table ?kitchen            (5)
Num of Replan: 4
Co-Model:
detect ?apple ?coffee_table ?living_room (1)
detect ?apple ?table ?kitchen            (2)
detect ?apple ?table ?kitchen            (3)
Num of Replan: 2
MCQA with Co-Model:
detect ?apple ?table ?kitchen (1)
detect ?apple ?table ?kitchen (2)
Num of Replan: 1
Concept demonstration

Open image ↗