Literature
Display
← All papers

Searching in Space and Time: Unified Memory-Action Loops for Open-World Object Retrieval

Taijing Chen · Sateesh Kumar · Junhong Xu · Georgios Pavlakos · Joydeep Biswas · Roberto Martín-Martín

· ICRA 2026 ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

This edition reviews arXiv v3 dated 20 November 2025. All six authors are affiliated with UT Austin Computer Science. The complete nine-page paper has no appendix; all figures, tables, formal expressions and references are retained.

STAR treats recalling an old observation and probing the current environment as actions in one decision loop. A language/vision model combines an indexed archive with a task-specific working memory, checking whether remembered objects remain where they were seen.

STARBench contains 360 simulated tasks. STAR improves over the compared one-shot retrieval baselines, with some ties. Real-robot evaluation tests guiding a person to the correct landmark, rather than physically picking and delivering the requested object.

I INTRODUCTION

A request can identify an object by its appearance, present relation or past location. The robot must determine which object the user means, locate it now and sometimes open a receptacle to access it. An earlier sighting can establish identity without establishing the current location.

STAR retains task-agnostic captions, poses, timestamps and raw views during patrols. Once a request arrives, its controller can retrieve evidence, navigate, perceive or manipulate, then use each outcome to choose again. This extends the archive approach of ReMEmbR with repeated interaction between memory and current evidence.

The contribution is a unified memory/action architecture and a benchmark for evolving households. “Open-world” denotes open-ended language and potentially unseen targets within the supplied sensors, memory and tool interface; the experiments use a small, controlled set of scenes and object classes.

Complete original Figure 1.
Fig. 1: A fundamental skill for a robot is to retrieve desired objects, often specified through a combination of spatial and temporal references (e.g., “bring the book that was on the desk yesterday”). STAR is a framework that integrates long-term memory with an agentic search engine, enabling the robot to decide whether to search in time (recall past observations) or in space (probe the current environment), producing action sequences (2 to 6) to obtain the necessary information to retrieve the correct object.

II Related Work

Object search: navigation methods find visible targets, while mechanical search can expose hidden objects. Dynamic and open-vocabulary scene representations broaden the setting, but the authors argue that accessible histories are needed for references to past states.

Benchmarks: embodied visual question answering, ALFRED and object-navigation suites evaluate related perception and interaction skills. STARBench adds changing scenes and instructions that distinguish objects using historical or habitual locations.

Language-model agents: Code as Policies, LM-Nav and later systems connect pretrained reasoning to callable robot skills. STAR exposes memory retrieval through that same tool interface, allowing recall and physical verification to alternate.

Memory: scene graphs and long-term semantic maps organize objects and relations; Embodied-RAG and ReMEmbR instead retrieve stored observations. STAR keeps raw views so task-relevant details omitted from captions can still be inspected. The broad novelty claims are the authors’ characterization of prior work, not a controlled comparison against every cited system.

III Problem Statement

The changing environment is only partially observed. Each patrol record contains a timestamp, observation and robot pose. Before the instruction is known, a memory-construction function indexes this history in a task-agnostic form; after task time T, the robot has at most K policy steps.

The policy conditions on the instruction, stored history and new observations. Old observations identify likely locations but can be stale, so successful search requires deciding when to recall and when to obtain fresh evidence. The objective is to reach the requested object within the budget, preferably using fewer steps. The formal history definitions and policy remain below.

Equations 1
ak∼π(a∣l,M,ST:T+k),\displaystyle a_{k}\sim\pi(a\mid l,M,S_{T:T+k}), (1)

t∈{0,1,2,…}t\in\{0,1,2,\ldots\}

et∈ℰe_{t}\in\mathcal{E}

st=(t,ot,xt)s_{t}=(t,o_{t},x_{t})

ot∈𝒪o_{t}\in\mathcal{O}

ℓ∈ℒ\ell\in\mathcal{L}

S≤T={si}i=0TS_{\leq T}=\{s_{i}\}_{i=0}^{T}

M=Φ⁡(S≤T)M=\Phi(S_{\leq T})

ST:T+k={si}i=TT+kS_{T:T+k}=\{s_{i}\}_{i=T}^{T+k}

k∈{1,…,K}k\in\{1,\ldots,K\}

IV STAR: SpatioTemporal Active Retrieval

STAR separates persistent experience from the evidence needed for the current request. Figure 2 shows the archive-building process, the controller’s tool interface and the action/outcome loop that updates working memory.

Complete original Figure 2.
Fig. 2: STAR system. The robot patrols dynamic environments over multiple days to build a non-parametric long-term memory of past observations (left). When the user requests “bring me the book that was on the study desk yesterday”, the agent initializes its working memory with the task. Guided by this working memory, STAR chooses actions from a unified space: recalling past observations (search in time) or probing the current environment through navigation, perception, and manipulation (search in space). Each outcome updates the working memory, and the loop continues until the robot successfully retrieves the target object.

IV-A System Overview

Putting the entire archive into a model prompt would exceed context limits and introduce irrelevant material. STAR instead retrieves records into working memory H, which also accumulates outcomes of actions in the current environment.

At every step the controller chooses either an archive query or an embodied action. Equation 2 expresses the intended approximation of full-history conditioning by working-memory conditioning; the paper does not prove that the selected memory is sufficient or lossless for every instruction.

Equation 2
ak∼π(a|ℓ,Hk)≊π(a|l,M,ST:T+k),\displaystyle a_{k}\sim\pi(a|\ell,H_{k})\approxeq\pi(a|l,M,S_{T:T+k}), (2)

IV-B Non-Parametric Long-term Memory

The non-parametric archive stores each observation with three searchable indices: a text embedding of its caption, its timestamp, and the robot pose. Text, time and location queries therefore retrieve different views of the same records.

Raw images remain attached to the indices. The agent can replay an image when its caption or embedding omits a distinguishing attribute. Adding another modality means adding an index rather than changing the record-storage design. Exact captioner/embedding versions, similarity metrics, retrieval count and storage scaling are not specified in this paper.

Equation

mt=(t,xt,embed​(ot),ot)m_{t}=(t,x_{t},\texttt{embed}(o_{t}),o_{t})

IV-C Working Memory Update with Spatio-temporal Actions

Working memory is an action/outcome trajectory. An action pairs a callable tool with arguments drawn from that tool’s parameter space. Temporal, spatial and semantic archive queries count as searches in memory; navigation, detection, grasping and other robot skills probe the present environment.

For an archive query, the tool converts the model’s text argument into the relevant vector representation, retrieves its top-r neighbors and returns the corresponding records. These results can establish what an object looked like, where it was seen and when. Nearest-neighbor retrieval is not a guarantee that every relevant historical event is returned.

A robot skill returns a raw observation, detections or a success/failure signal. The controller appends the executed action and outcome to working memory, then reasons again. Fresh evidence can invalidate an old location hypothesis without discarding the old observation’s value for identifying the target. The update is append-only in the described method; no learned compression or forgetting mechanism is evaluated.

Equations 3–4
yk={mi:i∈Ir​(θ,M)}≔f⁡(θ,M),\displaystyle y_{k}=\{m_{i}:i\in I_{r}(\theta;M)\,\}\coloneq f(\theta;M), (3)
Hk+1=Hk⊕(ak,yk)=(a1:k,y1:k)\displaystyle H_{k+1}=H_{k}\oplus(a_{k},y_{k})=(a_{1:k},y_{1:k}) (4)

Hk=(a1:k−1,y1:k−1)H_{k}=(a_{1:k-1},y_{1:k-1})

ℱ={f1,…,fN}\mathcal{F}=\{f_{1},\ldots,f_{N}\}

(Θ={Θ1,…,ΘN}(\Theta=\{\Theta_{1},\ldots,\Theta_{N}\}

ak=(f,θ)ka_{k}=(f,\theta)_{k}

fi∈ℱf_{i}\in\mathcal{F}

θi∈Θi\theta_{i}\in\Theta_{i}

𝒜≔(ℱ,Θ)\mathcal{A}\coloneqq(\mathcal{F},\Theta)

Ir​(θ,M)={i1,…,ir}I_{r}(\theta;M)=\{i_{1},\ldots,i_{r}\}

IV-D Action Selection with Working Memory

The controller receives the instruction, working memory and remaining step budget, selects a tool/argument pair and repeats until retrieval succeeds or the budget is exhausted. Budget awareness is supplied in context, rather than established by an optimal planning guarantee.

The printed budget formula is Rₖ=(T+K)−k, although earlier k denotes steps since task time T. Under that relative convention the remaining budget would be K−k; the paper’s expression instead corresponds to absolute time. The original formula is preserved, and the implementation convention is not clarified.

Equation

Rk=(T+K)−kR_{k}\;=\;(T{+}K)-k

ak∼π⁡(a∣ℓ,Hk,Rk)a_{k}\sim\pi(a\mid\ell,H_{k},R_{k})

V STARBench: A Benchmark for Spatiotemporal Object Search

STARBench uses VirtualHome apartment scenes and Unity3D interactions. Patrol histories cover three to six simulated days with 1,200–1,500 observations per day, including views, poses, times and captions. Scene-graph histories are also supplied for comparison.

Three scenes support 225 visible-search tasks, 90 interactive-search tasks and 45 commonsense-search tasks. Visible and interactive search each use five instruction families: class, attribute, current spatial relation, past spatial relation, and habitual spatial relation. Interactive search requires exposing a hidden object; commonsense search can require finding a target never observed in memory.

The benchmark has five object classes for visible search, two for interactive search and five for commonsense search. Thus it tests varied references and changes within controlled households rather than arbitrary real-world object diversity. Figure 3’s spatial-temporal caption says t=8 while that row’s first image is labeled t=13; both original labels are retained.

Complete original Figure 3.
Fig. 3: Task families in STARBench. Each row shows one task family with its instruction (left), the agent’s prior observations (middle), and the correct target object (right). Bounding boxes indicate locations of interest including target object positions. Curr. Observation (Current agent observation). In the class-based case, the folder seen at t=52t=52 is the target object. In the attribute-based case, the instruction is “find the green folder”; although a black folder is visible at t=5t=5, the correct target is the green folder first observed at t=22t=22. In the spatial case, the toy observed on the bed at t=74t=74 is the target. In the spatial-temporal case, the folder was seen one day earlier next to the grey chair at t=8t=8 later moved, and its new position at t=115t=115 is the target. In the spatial-frequentist case, the black folder most often found next to the grey chair but last observed at t=143t=143 in a new location is the target.
Task Type Visible Interactive Commonsense
No. of scenes 33 33 33
No. of Tasks 225225 9090 4545
No. of Object Classes 55 22 55
Task families 55 55 11
TABLE I: Summary of STARBench. The table shows the number of scenes, tasks, and objects for visible and interactive object search. We construct three task types with a total of 360360 tasks: Visible (Visible Object Search), Interactive (Interactive Object Search) and Commonsense (Commonsense Object Search)

VI Experimental Evaluation

Simulation success requires retrieving the requested object within twenty decisions. The reported backbone is OpenAI o3; each model call selects one action. Available simulated skills include navigate, detect, pick and open, alongside queries of past text/images.

Memory knowledge: Oracle construction supplies true visible-object labels and scene-graph relations, plus true node associations for the graph agent. Realistic construction uses predicted perception. However, both modes still use simulator segmentation masks for object detection during skill execution, so Realistic is not an entirely unprivileged end-to-end perception test.

Baselines: Random navigates randomly; SG+S uses scene-graph history and a single retrieval attempt; TR+S uses non-parametric recall followed by a one-shot plan. STAR interleaves recall and embodied actions with feedback. These comparisons evaluate the complete loop against those restricted one-shot designs, not an isolated change in memory representation.

Hardware: a TIAGo operates in one mock apartment with kitchen, living and study areas. There are nine tasks in each of six categories, fifty-four total. Success means leading the user to a correct landmark where the item can be found. Physical pickup, delivery and cabinet manipulation are not required by this hardware success criterion.

Complete original Figure 4.
Fig. 4: Mock Apartment for real-world evaluations. We construct a mock apartment with a kitchen, a living room, and a study area.

Experimental Results

Visible search: Figure 5 reports forty-five tasks per instruction family. The plotted STAR rates lead the compared methods, especially for attributes and temporal references. Graph history and temporal retrieval show complementary strengths. Figure 6 assigns 46.2% of STAR actions to physical actions versus 27.7% for TR+S; action proportions support a behavioral difference but do not alone establish its causal contribution or runtime cost.

Interactive search: in Oracle mode STAR scores 0.67/0.50/0.50/0.61/0.61 across class/attribute/spatial/past/frequentist tasks, versus 0.56/0.33/0.22/0.17/0.28 for TR+S. In Realistic mode STAR scores 0.67/0.61/0.50/0.28/0.44. The past-reference condition ties TR+S at 0.28 and falls from STAR’s Oracle 0.61; improvement is therefore not strict in every condition. The paper attributes gains to using past visual context to choose the correct receptacle, without a separate controlled ablation of that explanation.

Action efficiency: successful visible runs average 4.93 Oracle or 4.74 Realistic actions, versus a minimum of three. Interactive runs average 8.27 or 8.16, versus five. These are roughly 1.58–1.65 times the minima and exclude failed runs. The table includes perception calls alongside navigation and manipulation, so its “physical action” total is not simply robot motions or elapsed time. The 4.93 total differs by 0.01 from the sum of its rounded category means, consistent with rounding.

Unseen targets: commonsense search yields STAR 0.57 in both modes, versus TR+S 0.19 Oracle/0.43 Realistic and Random/SG+S 0.10 Oracle. The result supports searching likely locations beyond directly remembered sightings, but success remains limited to the benchmark’s objects and scenes.

Hardware: STAR succeeds at rates 0.67/0.78/1.00/0.56/0.56/0.33 across the six listed categories, compared with TR+S 0.44/0.67/1.00/0.56/0.44/0.33. It improves class, attribute and habitual-location tasks and ties the other three. With nine trials per category, the rounded rates correspond to 35/54 successes versus 31/54, about a 7.4-percentage-point aggregate gain. This is landmark guidance in one apartment, not a demonstrated end-to-end grasp-and-deliver rate.

Tables and plots preserve the reported uncertainty marks, but their interval definition and repeated-model-run protocol are not given. No independent archive-size, raw-image-replay, retrieval-count, backbone or closed-loop graph-baseline ablation is reported, and wall-clock latency, token cost and long-term memory growth are not quantified.

Complete original Figure 5.
Fig. 5: Execution success rates across five task types of Visible Object Search tasks in STARBench (45 tasks per type). Bars indicate approaches; hatching denotes the environment knowledge used to construct long-term memory. Oracle builds memory with ground-truth object class labels; Realistic builds memory from model predictions only. SG+S uses full scene-graph history for a one-shot attempt; TR+S queries non-parametric memory for a one-shot attempt; STAR (ours) combines temporal retrieval with spatial search and achieves the highest success across task types.
Complete original Figure 6.
Fig. 6: Distribution of executed actions for all tasks.
TABLE II: Execution success rates for interactive search tasks in STARBench. Task abbreviations: C = Class-based, A = Attribute-based, S = Spatial, ST = Spatial-Temporal, SF = Spatial-Frequentist.
Task TR+S STAR
Oracle Realistic Oracle Realistic
C 0.56 ±\pm 0.21 0.28 ±\pm 0.19 0.67 ±\pm 0.20 0.67 ±\pm 0.20
A 0.33 ±\pm 0.20 0.17 ±\pm 0.17 0.50 ±\pm 0.21 0.61 ±\pm 0.21
S 0.22 ±\pm 0.18 0.06 ±\pm 0.12 0.50 ±\pm 0.21 0.50 ±\pm 0.21
ST 0.17 ±\pm 0.17 0.28 ±\pm 0.19 0.61 ±\pm 0.21 0.28 ±\pm 0.19
SF 0.28 ±\pm 0.19 0.11 ±\pm 0.15 0.61 ±\pm 0.21 0.44 ±\pm 0.21
TABLE III: Average number of physical actions executed per successful run. “Optimal” indicates the ground-truth minimal #action required for success.
Visible Object Search
Method Env. Percep. Nav. Manip. Total
STAR Oracle 1.80 1.62 1.52 4.93
Realistic 1.70 1.60 1.44 4.74
Optimal – 1 1 1 3
Interactive Object Search
Method Env. Percep. Nav. Manip. Total
STAR Oracle 3.21 2.00 3.06 8.27
Realistic 3.18 2.11 2.87 8.16
Optimal – 2 1 2 5
TABLE IV: Execution success rates for common sense tasks.
Method Oracle Realistic
Random 0.10±0.130.10\!\pm\!0.13 –
SG+S 0.10±0.130.10\!\pm\!0.13 –
TR+S 0.19±0.160.19\!\pm\!0.16 0.43±0.190.43\!\pm\!0.19
STAR 0.57±0.190.57\!\pm\!0.19 0.57±0.190.57\!\pm\!0.19
TABLE V: Execution success rates for real-world deployment. (9 tasks per type; 54 tasks in total.)
Task Type Random TR+S STAR
Class-based 0.00 0.44 0.67
Attribute-based 0.00 0.67 0.78
Spatial 0.11 1.00 1.00
Spatial-Temporal 0.00 0.56 0.56
Spatial-Frequentist 0.00 0.44 0.56
Common-Sense 0.00 0.33 0.33

VII Conclusions

Treating memory recall and current-world interaction as actions in one loop improves the evaluated object-search tasks relative to the selected one-shot baselines. The evidence concerns three simulated scenes and one real mock apartment, with different simulation and hardware success criteria.

Future work includes pruning or compressing memory while retaining useful history, and distilling low-level action traces into reusable high-level skills. The source promises a public benchmark release upon acceptance; this edition does not independently establish the later release status.

Concept demonstration

Open image ↗