At a glance
HALO trains a visuomotor policy to retrieve relevant past observations and actions. Generated video question–answer supervision guides a shared memory backbone while imitation learning grounds the representation in control, and top-eight attention limits how much history enters each retrieval. Separate task-specific models are evaluated on memory-dependent simulated tasks and five physical tasks across stationary and mobile platforms. Average physical success rises from 36% to 55% over the standard transformer baseline; long timed episodes reach eight minutes, with substantial failures still remaining.
I Introduction
Partially observable tasks may require remembering an object’s prior location, the container it came from, a count of hidden objects, or when an appliance was activated. Simply adding long-context attention can let a policy exploit irrelevant correlations or repeatedly incorporate off-demonstration history.
HALO combines auxiliary questions about the past with action imitation and sparse retrieval. Its aim is a reusable retrieval mechanism across information types, rather than a single generalist policy: the implementation trains a separate model for each task. The eight-minute horizon includes timed heating tasks and should not be interpreted as eight minutes of continuous complex manipulation.
II Related Work
Related work spans memory under partial observability, foundation-model supervision, and long-context retrieval mechanisms.
II-A Imitation Learning under Partial Observability
Object-centric memory, semantic similarity, hand-written storage rules, and learned compressed states can each preserve useful information while discarding other details. HALO keeps an explicit encoded history and learns which entries to retrieve. This retains flexibility across the tested tasks, at the cost of storing and searching a growing episode buffer.
II-B Knowledge Distillation from Foundation Models
Vision–language models provide priors about scene events and what task information matters. HALO transfers these priors through question-answer training on episodic memory, while retaining action supervision to recover details needed for control. The language models are used to generate training supervision; the reported control policy does not require generating a new textual summary at every action.
II-C Memory Mechanisms for Long-Context Reasoning
Transformers provide content-based access to old observations, whereas recurrence and compressed states summarize them. Dense attention may also expose control to irrelevant historical features. HALO applies content-selected sparse attention; the experiments test whether this improves control, without proving that the selected memories are causally sufficient.
III HALO
The method combines shared encoders and a transformer memory backbone with separate action and answer heads, using joint action imitation and question-answer supervision.
III-A Problem Formulation and Overview
The policy receives a task instruction, current sensory input, and encoded past observations and actions. Observation and action encoders map these modalities into a common latent space. Query–key similarity retrieves history, which the backbone combines with the current observation before predicting an action chunk or text answer.
The implementation stores sampled, pooled embeddings rather than every raw frame or action. Within that encoded buffer, sparse retrieval selects entries without replacing the history with a single summary. The distinction matters when interpreting the paper’s statement that it stores all information.
Equations
III-B Distilling Vision-Language Model Priors via Video Question–Answering
A text head answers generated questions about past object locations, counts, relations, and event progress. Its encoder and memory backbone are shared with the action predictor, so answer supervision can shape representations used for control. The combined objective adds answer cross-entropy to action imitation loss. Auxiliary questions encourage useful recall, but successful question answering alone does not guarantee sufficient information for low-level action prediction.
Equations 1–4
| (1) |
| (2) |
| (3) |
| (4) |
III-C Generating VQA Data from Long-Horizon Trajectories
Trajectory processing first combines object-related visual information and activity descriptions into text. A language model then generates questions and answers for randomly selected frame indices, reducing its tendency to focus only on early events. A second model scores correctness and relevance, filtering weak pairs and about 20% of generated data. These are automatic quality judgments, not human-verified ground truth for every answer.
III-D Reducing Model Drift with Top- k Attention Sparsification
For each query, select the eight highest-scoring memory entries and normalize attention only over them. A straight-through estimator supplies training gradients through the discrete selection. This restricts which values influence the output, but the candidate memory must still be scored; fixed retrieval count does not imply constant storage or search cost as history grows. Sparsity is an empirical regularizer here, not a formal bound on closed-loop error accumulation.
Equations 5–6
| (5) |
| (6) |
IV Experiments
Experiments measure task success, compare memory designs, and probe retrieval behavior and action-distribution drift. Simulation and physical evaluations use different trial counts and should remain separate.
IV-A Evaluation Tasks.
The tasks require recalling a hidden object location, returning a bowl to its original plate, counting items placed out of sight, and remembering a stove activation time for four-, six-, or eight-minute delays. A physical collaboration task also requires counting cups placed by a human before the robot completes a specified total. Initial object arrangements or target counts vary across episodes. These controlled tasks expose distinct memory requirements rather than unrestricted household competence.
| Method | Retrieve Obj. | Return to Container | Store N Objs. | Heat Stove | Store N Objs. | Average |
|---|---|---|---|---|---|---|
| for mins. | Human-Robot | |||||
| (Spatial Info.) | (Relational Info.) | (Numerical Info.) | (Event-time Info.) | Spatial Info. | ||
| Standard Transformer | ||||||
| HALO |
IV-B Baselines.
Comparisons include object-location memory from SAM2Act++ [16], textual summaries from ReMemBer [3], hand-designed feature selection, Scene Memory Transformer [17], and token merging. Additional comparisons test recurrent/state-space models and fixed or gated attention patterns.
The main table includes a standard transformer and HALO without question-answer supervision, permitting separate comparisons for sparse attention and joint training. Not every alternative is evaluated on every physical task.
IV-C Results
Across four simulated tasks, HALO averages 41% success versus 22% for the standard transformer, 31% for sparse retrieval without question-answer training, 34% for scene-memory compression, 29% for token merging or hand-designed features, 20% for object-centric memory, and 18% for text summaries. Adding question-answer supervision therefore gives ten percentage points in Table I; sparse attention without that supervision gives nine points over the standard transformer. No method wins every task: hand-designed features lead object retrieval and object-centric memory slightly leads counting.
Across five physical tasks with twenty trials each, HALO averages 55% versus 36%. Its individual successes are 55%, 40%, 55%, 60%, and 65%, compared with 40%, 30%, 20%, 40%, and 50%. These are 100 physical rollouts per method across tasks, without reported confidence intervals or repeated training-seed variation.
Table III evaluates other memory mechanisms on only two simulated tasks. HALO scores 64%/32%, gated attention 45%/28%, LSTM 14%/12%, Mamba 20%/18%, and Transformer-XL 12%/20%. These task-matched numbers should be used instead of the prose’s inconsistent aggregate improvement claims for some recurrent baselines.
Separate design ablations favor a moderate retrieval budget and joint training: the reported k sweep peaks at k=8 with 52% success, and co-training reaches 64% versus 44% for answer-pretraining then action-finetuning and 42% without answer training. Their aggregation and task scope are not fully specified, so they are not interchangeable with the four-task 41% average. The paper also reports better generated-answer quality than a single long-video prompt, without supplying a complete evaluation sample protocol.
V Conclusion and Future Work
Joint question-answer and action training plus selective retrieval improves the tested long-memory control tasks. Remaining limits include per-task training, automatic supervision errors, modest absolute success, fixed retrieval size, and history-dependent latency. Adaptive retrieval, cross-episode learning, and speculative retrieval are proposed future work. The reported evidence does not establish unlimited context or safe autonomous household operation.
VI Acknowledgments
The acknowledgments identify feedback from the collaborating robotics labs and support from United States and Korean research funding programs.
VII Appendix
The appendix provides retrieval examples, generated questions, sensitivity visualizations, drift diagnostics, a toy retrieval study, implementation details, and the complete generation/filtering prompts.
VII-A Qualitative Analysis of Retrieved Information
Retrieved memories can include both where a target was observed and where it was absent, along with past actions and stage-progress cues. The two illustrated decisions involve navigation toward an object and returning an item to its original container. These examples suggest richer retrieval than a rule that stores only visible object locations, but they do not by themselves prove which memories caused the successful action.
VII-B Qualitative Visualization of Generated VQA Data
The retained question-answer examples cover locations, counts, timing, and task progress. Only four video frames are shown per example for readability, while questions refer to the underlying trajectory. The examples illustrate the supervision format rather than provide a comprehensive correctness audit.
VII-C Qualitative Visualization of Spurious Correlations
Grad-CAM-style sensitivity maps show HALO emphasizing the target bottle in an example where the standard transformer is sensitive to a distractor tissue roll. This supports the proposed explanation of more relevant retrieval, but attention or gradient visualizations are qualitative diagnostics rather than causal intervention tests.
VII-D Quantitative Analysis of Model Drift Errors
Action histograms from simulated policy rollouts and expert demonstrations are compared after identical initial-state resets. Top-k attention lowers the reported Jensen–Shannon divergence from 0.07 to 0.06. Because the measure pools discretized joint actions, it summarizes distributional similarity rather than temporal correspondence, error propagation, or task completion. The original bar chart is retained; matching expert action frequencies is not itself a success guarantee.
VII-E Quantitative analysis of the effect of k with context length N
The toy study varies how many useful bits are required and the context length. At fixed context size, task information demand matters more than modest changes in k; for example, the high-information row benefits from k=8 or 16 compared with k=2, while shorter-information rows change little. The context-length panel also declines as histories grow. These synthetic results motivate task-aware budgets but do not establish a universal k for robotic control.
VII-F Implementation Details
A pretrained frozen vision encoder and learned pooling produce observation embeddings, and an MLP embeds past actions. The backbone has four local-attention layers and two long-history retrieval layers. The action head predicts 32 future actions, while a small autoregressive text head uses the Qwen3 vocabulary. Separate models are trained per task.
The buffer is described as 192–3,840 entries depending on horizon, sampled every eight interaction steps, with k=8 selected by cosine similarity. The task table lists smaller particular capacities and omits a separate human-collaboration row. Imitation uses L1 loss, answer training uses cross-entropy, and the auxiliary weight is selected from 0.1 and 1.0.
SAM3 and GPT-4o-mini generate visual/event summaries and candidate answers; GPT-4o judges correctness and relevance. The final datasets contain about 20,000–30,000 question-answer pairs per task. Training uses AdamW, learning rate 0.0005, effective batch size 32, and up to 200 epochs with early stopping. Evaluation uses fifty simulated and twenty physical episodes per task. Table IV preserves the reported demonstration counts, typically fifty in simulation and fifty to one hundred in the listed physical tasks.
| Task Name | Domain | Max Memory Storage | # Cameras | # Expert Demos |
|---|---|---|---|---|
| Retrieve Object | Simulation | |||
| Retrieve Object | Real-world | |||
| Return to Same Container | Simulation | |||
| Return to Same Container | Real-world | |||
| Store Objects | Simulation | |||
| Store Objects | Real-world | |||
| Heat Stove for mins | Simulation | |||
| Heat Stove for mins | Real-world |
| Config | Value |
|---|---|
| Optimizer | AdamW |
| Base learning rate | |
| Effective batch size | |
| Weight decay | |
| Warmup epochs | |
| Total epochs | |
| Early stopping epochs | |
| Action prediction horizon | |
| Proprioception noise (std) | |
| Brightness augmentation | |
| Contrast augmentation |
VII-G Prompts
The complete original prompts specify frame-indexed, task-relevant questions, short grounded answers, and structured output. Filtering keeps scores of three or higher on a five-point scale; its rubric permits a correct but non-memory-dependent question at score three, so not every retained pair is guaranteed to require recall. The filtering prompt also includes a reasoning field in its example schema while later excluding fields beyond id, score, and decision. These original prompt inconsistencies are preserved, along with exact line breaks and indentation.
You are given a task instruction, task's description, summary of task events labeled by frame numbers, and a list of information provided per-frame and per-camera about the visible objects in the scene, their locations, and robot actions.
You are given a separate list of QUERY-FRAME INDEXES to generate the query and answer.
Your job:
1. Identify events relevant to the task.
Use the task description, and summary of task events to understand the scene in the current episode.
Using these information, describe what plausibly happens in the task for each of the given QUERY-FRAME INDEXES.
2. Write ONE memory-based query important for the task for each of the given QUERY-FRAME INDEXES
The question must require recalling earlier information provided in the prompt for each of the given QUERY-FRAME INDEXES. It may include:
- object location seen earlier in the QUERY-FRAME INDEX (e.g., bounding box coordinates, etc.)
- number of objects of a particular type seen earlier in the QUERY-FRAME INDEX (e.g., number of sponges, etc.)
- few words about the task events achieved upto the QUERY-FRAME INDEX using the provided summary of task events (e.g., picked two sponges, placed one in sink)
- time-frame of the task events which includes the QUERY-FRAME INDEX (e.g., frame 10 to frame 20)
- important object relations to remember (e.g., object X is to the left of object Y)
Rules (VERY IMPORTANT):
- The answer should be relevant to the provided task instruction and help in solving the task.
- The questions should be about the past events or observations upto the QUERY-FRAME INDEX, not the future events.
- Include when, what, where, how much, how many, description, etc. in the question, especially depending on the task demands.
- Questions should NOT include generic references like 'first frame,' 'last frame' but rather use the QUERY-FRAME INDEX provided in the prompt to indicate the frame number in the question.
- Do NOT include questions not used or important for completing the task.
Example 1 (with QUERY-FRAME INDEX: 19), Query: "In the eye-in-hand camera, how many sponges were there in frame 19?"
Example 2 (with QUERY-FRAME INDEX: 12), Query: "What was the location of the blue sponge in frame 12 in the agent-view camera?"
Example 3 (with QUERY-FRAME INDEX: 38), Query: "Describe the task events up until frame 38 from both cameras?"
Example 4 (with QUERY-FRAME INDEX: 24), Query: "What was the location all the sponges in frame 24 in the agent-view camera?"
Example 5 (with QUERY-FRAME INDEX: 13, between frame 10 and frame 20), Query: "When did the robot pick up the blue sponge?"
3. Answer the query
Provide the single best answer using only the provided information provided for each of the given QUERY-FRAME INDEXES and your event notes.
The answer should be appropriate for the referenced QUERY-FRAME INDEX and the camera. If there's no answer, provide N/A
The answer should be concise using very few words. Make sure if there are multiple answers to the query, provide all the answers.
Example 1, 2 sponges
Example 2, Bbox: (x_min, y_min, x_max, y_max) (if there are multiple bboxes, provide all the bboxes separated by a commas)
Example 3, picked one sponge, placed one in trash, picked another sponge
Example 4, Bbox: (x_min, y_min, x_max, y_max), (x_min, y_min, x_max, y_max), ...
Example 5, Frame: 10 to frame 20
OUTPUT FORMAT:
The JSON should contain a single "results" field with a list of dictionaries, each containing the three required fields for each of the specified QUERY-FRAME INDEX:
json{{
"results": [
{{
"description-of-query": "counting / task-events / location / etc, camera-name",
"query": "<one memory-based question for the first QUERY-FRAME INDEX>",
"answer": "<concise answer grounded in the frames for the first QUERY-FRAME INDEX>"
}},
...
}}
TASK INSTRUCTION: \textit{task_language}
TASK DESCRIPTION: \textit{task_description}You are evaluating multiple candidate outputs for a single robot episode.
You will be given the SAME EPISODE CONTEXT as in the base prompt:
- A task instruction, task's detailed description, summary of task events labeled by frame numbers, and a list of information provided per-frame and per-camera about the visible objects in the scene, their locations, robot actions, etc.
After the task information, you will also be given a BUNDLE of candidate items to evaluate.
Each candidate item includes:
- id (MUST be an integer)
- 'query' (a memory-based question),
- 'answer' (the model's answer),
Your job (per item):
1) Task alignment: Do query, instruction, and answer all help in solving the task as described in the task instruction and task description?
2) Grounding: Is the answer specific and consistent with the episode's referenced frame(s) indices and camera(s)? Is the answer correct? Verify for both.
3) Memory dependence: Does the query REQUIRE recalling earlier information about the visible objects in the scene, their locations, robot actions, etc.
4) Instruction quality: Does the instruction briefly prime what to remember?
Answer should be correct for high scores 3--5, both inclusive.
Scoring (integers only, 1--5):
5 = Excellent: query and answer clearly help in solving the task, grounded and answers are correct, memory-dependent question
4 = Good: query and answer help in solving the task; grounding and answer are correct; partially memory-dependent question
3 = Fair: query and answer help in solving the task; ground and answer are correct; not memory-dependent question
2 = Weak: the question-answer pair is not useful to remember for the task (e.g., asking about distractor objects, not relevant to the task); answer is correct;
1 = Poor: irrelevant to the task; answer is incorrect;
Decision rule:
- If score >= 3 \rightarrow decision = ``keep''
- If score <= 2 \rightarrow decision = ``skip''
INPUT FORMAT
You will receive:
1) The task information: a task instruction, task's detailed description, and a list of information provided per-frame and per-camera about the visible objects in the scene, their locations, robot actions, etc.
2) Then a bundle of candidate items:
[
{{
"id": "<integer>",
"query": "<string>",
"answer": "<string>",
}},
...
]
STRICT OUTPUT FORMAT (no prose, no extra fields in the JSON). Mention one entry per ID for each entry in the candidate items. Do NOT skip any IDs:
json{{
"results": [
{{"reasoning": "<explain briefly why it will be useful to remember for the task>", "id": <integer>, "score": <integer>, "decision": "keep"|"skip"}},
...
]
}}
Important:
- 'id' in output MUST match the integer 'id' from input.
- Base your judgments ONLY on the provided task information (task instruction, task description, and provided information) and the candidate fields.
- Do NOT include explanations, intermediate notes, or any fields other than id, score, and decision in the JSON.
TASK INSTRUCTION: {task_language}
TASK DESCRIPTION: {task_description}