Literature
Display
← All papers

Revisiting Open-Loop Execution in Robotics: Toward Reactive, Higher-Performing Policies

Michael Zeng · Abhinav Agarwal · Ajay Bati · Brian Lee · Siddharth Ancha · Russ Tedrake

Preprint posted · arXiv (preprint) ↗

Robotics13 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the complete 31-page arXiv v2 paper, including all nine appendices. It distinguishes predicting a long action sequence from executing that sequence without feedback. The central result is empirical and data-dependent: longer visual context often makes shorter execution prefixes work better. All 18 numbered figures and their five nested panels, five numbered tables, four original callout diagrams, two displayed equations and 75 references remain.

At a glance

Why do imitation policies often benefit from executing several predicted actions before observing again? The paper argues that a major reason is insufficient history for imitating demonstrations whose actions depend on hidden plans, elapsed time or earlier observations. Experiments on four simulation tasks and two hardware tasks show that increasing context often shifts the best execution horizon toward reactive control. Data-coverage interventions also matter, especially at low data. The conclusion supports testing long-context policies with short execution prefixes, rather than abandoning action-sequence prediction.

1 Introduction

Action chunking combines two choices: how many future actions to predict and how many to execute before replanning. This study holds the future prediction horizon fixed while varying execution. Long open-loop prefixes can preserve a coherent behavior when a short-history imitator cannot infer the demonstrator’s latent state, but they also delay responses to contact or control errors. The proposed remedy is enough observation history to infer that state while continuing frequent replanning. The paper adds a double-encoder architecture and scripted expert variants to study this tradeoff.

Our key contributions are: 1. We show that non-Markovianity of the expert demonstrator is a key reason long execution horizons are necessary for short-context policies. 2. We directly compare expert non-Markovianity with compounding errors and find that, while compounding errors matter, expert non-Markovianity has a stronger influence on the optimal execution horizon than compounding errors in practice. 3. We show that increasing context length reduces reliance on long execution horizons; sufficiently long context removes the benefit of long execution horizons entirely; and increasing context length presents a promising direction for training highly reactive policies that achieve higher task performance.

2 Related Works

ACT and Diffusion Policy popularized chunked robot control. Prior explanations include reduced compounding error, inference latency, smoothness and mode commitment. Zero-latency simulation isolates effects that cannot be attributed to inference delay alone. Earlier compounding-error analysis remains relevant, particularly with little data; this paper argues that expert history dependence is stronger in its moderate/high-data settings. Adaptive horizons, chunk consistency and reactive refinement heads address related tradeoffs, but do not by themselves identify why a short-context policy fails.

3 Preliminaries

The analysis separates observation history, future prediction length and execution prefix, then distinguishes distribution shift from hidden information in the demonstrator. These mechanisms can coexist.

3.1 Imitation learning and Diffusion Policy

Behavior cloning learns an action-chunk distribution from observation/action demonstrations. The context length counts observed frames; the prediction horizon counts future actions; the execution horizon counts actions used before a fresh policy query. A success–horizon curve varies only the execution prefix for a given trained setup. With a fixed sampling period, the prefix length also determines nominal open-loop duration. Most comparisons predict 15 future actions; short-context baselines observe two frames.

3.2 Compounding Errors in Imitation Learning

Small policy errors can move the robot away from demonstrated states, where further errors accumulate. Longer execution chunks may reduce the number of feedback decisions exposed to that mismatch. DAgger instead labels policy-visited states with expert actions. The paper uses data scale and human-gated DAgger as interventions intended to reduce this source of failure, while acknowledging that they do not perfectly isolate it.

3.3 Non-Markovian experts

An expert is Markovian relative to available observations when current observations suffice to determine its action distribution. A non-Markovian expert also uses earlier states or a latent memory such as a plan or counter. The exposition approximates camera observations as Markovian environment states; real images need not satisfy that assumption. Expert memory and physical partial observability are therefore related but distinct concerns, and “Markovian” must be interpreted relative to the learner’s actual inputs.

4 Expert Markovianity Dominates the Success-Horizon Curve

Expert comparisons use image-based Diffusion Policies with two observation frames, 15 future action predictions and execution prefixes from one to 15. Tasks cover furniture assembly, planar pushing, gear insertion and kitchen manipulation. Most datasets contain at least 200 demonstrations. Push-T uses ManiSkill for expert comparisons and Drake for the main context study; their contact models and success conditions differ. Kitchen additionally supplies a subtask-sequence label only for the scripted-expert condition, an important qualification to an otherwise matched comparison.

Refer to caption
Figure 1: Benchmark Tasks. FurnitureSimOneLeg [51]: pick up table leg and insert and screw into table top to assemble the table. Push-T [52]: Push T-shaped slider to target pose, then return robot to reset pose. GearInsertion: Pick up large gear and insert onto gear carrier, meshed with other gears. Kitchen [53]: Complete any 4 kitchen subtasks (e.g., open microwave, move kettle, turn top/bottom knob, flip switch, open slide door, open hinge door), in any order; the dataset contains random subtask permutations. All tasks use image-based observations.

4.1 Policies Trained on Markovian Expert Demonstrations Exhibit Optimal Execution Horizon T exec ∗ = 1

The experiment asks whether replacing history-dependent demonstrations with observable scripted or learned experts changes which execution horizon performs best. The title’s one-step optimum describes these tested datasets, rather than a universal property of all Markovian-expert imitation.

4.1.1 Experimental Setup

FurnitureSim’s scripted baseline derives its control state from current and previous environment states, so it is observable to a two-frame learner. A second version adds latent waypoint offsets, sticky state transitions lasting up to 16 steps and counted alignment maneuvers lasting up to 30 frames. These introduce deliberately hidden time/plan information. Human teleoperation supplies another history-dependent dataset. Other tasks use deterministic reinforcement-learning or finite-state-machine experts, with details and observation caveats in Appendix A.

Refer to caption
Figure 2: Success-horizon curves for Furniture-Sim one-leg. The policy trained on human non-Markovian (NM) expert data (Left) exhibits the typical inverted U-shape curve; the policy trained on Markovian (M) scripted expert data (Center) exhibits an approximately monotonically decreasing curve; adding non-Markovian behaviors to the scripted expert (Right) restores the inverted U-shape.
Refer to caption
Figure 3: Success-horizon curve with non-Markovian (NM) human vs scripted Markovian (M) expert demonstrations for PushT-M (Top Left), GearInsertion (Top Right), and Kitchen (Bottom) tasks. All exhibit the typical inverted U-shaped curve when trained on human data, but all become monotonically decreasing when trained on Markovian expert data.

4.1.2 Results and Discussion

Human demonstrations produce the familiar inverted-U success curves: intermediate execution prefixes work best. Scripted/learned observable experts instead favor the shortest prefixes, with roughly decreasing performance as execution grows. Injecting hidden state into the FurnitureSim expert restores the inverted U. Failure videos show repeated alignment, jitter or switching between locally plausible behaviors without committing to a task transition. These support the hidden-state explanation in the studied settings. Longer open-loop chunks reduce opportunities to change modes, while reactive control helps correct missed insertions and unexpected contact motion.

Refer to caption
(a)
Refer to caption
(b)
Refer to caption
(c)
Refer to caption
(d)
Refer to caption
(e)
Figure 4: Illustration of common failure modes of short-context policies using short execution horizons. (a): FurnitureSimOneLeg: robot cycles between coherent alignment motions but fails to transition to insertion, causing timeout. (b): FurnitureSimOneLeg: robot performs repeated alignment motions with the table leg, but fails to transition to grasp. (c): GearInsertion: robot repeatedly aligns gear with shaft. (d): Push-T: the robot jitters until timeout, oscillating between two modes (pushing the T rightward versus moving around to push it upward). (e): Kitchen: robot approaches light switch but abruptly changes course toward the top knob and collides.
Key Takeaway. Across tasks, longer execution horizons are beneficial when short-context policies imitate non-Markovian demonstrations. When the expert is Markovian, the optimal execution horizon shifts to Texec∗=1T_{\text{exec}}^{*}=1, and the most reactive policies perform best. Expert non-Markovianity is therefore a key driver of the success-horizon curve.

4.2 Investigating the Role of Compounding Errors

Compounding-error interventions are tested on FurnitureSimOneLeg. With only 100 Markovian demonstrations, the optimum is still a four-action prefix; with 200–1,000, increasing data produces a smaller shift toward reactivity. Thus the findings explicitly retain a low-data case where nontrivial open-loop execution helps despite an observable expert.

Three human-gated DAgger rounds add 40 correction demonstrations per round for each independently trained horizon branch. The tested prefixes are 1, 3, 6, 10 and 15. Corrections modestly favor shorter prefixes for the Markovian expert but do not noticeably reshape the non-Markovian curve, where corrections retain hidden-state ambiguity. This compares specific interventions on one task; it does not measure or eliminate all compounding error, nor establish the same causal ranking on every benchmark.

Refer to caption
Figure 5: Success-horizon curve at dataset sizes ∈{100,200,400,1000}\in\{100,200,400,1000\} demonstrations, using both Markovian (Right) and non-Markovian (Left) scripted expert data. Only at the lowest data scale (100 demonstrations), we find compounding errors to play a significant role in shaping the curve, causing even the policy trained on Markovian expert data to prefer a long execution horizon. Above 200 demonstrations, we find increasing dataset size only modestly shifts preference toward lower execution horizons.
Refer to caption
Figure 6: Success-horizon curves after applying HG-DAgger for Scripted Markovian (Right) and non-Markovian (Left) expert-trained policies. We find no noticeable shift for the non-Markovian expert-trained policy, and a slight shift favoring lower execution horizons for the Markovian expert-trained policy.
Key Takeaway. Across interventions designed to mitigate compounding errors: (i) increasing dataset size and (ii) HG-DAgger, we observe at most modest shifts toward shorter execution horizons, even when the interventions improve overall success. Within FurnitureSimOneLeg and the interventions tested, expert Markovianity therefore appears to have a stronger influence on the success-horizon curve than compounding errors.

5 Long Context Reduces the Need for Long Execution Horizons

The next experiments increase visual history to make more of the expert’s latent state inferable. They evaluate both the relative preference for short execution and absolute task success: an improved horizon curve does not necessarily mean the overall policy improves.

5.1 Methodology

Furniture assembly, gear insertion and hardware tasks use a cross-attention U-Net with separate long-range and short-range image encoders. Both encode the newest two frames; only the long-range encoder sees older frames. During training, the short stream is dropped with probability 0.3, encouraging use of history. PushT-D instead freezes a short-context-pretrained encoder in a FiLM-conditioned U-Net; Kitchen uses a fully tuned FiLM model. The shared qualitative finding spans these choices, while exact gains are not an architecture-independent controlled effect.

5.2 Results and Discussion

On 200 human furniture demonstrations, moving from two frames to 16–20 shifts the best execution prefix from about six to one or two. Scripted non-Markovian data show stronger absolute results at 1,000 demonstrations: the best long-context/reactive cell reaches 93.2%, compared with 90.6% for the best two-frame policy. Some latent variables remain unobservable even with more history. PushT-D, gear insertion and Kitchen show similar shifts, while Appendix F reports an absolute-performance failure case in PushT-M.

Hardware tests use 516 pill-dispensing and 540 bag-insertion demonstrations. With two-frame context, eight-action prefixes give 75% and 80% success; with eight-frame context, two-action prefixes give 90% and 85% (18/20 and 17/20 trials). These are small independent evaluation sets with broad confidence intervals, not established gains of equal statistical strength. Hardware never tests a one-action prefix. Twelve-frame context worsens the best results in both tasks, showing that more history is not always better at fixed data scale.

Refer to caption
Figure 7: Success-horizon curves of policies trained across a range of context lengths for FurnitureSimOneLeg, on 200 human demonstrations. As context length increases, peak performance shifts toward shorter execution horizons, with the longest-context policies performing best at execution horizons of 1 or 2.
Refer to caption
Figure 8: Success-horizon curves of policies trained across a range of context lengths for FurnitureSimOneLeg, on 200 and 1000 non-Markovian scripted expert demonstrations. We find increasing dataset size disproportionately improves performance of long-context policies at low execution horizons. With 1000 demonstrations, the highest performing policies use long context length ToT_{o} of 12 – 20 and short execution horizon Texec<4T_{\text{exec}}<4 and achieve 93.2% success, outperforming the best short-context (To=2T_{o}=2) policy at Texec=6T_{\text{exec}}=6 with 90.6% success.
Refer to caption
Figure 9: Success-horizon curves of policies trained across a range of context lengths for PushT-D, GearInsertion, and Kitchen on human demonstrations. Dataset sizes are 160 demonstrations for PushT-D, 200 for GearInsertion, and 581 for Kitchen. With sufficiently long context length, Texec=1T_{\text{exec}}=1 becomes optimal or near-optimal in all tasks.
Refer to caption
Refer to caption
Figure 10: Real-world SinglePillDispense (Left) and SlipIntoBaggie (Right) tasks. In SinglePillDispense, a successful rollout dumps a single pill out of the bottle onto the cap and then rights the bottle. In SlipIntoBaggie, a successful rollout fully inserts the paper slip into the baggie without crumpling the paper.
Table 1: SinglePillDispense and SlipIntoBaggie success rates over 20 trials, with 95% confidence intervals.
SinglePillDispense SlipIntoBaggie
Texec=2T_{\mathrm{exec}}=2 Texec=4T_{\mathrm{exec}}=4 Texec=8T_{\mathrm{exec}}=8 Texec=2T_{\mathrm{exec}}=2 Texec=4T_{\mathrm{exec}}=4 Texec=8T_{\mathrm{exec}}=8
To=2T_{o}=2 50% [29.9, 70.1] 60% [38.7, 78.1] 75% [53.1, 88.8] 60% [38.7, 78.1] 40% [21.9, 61.3] 80% [58.4, 91.9]
To=8T_{o}=8 90% [69.9, 97.2] 80% [58.4, 91.9] 60% [38.7, 78.1] 85% [64.0, 94.8] 70% [48.1, 85.5] 55% [34.2, 74.2]
To=12T_{o}=12 75% [53.1, 88.8] 70% [48.1, 85.5] 60% [38.7, 78.1] 75% [53.1, 88.8] 65% [43.3, 81.9] 65% [43.3, 81.9]
Key Takeaway. Increasing context length shifts peak performance toward shorter execution horizons, with sufficiently long context lengths reaching optimal Texec∗T^{*}_{\text{exec}} of 1 – 2. Long-context, reactive control also scales favorably with data, outperforming short-context, long-execution-horizon baselines at sufficiently high data scales.

6 Discussion and Conclusion

The practical implication is to tune context and execution together, with adequate data and checkpoint selection, rather than assume long open-loop execution is essential. The evidence concerns these tasks and policy families; visual memory must support local contact control as well as semantic task recall. Limitations include imperfect observability, task/architecture differences, limited hardware trials and training-selection uncertainty. Long-context inputs increase sample complexity, and their benefits may saturate or reverse.

Acknowledgments

The authors acknowledge research and teleoperation advice, Mundane Systems hardware support, funding from Amazon, the National Science Foundation and Toyota Research Institute, and MIT computing resources.

Appendix A Experimental Setup

The appendix documents task definitions, expert construction, action spaces and hardware evaluation. These details explain which observations make an expert observable and why results across task variants are not interchangeable.

A.1 Benchmark Simulation Task Details

FurnitureSimOneLeg uses low-randomization furniture poses, two cameras and 200 demonstrations per expert. PushT-M uses 300 per expert; the 160-demo PushT-D context study adds a return-to-start success condition and different contact physics. GearInsertion uses three cameras; its “human” data combine 100 teleoperated and 100 MimicGen-generated demonstrations, compared with 200 scripted successes. Kitchen uses 581 human and 581 scripted demonstrations, with any four of seven subtasks required. Tables 2–3 retain each proprioceptive/action space and the action frequencies: 10 Hz furniture/pushing, 30 Hz gear insertion and 12.5 Hz Kitchen.

Table 2: Benchmark Tasks – Proprioceptive Inputs and Action Spaces
Task Proprioceptive State Action Space
FurnitureSimOneLeg 16D: end-effector pose, velocity, and gripper width 10D: delta end-effector command (3D translation, 6D rotation, 1D gripper)
Push-T 3D: planar end-effector position and orientation 2D: delta end-effector position in the XY plane
GearInsertion 9D: 3D end-effector position, 4D orientation quaternion, 2D gripper state 7D: delta end-effector pose and gripper command
Kitchen 9D: seven arm-joint positions and two gripper-finger positions 9D: velocity commands for the seven arm joints and two gripper joints
Table 3: Benchmark Tasks – Policy Action Prediction Frequencies
Task Predicted-Action Frequency
FurnitureSimOneLeg 10 Hz
Push-T 10 Hz
GearInsertion 30 Hz
Kitchen 12.5 Hz

A.2 Markovian Experts for Push-T , GearInsertion , and Kitchen

PushT-M’s observable expert is a deterministic evaluation of a PPO-trained Gaussian policy. Gear insertion uses a state-based controller with small executed/recorded action noise to broaden supervision. Kitchen samples a four-subtask sequence and exposes it to both the scripted expert and its learner; human-data policies do not receive that label. Scripted observations also receive noise to match the human collection setup. Only successful expert episodes are retained. Success conditioning can itself induce history dependence; the authors expect the effect to be small because expert success exceeds 90%, but do not prove it absent.

A.3 Hardware Experiment Details

Two bimanual tasks test precise feedback under real sensing and inference delay. Their timing, success rules and checkpoint-selection protocol are specified below.

SinglePillDispense .

Starting with bottle and cap already held, the robot must dispense exactly one pill into the cap within 30 seconds; zero or multiple pills fail. The timeout excludes prolonged incoherent shaking that eventually succeeds by chance. Pills weigh approximately 0.6 g; the bottle holds up to 24 and measures 1.7 inches in diameter by 2.8 inches long.

SlipIntoBaggie .

The robot must pick up a bag and paper slip and fully insert the uncreased slip within two minutes. Creasing terminates the attempt as failure. The slip is approximately 83.7×73.8 mm, with ±2 mm variation, and the bag is 127×76.2 mm, making alignment and deformation relevant.

Hardware and control implementation.

Two ARX-5 arms use wrist D405 cameras and an overhead D435. The policy predicts delta end-effector actions at 10 Hz; a 125 Hz interpolator feeds inverse kinematics and joint-position PD control. Observation embeddings are cached. To handle inference delay, execution interpolates from the preceding chunk’s final action to the new chunk’s action 180 ms ahead, an empirically tuned splice. Thus the hardware result includes latency management; a two-action prefix nominally spans 0.2 seconds at the policy action rate.

Evaluation procedure.

Training saves the five checkpoints with lowest validation DDIM mean squared error. Preliminary hardware rollouts screen task performance and qualitative stability, then the selected checkpoint receives an independent 20-trial evaluation. This is distinct from the simulation protocol’s per-horizon best-of-five reporting.

Note on lighting variation.

Color jitter was unintentionally omitted. Evaluations were restricted to the same five-hour daily window, while demonstrations span varying illumination. The suggested benefit of stronger lighting augmentation is a hypothesis, not an evaluated result.

Appendix B Training and Evaluation Methods

Training uses conditional diffusion over action sequences, with task-dependent visual encoders. The validation and checkpoint protocol is essential for interpreting both success curves and uncertainty bars.

B.1 Diffusion Policy

The encoder maps observation history into features, and a conditional U-Net learns to predict injected Gaussian noise at each diffusion noise level. Inference starts from a noisy action chunk and denoises it iteratively. The two original displayed equations retain the corruption process and training objective. Execution uses only the selected prefix before obtaining fresh observations.

Equations
Atk=αk​At+σk​ϵ,ϵ∼𝒩⁡(0,I),A_{t}^{k}=\alpha_{k}A_{t}+\sigma_{k}\epsilon,\qquad\epsilon\sim\mathcal{N}(0,I),
minθ⁡𝔼(Ot,At)∼𝒟,;k,;ϵ​[|ϵ−ϵθ​(Atk,k,Ot)|2].\min_{\theta}\mathbb{E}_{(O_{t},A_{t})\sim\mathcal{D},;k,;\epsilon}\left[\left|\epsilon-\epsilon_{\theta}(A_{t}^{k},k,O_{t})\right|^{2}\right].

B.2 Training and Evaluation Parameters

Simulation uses ResNet-18, usually R3M-initialized; Push-T uses ImageNet initialization with GroupNorm, while hardware uses DINOv3 at one-tenth the policy learning rate. Cameras have separate encoders, sharing weights over time. Tables 4–5 retain batch size 64, peak policy rate 10−4, cosine scheduling, 500 warmup steps, weight decay 10−6, random crops, 128-dimensional projected features, a 69M-parameter U-Net and ten DDIM inference steps.

Validation holds out 5% of demonstrations, or 2% on hardware, so stated dataset counts include the holdout. Simulation saves checkpoints every five epochs, chooses five by validation loss, evaluates each at each horizon for 500 episodes, and reports the highest-performing checkpoint. Timeouts are approximately 120% of the longest demonstration. Wilson intervals describe finite-rollout uncertainty for the reported checkpoint, without training-seed variability or correction for checkpoint/multiple-policy selection. The authors leave formal multiple-comparison testing to future work.

Table 4: Training Hyperparameters Shared for All Models
Parameter Value
Max LR 10−410^{-4}
LR Scheduler Cosine
Weight Decay 10−610^{-6}
Warmup Steps 500
Batch Size 64
Image Size Input Ncameras×320×240×3N_{\text{cameras}}\times 320\times 240\times 3
Image Size Encoder Ncameras×224×224×3N_{\text{cameras}}\times 224\times 224\times 3
Image Augmentations 95% random crop
Encoder Feature Projection Dim 128
Table 5: Diffusion Model Hyperparameters
Parameter Value
U-Net Down dims [256,512,1024][256,512,1024]
U-Net Parameters 69 million
DDIM Inference Steps 10

Appendix C Understanding the Benefits of Long Execution Horizons and Long Context

The following diagrams provide an informal mechanism for the observed curves: limited histories can alias distinct expert states, and action commitment or better context can reduce the resulting ambiguity.

C.1 Why Long Execution Horizons Prevent Failure

An idle period can look identical at every step to a short-context policy, even when the expert knows when to exit. Across demonstrations, locally valid transitions can also combine into a loop that no individual demonstration contains. Long prefixes can cross these ambiguous regions without resampling a behavior. The appendix’s 1/execution-horizon replanning probability assumes phase is uncorrelated with the aliased state. Its language about immunity to short cycles is an illustrative argument, not a general guarantee for arbitrary learned chunk distributions, dynamics or initial phases; model errors and longer ambiguous regions remain possible.

Refer to caption
Figure 11: Illustration of hidden-state aliasing in an idle demonstration. While making high precision maneuvers in tasks like Push-T, humans often pause to align, observe, and think. Such pauses show up as a sequence of idle actions o1k→o2k→⋯→o10ko^{k}_{1}\rightarrow o^{k}_{2}\rightarrow\cdots\rightarrow o^{k}_{10} in the training data. In this example, the full-history conditioned expert πE\pi_{E} advances after it observes the 10th idle state. The imitator πθ\pi_{\theta} only observes the current state o0o_{0} and cannot infer idle count, so both idling and advancing appear valid. The imitator is therefore unlikely to reproduce the expert’s exact sequence of 10 idle actions before advancing, and may idle for much longer than desired, resulting in task timeout.
Refer to caption
Figure 12: Illustration of cycling. Two demonstrations jointly induce a cycle in the aggregated dataset. When observing only o0o^{0} or o2o^{2}, a short-context imitator cannot determine whether to continue or exit the cycle, since both actions appear valid in the training data. It may therefore repeatedly switch between the demonstrated trajectories and enter a cycle: o0→o1→o2→o3→o0o^{0}\rightarrow o^{1}\rightarrow o^{2}\rightarrow o^{3}\rightarrow o^{0}. Longer execution horizons reduce the chance of entering such a cycle: a chunk of actions originating at o0o^{0} toward o1o^{1} and executed open-loop with Te​x​e​c≥3T_{exec}\geq 3 will cross beyond o2o^{2}, whereas re-planning at o2o^{2} can increase the chance of returning to o3o^{3} and entering a cycle.

C.2 Why Long Context Lengths are a Principled Substitute for Long Execution Horizons

A sufficiently long history can include the earlier observation that distinguishes two otherwise identical present situations, allowing coherent behavior while still replanning frequently. This directly addresses aliasing when the relevant information is observable in history. It cannot recover genuinely unobserved latent choices merely by extending the window.

Refer to caption
Figure 13: Illustration of how context length disambiguates the expert action distribution. The observations o2,…,oko^{2},\ldots,o^{k} form an aliased sequence: conditioned only on observations within this sequence, a short-context policy cannot identify the expert’s intended behavior (whether to imitate demonstration 1 or 2). A long execution horizon reduces the probability of re-planning within the sequence, but ambiguity remains whenever re-planning occurs there. By contrast, a sufficiently long context includes the preceding observation o0o^{0}, which disambiguates the correct expert action (a1a^{1}) distribution throughout the sequence, regardless of re-planning phase.

Appendix D HG-DAgger Implementation Details

Each expert/horizon branch maintains its own correction dataset. In each of three rounds, the best preceding checkpoint is rolled out until 50 failures are collected. One annotator marks the earliest consequential deviation and, for hidden-state experts, the scripted controller state. Simulation resets there; successful expert corrections are collected, discarding episodes after ten failed correction attempts. Forty successful corrections are added when available, followed by 50,000 training steps. Five horizons and two expert types require 1,500 annotated failures. Corrections are never shared across horizons.

Appendix E Exploratory Noise-Injection Experiment

Exploratory data mix clean trajectories with noisy executed actions while recording clean expert labels. The sweep varies clean fraction 0.6–0.9 and state-dependent noise scale 1/16–1. Some settings improve absolute performance at both short and long execution horizons, without a consistent leftward optimum shift. This is not a clean causal test: the non-Markovian expert can add latent-state ambiguity, while later chunk labels are paired with states reached by noisy rather than clean preceding actions. Those biases can favor opposite horizons. The experiment also reduces context to one frame and removes velocity inputs to avoid perturbation/history confounding.

Refer to caption
Figure 14: Success-horizon curves of policies trained on scripted non-Markovian expert data under exploratory noise injection. Left: ablation over the action-noise scale σu\sigma_{u}, with fraction of clean trajectories fixed at α=0.8\alpha=0.8. Right: ablation over clean-trajectory fraction α\alpha, with σu=14\sigma_{u}=\frac{1}{4}. We use To=1T_{o}=1 in this experiment for reasons described below.

Appendix F Additional Details on Long-Context Training

The appendix specifies the double-encoder implementation and separates its architectural effects from the task-dependent benefits and failures of longer context.

F.1 Double Encoder Method Details

Independent long- and short-range encoders produce a combined sequence of context-length-plus-two tokens. The newest two frames appear in both streams, tagged by temporal and learned range embeddings before cross-attention. With probability 0.3 per training sample, the short stream is replaced by a learned null representation; dropout is disabled at inference. Past-action prediction is also trained: although 15 future actions are fixed, total predicted sequence length grows to 15 plus context length minus one. Context scaling therefore changes the full prediction workload.

F.2 Double Encoder Method Analysis

FurnitureSim policies with and without the double encoder show comparable absolute performance and the same qualitative context/horizon trend. The double encoder produces smoother variation across execution horizons. The suggested explanation—redundant immediate-state representations reducing brittleness—is unverified.

Refer to caption
Figure 15: FurnitureSimOneLeg success-horizon curves for To∈{2,4,8,12,16,20}T_{o}\in\{2,4,8,12,16,20\} for policies trained with double encoder cross attention U-Net and cross attention U-Net (no double encoder) architectures on 200 human teleop demonstrations. Both plots exhibit similar trends (shorter TexecT_{\text{exec}} perform better as ToT_{o} increases) and comparable absolute performance, but the double encoder architecture exhibits much less variation in performance across execution horizons, producing smoother and more interpretable curves.

F.3 Context Length Ablations for PushT-M

PushT-M is a counterexample to an unconditional “more context improves performance” rule: longer histories favor shorter execution, yet absolute success falls substantially. Sparse visual rendering and causal confusion are proposed explanations, not established causes. With the same two-frame data, the double encoder improves one-step performance relative to the FiLM architecture, but the preferred horizon remains around four or five. Architecture affects the curve as well as context.

Refer to caption
Figure 16: Success-horizon curves from long-context policies trained in PushT-M.

Appendix G Additional Discussion: Multimodality and Smooth Execution

The smoothness discussion assumes perfect recovery of the expert conditional distribution. With enough expert-relevant history, sampled trajectories should preserve expert smoothness; with shorter context, recent samples may constrain only local discrete-time smoothness. These idealized arguments do not establish smooth or safe execution for finite-data, imperfect policies, and learning-error analysis is deferred.

Appendix H Additional Discussion: Benefits of Reactivity in Our Experiments

Reactive feedback helps correct table motion during screwing, missed gear insertion, unexpected T-block motion and imprecise kitchen grasps. These observed failures explain why feedback can matter even in nominally static tasks, without requiring external perturbations.

Appendix I Additional Discussion: The Importance of Checkpoint Selection for Long-Context, Reactive Policies

Shorter execution prefixes often favor earlier checkpoints. Longer context also favors earlier checkpoints in furniture assembly and both hardware tasks, but not consistently in Push-T, gear insertion or Kitchen. The plots normalize the best checkpoint epoch by the latest evaluated epoch, and pool heterogeneous settings. Early stopping may reduce distribution-shift sensitivity, but architecture and other variables are confounded. The recommendation is to evaluate checkpoints carefully; stronger claims about large-data reactive policies remain hypotheses.

Refer to caption
Figure 17: Distribution of the optimal training duration (calculated as the fraction of the highest-performing checkpoint’s epoch divided by the latest evaluated checkpoint’s epoch) as a function of execution horizon, pooled across all short-context (To=2T_{o}=2) scripted- and human-expert experiments (individual points represent individual experiments; black diamonds/bars represent median and interquartile range). We observe a consistent trend across almost all experiments that lower execution horizons prefer earlier checkpoints.
Refer to caption
Figure 18: Distribution of the optimal training duration as a function of context length for all FurnitureSimOneLeg, SinglePillDispense, and SlipIntoBaggie context length experiments (including all execution horizons). We observe policies with longer context lengths often perform better at earlier training checkpoints.
Concept demonstration

Open image ↗