Literature
Display
← All papers

RoboSSM: Scalable In-context Imitation Learning via State-Space Models

Youngju Yoo · Jiaheng Hu · Yifeng Zhu · Bo Liu · Qiang Liu · Roberto Martín-Martín · Peter Stone

· IROS 2026 ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

RoboSSM uses the Longhorn state-space model to infer manipulation tasks from demonstration prompts without updating policy weights at deployment. On selected LIBERO splits, it tolerates prompts much longer than those used for training and runs faster than the matched transformer baseline at large context lengths. The evidence is simulated, with modest long-horizon performance and task-dependent gains; it does not establish general real-robot adaptation.

I INTRODUCTION

In-context imitation learning supplies demonstrations as context instead of fine-tuning a policy for every new task. Transformer cost and length generalization can become limiting when prompts grow. RoboSSM replaces attention-based sequence modeling with recurrent state updates and investigates a test-time memory weighting. The central experiment trains with two demonstrations and tests up to 32, separating adaptation to new tasks from adaptation to longer prompts.

Refer to caption
Fig. 1: Overview of RoboSSM. Training (left): Longhorn receives NtrainN_{\mathrm{train}} trajectories from 𝒫train\mathcal{P}_{\mathrm{train}} and a query trajectory for the same task. Inference (right): Given NtestN_{\mathrm{test}} trajectories from 𝒫test\mathcal{P}_{\mathrm{test}} containing unseen tasks, the model predicts actions and updates the environment iteratively from the initial observation embedding.

II Related Work

Prior work supplies the demonstration-prompting setup and the recurrent sequence-modeling backbone.

II-A In-Context Imitation Learning

Existing in-context robot policies encode observations and actions or action tokens in causal transformers, often evaluating near the training prompt length. RoboSSM specifically tests longer contexts, additional demonstrations, and longer task episodes. Its input design differs: demonstration actions and task language are excluded in the main setup.

II-B State-Space Models

State-space models update a hidden state through recurrence, enabling linear-time processing at fixed state size. Modern variants make transition rules input dependent and support parallel training. Longhorn provides the sequence backbone here; the contribution is its use and evaluation for in-context imitation, not a new general state-space architecture.

Equation 1
st=A​st−1+B​xt,s_{t}=As_{t-1}+Bx_{t}, (1)
Refer to caption
Fig. 2: We evaluate RoboSSM on challenging manipulation tasks from the LIBERO benchmark. In LIBERO-90, the Study and Living Room suites each comprise 4 different scenes, while the Kitchen suite comprises 10 different scenes. Across these suites, the tasks cover diverse manipulation types, including pick-and-place tasks involving diverse objects and target locations (e.g., pick up the left/right bowl and place it in the tray and pick up the book and place it in the back/front compartment of the caddy), and drawer opening and closing tasks. We also evaluate on LIBERO-Long, which contains long-horizon, multi-stage tasks (e.g., put both the alphabet soup and the cream cheese box in the basket).

III Method

The policy combines multimodal observation encoding, Longhorn recurrence, and supervised action prediction over prompted trajectories.

III-A Architecture

Each time step becomes one observation embedding. The recurrent backbone processes the prompt followed by the current rollout and outputs actions through a prediction head.

III-A 1 Input Encoding

Convolutional encoders process front and wrist images; multilayer perceptrons encode joint and gripper state. Concatenated features are projected into an observation token. Inputs deliberately omit demonstration actions to prevent copying and omit task instructions to require use of visual/proprioceptive context. Ground-truth actions still supervise training.

III-A 2 Longhorn state-space block

Longhorn maintains a matrix-valued memory. Input-derived keys and weighting vectors control state retention and outer-product updates; a query projects the state into the context used for action prediction. The retained equations specify these operations. Fixed-size recurrent state avoids storing every prompt token for future attention, though it necessarily compresses the past.

Equations 2–4
st=At⊙st−1+Bt,s_{t}=A_{t}\odot s_{t-1}+B_{t}, (2)
At\displaystyle A_{t} =(𝟏d×m−εt⊗kt⊙2),\displaystyle=(\mathbf{1}_{d\times m}-\varepsilon_{t}\otimes k_{t}^{\odot 2}), (3a)
Bt\displaystyle B_{t} =(εt⊗𝟏m)⊗kt,\displaystyle=(\varepsilon_{t}\otimes\mathbf{1}_{m})\otimes k_{t}, (3b)
εt,i\displaystyle\varepsilon_{t,i} =βt,i1+βt,i​kt⊤​kt.\displaystyle=\frac{\beta_{t,i}}{1+\beta_{t,i}\,k_{t}^{\top}k_{t}}. (3c)
rt=st​qt∈ℝd,r_{t}=s_{t}q_{t}\in\mathbb{R}^{d}, (4)

III-A 3 Longhorn for In-Context Imitation Learning

An online-regression interpretation balances staying near the prior state against fitting the current key–value pair. A sigmoid-derived beta vector controls this tradeoff. At test time, scaling beta by a factor between zero and one reduces the weight of new observations to encourage retention of demonstration context. The best scale is task dependent, so this is an additional inference hyperparameter.

Equation 5
st=arg⁡mins∈ℝd×m​{‖s−st−1‖F2+‖s​kt−xt‖diag⁡(βt)2},s_{t}=\arg\min_{s\in\mathbb{R}^{d\times m}}\left\{\|s-s_{t-1}\|_{\mathrm{F}}^{2}+\|sk_{t}-x_{t}\|_{\mathrm{diag}(\beta_{t})}^{2}\right\}, (5)
Refer to caption
Fig. 3: Comparison of RoboSSM and ICRT across test-time demonstrations (NtestN_{\mathrm{test}}), with both models trained with Ntrain=2N_{\mathrm{train}}{=}2. ICRT’s performance drops sharply once Ntest>NtrainN_{\mathrm{test}}{>}N_{\mathrm{train}}.
Refer to caption
Fig. 4: Results with a fixed test-time prompt Ntest=8N_{\mathrm{test}}=8 across models trained with different numbers of demonstrations (NtrainN_{\mathrm{train}}).
Refer to caption
Fig. 5: Results with a fixed test-time prompt Ntest=32N_{\mathrm{test}}=32 across models trained with different numbers of demonstrations (NtrainN_{\mathrm{train}}). ICRT yields zero success across all NtrainN_{\mathrm{train}}.

III-B In-Context Imitation Learning

Prompt and query demonstrations come from the same task during training, and action outputs on both are supervised. At test time, demonstration observations initialize the recurrent context; the robot then acts from its initial observation and appends each new observation. No policy parameters are updated, but demonstrations of the target task are still required.

Equations 6–8
𝒫=[τ1,τ2,…,τN],\mathcal{P}=\left[\tau_{1},\tau_{2},\dots,\tau_{N}\right], (6)
τi={o1(i),o2(i),…,oTi(i)},\tau_{i}=\left\{o_{1}^{(i)},o_{2}^{(i)},\dots,o_{T_{i}}^{(i)}\right\}, (7)
at=πθ(𝒫;o1q,…,otq),t=1,…,Tq.a_{t}=\pi_{\theta}\!\left(\mathcal{P};o_{1}^{q},\ldots,o_{t}^{q}\right),\quad t=1,\ldots,T_{q}. (8)
Refer to caption
Fig. 6: Comparison of RoboSSM and ICRT when evaluated on test-time prompts with temporally dilated demonstrations.
Refer to caption
Fig. 7: Results of RoboSSM and ICRT on in-context imitation learning, where the prompt consists of the same number of demonstrations during training and testing (Ntrain=NtestN_{\mathrm{train}}=N_{\mathrm{test}}).

IV Empirical Results

Experiments separately test longer prompts, matched prompt lengths, comparison with language-conditioned multitask policies, memory scaling, latent representations, and runtime.

TABLE I: Comparison of RoboSSM and ICRT on the LIBERO-Long living room task suite, which consists of tasks of the form: “Put both Object A and Object B in Object C.” Success rates are computed by assigning 0.5 for each correctly placed object and 1.0 for complete success.
Method Success rate (%)
Ntest=2N_{\text{test}}=2 Ntest=4N_{\text{test}}=4
ICRT [17] 0.0 ±\pm 0.0 0.0 ±\pm 0.0
RoboSSM 16.1 ±\pm 0.6 17.4 ±\pm 0.9

IV-A Experimental Setup

All reported evaluations use LIBERO simulation with disjoint training and test tasks.

IV-A 1 Datasets

LIBERO-Object and scene-based subsets of LIBERO-90 provide eight training tasks per group, except living room with fourteen; each group has two held-out tasks. Each task supplies 50 demonstrations. Two LIBERO-Long tasks test transfer from a corresponding short-task group. These restricted splits should not be confused with average performance over the entire benchmark.

IV-A 2 Baselines

The ICRT comparison uses a four-block LLaMA2-style transformer matched approximately in parameter count to a four-block Longhorn model. Transformer hidden size is 512 with six heads; Longhorn value and key/query dimensions are 512 and 16. Multitask controls use the same backbone families but receive language rather than demonstrations. This is a controlled reimplementation comparison, not a test of every possible transformer configuration.

TABLE II: Comparison of RoboSSM and ICRT with multi-task learning (MTL) policies using their respective backbones. w/ lang denotes that language instructions are included in the input, and w/o lang denotes that language instructions are excluded. Both ICIL frameworks consistently outperform the MTL baselines, demonstrating the capabilities of in-context imitation learning methods.
LIBERO-90
Method LIBERO-Object Study Scene Living Room Scene Kitchen Scene
MTL-TF 24.6 ±\pm 1.7 14.6 ±\pm 6.4 2.5 ±\pm 3.5 0.0 ±\pm 0.0
MTL-SSM 20.4 ±\pm 3.3 15.0 ±\pm 4.3 0.4 ±\pm 0.9 0.0 ±\pm 0.0
ICRT [17]
   w/o lang 66.3 ±\pm 3.8 22.5 ±\pm 6.1 23.3 ±\pm 2.8 0.0 ±\pm 0.0
   w/ lang 48.3 ±\pm 5.9 21.2 ±\pm 3.8 27.1 ±\pm 3.7 0.0 ±\pm 0.0
RoboSSM
   w/o lang 57.9 ±\pm 3.5 34.2 ±\pm 3.7 33.8 ±\pm 4.5 3.8 ±\pm 1.3
   w/ lang 45.4 ±\pm 2.3 35.4 ±\pm 4.7 29.2 ±\pm 5.3 0.0 ±\pm 0.0

IV-A 3 Implementation Details

Training uses one A100, 200 epochs, batch size four, AdamW, and cosine learning-rate decay. Images receive color jitter and random masks. Evaluation runs 20 rollouts per held-out task with at most 200 steps, reporting averages over six seeds. Joint angles and gripper state accompany both camera views.

IV-B Prompt Length Generalization

Prompt length is increased through more demonstrations, longer-horizon tasks, or repeated frames. These probe different forms of extrapolation and need not imply the same kind of generalization.

IV-B 1 Number of demonstrations

Models trained on two demonstrations are tested with one to 32. RoboSSM generally improves or plateaus, with LIBERO-Object strongest at 32; the matched ICRT degrades beyond its training length. Further sweeps vary training counts while testing at eight or 32. The graphs support improved length robustness for this setup; they do not show that more demonstrations always help or that all attention-based policies must fail.

IV-B 2 Long-horizon task adaptation

On two long-horizon living-room tasks, RoboSSM scores 16.1% with two demonstrations and 17.4% with four, versus zero for ICRT. Crucially, Table I awards half credit for placing one of two objects. These values are partial-completion scores, despite the “success rate” label, and should not be interpreted as rates of completing the whole task. The result establishes some transfer but leaves most of the task unsolved.

IV-B 3 Time dilation

Temporal dilation repeats each demonstration embedding up to sixteen times. RoboSSM retains more performance than ICRT as prompts lengthen. This tests robustness to redundant repeated inputs, not realistic changes in contact dynamics, camera motion, or operator timing. Real-robot deployment remains untested.

Refer to caption
Fig. 8: Effectiveness of test-time β\beta-scaling across task suites.
Refer to caption
(a) ICRT
Refer to caption
(b) RoboSSM
Fig. 9: Visualization of latent representations with Ntest=8N_{\mathrm{test}}=8 (left) and Ntest=16N_{\mathrm{test}}=16 (right). Blue corresponds to prompt demonstrations and red to the predicted trajectory.

IV-C In-distribution ICIL

When training and testing use equal demonstration counts, RoboSSM performs comparably or better in many plotted settings, especially study and living-room scenes. This complements its stronger length extrapolation, but performance is not uniformly higher across every suite and prompt configuration.

IV-D Comparison to Multi-Task Learning

Demonstration-conditioned models generally exceed language-only multitask baselines. At four demonstrations, RoboSSM without language scores 57.9/34.2/33.8/3.8% across Object/Study/Living Room/Kitchen; ICRT scores 66.3/22.5/23.3/0%. Thus ICRT is better on Object in this comparison, while RoboSSM improves the other three. Adding language does not consistently help and reduces several scores. Kitchen remains particularly difficult.

Refer to caption
Fig. 10: Inference runtime on LIBERO-Object for ICRT and RoboSSM.

IV-E β -Scaling of Longhorn for ICIL

A beta-scale sweep from 0.4 to 1.0 uses two training demonstrations and eight at test time. Different suites prefer different scales. The ablation supports the importance of the state-retention tradeoff but does not identify a universally optimal setting.

IV-F Latent Space Analysis

Two-dimensional t-SNE plots compare prompt and rollout embeddings. RoboSSM rollouts stay closer to prompt clusters than the transformer examples. Such projections are qualitative diagnostics; proximity in a visualization is not proof of correct actions or a causal explanation of success.

IV-G Runtime Analysis

Runtime includes prompt processing and 200 action updates. RoboSSM’s recurrent work scales with prompt plus rollout length at fixed model size. Transformer prompt attention has quadratic prefill cost, and cached decoding still attends to retained context; its cost is not independent of prompt length merely because keys and values are cached. The measured runtime gap grows for long prompts in this implementation.

V Conclusion

Longhorn is a useful backbone for demonstration-conditioned imitation with long prompts. The strongest evidence is improved prompt-length robustness and runtime in selected simulated tasks. Broader task corpora, higher long-horizon completion, and real-robot evaluation remain necessary to support lifelong or general-purpose adaptation claims.

Concept demonstration

Open image ↗