At a glance
RoboSSM uses the Longhorn state-space model to infer manipulation tasks from demonstration prompts without updating policy weights at deployment. On selected LIBERO splits, it tolerates prompts much longer than those used for training and runs faster than the matched transformer baseline at large context lengths. The evidence is simulated, with modest long-horizon performance and task-dependent gains; it does not establish general real-robot adaptation.
I INTRODUCTION
In-context imitation learning supplies demonstrations as context instead of fine-tuning a policy for every new task. Transformer cost and length generalization can become limiting when prompts grow. RoboSSM replaces attention-based sequence modeling with recurrent state updates and investigates a test-time memory weighting. The central experiment trains with two demonstrations and tests up to 32, separating adaptation to new tasks from adaptation to longer prompts.
II Related Work
Prior work supplies the demonstration-prompting setup and the recurrent sequence-modeling backbone.
II-A In-Context Imitation Learning
Existing in-context robot policies encode observations and actions or action tokens in causal transformers, often evaluating near the training prompt length. RoboSSM specifically tests longer contexts, additional demonstrations, and longer task episodes. Its input design differs: demonstration actions and task language are excluded in the main setup.
II-B State-Space Models
State-space models update a hidden state through recurrence, enabling linear-time processing at fixed state size. Modern variants make transition rules input dependent and support parallel training. Longhorn provides the sequence backbone here; the contribution is its use and evaluation for in-context imitation, not a new general state-space architecture.
Equation 1
| (1) |
III Method
The policy combines multimodal observation encoding, Longhorn recurrence, and supervised action prediction over prompted trajectories.
III-A Architecture
Each time step becomes one observation embedding. The recurrent backbone processes the prompt followed by the current rollout and outputs actions through a prediction head.
III-A 1 Input Encoding
Convolutional encoders process front and wrist images; multilayer perceptrons encode joint and gripper state. Concatenated features are projected into an observation token. Inputs deliberately omit demonstration actions to prevent copying and omit task instructions to require use of visual/proprioceptive context. Ground-truth actions still supervise training.
III-A 2 Longhorn state-space block
Longhorn maintains a matrix-valued memory. Input-derived keys and weighting vectors control state retention and outer-product updates; a query projects the state into the context used for action prediction. The retained equations specify these operations. Fixed-size recurrent state avoids storing every prompt token for future attention, though it necessarily compresses the past.
Equations 2–4
| (2) |
| (3a) | ||||
| (3b) | ||||
| (3c) | ||||
| (4) |
III-A 3 Longhorn for In-Context Imitation Learning
An online-regression interpretation balances staying near the prior state against fitting the current key–value pair. A sigmoid-derived beta vector controls this tradeoff. At test time, scaling beta by a factor between zero and one reduces the weight of new observations to encourage retention of demonstration context. The best scale is task dependent, so this is an additional inference hyperparameter.
Equation 5
| (5) |
III-B In-Context Imitation Learning
Prompt and query demonstrations come from the same task during training, and action outputs on both are supervised. At test time, demonstration observations initialize the recurrent context; the robot then acts from its initial observation and appends each new observation. No policy parameters are updated, but demonstrations of the target task are still required.
Equations 6–8
| (6) |
| (7) |
| (8) |
IV Empirical Results
Experiments separately test longer prompts, matched prompt lengths, comparison with language-conditioned multitask policies, memory scaling, latent representations, and runtime.
| Method | Success rate (%) | |
| ICRT [17] | 0.0 0.0 | 0.0 0.0 |
| RoboSSM | 16.1 0.6 | 17.4 0.9 |
IV-A Experimental Setup
All reported evaluations use LIBERO simulation with disjoint training and test tasks.
IV-A 1 Datasets
LIBERO-Object and scene-based subsets of LIBERO-90 provide eight training tasks per group, except living room with fourteen; each group has two held-out tasks. Each task supplies 50 demonstrations. Two LIBERO-Long tasks test transfer from a corresponding short-task group. These restricted splits should not be confused with average performance over the entire benchmark.
IV-A 2 Baselines
The ICRT comparison uses a four-block LLaMA2-style transformer matched approximately in parameter count to a four-block Longhorn model. Transformer hidden size is 512 with six heads; Longhorn value and key/query dimensions are 512 and 16. Multitask controls use the same backbone families but receive language rather than demonstrations. This is a controlled reimplementation comparison, not a test of every possible transformer configuration.
| LIBERO-90 | ||||
| Method | LIBERO-Object | Study Scene | Living Room Scene | Kitchen Scene |
| MTL-TF | 24.6 1.7 | 14.6 6.4 | 2.5 3.5 | 0.0 0.0 |
| MTL-SSM | 20.4 3.3 | 15.0 4.3 | 0.4 0.9 | 0.0 0.0 |
| ICRT [17] | ||||
| w/o lang | 66.3 3.8 | 22.5 6.1 | 23.3 2.8 | 0.0 0.0 |
| w/ lang | 48.3 5.9 | 21.2 3.8 | 27.1 3.7 | 0.0 0.0 |
| RoboSSM | ||||
| w/o lang | 57.9 3.5 | 34.2 3.7 | 33.8 4.5 | 3.8 1.3 |
| w/ lang | 45.4 2.3 | 35.4 4.7 | 29.2 5.3 | 0.0 0.0 |
IV-A 3 Implementation Details
Training uses one A100, 200 epochs, batch size four, AdamW, and cosine learning-rate decay. Images receive color jitter and random masks. Evaluation runs 20 rollouts per held-out task with at most 200 steps, reporting averages over six seeds. Joint angles and gripper state accompany both camera views.
IV-B Prompt Length Generalization
Prompt length is increased through more demonstrations, longer-horizon tasks, or repeated frames. These probe different forms of extrapolation and need not imply the same kind of generalization.
IV-B 1 Number of demonstrations
Models trained on two demonstrations are tested with one to 32. RoboSSM generally improves or plateaus, with LIBERO-Object strongest at 32; the matched ICRT degrades beyond its training length. Further sweeps vary training counts while testing at eight or 32. The graphs support improved length robustness for this setup; they do not show that more demonstrations always help or that all attention-based policies must fail.
IV-B 2 Long-horizon task adaptation
On two long-horizon living-room tasks, RoboSSM scores 16.1% with two demonstrations and 17.4% with four, versus zero for ICRT. Crucially, Table I awards half credit for placing one of two objects. These values are partial-completion scores, despite the “success rate” label, and should not be interpreted as rates of completing the whole task. The result establishes some transfer but leaves most of the task unsolved.
IV-B 3 Time dilation
Temporal dilation repeats each demonstration embedding up to sixteen times. RoboSSM retains more performance than ICRT as prompts lengthen. This tests robustness to redundant repeated inputs, not realistic changes in contact dynamics, camera motion, or operator timing. Real-robot deployment remains untested.
IV-C In-distribution ICIL
When training and testing use equal demonstration counts, RoboSSM performs comparably or better in many plotted settings, especially study and living-room scenes. This complements its stronger length extrapolation, but performance is not uniformly higher across every suite and prompt configuration.
IV-D Comparison to Multi-Task Learning
Demonstration-conditioned models generally exceed language-only multitask baselines. At four demonstrations, RoboSSM without language scores 57.9/34.2/33.8/3.8% across Object/Study/Living Room/Kitchen; ICRT scores 66.3/22.5/23.3/0%. Thus ICRT is better on Object in this comparison, while RoboSSM improves the other three. Adding language does not consistently help and reduces several scores. Kitchen remains particularly difficult.
IV-E β -Scaling of Longhorn for ICIL
A beta-scale sweep from 0.4 to 1.0 uses two training demonstrations and eight at test time. Different suites prefer different scales. The ablation supports the importance of the state-retention tradeoff but does not identify a universally optimal setting.
IV-F Latent Space Analysis
Two-dimensional t-SNE plots compare prompt and rollout embeddings. RoboSSM rollouts stay closer to prompt clusters than the transformer examples. Such projections are qualitative diagnostics; proximity in a visualization is not proof of correct actions or a causal explanation of success.
IV-G Runtime Analysis
Runtime includes prompt processing and 200 action updates. RoboSSM’s recurrent work scales with prompt plus rollout length at fixed model size. Transformer prompt attention has quadratic prefill cost, and cached decoding still attends to retained context; its cost is not independent of prompt length merely because keys and values are cached. The measured runtime gap grows for long prompts in this implementation.
V Conclusion
Longhorn is a useful backbone for demonstration-conditioned imitation with long prompts. The strongest evidence is improved prompt-length robustness and runtime in selected simulated tasks. Broader task corpora, higher long-horizon completion, and real-robot evaluation remain necessary to support lifelong or general-purpose adaptation claims.