Literature
Display
← All papers

Learning Composable Skills by Discovering Spatial and Temporal Structure with Foundation Models

Neil Nie · Wenlong Huang · Jiayuan Mao · Fei-Fei Li · Weiyu Liu · Jiajun Wu

· ICRA 2026 ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

STACK learns reusable robot skills from five or ten demonstrations per domain. Foundation models segment demonstration videos and identify relevant objects; diffusion models learn object-relative trajectories and effects. A planner composes skills while checking predicted geometry. The real-robot study covers new layouts, added obstacles, and longer sequences. Main results measure partial task completion, not full-task success. Evaluation is primarily open loop, and the method assumes rigid or articulated geometry rather than deformable objects.

Figure 1. STACK overview: demonstration, new scene layouts, new geometric constraints, and longer task horizons.
Figure 1. STACK overview: demonstration, new scene layouts, new geometric constraints, and longer task horizons.

1. Introduction

Temporal decomposition allows learned skills to be reused in new sequences, while object-relative representations reduce dependence on background and absolute placement. STACK obtains both structures from foundation models instead of manually labeling each demonstration. It learns both how to execute each skill and how the relevant geometry will change, allowing later actions to account for earlier placements. [1], [2], [4], [5], [6], [7].

3. Problem formulation

Demonstrations contain a language goal, partial point clouds, and end-effector actions. Each discovered skill has a description, relevant entities, a trajectory sampler, and an effect predictor. Planning chooses a skill sequence and trajectories that approach the goal while satisfying transition, kinematic, and collision constraints. Entities may be entire objects or task-relevant parts. The displayed optimization is unnumbered in the original paper.

Equation unnumbered
(unnumbered)Original unnumbered planning objective and feasibility constraints.

4. Method

The pipeline first discovers temporal skills and their relevant geometry, then trains a sampler and effect model for each skill. At test time, language-guided candidate sequences are instantiated through geometric search.

Figure 2. Skill discovery, spatial segmentation, sampler/effect learning, and composition.
Figure 2. Skill discovery, spatial segmentation, sampler/effect learning, and composition.

4.1 Discovering skills

A video-language model receives the goal, demonstration frames, gripper state, and joint-torque cues, with arm-color overlays for bimanual tasks. Proposal and refinement stages identify skill boundaries and descriptions. The model names relevant entities; open-vocabulary detection and SAM2 produce masks, tracking, and reconstructed RGB-D point clouds in a common frame. Outlier filtering and farthest-point sampling clean the observations. [69], [70], [79].

4.2 Learning composable skills

A conditional diffusion sampler maps relevant point clouds and, when needed, the previous gripper pose to a short end-effector trajectory. The learned effect model predicts each entity’s rigid transform. ICP alignment between segment endpoints supplies training labels. Object-relative augmentation transforms point clouds, trajectories, and effects together, with shared rotation/translation settings across domains. Simulated viewpoint occlusion reduces overfitting to specific observations. [10], [71].

4.3 Planning with learned skills

A vision-language model proposes skill sequences. Tree search samples trajectories, predicts entity motion, rejects infeasible or colliding candidates, and advances using updated scene geometry. Partial-cloud distances provide inter-entity collision checks. Transit and rigidly attached-object move skills connect learned actions; nvblox builds scene distance fields and cuRobo generates collision-free connecting motion. [72], [73].

5. Experiments

5.1 Domains and generalization

Mug Tree uses five demonstrations; Bookshelf and Kitchen each use ten. Tasks involve hanging mugs, bimanual book insertion with pushing, and washing/stowing plates with an articulated faucet. SC changes initial object poses, GC adds obstacles or kinematic restrictions, and LH requires multiple objects beyond the demonstrated horizon. Franka robots, calibrated RealSense cameras, and GELLO provide execution, perception, and demonstrations. [74].

Figure 3. Mug Tree, Bookshelf, and Kitchen demonstration domains.
Figure 3. Mug Tree, Bookshelf, and Kitchen demonstration domains.

5.2 Segmentation results

Temporal evaluation compares segment structure, framewise labels, and missing/extra segments against manual ground truth. STACK reports zero missing/extra segments in all three domains and matches the zero-shot video model on Mug Tree while improving Bookshelf and Kitchen. Spatial entity segmentation reaches 97% mAcc and 91% mIoU. These annotation metrics do not imply flawless perception during execution. [75], [76], [77].

Table I. Temporal segmentation metrics: Edit Score, Mean over Frames, and missing/extra segments.
Table I. Temporal segmentation metrics: Edit Score, Mean over Frames, and missing/extra segments.

5.3 Manipulation results

Each method/domain/generalization cell has two cases with five trials each, totaling 540 real-world trials. Partial success measures the fraction of key steps completed. STACK scores 90.0/80.0/77.4% on Mug Tree SC/GC/LH, 90.0/93.3/86.1% on Bookshelf, and 85.0/75.0/71.6% on Kitchen. BLADE ties SC in Bookshelf and Kitchen but is weaker under geometric constraints and longer horizons. DP3-Long scores zero throughout; DP3-Short uses discovered segments and an oracle skill sequence but achieves nonzero scores only on Mug Tree. Removing spatial augmentation or effect prediction reduces aggregate performance. Full-task completion is considerably lower on some longer tasks; Appendix Table A1 retains those counts. The main text attributes 51.5% of failures to execution, 27.3% to detection, and 21.2% to segmentation, with accumulation implicated in 13.7% of failures. The supplement’s segmentation claim conflicts with this breakdown. [5], [10].

Table II. Mean partial success percentages over ten trials per method/domain/generalization setting.
Table II. Mean partial success percentages over ten trials per method/domain/generalization setting.
Figure 4. Training/testing layouts, obstacle-aware trajectories, and longer task sequences.
Figure 4. Training/testing layouts, obstacle-aware trajectories, and longer task sequences.

6. Conclusion and limitations

The evidence supports combining discovered structure, diverse learned trajectories, and predicted geometry in these few-demonstration tasks. STACK does not model cloth or other deformable objects. Evaluations emphasize open-loop composition; reported 2–5 Hz sampler speed is not a demonstrated closed-loop controller. Effects are used deterministically despite diffusion-based parameterization, so uncertain outcomes remain a limitation. Longer plans increase search cost and may require hierarchical or interleaved planning/execution. [78].

Supplementary material

The linked supplement supplies implementation details, extra experiments, failure examples, planning pseudocode, and six complete prompts. The public PDFs are anonymized submission copies; author names and ICRA 2026 attribution are taken from the authors’ project record. The supplement contains unresolved author notes and a failure-accounting inconsistency, identified below.

A. Implementation details

A.1 Temporal segmentation

The reported model is gemini-2.5-pro-preview-03-25. Gripper widths become binary open/closed indicators; joint torques become averaged, rounded scalars. Both are timestamped at one frame per second. Robot-model overlays distinguish arms. Bimanual and single-arm proposals are aggregated before cross-demonstration refinement, then an entity-naming prompt identifies anchor and moving objects. Free-space motion may still contaminate contact-rich skill segments.

A.2 Spatial segmentation

OWLv2 supplies bounding boxes, with Lang-SAM used when no relevant object is detected; SAM2 produces masks. Masked depths from available cameras reconstruct and merge entity point clouds. Training processes each frame, and testing processes each new observation. Specific descriptive names help reduce irrelevant detections, but missing or incorrect boxes remain a failure source. [69], [70], [79].

A.3 Trajectory sampler

Inputs are downsampled to 2,048 points; every skill trajectory has 32 steps. Grasp-dependent skills also receive the initial end-effector pose. Each arm outputs translation, a continuous 6D rotation representation, and gripper width. A point-cloud encoder conditions a 1D U-Net diffusion model. Reported training uses AdamW at 7×10⁻⁵, batch size 612, 700 epochs, and one 24 GB RTX A5000; DDIM uses 50 steps. The baseline is trained for 300 epochs, so comparisons are not identical-training-budget ablations. [10], [80], [81], [82].

A.4 Effect model

The model predicts entity SE(3) transforms from the skill trajectory and point clouds. Small-motion labels use direct ICP; larger motions initialize registration from an assumed rigid gripper attachment, then refine it. The effect network shares the sampler’s diffusion architecture and training settings but outputs translation and rotation without gripper state. The paper nevertheless treats effects deterministically in planning. [71].

A.5 Spatial augmentation

Single-object skills use that object as anchor; two-object skills distinguish an anchor and moving entity. Anchor perturbations include ±10° yaw and ±5 cm translations. Moving objects additionally sample reachable planar locations, ±36° yaw, and ±5 cm translations, with matching changes to the initial gripper pose. Pseudo-camera projections remove occluded/out-of-view points through a z-buffer.

A.6 Planning algorithm

The preserved depth-first search instantiates language-generated skeletons, samples skill trajectories, checks connecting transit/move paths, predicts effects, and rejects collisions. It backtracks when a candidate cannot continue. cuRobo and nvblox construct collision-free connections from depth observations and robot configurations. The algorithm returns a complete instantiated sequence or failure. [72], [73].

Algorithm 1. Planning with learned composable skills.

Algorithm 1. Planning with learned composable skills.

B. Experimental details

B.1 Robot setup

One or two Franka Panda arms use RealSense RGB-D cameras. PyBullet inverse kinematics converts sampled end-effector poses into smooth joint sequences; Deoxys supplies joint impedance control. An AprilTag-board procedure aligns cameras and robot bases, using a wrist camera and known kinematics. [35], [83].

Figure A1. Single-arm and bimanual robot setups, RGB-D cameras, and GELLO teleoperation.
Figure A1. Single-arm and bimanual robot setups, RGB-D cameras, and GELLO teleoperation.

B.2 Task designs

The mug tree has six branches, five reachable; longer tasks hang three or four mugs. Bookshelf demonstrations combine coordinated grasping with pushing a preexisting book, and longer tasks place two books. Kitchen demonstrations wash and rack plates, with longer tasks handling two plates. Added obstacles block shelves, pegs, or rack space. The supplement leaves an author note requesting clarification of domain-specific partial-success definitions; the main paper provides the general key-step rule.

B.3 Baseline implementation

DP3-Long uses 2,048 points, 32-step actions, no proprioceptive input, 50 DDIM steps, and 300 training epochs. DP3-Short adds predicted task progress for skill termination. BLADE uses manually specified PDDL predicates, GPT-4o image-based predicate classification, an FF planner, and STACK’s learned samplers. This isolates the planning representation imperfectly because the compared systems also differ in conditioning and training. A placeholder for temporal-baseline details remains in the source. [10], [84], [85].

C. Additional results

C.1 Complete-task success

Table A1 counts fully successful trials out of ten, unlike the main partial-success table. STACK completes 9/8/3 Mug Tree SC/GC/LH trials, 9/8/7 Bookshelf trials, and 8/7/2 Kitchen trials. Thus 71.6% Kitchen LH partial success corresponds to only 2/10 fully completed trials. Training-configuration partial success is 90%, 88%, 72%, and 90% for Mug Tree, Bookshelf, Kitchen, and Cabinet.

Table A1. Fully successful trials out of ten; these are binary counts, not partial-success percentages.
Table A1. Fully successful trials out of ten; these are binary counts, not partial-success percentages.

C.2 Articulated cabinet

The Cabinet extension tests geometric effects of opening an articulated door. Reported average partial success is 85%, 80%, and 76% for SC, GC, and LH. These scores extend the geometric setting without addressing deformable-object dynamics.

Figure A2. Articulated cabinet task and demonstration steps.
Figure A2. Articulated cabinet task and demonstration steps.

C.3 Entity segmentation

Against expert labels, entity point clouds attain 97% mAcc and 91% mIoU across demonstration frames, including the cluttered Kitchen domain. This repeats the main segmentation result rather than providing an independent replication.

C.4 Alternative trajectory model

Replacing diffusion with a RoboMimic LSTM/GMM policy in Mug Tree fails to produce meaningful task performance with or without augmentation, although it can fit one demonstration. The authors attribute this to trajectory diversity. The limited ablation supports their model choice in this setup, not a general impossibility result for recurrent mixture policies. [86].

D. Failure analysis

The supplement’s prose says segmentation succeeded in all trials, conflicting with the main text’s 21.2% segmentation-failure share. Its failure figure also shows an incompletely detected faucet/handle, making detector-versus-segmenter categorization ambiguous. Both versions identify execution and perception weaknesses, difficult faucet-turning and mug-hanging skills, and 13.7% failure involvement from accumulated errors.

Figure A4. Per-skill success and failure-category breakdown, as reported in the supplement.
Figure A4. Per-skill success and failure-category breakdown, as reported in the supplement.
Figure A5. Imprecise bimanual book grasp and incomplete faucet detection/segmentation.
Figure A5. Imprecise bimanual book grasp and incomplete faucet detection/segmentation.

F. Additional limitations

The effect model describes geometry, not changes such as a plate becoming clean or water boiling. Correct non-geometric task logic therefore depends on the language model’s proposed skeletons. Integrating symbolic effects is suggested as future work. Generalization to new object instances/categories is not directly evaluated; similar shapes may transfer, but broader affordance variation would require more data or stronger 3D priors. [5].

G. Prompts

The six original prompt listings below preserve the source wording and examples. They specify segmentation, aggregation, refinement, entity naming, and candidate-plan generation. Prompt instructions belong to the paper’s experimental setup. Their heuristics include inconsistent minimum-duration thresholds and assumed cross-demonstration segment counts, which should be considered when interpreting claims of automatic discovery.

Listing 1. Prompt for Proposing Coarse Temporal Segments (Single Arm)

Proposes single-arm contact-rich segments using visual context, gripper state, torque, and examples.

You are a robot arm with 2 cameras and a gripper. The left
and right camera can see the environment from the third
person point of view.
The robot is trying to {task_name} once in this video.
To help you perform this task, here I have provided you
with the gripper width data. The gripper width data is a 1
D array that contains the gripper width at each timestamp.
When the gripper is closed, the robot could be grasping an
object. BUT THE ROBOT COULD ALSO BE NOT GRASPING ANYTHING
when the gripper is closed.
You must use the video frames and additional information
to verify if the robot is actually grasping an object or
not.
{gripper_widths}
To further help you perform this task, here I have
provided you with the robot’s overall joint torque data.
The joint torque data is a 1D array that contains the
joint torque at each timestamp.
The higher the joint torque, the more force the robot is
exerting.
{joint_torque}
Please look at the video content very carefully. Make sure
the segmentation you come up with is very fine-grained,
and have meaningful segments and describe each segment.
Your job is to segment the video into meaningful **
temporal segments **.
Here is the general instruction (with the detailed
instructions below):
**Step 1**: Rough Segmentation:
Each segment should be open gripper --> close gripper -->
open gripper. This is a new segment.
Each segment should be a dictionary with the following
keys.
- approach_start (format mm:ss):
A second or two before the robot arm comes into contact
with the target object / environment
OR a second or two seconds before the gripper closes
and marks the start of free-space motion.
Both of these cases are acceptable, but you should
choose the one that comes last.
- approach_end (format mm:ss):
the end time of the the robot arm fully grasped the
target and the gripper is closed.
This segment MUST END when the gripper is closed.
- interaction_start (format mm:ss):
- The start of the meaningful interaction, when the
gripper is holding / touching / releasing / grasping an
object.
- This can occur after a free-space motion period
following the gripper closure.
- This is when the gripper is first closing, and AFTER
FREE-SPACE MOTION IS COMPLETE IF THERE’S FREE-SPACE
MOTION.
- interaction_end (format mm:ss):
- The end of the meaningful interaction.
- If grasping, the moment AFTER the robot releases the
target object (the gripper is open).
- If interacting without grasping, after the gripper
opens and the robot is no longer in contact with the
target object.
- In some cases, the interaction_end may occur before
the gripper is fully open (if justified, please explain
).
Here are a few examples of the rough segmentation based on
gripper open and close signal, and the torque signals.
Example 1: pour tea
1. approach_start: just before the robot grasps the teapot
2. approach_end: after the gripper closes
3. interaction_start: just after the gripper closes
4. interaction_end: the gripper releases the teapot after
placing it on the table
First, watch the video from start to finish and determine
how many segments there are in the video by counting the
number of times the gripper opens --> closes --> opens.
** Detailed Instructions for Step 1 Rough Segmentation **
You must follow these important guidelines very very
carefully.
1. Watch the video from the beginning to the end first
before making any segmentations or before answering any
questions. This step is very important.
2. Count the number of times the gripper opens --> closes
--> opens. Let the count be N. The number of output
temporal segments should be at least N segments.
3. For each of the segments, explain WHAT and WHY the
robot is doing what it is doing. Consider the overall
context of the video.
4. Try not to include free-space motion in your
segmentation. This is very important. Free-space motion is
when the robot is holding the object the robot is grasing
is not in contact with the environment. In other words,
pure transport motion. If the robot is not in contact with
the environment the torque should be low.
If you determine the segment contains free-space motion,
but the torque is high during the segment, then the high
torque part is not free space motion, so you must include
this segment.
5. If the robot is trying to use the object in the robot
gripper, or the gripper itself to interact with some other
object, that is NOT free space motion.
6. If the segment begins with the robot grasping and
object, BE SURE TO INCLUDE THE GRASPING time segment in
your segment as well. And explain this.
7. Remember WHICH ARM YOU ARE LOOKING AT, and don’t let
the other arm distract you from returning the correct
segmentations.
8. The interaction start to end duration should be at
least 4 or more seconds. This is very important.
9. If the segment is grasping an object, interaction_start
should be RIGHT BEFORE the gripper closes!
**Step 2**: Segmentation Refinement:
For each segment, determine if it’s necessary to split the
segment into new chunks.
Usually, if a segment is longer than 15 seconds and
interacts with more than 1 object, then you should split
the segment.
If you should split, ALWAYS split the segment into THREE
NEW segments.
Here are a few examples:
Example 1:
If the segment is grasp the tea pot, pour tea into the cup
, and return the tea pot to the table, then you should
split this into three segments:
1. grasp the tea pot
2. pour tea into the cup
3. return the tea pot to the table
Example 2:
If the segment is grasp the plate from table, hold the
plate wash it under water, and place the plate onto the
drying rack, then you should split this into three
segments:
1. grasp the plate from table
2. hold the plate wash it under water
3. place the plate onto the drying rack
Example 3:
If the segment is pouring water from a bowl to another
bowl, and then placing the bowl back on the table, then it
should be split into three segments
1. grasping the bowl
2. pouring water from a bowl to another bowl
3. placing the bowl back on the table
where you shouldn’t split:
Example 4:
If the segment is place book onto the bookshelf, then it
SHOULD NOT be split, because placing the book onto the
bookshelf is a single action.
Example 5:
If the segment is turn on the light by flipping a switch,
then it SHOULD NOT be split, because flipping the light
switch is a single action.
Example 6:
If the segment is using the arm to wipe the table
repeatedly, then it SHOULD NOT be split, because wiping
the table is a single action.

Example 7:
If the left arm is grasp the plate and transporting it to
the dishrack, this should be split into
- grasping the plate
- transporting the plate
- and placing the plate, and make sure to capture to the
full insertion motion
This is very important to split this segment into these
meaningful segments
** Detailed Instructions for Step 2 Segmentation
Refinement **
- For each segment, determine if it’s necessary to split
the segment into new chunks. Look at the examples and
justify your reasoning by describing how it relates to the
examples.
- If you can confidently determine the robot is grasping
something as part of the rough segment, then the grasping
phase must be split into its own segment.
- Explain your reasoning on why to split it or not
- If a segment is long, and similar to the examples above
where splitting is reasonable, then it should probably be
split.
- Split the segment by providing a list of new
interaction_start and interaction_end timestamps.
**Output Validation**
- First, follow the instruction above and complete step 1
and step 2 very carefully. Consolidate all of your
segments.
- You are a reasoning model so make sure to reason. Please
explain your reasoning and think carefully step by step
before answering.
- For each segment you generate in the output, make sure
the segment is at least 4 seconds long.
- If no segment is found but you see the gripper state
changed in the video, you want to double check everything
from start to finish to re-propose segments
- Verify for each segment, from approach_start to
interaction_end, the gripper state changes from open to
close to open. This is very important
**Error Checking**
If no segments found, re-watch the video and re-evaluate
everything from the first instruction to the last
instruction.
** Output Format **
Your output should contain your reasoning, and 1 code
block with all of the segments in the following format:
No need to include approach and reset away segments.
‘‘‘
[
{
"interaction_start": "xxx",
"interaction_end": "xxx",
"skill_description": "description of the skill. The
description should only include the specific arm.
LEFT or RIGHT arm.",
"skill_name": "snake_case_name_for_the_skill"
},
...
]
‘‘‘

Listing 2. Prompt for Proposing Coarse Temporal Segments (Bimanual)

Identifies coordination requiring both arms, using arm overlays, contact, torque, and task context.

The robot is trying to complete the following task:
{task_name}
** Your Primary Objective: **
Temporally segment video into bimanual coordination phases
using sensor data and visual context.
**Robot Capabilities:**
- Two arms (left=black gripper, right=light green gripper)
with parallel jaw grippers
- One camera: Center camera provide third-person view
The left arm is masked in a light blue color, and the
right arm is masked in red light red color.
Remember this crucial information when looking at the
video.
This information can help you disambiguate between the
left and right arm.
**Data Stream Definitions:**
1. **Gripper Widths** (Left/Right):
- Closing/opening = object interaction (grasp/release).
2. **Joint Torque** (Left/Right):
- Large value might indicate forceful coordination.
- Synchronized torque spikes might indicate
collaborative manipulation.
Analyze the following data streams with these guidelines:
- Left gripper open / closed, in [MM:SS, o/c ... ] format.
o is for open and c is for closed. Here is the data: {
gripper_widths_left},
- Left torque, in [MM:SS, torque value ...] format. Here
is the data {joint_torque_left}
- Right gripper open / closed, in [MM:SS, o/c ... ] format
. o is for open and c is for closed. Here is the data: {
gripper_widths_right}
- Right torque, in [MM:SS, torque value ...] format. Here
is the data {joint_torque_right}
**Objective:**
Analyze a robot manipulation video and identify motion
segments that should be categorized as bimanual
coordinated tasks. Return timestamps in "mm:ss" format.
**Bimanual Coordination Manipulation Definition and
Guidelines**:
- When trying to determine if the motion is bimanual
coordinated manipulation, ask yourself the question, "
Is this possible without the second arm?" "Can I still
achieve the desired outcome with a single arm?"
- Look at the surrounding context of the video.
- Both arms must be actively contributing to the task
through the following criteria. These are suggestions,
not strict rules, not exhaustive list:
- both arms are in contact with the same object
- force-coupled motion (torque synchronization)
- and/or spatial coordination (working in shared
workspace region)
- and/or complementary roles (one stabilizes while
other manipulates)
- anything that can’t be done with a single arm
- Some examples of bimanual coordinated manipulation:
- Left arm holding a bowl while the right arm is
stirring the soup. Both arms are actively
contributing to the task of stirring the soup.
- The left arm is stabilizing the plate/book/object
while the right arm is trying to grasp the plate/
book/object.
- Some examples of not bimanual coordinated
manipulation:
- One arm is doing something while the other arm is
just holding an object, not really doing anything.
- MOVING the object around is not bimanual
coordinated manipulation. It’s free-space motion.
- DO NOT INCLUDE IF the two arms are MOVING AWAY from
each other
**Identification Guidelines:**
- **Start Time:** First frame showing bimanual coordinated
motion is starting (not exhaustive list).
- Increasing torque values on both arms is usually a
good indicator of bimanual coordinated motion.
- The start time should be the GRIPPER CLOSING TIME of
ONE OF THE ARMS, or when one of the arms comes into
contact with the target object.
- Both arms are APPROACHING the target object.
- You can give a 2 second buffer by subtracting 2
seconds from the start time, this is because you want
to include some of the approach motion.
- If the segment ends with the grippers all closed,
then the segment should start with when the grippers
are open.
- **End Time:** Last frame before either:
1) Object fully grasped or placed down
2) Transition to the next motion segment
3) There is no longer two arm coordinated motion. Only

one arm is doing moving motion. Pay attention to the
context of the video.
4) Torque values dropping significantly
5) The robot is clearly doing something else now. Pay
attention to the context of the video.
- **Minimum Segment Duration:** 3 seconds for meaningful
actions
- **Maximum Segment Duration:** 10 seconds. If the segment
is longer than 10 seconds, then it’s probably not a
bimanual coordinated manipulation.
**Processing Instructions:**
1. **Video Analysis:**
a) Perform full video scan (00:00 - end). When scanning
, as part of your output, explain very carefully not
just what you are seeing, and why you think the robot
is doing what it is doing, and why this motion
contribute to the overall goal of the task.
b) Identify all potential bimanual coordinated motion
segments. Find everything you possibly can, don’t miss
anything.
c) For each candidate segment, make sure to explain why
it is bimanual coordinated manipulation, and why the
robot is doing what it is doing, not just what you are
seeing.
d) For each segment, perform the per-segment evaluation
.
2. **Per-Segment Evaluation:**
For each candidate segment:
a) Review the motion definition very very carefully,
follow the definition make sure the segment is bimanual
coordinated manipulation.
First, look at the left arm. Then, look at the right
arm. Then look at them together.
Explain WHY it is bimanual coordinated manipulation,
and why the robot is doing what it is doing, not just
what you are seeing.
b) Check both arm gripper state, torque values.
c) Analyze object interaction context
d) Verify the segment is in fact bimanual coordinated
manipulation. If not, discard it. Do not include it in
the output.
e) Make sure the end time is not too late. We don’t
want to include too much free-space motion. The segment
shouldn’t be too long.
**Output Format:**
Return list of dictionaries with strict structure:
‘‘‘
[
{
"interaction_start": "mm:ss",
"interaction_end": "mm:ss",
"skill_name": "snake_case_name_for_the_skill",
"skill_description": "description of the skill."
},
...
]
‘‘‘
**Critical Requirements:**
1. Explicitly state conflict resolutions when:
- Arm movements are ambiguous
- Torque values border threshold
- Spatial relationships are unclear
2. Prioritize task context over raw motion metrics
3. There should usually be at least 1 segment. If there’s
no bimanual segment then you should really double check
everything.
Think extremely carefully, explain your reasoning step by
step. In your explanation, do not include code blocks.
Explain your reasoning in the output in great detail in
the output.

Listing 3. Prompt for Aggregating Proposed Temporal Segments

Selects and orders proposed segments without modifying timestamps, resolving overlap and minimum-duration rules.

You are watch a bimanual robot video.
**Scene configuration:**
- 2 arms with parallel jaw grippers (LEFT has BLACK
gripper)
- 1 camera: Center camera provide third-person view. This
is the view you are watching.
- Tabletop environment
- The robot is trying to {task_name}
- The left robot is masked in blue, and the right robot is
masked in red. Remember this crucial information when
looking at the video.
**Proprioceptive data as context:**
- Left gripper open / closed, in MM:SS : open/closed
format. {gripper_widths_left},
- Left torque, in MM:SS : torque value format. {
joint_torque_left}
- Right gripper open / closed, in MM:SS : open/closed
format. {gripper_widths_right}
- Right torque, in MM:SS : torque value format. {
joint_torque_right}
You might jointly use the video and the proprioceptive
data to determine if the temporal segments are correct.
**Temporal Segmentation Explanation:**
- You are provided with a list of temporal segments.
- Conceptually, a temporal segment is defined as the
period of time when the robot is doing meaningful
manipulation, towards the goal of the task: {task_name}.
- Each temporal segment is a dictionary containing at
least the following keys:
- "interaction_start": "mm:ss", the start time of the
segment.
- "interaction_end": "mm:ss", the end time of the
segment.
- There could be multiple temporal segments in the video.
- The temporal segments that you are given might contain
overlaps, might contain gaps, and might contain semantic
errors.
The temporal segments sorted in time:
{temporal_segments}
**Task:**
You will watch the video and the corresponding context
information from start to end.
You will look at each of the temporal segments and
determine if they are correct.
The goal is to return a new list of temporal segments that
are correct. If you look at the
new list of temporal segments, sequentially along with the
video, they should tell a complete story and make sense.
**Guidelines:**
1. Start by watching the video from start to end while
considering the sensor data as well as the temporal
segments.
2. For each temporal segment, think about and explain why
the robot is doing what it is doing. Then look at the
video as a whole, and think very very deeply about if this
temporal segment is truly correct or not.
3. Identify overlapping temporal segments.
If the overlapping amount is small, and it makes sense to
keep both segments, keep both segments. Especially if the
overlap is between two single arm segments.
If the overlapping amount is significant (3+ seconds),
then you need to make a decision on which segment to keep.
If the overlapping segments are one bimanual one single
arm, and the overlapping amount is greater than 2 seconds,
then you probably want to keep the bimanual segment.
you need to make a decision on which segment to keep.
Remember, if segments are overlapping, you can only keep
1.
4. DO NOT MODIFY the start and end time of the temporal
segments. Only determine if the temporal segment is
correct or not.
5. Look at the output temporal segments, and explain why
you made the decisions you made. Finally, order the
temporal segments by the start time.
6. Discard any segment that is less or equal to 3 seconds
long. If a segment is too short, that’s usually an
indicator that the segment is not meaningful and incorrect
.
7. Under absolutely no circumstances should you modify the
segment starting and end time. You should only determine
if the segment is correct or not.

8. The segments MUST BE LONGER THAN 2.5 SECONDS!! IF NOT
DISCARD IT!
9. The skill and skill description fields could be
inherited from the input, if the segment is correct. Read
the skill description and skill name fields very carefully
and understand them.
**Output Format:**
list of dictionaries in a code block with the following
keys:
"interaction_start": "mm:ss",
"interaction_end": "mm:ss",
"skill_name": "a short name for the skill, such as
pick_up_book_bimanul, or turn_off_faucet_left_arm, or
pick_up_mug. The name should be snake case"
"skill_description": "a description of this skill and how
this skill is used in the demonstration video. If this is
a single arm skill, only mention the arm that is being
used."
Final Instructions:
- Follow the above guidelines very very carefully. For
each segment, follow the guidelines EXACTLY and EXPLAIN
- The output should be a valid JSON object, with valid
explanations.
- Be sure to explain your reasoning in great detail.
Provide your reasoning steps in the output!

Listing 4. Prompt for Refining Aggregated Temporal Segments

Checks segmentation consistency across demonstrations and reproposes inconsistent segments.

Ultra-deep thinking mode. Greater rigor, attention to
detail, and multi-angle verification.
Start by outlining the task and breaking down the problem
into subtasks. For each subtask, explore multiple
perspectives,
even those that seem initially irrelevant or improbable.
Purposefully attempt to disprove or challenge your own
assumptions at every step.
Triple-verify everything. Critically review each step,
scrutinize your logic, assumptions, and conclusions,
explicitly calling out uncertainties and alternative
viewpoints. Independently verify your reasoning using
alternative methodologies or tools,
cross-checking every fact, inference, and conclusion
against external data, calculation, or authoritative
sources.
Deliberately seek out and employ at least twice as many
verification tools or methods as you typically would.
Use mathematical validations, web searches, logic
evaluation frameworks, and additional resources explicitly
and liberally to cross-verify your claims.
Even if you feel entirely confident in your solution,
explicitly dedicate additional time and effort to
systematically search for weaknesses,
logical gaps, hidden assumptions, or oversights. Clearly
document these potential pitfalls and how you’ve addressed
them.
Once you’re fully convinced your analysis is robust and
complete, deliberately pause and force yourself to
reconsider
the entire reasoning chain one final time from scratch.
Explicitly detail this last reflective step.
<task>
** Setup **
You are given a list of videos. Each video is a robot
demonstration.
The video contains a robot performing a task with goal {
task_name}.
** Input **
You are given {num_videos} videos. Associated with each is
a list of temporal
segements. A temporal segment is defined as a period of
time (mm:ss format) such that
the robot is doing something meaningful. For example, if
the video shows the robot
grasping a cup, then placing the cup on a shelf, you would
expect to have two temporal segments,
one for grasping the cup, and one for placing the cup on a
shelf.
The expected number of temporal segments for each
demonstration should be exactly the same
as the other demonstrations.
** Your Task **
Some of the videos may have inconsistent temporal segments
compared to others.
Detect which of the videos have inconsistent temporal
segments compared to others?
Think very carefully step by step before answering.
It is possible that all videos have consistent temporal
segments. In that case,
provide an empty list. But you still have to explain your
reasoning in great detail.
First, identify the videos that have consistent temporal
segments.
Second, understand what is the correct temporal
segmentation by looking at all of the input.
At least one or several of the videos should have the
correct temporal segmentation. Therefore
start with those videos and understand the task, videos,
context, and everything in the input prompt.
Third, re-propose the temporal segments for the videos
that have inconsistent temporal segments.
** Output **
1. Provide the list of videos that have inconsistent
temporal segments.
2. For each video that you believe that have inconsistent
temporal segments, do the following:
1. Re-propose the temporal segments for this video.
2. Check and verify the re-proposed temporal segments,
such that they are consistent with the other videos.
3. Explain in great detail why the re-proposed
temporal segments are consistent with the other videos
.
3. For the newly proposed temporal segments, provide them
in JSON format, in code.
‘‘‘json
[
open curly brackets
"video_id": the integer id of the video (remember
to use zero-indexing)
"segments": the list of the NEW temporal segments
close curly brackets
]
‘‘‘
** Verification and Check **
1. For each video that you believe that have inconsistent
temporal segments, verify that the re-proposed temporal
segments are consistent with the other videos.
2. If the re-proposed temporal segments are consistent
with the other videos, then you have successfully detected
the inconsistent temporal segments.
3. If the re-proposed temporal segments are not consistent
with the other videos, then you have failed to detect the
inconsistent temporal segments.
4. For each video that you believe that have consistent
temporal segments, verify that the temporal segments are
consistent with the other videos.
5. If the temporal segments are consistent with the other
videos, then you have successfully detected the consistent
temporal segments.
6. If the temporal segments are not consistent with the
other videos, then you have failed to detect the
consistent temporal segments.
** Final Instructions **
- Think very carefully step by step
- Provide your detailed reasoning
- Read every instruction carefully
- Watch all the videos and read the temporal segments
carefully before answering
</task>

Listing 5. Prompt for Naming Relevant Entities in Each Skill Segment

Names a mandatory target/anchor and an optional distinct moving object, with descriptive attributes.

You are given a list of videos of the robot doing a task.
These videos are
all demonstrating the same task {task_name}.
You are also given a list of temporal segments. One
segment per video. Please focus on the temporal segment,
but also
pay attention to the broader context of the video.
Your task is to identify the object of interest in the
temporal segment. The object(s) of interest is the object
that is
most relevant to the task and the temporal segment. You
must also identify the parameters for the object.
Here are some examples with explanations:
** Example 1 **:
Task: Grasping a orange cut that is sitting on the table.
Target object: cup
Target object parameters: orange
Moving object: None
Moving object parameters: None
Explanation: because the robot is just grasping the cup,
there is no moving object.
By the end of this segment, the robot is expected to be
grasping the mug.
** Example 2 **:
Task: placing a blue bow onto the wooden shelf
Target object: shelf
Target object parameters: wooden color
Moving object: bow
Moving object parameters: blue
Explanation: because the robot is placing the bow onto the
shelf, the bow is the moving object.
In this case, the robot should be clearly holding the bow
in its hand in the beginning of this segment.
** Example 3 **:
Task: grasping green bow using both robot arms from a
countertop before washing it in the bathroom
Target object: bow
Target object parameters: green
Moving object: None
Moving object parameters: None
Explanation: because the two robot arms are both working
together to grasp the bow, there is no moving object.
In this case, the robot should be clearly not holding
anything in the beginning of the segment.
By the end of the segment, the robot should be holding
something.
** Example 4 **:
Task: washing a black frying pan under the kitchen sink
faucet with running water.
Moving object: frying pan
Moving object parameters: black
Target object: sink faucet hand and handles
Target object parameters: silver
Explanation: because the robot is washing the frying pan,
the frying pan is the moving object.
We specify the target object as the sink faucet because
the robot is washing the frying pan under the faucet.
This is very important. No other object is relevant
besides the position of the faucet.
** Additional Instructions **:
- The object of interest names must be as descriptive as
possible.
- You must watch all of the videos and the temporal
segments before answering.
- For each skill, you must identify the target object, and
the moving object.
- There MUST be a target object.
- To determine the target object, focus on the object
that the position matters the most for the robot to
pay attention to. Look at the examples above carefully
.
- If the interaction is between the robot arm(s) and an
object, then there could
be no moving object. So it’s possible that there’s no
moving object!
- You must consider all of the videos and provide one
common proposal for one moving object (possibly none) and
one target object.
- The moving and target cannot be the same object.
Therefore, if the moving and target are the same, keep the
target object
and set the moving object to null.
** Output format **
Output the list of objects as a python code block.
‘‘‘json
open_curly_bracket
"moving": "object name",
"moving_parameters": "something about the object,
focus on color, shape position",
"target": "object name",
"target_parameters": "something about the object,
focus on color, shape position"
close_curly_bracket
‘‘‘
Watch the videos carefully and think step-by-step. You
must also provide your detailed reasoning.

Listing 6. Prompt for Generating Candidate Skill Skeletons

Generates at least two candidate plans using only discovered actions; geometric feasibility is checked later.

You are a helpful robot agent trying to generate a list of
possible task plans. Read this prompt very carefully
and following the instructions.
** Background Information **
You are in an environment with robot arm(s). The robot
arms is trying to following the human instruction to
complete
some tasks. You are given an video of the robot going
these tasks, as well as the timestamps where the robot
arms
are completing certain actions.
You are given {video_count} demo videos above. Each video
is a demonstration of the robot arms
** Viable Actions **
You are given a list of actions the robots can take. Each
action has a name, and a short description of the purpose
of such
actions. The format of the actions is a list of
dictionaries with the following keys:
"interaction_start": "MM:SS", the start time of this
specific action in the video
"interaction_end": "MM:SS", the end time of this specific
action in the video
"reasoning": "why this action is important...",
"skill_name": "snake case name for the action",
"skill_description": "description of the action"
** Viable Actions From Demonstrations **
You are provided with several demonstrations. You are
provided a list of actions for each demonstration.
Here is the list of actions in JSON format:
{actions}
** Completing a Task with the Robots **
The human is interested in completing this task {task}.
You must think about how you can complete this task
with the given set of actions. You need to look at the
demonstration video and the actions, and think about how
the actions can be used to complete the task.
** Plan Generation **
- First, you need to list out all of the unique actions
from all of the actions provided. You can merge actions
that are similar.
To perform this step, you need to look at the videos
carefully, and also correspond the actions to the videos.
Then identify the unique actions and merge them.
- Read the actions very carefully, understand each action.

- Watch the video and understand the task. Explain how
each action is used in the video.
- Think about the task very carefully. For each action
provided, explain how the action contributes to completing
the task.
- Generate a list of possible task plans with the given
set of possible actions. Each plan should be able to
complete the task upon plan verification.
- For each plan, explain why you generated this plan.
** Output Format **
- You must output a list of possible task plans.
- Each task plan must be a list of actions. The output
must be a JSON code block. the list should look like this:
‘‘‘
[
[ # this is the start of the first plan
"action_1",
"action_2",
...
],
[ # this is the start of the second plan...
"action_1",
"action_2",
...
],
]
‘‘‘
- Each action in the task plan must be a snake case name
for the action.
- You must also provide a short explanation of why you
generated this plan.
- There must only be 1 code block in your output.
** Additional Instructions **
- Please propose at least 2 task plans.
- Try to change things around, you can try to skip an
action, or repeat a sequence of actions, if it makes sense
.
- You must not use any other actions that are not provided
in the input.
- You must use the actions provided in the input.
- You must not use any other actions that are not provided
in the input.
- You must think extremely carefully before answering.
"""
Concept demonstration

Open image ↗