Overview
Project page. This edition follows the complete twenty-one-page, 15 June 2026 v1. All fifty-four illustrated task definitions, eight figures, twelve numbered tables and both diagnostic equations are retained. Author spelling and order follow the PDF title page.
At a glance
ATOM-Bench fine-tunes manipulation policies on atomic tasks, then tests whether they can reuse those skills in unseen task combinations on physical robots. Each of two platforms has fifteen atomic and twelve compositional tasks; 3,000 human demonstrations support adaptation and five policies undergo 2,700 main evaluation rollouts. Simple instruction grounding is generally stronger than precise motor execution, and compositional completion remains low even for the strongest adapted policy. The proposed scores diagnose gaps, but do not by themselves identify the causal fraction of failures due to composition.
1 Introduction
Successful task demonstrations and broad benchmark averages do not reveal whether a policy acquired reusable skills. ATOM-Bench controls which atomic tasks appear during fine-tuning and withholds their tested combinations. This supplies a more explicit adaptation/evaluation split than a general task-suite ranking.
The contribution is a paired physical benchmark, not a new control policy. Its controlled holdout concerns this fine-tuning set; the study does not establish that similar skills or combinations were absent from every foundation model’s pretraining data.

2 Related Work
Simulation suites such as VIMA, LIBERO and VLABench enable controlled variation; LIBERO-PRO and RoboHiMan probe generalization. Physical benchmarks including RoboArena, AutoEval, RoboChallenge and GM-100 emphasize ranking, scale or task diversity.
ATOM-Bench adds an explicit atomic-adaptation/composition-evaluation protocol. Its policy comparison includes π₀.₅, LingBot-VLA, GR00T N1.6, SmolVLA and Motus, under the authors’ fine-tuning recipes rather than a training-free test of base checkpoints.
3 ATOM-Bench
3.1 Task Factorization and Task Groups
Six motor families describe physical operations: pick-and-place, reorientation, pushing, stacking, pouring and articulated access. Seven instruction atoms describe color, shape, size, source relation, destination, counting and exclusion. Each platform instantiates these in eight motor tasks, seven instruction tasks and twelve withheld compositions.
Instruction atoms use a simple executable carrier action rather than a language-only test. The paired dual-arm tasks add roles, coordination and ordered stages. These are practical functional categories, not a proof that every task varies only one independent factor; for example, an access sequence can contain placing an object inside a drawer.
3.2 Data Collection
The single-arm setup uses a seven-joint Franka Panda, Robotiq 2F-85 gripper and left/front/wrist RealSense views. Cobot Magic uses two arms with left/right/front views and master-arm teleoperation, following the Mobile ALOHA hardware lineage. Franka demonstrations use a SpaceMouse.
One hundred demonstrations per atomic task yield 1,500 per platform and 3,000 total, recorded at 30 Hz. Collection takes about nine hours for Franka and 12.5 hours for Cobot Magic. Composition demonstrations are excluded from fine-tuning.
3.3 Metrics
SR counts complete task success. PSR averages the fraction of annotated intermediate score points achieved, so it can be high when a policy repeatedly completes only early steps. Each task uses ten shared physical test seeds and reported aggregates are task macro-averages.
AS is the mean isolated-task PSR over the atoms required by a composition, averaging representative tasks when an atom has several. CFS normalizes the positive AS-minus-composition-PSR gap by remaining process failure. AS is an empirical baseline, not a proven ceiling on composed success; CFS is an algebraic diagnostic, not an identified causal share. Sequential attrition, context changes and different score-point structures can create a gap without isolating a new composition mechanism. The PSR=1 case needs a denominator convention that is not specified.
CFS is computed per task before averaging, so applying the nonlinear formula to table-level mean AS/PSR need not reproduce the reported mean CFS. Transfer Gap is single-task SR minus joint-atomic SR: positive favors specialization and negative favors joint training.
Equations
4 Experiments
The experiments separate atomic acquisition, withheld composition, observed failure types and the effect of joint versus single-task adaptation. The main total equals five policies × two platforms × twenty-seven tasks × ten trials = 2,700 rollouts; the specialization and robustness studies are additional protocols.
4.1 Experimental Setup
For each platform, each policy is jointly fine-tuned on all fifteen atomic tasks. Its checkpoint is then tested on those tasks and the twelve withheld compositions. The single-task comparison trains a separate checkpoint on one hundred demonstrations of each task and is reported only for π₀.₅.
A mask overlay helps operators reproduce the same ten starting arrangements across policies, but placement is approximate rather than metrologically exact. These are physical scene seeds, not evidence of ten independent training runs. Model-specific optimization budgets and adaptation modes remain part of the comparison.
4.2 Atomic Skill Acquisition
π₀.₅ leads Table 1: instruction SR is 94.3% on Franka and 71.4% on Cobot Magic, versus motor SR 46.2% and 45.0%. Motus is generally second; LingBot-VLA slightly exceeds Motus on Franka motor tasks but falls behind on the dual-arm track. PSR often exceeds SR, exposing incomplete execution.
The qualitative pattern is clear—pick-and-place and simple attributes are easier than fine control, counting and exclusion—but the per-atom means differ between text and Figure 3. For pick_place/pour/push/reorient/access, the prose gives 74/20/16/11/9%, while the chart labels show 69/16/14/10/8%. Both are preserved rather than silently reconciled. Instruction tasks also use simpler carrier actions, so their stronger results do not isolate reasoning from motor difficulty.
Original selectable table
| Franka Panda | Cobot Magic | |||||||
|---|---|---|---|---|---|---|---|---|
| Motor | Instruction | Motor | Instruction | |||||
| Model | SR | PSR | SR | PSR | SR | PSR | SR | PSR |
| Pi0.5 | 46.2 | 56.2 | 94.3 | 95.7 | 45.0 | 72.0 | 71.4 | 83.2 |
| Motus | 36.2 | 46.2 | 67.1 | 78.7 | 35.0 | 59.3 | 50.0 | 67.5 |
| LingBot-VLA | 37.5 | 46.5 | 54.3 | 60.5 | 26.2 | 42.8 | 21.4 | 47.1 |
| GROOT N1.6 | 28.8 | 47.1 | 57.1 | 69.5 | 23.8 | 41.0 | 27.1 | 52.1 |
| SmolVLA | 17.5 | 29.8 | 11.4 | 31.2 | 10.0 | 32.2 | 5.7 | 29.3 |

4.3 Compositional Generalization
π₀.₅ has AS 83.3%/79.5% but compositional SR only 15.8%/16.7% on Franka/Cobot Magic; its compositional PSR is 30.4%/42.6%. Motus achieves SR 10.8%/7.5%, and the remaining policies are lower. Thus isolated partial-progress competence does not translate directly into reliable completion of these new combinations.
The reported CFS values are 73.7%/56.8% for π₀.₅ and about 49%/48% for Motus. These summarize normalized process gaps under the proposed metric, rather than proving that those fractions of rollouts failed solely because of composition. Comparison of AS with SR also mixes partial-progress and full-completion measures; Table 2 retains both PSR and SR for that reason.
Original selectable table

4.4 Failure Analysis
Failed rollouts are labeled motor failure, reference failure, constraint failure, safety abort or timeout. The authors report that motor/reference/timeout account for over 90%, without a complete per-category count table here. The examples distinguish selecting the right object but failing to reorient it from choosing the wrong object despite executing a grasp.
In the paired pouring example, the single arm succeeds while the dual-arm policy fails to hold the receiving bowl, preventing the later pour. This is a concrete dependency failure. Platform comparisons also change embodiments, action sequences and coordination demands, so they do not isolate an arm-count effect.


4.5 Atomic-transfer vs. Single-task Atomic Fine-tuning
Most Franka task pairs favor joint atomic training for π₀.₅, while Cobot Magic is mixed: instruction tasks often benefit but several motor tasks favor specialization. Shared grasp/move/place experience is a plausible explanation, not a separately isolated causal test.
The joint checkpoint sees 1,500 demonstrations across tasks, whereas a single-task checkpoint sees one hundred. The comparison therefore combines transfer with additional data and variation; it is not a matched-total-data verdict that multitask training is universally better. Figure 4 preserves every paired task point.
5 Conclusion
The benchmark shows that five adapted policies still struggle with precise physical operations and held-out skill combinations in this controlled tabletop setting. Its main value is making the training/task relationship and partial-progress measurements explicit, so failures can be inspected beyond a single overall success number.
6 Limitations
The study covers five policies and common tabletop operations, excluding deformables, tools, mobile manipulation and very long-horizon tasks. Manual placement, safety monitoring and score-point annotation remain expensive. Ten physical trials per task provide coarse 10-percentage-point task-level SR increments; uncertainty and training-seed variability are not separately characterized here.
Appendix A Task Design
Tables 3–8 retain all fifty-four task prompts, atom assignments and scene images. Franka uses m/i/x identifiers and Cobot Magic dm/di/dx. Motor tasks include two pick-place instances and two access instances; each track’s seven instruction tasks test attributes, relations, quantities or filtering.
Compositions pair motor operations with multiple constraints—for example, selecting exactly two red cubes, accessing a drawer before relation-based selection, or choosing a receiving bowl before pouring. The dual-arm counterpart may add meaningful requirements beyond coordination: dx9 stacks three specified cubes and dx10 adds color/shape/exclusion. Some listed counterparts, such as dx7/dx8, do not explicitly prescribe two-hand roles. Paired does not mean an otherwise identical task. The original motor-table column heading says Instruction_atom; its entries are motor labels and are retained as printed.
Original selectable table and full-resolution task images
| Task_id | Prompt | Instruction_atom | Image |
|---|---|---|---|
| m1 | Pick up the cube and place it into the basket. | pick_place |
|
| m2 | Pick up the ball and place it into the basket. | pick_place |
|
| m3 | Place the bottle upright on the tray. | reorient |
|
| m4 | Push the cube into the marked area. | push |
|
| m5 | Pour the coffee beans from the cup into the bowl. | pour |
|
| m6 | Open the drawer, then close it. | access |
|
| m7 | Stack the two cubes. | stack |
|
| m8 | Open the lid, then close it. | access |
|
Original selectable table and full-resolution task images
| Task_id | Prompt | Instruction_atom | Image |
|---|---|---|---|
| i1 | Pick up the blue cube and place it into the basket. | color |
|
| i2 | Pick up the triangular block and place it into the basket. | shape |
|
| i3 | Pick up the largest block and place it onto the tray. | size |
|
| i4 | Pick up the object to the left of the cylinder and place it into the basket. | source_relation |
|
| i5 | Pick up exactly two blocks and place them into the basket. | count |
|
| i6 | Pick up the red cube and place it into the right basket. | goal_destination |
|
| i7 | Place all non-red objects into the basket. | exclusion |
|
Original selectable table and full-resolution task images
| Task_id | Prompt | Motor_atom | Instruction_atoms | Image |
|---|---|---|---|---|
| x1 | Pick up exactly two red cubes and place them into the basket. | pick_place | color + count |
|
| x2 | Pick up the non-red object to the right of the cylinder and place it into the basket. | pick_place | source_relation + exclusion |
|
| x3 | Open the drawer, pick up the red object to the left of the cylinder, place it into the basket, then close the drawer. | access + pick_place | color + source_relation |
|
| x4 | Open the drawer, pick up exactly two blocks, place them into the right basket, then close the drawer. | access + pick_place | count + goal_destination |
|
| x5 | Place the two smallest bottles upright into the tray. | reorient | count + size |
|
| x6 | Place the green bottle upright onto the right tray. | reorient | color + goal_destination |
|
| x7 | Push the non-red block into the right marked area. | push | exclusion + goal_destination |
|
| x8 | Push the triangular block to the left of the cylinder into the marked area. | push | shape + source_relation |
|
| x9 | Stack the smallest red cube on top of the green cube. | stack | color + size |
|
| x10 | Stack the triangular block that is not purple on top of the cube. | stack | shape + exclusion |
|
| x11 | Pour the beans from the bean dish into the small bowl on the right. | pour | size + goal_destination |
|
| x12 | Pour the beans from the brown bean dish on the left into the bowl. | pour | color + source_relation |
|
Original selectable table and full-resolution task images
| Task_id | Prompt | Instruction_atom | Image |
|---|---|---|---|
| dm1 | Pick up the basket with the left hand and place the cube into the basket with the right hand. | pick_place |
|
| dm2 | Pick up the basket with the left hand and place the ball into the basket with the right hand. | pick_place |
|
| dm3 | Place the bottles upright on the tray. | reorient |
|
| dm4 | Push the two cubes into the marked area. | push |
|
| dm5 | Pick up the bowl with the left hand and pour the coffee beans from the cup into the bowl with the right hand. | pour |
|
| dm6 | Open the drawer with the left hand, place the block into the drawer with the right hand, then close the drawer with the left hand. | access |
|
| dm7 | Pick up one cube with the left hand and another cube with the right hand. First place the cube in the right hand at the target position, then stack the cube in the left hand on top. | stack |
|
| dm8 | Open the box lid with the left hand, place the block into the box with the right hand, then close the box lid with the left hand. | access |
|
Original selectable table and full-resolution task images
| Task_id | Prompt | Instruction_atom | Image |
|---|---|---|---|
| di1 | Pick up the basket with the left hand and place the blue cube into the basket with the right hand. | color |
|
| di2 | Pick up the basket with the left hand and place the triangular block into the basket with the right hand. | shape |
|
| di3 | Pick up the basket with the left hand and place the largest block into the basket with the right hand. | size |
|
| di4 | Pick up the basket with the left hand and place the object to the left of the red cylinder into the basket with the right hand. | source_relation |
|
| di5 | Pick up one cube with each hand and place exactly two cubes into the same basket, leaving the remaining cube outside. | count |
|
| di6 | Pick up one cube with the left hand and place it into the upper basket. Pick up one cube with the right hand and place it into the lower basket. | goal_destination |
|
| di7 | Place all geometric objects except the red cube into the basket. | exclusion |
|
Original selectable table and full-resolution task images
| Task_id | Prompt | Motor_atom | Instruction_atoms | Image |
|---|---|---|---|---|
| dx1 |
Pick up one red cube with each hand and place exactly two red cubes into the basket.
|
pick_place | color + count |
|
| dx2 |
Pick up the basket with the left hand and place the all non-red objects to the right of the cylinder into the basket with the right hand.
|
pick_place | source_relation + exclusion |
|
| dx3 |
Open the drawer with the left hand, pick up the red cube to the left of the cylinder with the right hand, place it into the basket, then close the drawer with the left hand.
|
access + pick_place | color + source_relation |
|
| dx4 |
Open the drawer with the left hand, pick up one block with each hand, place exactly two blocks into the right basket, then close the drawer with the left hand.
|
access + pick_place | count + goal_destination |
|
| dx5 |
Place the two smallest bottles upright into the tray.
|
reorient | count + size |
|
| dx6 |
Pick up the two green bottles lying down, one with each hand, and place them upright in the right target area, leaving the bottles of other colors in place.
|
reorient | color + goal_destination |
|
| dx7 |
Push the block that is not red into the right marked area.
|
push | exclusion + goal_destination |
|
| dx8 |
Push the triangular block to the left of the cylinder into the marked area.
|
push | shape + source_relation |
|
| dx9 |
Place the green cube on top of the largest blue cube with your left hand, and put the smallest red cube onto the green cube with your right hand.
|
stack | color + size |
|
| dx10 |
Pick up the triangular block with the left hand and pick up a cube that is not red with the right hand. First place the cube at the target position, then stack the triangular block on top of it.
|
stack | color + shape + exclusion |
|
| dx11 |
Pick up the smaller of the two bowls with the left hand, pick up the bean dish containing beans with the right hand, and pour the beans from the bean dish into that smaller bowl.
|
pour | size + goal_destination |
|
| dx12 |
Pick up the bowl with the left hand, pick up the brown bean dish with the right hand, and pour the beans from the brown bean dish into the bowl.
|
pour | color + source_relation |
|
Appendix B Physical Test Seed Placement
Before every rollout, the operator aligns objects with a semi-transparent reference image in the camera view. Figure 7 preserves both front and side examples. The method improves consistency across policies without claiming exact repeated poses or eliminating physical reset variability.

Appendix C Robustness to Language and Visual Perturbations
Robustness reuses the atomic-transfer checkpoint without adaptation and selects tasks it already performs well on. The reported experiment is specifically π₀.₅ on Cobot Magic, four atomic and two compositional tasks; it is not an all-model or all-task robustness survey.
Visual perturbations (Perturbed-Vis).
The visual change replaces the white tabletop cloth with brown, retaining objects, arrangement, lighting, task rubric and physical seeds. This is one controlled background shift, not a test of broad visual distribution changes.
Language perturbations (Perturbed-Lan).
One authored paraphrase per selected task changes verbs, role phrasing or sentence structure while preserving atom-defining words, left/right roles and temporal order. Table 9 keeps every original/paraphrased pair. Because key constraint tokens remain fixed, the test does not cover arbitrary language reformulation or new semantics.
Original selectable table
| Task | Original prompt | Perturbed-Lan prompt |
|---|---|---|
| di1 | Pick up the basket with the left hand and place the blue cube into the basket with the right hand. | Use your left hand to hold the basket, and use your right hand to drop the blue cube into it. |
| di2 | Pick up the basket with the left hand and place the triangular block into the basket with the right hand. | Hold the basket with the left hand; with the right hand, put the triangular block inside. |
| dm2 | Pick up the basket with the left hand and place the ball into the basket with the right hand. | Grasp the basket using the left hand, then drop the ball into it using the right hand. |
| dm6 | Open the drawer with the left hand, place the block into the drawer with the right hand, then close the drawer with the left hand. | Pull the drawer open with the left hand, put the block inside with the right hand, then push the drawer shut with the left hand. |
| dx1 | Pick up one red cube with each hand and place exactly two red cubes into the basket. | Use each hand to grasp one red cube, then place exactly two red cubes into the basket. |
| dx12 | Pick up the bowl with the left hand, pick up the brown bean dish with the right hand, and pour the beans from the brown bean dish into the bowl. | Lift the bowl with the left hand and the brown bean dish with the right hand, then tip the beans from the brown bean dish into the bowl. |
Results.
The four atomic tasks remain at 90–100%. Composition dx1 falls from 60% to 40% under both shifts; dx12 falls from 50% to 40% with the cloth change and 20% with the paraphrase. Across the six tasks, mean SR is 78.3% baseline, 76.7% visual and 71.7% language.
These are small samples with ten trials per condition, selected partly for prior success. They suggest brittleness in the two tested compositions, but do not prove that the atomic representation is generally disentangled or that paraphrasing is always a stronger stress test. Improvements on some atomic trials partly offset composition losses in the six-task average.

Appendix D SFT Configurations for Models in ATOM-Bench
All runs use H100 GPUs and official model code/recipes where possible. π₀.₅, LingBot-VLA, GR00T and Motus use full fine-tuning; SmolVLA uses LoRA. AdamW learning rates, schedules, batch sizes and epochs vary across models and, for several models, across platforms; Table 10 retains every setting.
Examples include three epochs for π₀.₅ on both platforms, three/six for GR00T and nine/five for SmolVLA. This is a practical recipe-based comparison, not equalized optimization compute. Joint adaptation uses 1,500 demonstrations per platform; the π₀.₅ specialization baseline uses one hundred per checkpoint.
Original selectable table
| Model | Base ckpt. | Mode | Optimizer | LR | Batch | Steps |
| Franka Panda | ||||||
| Pi0.5 | pi05_base | full | AdamW | 2.0e-5 (cosine, warmup steps 1000) | 256 | 3 epochs |
| LingBot-VLA | lingbot-vla-4b | full | AdamW | 2.5e-5 (cosine, warmup ratio 0.25) | 128 | 3 epochs |
| GROOT N1.6 | GR00T N1.6 | full | AdamW | 1.0e-4 (cosine, warmup ratio 0.05) | 256 | 3 epochs |
| SmolVLA | smolvla_base | LoRA | AdamW | 1.0e-4 (cosine, warmup steps 1000) | 256 | 9 epochs |
| Motus | Motus | full | AdamW | 2.0e-5 (linear, warmup steps 1000) | 64 | 6 epochs |
| Cobot Magic | ||||||
| Pi0.5 | pi05_base | full | AdamW | 2.0e-5 (cosine, warmup steps 1000) | 256 | 3 epochs |
| LingBot-VLA | lingbot-vla-4b | full | AdamW | 5.0e-5 (constant) | 256 | 3 epochs |
| GROOT N1.6 | GR00T N1.6 | full | AdamW | 1.0e-4 (cosine, warmup ratio 0.05) | 256 | 6 epochs |
| SmolVLA | smolvla_base | LoRA | AdamW | 1.0e-4 (cosine, warmup steps 1000) | 256 | 5 epochs |
| Motus | Motus | full | AdamW | 2.0e-5 (linear, warmup steps 1000) | 64 | 5 epochs |
Appendix E Atom Taxonomy and Design Rationale
E.1 Motor Atoms
The six motor families draw on Meta-World, FMB and manipulation motion taxonomies. Stacking is separated from transport because relative pose and stable contact matter; access groups opening, maintaining and closing articulated containers. These functional categories are useful abstractions rather than irreducible elementary motions.
Original selectable table
| Motor Atom | Definition | Design Rationale |
|---|---|---|
| pick_place | Grasp an object and place it into a target container or region. | Foundational tabletop operation and default carrier motor for many instruction atoms. |
| reorient | Change an object’s pose or orientation before completing placement, such as placing a bottle upright. | Tests object pose control beyond basic transport. |
| push | Move an object along a support surface to a target region without necessarily grasping it. | Captures canonical non-prehensile tabletop manipulation. |
| stack | Place one object on top of another object. | Tests relative-pose precision and contact stability. |
| pour | Transfer granular or liquid-like contents from one container to another. | Tests functional container use and material transfer. |
| access | Open, maintain, or close an articulated container, such as a drawer, box, or lid. | Tests articulated-object interaction and phase transitions. |
E.2 Instruction Atoms
Instruction factors draw on ReferItGame, unambiguous object descriptions, CLEVR and embodied benchmarks. Color, shape, size, spatial source relation and exclusion select what to manipulate; count specifies how many; goal_destination binds where to deliver it. Table 12 preserves the exact definitions and design rationale for all seven.
Original selectable table
| Instruction Atom | Definition | Design Rationale |
|---|---|---|
| color | Select the target object according to color. | Basic attribute for object-level referring expressions. |
| shape | Select the target object according to geometric shape. | Stable visual attribute for tabletop object selection. |
| size | Select the target object according to relative size, such as the largest or smallest object. | Tests comparative attribute grounding. |
| count | Manipulate an exact number of objects. | Tests set-level reasoning and multi-object selection. |
| exclusion | Select objects by excluding a specified attribute or category. | Tests logical filtering over candidate objects. |
| source_relation | Select the target object according to its spatial relation to another reference object. | Tests relation-based object reference. |
| goal_destination | Bind the action endpoint to one among multiple candidate destinations or target regions. | Tests binding between instruction and action endpoint. |