At a glance
Chain-of-Modality (CoM) extracts manipulation plans and action parameters from a human demonstration by analyzing muscle activity or audio, hand poses, and video in sequence. The language model then translates its analysis into calls to a supplied robot skill library. This exposes effort and timing cues that can be difficult to infer from images alone.
Experiments test four demonstration-analysis tasks and several physical robot settings using Gemini 1.5 Pro and GPT-4o. The benefits depend on modality, task, and model; the resulting robot programs remain open loop. The project page links videos and code.
The complete nine-page source contains five figures, two numbered tables and three program examples. It refers to an appendix for signal processing and robot APIs, but that appendix is absent from this arXiv version. The preserved code is illustrative source material, not a complete standalone robot implementation.
I INTRODUCTION
A manipulation video can show what moves while concealing how hard the demonstrator grasps, pushes, or strikes. Muscle activity and impact loudness supply extra cues about effort and action boundaries. CoM first uses those cues to segment a demonstration, adds fingertip motion to infer actions and directions, and finally uses images to identify the objects involved.
A second model call converts this analysis into a program using predefined perception and control APIs. These APIs provide the embodiment-specific execution layer. The paper demonstrates adaptation across two robot platforms and several object arrangements, rather than learning new low-level controllers directly from a video.
II RELATED WORK
Video understanding has progressed from action classification to temporal localization and task-plan extraction. CoM adds nonvisual effort signals to this analysis. It also builds on Code as Policies, where language models compose robot skills, while replacing a language-only task specification with a multimodal demonstration.
Other human-video methods learn affordances, trajectories, dynamics, rewards or video-conditioned policies. This paper instead investigates how prompting a pretrained model can combine modalities to recover a discrete plan and its parameters. Its available skills and their meanings are supplied in the prompt.
III Learning from Multimodal Human Videos
The pipeline has three stages: collect synchronized demonstration modalities, progressively interpret them with a vision-language model, and generate a Python program using the robot’s existing APIs. The same model family supplies analysis and code generation.
III-A Multimodal Human Demonstration Video
Each demonstration combines RGB frames, either armband electromyography (EMG) or recorded interaction sound, and vision-estimated fingertip positions. Muscle activation and audio loudness serve as proxies for effort and timing; they are not direct calibrated measurements of contact force. Hand pose gives motion cues that the general-purpose model may miss in raw frames.
III-B Chain-of-Modality
CoM analyzes the modalities sequentially and conditions each stage on the preceding analysis. In the bottle example, force peaks suggest repeated effort; fingertip motion identifies grasping, release and rotation direction; images identify the bottle and cap. The final analysis names actions, targets and parameters.
The prompt specifies data formats, available actions and parameters, and one example demonstration paired with an analysis. Its objects and full task plan differ from the test tasks, but it does demonstrate the primitive skills and expected output format. Comparisons with merged and separately processed inputs test the value of this progressive reasoning structure.
III-C Writing Robot Code
The model receives the analyzed action sequence plus API descriptions and emits executable calls. For bottle opening, it adds localization and approach steps, holds the bottle with one arm, and loops three times over grasping, counterclockwise twisting, releasing and clockwise reset with the other.
The plug example uses force-like parameters on a 0–100 API scale: the retained code grasps at 100, moves against a box at 20 to reorient the plug, and inserts at 100. Those numbers are API commands, not forces in newtons. The source snippets retain their original imports, names, comments and indentation; the plug snippet uses Move_to although its local import line lists Push_towards.
from skills import Grasp, Release, Twist, Find, Move_to # Based on video analysis and APIs, generate python code:
Move_to('left', Find('bottle'))
Grasp('left')
Move_to('right', Find('bottle_cap'))
for _ in range(3):
Grasp('right')
Twist('right', 'counterclockwise', 180)
Release('right')
Twist('right', 'clockwise', 180)from skills import Grasp, Push_towards, Insert
Grasp('right', 'plug', 100) # force range from [0, 100]
Move_to('right', 'box', 20) # rotate plug in-hand
Insert('right', 'power_strip', 100)III-D Implementation Details
Eight EMG channels are sampled at 200 Hz, downsampled to the camera’s 60 Hz rate, and reduced to the maximum channel value at each time step. Audio is reduced to loudness. HaMeR supplies fingertip pixel coordinates; prompts describe thumb and middle fingertips and normalized scalar effort signals.
For object localization, Gemini 1.5 Pro predicts a bounding box from an RGB-D observation and object name. Depth and camera calibration produce the corresponding point cloud, whose mean gives the target location. Predefined control functions then execute the generated API calls. The source does not include the promised appendix with fuller processing and API details.
IV Experiments
Evaluation separates understanding the demonstration from executing the generated program. Ablations vary the available modalities and how the model combines them; real-robot trials test task execution, altered objects or layouts, and cross-platform deployment.
IV-A Experiment Setups
Demonstration analysis covers pressing a cube, opening a bottle, inserting a plug and playing a drum, with ten test videos per task and three model queries per video. Metrics are exact plan-and-parameter accuracy and a longest-common-string similarity score. The paper does not fully specify the similarity normalization or report confidence intervals.
Input ablations remove images, effort signals or hand poses. Reasoning baselines merge inputs and/or outputs, separate each modality before a final answer, or use CoM’s progressive conditioning. All methods within a task receive the same example demonstration. Figure 5 reports the reasoning comparison on three tasks; Table I includes all four modality-ablation tasks.
Robot tests use 20 trials per reported condition: bottle opening on ViperX and KUKA, plug insertion, two marker conditions for board wiping, and drumming. Generalization varies bottle types, object positions, marker shapes and beats. The setup describes seven bottle types, six unseen; the later results text instead mentions eight unseen bottles, so the exact bottle inventory is inconsistent.
| Pressing Cube | Opening Bottle | Inserting Plug | Playing Drum | ||
|---|---|---|---|---|---|
| Gemini 1.5 Pro | Image-only | 0.00/0.68 | 0.00/0.32 | 0.00/0.80 | 0.00/0.31 |
| w.o. img | 0.00/0.00 | 0.00/0.45 | 0.00/0.00 | 0.00/0.00 | |
| w.o. force | 0.00/0.68 | 0.00/0.64 | 0.00/0.72 | 0.00/0.03 | |
| w.o. hand | 0.70/0.96 | 0.00/0.49 | 0.47/0.96 | 0.57/0.90 | |
| All | 0.67/0.92 | 0.37/0.75 | 0.53/0.93 | 0.80/0.96 | |
| GPT-4o | Image-only | 0.00/0.48 | 0.00/0.43 | 0.00/0.77 | 0.00/0.37 |
| w.o. img | 0.00/0.00 | 0.00/0.53 | 0.00/0.00 | 0.00/0.00 | |
| w.o. force | 0.00/0.13 | 0.00/0.20 | 0.00/0.55 | 0.00/0.22 | |
| w.o. hand | 0.43/0.76 | 0.00/0.54 | 0.30/0.81 | 0.34/0.84 | |
| All | 0.40/0.84 | 0.00/0.46 | 0.33/0.79 | 0.43/0.86 |
| Opening Bottle (ViperX) | Opening Bottle (KUKA) | Insert Plug | Wiping Board (red marker) | Wiping Board (blue marker) | Playing Drum | |
|---|---|---|---|---|---|---|
| Ours | 12/20 | 15/20 | 15/20 | 16/20 | 14/20 | 16/20 |
| Oracle | 16/20 | 20/20 | 18/20 | 20/20 | 16/20 | 20/20 |
IV-B Results
Sequential reasoning improves the comparisons in Figure 5. The authors report advantages over Sep-Sep exceeding 19% with Gemini and 17% with GPT, but do not specify whether these are relative or percentage-point aggregates. The introductory 60% versus 17% summary should not be read as a uniform result across both models and every task.
Table I shows zero exact accuracy without effort signals, although similarity can remain high. Full-modality Gemini achieves 0.67, 0.37, 0.53 and 0.80 across cube, bottle, plug and drum tasks; GPT-4o achieves 0.40, 0.00, 0.33 and 0.43. Hand pose is crucial for the successful Gemini bottle setting, but adding it does not improve every table entry. GPT still has zero exact bottle accuracy with every tested modality set.
Table II records 88 successes in 120 robot trials (73.3%), versus 110/120 (91.7%) for manually authored oracle code. This supports the paper’s rounded 73% and 92% summaries while exposing the gap from execution and perception limits. Bottle success is 12/20 on ViperX and 15/20 on KUKA. The oracle comparison gives an empirical reference for the supplied robot system, rather than a guaranteed performance ceiling.
V Conclusion and Limitations
CoM shows that separating and progressively combining modalities can help pretrained models recover manipulation plans and API parameters from one demonstration. The result still relies on an action library, example prompts, signal preprocessing and robot-specific control primitives.
The audio representation uses impact volume rather than pitch or frequency structure. Programs execute without feedback-driven replanning, limiting recovery from unexpected events. Extending audio representations and generating closed-loop programs are proposed future directions; the small task set and absent appendix leave broader transfer and implementation details unresolved.