Literature
Display
← All papers

ForceBand: Learning Forceful Manipulation with sEMG

Botao He · Zhi Wang · Linna Kuang · Ishaan Ghosh · Jitendra Malik · Cornelia Fermuller · Tingfan Wu · Jiayuan Mao · Ruoshi Liu · Haozhi Qi · Yiannis Aloimonos

Preprint posted · CoRL 2026 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Reading guide

Read this as force supervision from calibrated human wearables plus robot force feedback. The edition covers all seventeen pages and Appendices A–H, keeping thirteen complete figures, two numbered tables with every subtable/note, all symbolic definitions and 46 references. The current official release differs from the paper’s described backbone/data packaging; source-version details appear with the model notes. Original project videos are retained separately from evaluation counts.

At a glance

ForceBand estimates per-finger force from eight wrist sEMG channels and inertial signals, then labels human demonstrations for a robot policy that predicts motion and grip force together. A ten-hour collection and roughly fifteen minutes of user calibration provide supervision. The reported benefit is object-specific squeezing that aperture-only policies cannot reproduce. Target-task human demonstrations remove fingertip sensors after calibration; robot execution still uses fingertip force sensors and a controller. The headline 87% success needs the table’s separate pick/place and squeeze denominators to interpret.

System at a glance

The system joins wearable acquisition, force estimation, visual/motion retargeting and force-aware robot control. ‘Robot-data-free’ refers to learning from human task demonstrations; it does not remove human force-sensor calibration, perception/retargeting tools, robot hardware setup or execution-time force feedback. Force traces estimated from wrist signals are proxies for fingertip measurements, not direct contact-force readings.

Complete original Figure 1: ForceBand learns force-aware robot policies from human demonstrations with wrist sEMG. A human performs natural manipulation while wearing a muscle-aware surface electromyography (sEMG) wristband (left) . The EMG2Force model converts muscle signals into per-finger force traces (middle) , which are synchronized with human video to create force-enriched demonstrations. These demonstrations are retargeted to robot embodiments and used to train a force-aware policy that predicts both action and force trajectories for downstream manipulation (right) .
Figure 1: ForceBand learns force-aware robot policies from human demonstrations with wrist sEMG. A human performs natural manipulation while wearing a muscle-aware surface electromyography (sEMG) wristband (left). The EMG2Force model converts muscle signals into per-finger force traces (middle), which are synchronized with human video to create force-enriched demonstrations. These demonstrations are retargeted to robot embodiments and used to train a force-aware policy that predicts both action and force trajectories for downstream manipulation (right).

1 Introduction

Motion and video demonstrations omit contact loads that distinguish gripping from purposeful squeezing. The proposed alternative instruments the wrist during task collection, while using fingertip sensors beforehand to learn/calibrate the force mapping. Muscle-aware electrode placement and time/frequency modeling improve the force proxy; a retargeted policy transfers it to a parallel-jaw robot. The contribution is a supervised sensing-and-control pipeline, not evidence that arbitrary internet video alone reveals force or that calibration-free transfer is established.

Complete original Figure 2: ForceBand system architecture. Our method predicts per-finger force traces from wrist sEMG and IMU signals by combining time domain and frequency domain representations. In parallel, human videos are transformed into robot-compatible observations. A flow matching policy is then trained to predict both action and force trajectories for forceful robot manipulation.
Figure 2: ForceBand system architecture. Our method predicts per-finger force traces from wrist sEMG and IMU signals by combining time domain and frequency domain representations. In parallel, human videos are transformed into robot-compatible observations. A flow matching policy is then trained to predict both action and force trajectories for forceful robot manipulation.

2 Wearable Force Sensing Design

The wearable design balances signal quality, repeatability and cost, using commodity acquisition and a printable structure. The quoted approximately $300 is a supplier-dependent band cost, not the complete robot, cameras, calibration sensors and compute setup. Keep the sensing hardware, fingertip ground-truth hardware and robot force-feedback hardware distinct.

Design Objectives.

Accessibility means reproducing the wearable from listed parts and ordinary fabrication; accuracy comes from low-noise acquisition, bipolar measurement and anatomically chosen sites. Cost and replication are design objectives, while the experiments test a particular eight-channel configuration. Anatomical fitting and calibration still require setup effort.

Complete original Figure 3: Hardware design of ForceBand . Our method combines anatomically guided wrist sEMG sensing with an IMU to capture muscle and motion signals relevant to finger-level force. Fingertip force sensors are used during dataset collection and calibration to provide ground-truth force supervision, and are removed during target-task demonstration collection.
Figure 3: Hardware design of ForceBand. Our method combines anatomically guided wrist sEMG sensing with an IMU to capture muscle and motion signals relevant to finger-level force. Fingertip force sensors are used during dataset collection and calibration to provide ground-truth force supervision, and are removed during target-task demonstration collection.

2.1 Muscle-aware sEMG Wristband Design

An OpenBCI Cyton with an ADS1299 front end measures differential electrode pairs to suppress common-mode noise. Seven channels target finger-related muscle regions and one wrist flexion; placement is checked with gestures after donning. This layout captures correlated muscle activity rather than isolating each tendon or measuring external fingertip load mechanically. User anatomy, placement, motion and co-contraction can alter the mapping, motivating per-user calibration.

Original mathematical representation and force-fusion details

For the data-collection board, we adopt the OpenBCI Cyton, an open-source multi-channel biosensing board built around the ADS-1299 chip, which enables low-noise acquisition (0.14​μ​Vrms0.14\mu V_{\mathrm{rms}} and an SNR of 119.5119.5). The bipolar design, as shown in Figure 3B, places a differential electrode pair at each site to measure the local potential difference over the target muscle region, suppressing common-mode noise and motion artifacts while preserving signal fidelity during natural manipulation. Finally, as shown in Figure 3A, rather than distributing electrodes uniformly, we position them based on a non-uniform, anatomically guided pattern, by concentrating coverage over forearm regions that provide informative signals for grasp-related finger motions, especially thumb, index, and middle finger activation. This muscle-specific placement increases the information content per channel and reduces the number of sensors required to improve wearability and efficiency without sacrificing performance, and allows the system to learn fine-grained human finger interactions and transfer the resulting force-aware representations to robot manipulation. In total, we use eight channels: seven over muscles controlling finger movement and one over a muscle controlling wrist flexion. We validate correct positioning with a short series of calibration gestures after donning the band; the full electrode sites and calibration protocol are detailed in Appendix A. The custom hardware is also flexible for modular expansion, as shown in Appendix B.

2.2 Texture-Preserving Ground-Truth Fingertip Force Sensor

Thin-film force-sensitive resistors inside transparent finger cots supply labels for pretraining/calibration while preserving camera access to hand appearance. Wires route along the palm toward the wrist. These labels are sensor-dependent proxies for normal fingertip contact, not a full multi-axis contact-wrench/pressure distribution. Removing the sensors applies to subsequent human demonstration collection; the model is still trained and adapted with measured force.

3 EMG2Force: Dataset and Force Prediction Model

The ten-hour collection covers four action categories and gesture distributions, with four users disclosed in the limitations. Inputs are eight EMG plus ten IMU channels over five-second windows sampled at 250 Hz, or 1,250 samples. A one-dimensional encoder and a DINOv3 spectrogram branch feed a transformer that estimates all five finger traces over the same window. The text does not establish a causal low-latency streaming bound, complete training hyperparameters, or a subject/session-disjoint evaluation protocol.

The official code release checked on 11 November 2026 ships a DINOv2 checkpoint with previous-window force conditioning and describes 8.6 cleaned hours, 6.4 with video, plus teacher-forced held-out-session evaluation. These are later released-artifact details, not replacements for the paper’s DINOv3/ten-hour description or its reported figures. Stored data are described at 1 kHz while the paper’s model windows use 250 Hz. See the official release; the project page’s ‘coming soon’ dataset label lags that release.

Complete original Figure 4: Dataset statistics. A dataset of synchronized egocentric video, sEMG, IMU, and per-finger force. (A) Action distribution and (B) gesture distribution, spanning atomic grasps and in-the-wild interactions with varied object shape, weight, and size.
Figure 4: Dataset statistics. A dataset of synchronized egocentric video, sEMG, IMU, and per-finger force. (A) Action distribution and (B) gesture distribution, spanning atomic grasps and in-the-wild interactions with varied object shape, weight, and size.
Original mathematical representation and force-fusion details

Fingertip Force Prediction Model. As shown in Figure 2, the model takes as input sEMG time series 𝐗EMG∈ℝC×T\mathbf{X}_{\mathrm{EMG}}\in\mathbb{R}^{C\times T} and an IMU time series 𝐗IMU∈ℝD×T\mathbf{X}_{\mathrm{IMU}}\in\mathbb{R}^{D\times T}, where C=8C=8 is the number of EMG channels, D=10D=10 is the IMU dimension (3 acceleration, 3 gyroscope, 4 orientation), and T=5​sT=5s is the length of the time window. All sensors are sampled at 250​Hz250\,\mathrm{Hz}, so each window spans 12501250 samples. We first concatenate the two modalities along the channel dimension, 𝐗=[𝐗EMG;𝐗IMU]∈ℝ(C+D)×N\mathbf{X}=[\mathbf{X}_{\mathrm{EMG}};\mathbf{X}_{\mathrm{IMU}}]\in\mathbb{R}^{(C+D)\times N}, and compute a short-time Fourier transform (STFT), which applies Fourier transforms over sliding temporal windows to obtain a time-frequency representation: 𝐒=STFT⁡(𝐗)∈ℝ(C+D)×F×T,\mathbf{S}=\mathrm{STFT}(\mathbf{X})\in\mathbb{R}^{(C+D)\times F\times T}, where FF is the number of frequency bins and TT is the number of time frames. The spectrogram representation helps the network capture frequency features of muscle activity, and its 2-D image-like structure allows us to leverage pretrained vision encoders. As shown by the ablation in Appendix D, this spectrogram branch improves performance in our setup.

Original mathematical representation and force-fusion details

The time series input 𝐗\mathbf{X} and spectrogram 𝐒\mathbf{S} are then encoded separately. For the time series, we use a one-dimensional convolutional encoder, and for the spectrogram, we use a pretrained DINOv3 [27] encoder. The resulting features are concatenated and passed to a transformer decoder, which predicts the fingertip force Fftp∈ℝ5×NF_{\text{ftp}}\in\mathbb{R}^{5\times N} for all five fingers over the same temporal window. The estimated fingertip forces Fftp∈ℝ5×NF_{\text{ftp}}\in\mathbb{R}^{5\times N}, paired with human video, are used to train robot policies.

4 Force-Aware Policy Learning from Human Demonstrations

Human hand/object tracking becomes an end-effector pose, aperture and entity-relative scene representation; arm inpainting and virtual-gripper rendering reduce the appearance gap. A flow-matching policy predicts fifty-step chunks with position, six-dimensional rotation, aperture and scalar force: eleven action components. Human thumb/index force averages become the robot grip-force channel, so retaining five-finger sensing does not imply the demonstrated robot independently controls five fingers. At execution, measured robot force closes the loop through the controller described in Appendix F.

Original mathematical representation and force-fusion details

From Video to Embodied Observations. We retarget per-frame human hand-object trajectories into an embodiment-agnostic representation that a parallel-jaw robot can execute. Hand keypoints define an SE⁡(3)\mathrm{SE}(3) end-effector pose Te​eT_{ee} and a 1-DoF aperture gg, while each manipulated object is tracked and lifted to a 6-DoF pose. On the visual side, we segment and inpaint the human arm, then overlay the virtual gripper and tracked object keypoints, producing an RGB observation that hides human appearance while preserving scene layout. Pipeline and tool details are given in Appendix C.

Original mathematical representation and force-fusion details

Flow Matching with Force-Augmented Action. The visual frame and a set of spatial-relation tokens jointly condition a transformer decoder that predicts an action chunk over a K=50K{=}50-step horizon. We augment each action with grip force as an additional prediction target: at=[p;r6​D;g;f]∈ℝ11a_{t}=[\,p\,;\,r_{6D}\,;\,g\,;\,f\,]\in\mathbb{R}^{11} where pp is the 3-D end-effector position, r6​Dr_{6D} is a continuous 6-D rotation representation, gg is the grasp aperture, and ff is the desired grip force. We train the decoder with conditional flow matching [17]: it regresses a velocity field that transports a Gaussian prior to the target action under per-component weights for pose, grasp, and force. Because force appears in both the policy input and prediction target, the network learns a closed-loop relationship between the current contact state and future force commands, rather than treating force as a passive observation.

5 Experiments

The experiments separately assess electrode placement, force/contact estimation and force-aware task execution. Read each comparison with its own data and metric. Table1 explicitly warns that the channel-masking ablation and separate placement datasets are not directly comparable. Policy success separates pick/place from squeeze; force-estimation errors do not themselves prove task safety or transferable absolute force calibration.

5.1 Ablation on Electrode Placement

Masking channels in the existing dataset reduces MAE from 1.89 N with one channel to 0.85 N with eight. The even-versus-muscle-aware comparison instead uses two separately collected thirty-minute datasets under the same ten-object/gesture protocol. There, 0.94→0.77 N is about an 18.1% MAE reduction; RMSE changes 1.77→1.33 N. The layout effect is not a paired simultaneous-electrode measurement, and uncertainty/session split details are not supplied. Do not compare 0.85 and 0.77 as the same dataset.

Complete original Table 1: Average force prediction errors with different electrode placements., all values and footnotes.
Table 1: Average force prediction errors with different electrode placements.
Accessible table, exact original values and notes
Hardware MAE (N) RMSE (N)
1 ch 1.89 3.69
2 ch 1.76 3.06
4 ch 0.93 2.00
8 ch ours 0.85 1.92
Hardware MAE (N) RMSE (N)
8 ch even 0.94 1.77
8 ch ours 0.77 1.33

Note: The left and right sub-tables should not be directly compared as they are using different data.

5.2 Contact and Force Estimation

Contact uses a 0.8 N threshold. Figure5 reports hand MAE 0.80 versus FEEL’s 1.58 N, and RMSE 1.36 versus 2.80 N—about 49% and 51% lower respectively. Per-finger force errors favor sEMG most on weak/occluded contacts. Precision–recall area matters because positive prevalence drops to 7% for the pinky.

The prose gives ring PR AUC 0.763/0.398 and pinky 0.590/0.314, whose ratios are about 1.92 and 1.88, despite claiming both exceed twofold. Figure5 instead labels the corresponding blue bars 0.38 and 0.27 and orange bars 0.76 and 0.59. Preserve both source versions rather than treating them as interchangeable. These contact metrics and selected qualitative traces do not establish fully characterized cross-user accuracy or external force generalization.

Complete original Figure 5: Quantitative force estimation results.
Figure 5: Quantitative force estimation results.
Original mathematical representation and force-fusion details

Baselines. We compare against two vision-based estimators. VISOR-HOS [7] is pre-trained on EPIC-KITCHENS; we evaluate both the released checkpoint (raw) and a version fine-tuned on our data (ft.). FEEL [8] trains a 2-layer MLP on frozen DINOv3 [27] features for binary contact detection (BCD) and per-finger force regression. The contact threshold is 0.80.8 N for all binary metrics.

Original mathematical representation and force-fusion details

Results (Fig. 5). ForceBand outperforms every vision baseline on hand-level BCD, and on hand-level force regression sEMG roughly halves both error metrics, as shown in hand-level results (Fig. 5 A,B). The advantage compounds at finger level (Fig. 5 C,D), where sEMG recovers load on muscles a camera cannot see. We report PR AUC, the area under the precision-recall curve, to summarize contact-detection quality under class imbalance. On the ring (0.7630.763 vs. 0.3980.398) and pinky (0.5900.590 vs. 0.3140.314), ForceBand exceeds FEEL by more than 2×2\times and the prevalence-matched random baseline by over 6×6\times, with consistent gains across all five fingers. PR AUC is the more informative metric in this imbalanced setting: positive prevalence falls from 45%45\% (thumb) to 7%7\% (pinky), and ROC AUC compresses above 0.850.85, masking precisely the regime in which vision breaks down. Overall, vision provides useful contact cues, but it cannot reliably infer hidden force magnitudes. The wrist-worn sEMG band complements vision by capturing muscle activity directly, producing per-finger force traces suitable for downstream force-aware policy learning.

5.3 Force-aware Policy Learning

The UR-5/Robotiq setup uses fifteen human demonstrations per object and ten evaluations per object/condition, across nine listed objects. Table2 reports separate pick/place and squeeze counts; chips have no squeeze task. Summing those entries gives ForceBand 80/90 pick/place and 72/80 squeeze, versus binary gripper 84/90 and 0/80. Thus force-aware control supplies squeezing but does not dominate binary control on pick/place alone. The headline 87% is retained as reported, without inventing a joint-success aggregation.

Squeeze success is judged by distinguishable object-specific force behavior, rather than a fully specified numerical force-profile tolerance. The project comparison marks small mustard, chocolate and ketchup as an OOD sequence; the paper’s fifteen-demonstrations-per-object wording leaves its precise training split recipe unclear. Uncertainty, repeated training seeds and per-shift trial counts are not fully enumerated. The physical policy uses four fingertip force sensors, a fixed 5 N pre-grasp adjustment with the policy paused, then PD force tracking; these controller aids matter when interpreting human-only policy learning and transfer.

Complete original Table 2: Policy success on pick, squeeze, and place tasks., all values and footnotes.
Table 2: Policy success on pick, squeeze, and place tasks.
Accessible table, exact original values and notes
Object Mustard Face Wash Toothpaste BBQ Small Chips BBQ Mustard Small Chocolate Ketchup
Weight (g) 650 280 176 290 43 580 244 210 410
Grasp Width (mm) 49 ∼\sim 54 18 ∼\sim 58 9 ∼\sim 37 54 ∼\sim 56 1 ∼\sim 68 67 ∼\sim 72 37 ∼\sim 40 19 ∼\sim 22 44 ∼\sim 46
Continuous gripper 7-0 6-4 4-2 6-0 3-N/A 8-0 3-0 0-0 7-0
Binary gripper 10-0 10-0 8-0 9-0 10-N/A 7-0 10-0 10-0 10-0
ForceBand 10-10 10-10 9-9 8-8 7-N/A 6-6 10-10 10-9 10-10

Note: Each entry reports pick-and-place and squeeze success out of 10 trials. N/A indicates that squeezing is not required for the task. Grasp Width denotes the gripper aperture range from initial contact to 20N of applied force.

Complete original Figure 6: Force-aware robot policy rollouts. ForceBand predicts object-specific force for pick, squeeze, and place tasks across objects with grasp widths from 1 to 72 mm, including both ID and OOD objects. In each force plot, different background colors indicate different task stages: pick, squeeze, and place. Across objects, the policy produces distinct peak forces from 3.2 N to 19.3 N, enabling squeeze behaviors that baselines fail to reproduce.
Figure 6: Force-aware robot policy rollouts. ForceBand predicts object-specific force for pick, squeeze, and place tasks across objects with grasp widths from 1 to 72 mm, including both ID and OOD objects. In each force plot, different background colors indicate different task stages: pick, squeeze, and place. Across objects, the policy produces distinct peak forces from 3.2 N to 19.3 N, enabling squeeze behaviors that baselines fail to reproduce.

6 Related Work

The related work contrasts visual contact inference, instrumented gloves, wrist EMG control and human-video policy transfer. Visual ambiguity and glove occlusion motivate the wearable, while prior force-supervised robot learning provides context. ForceBand combines those ideas with a calibrated estimator; its comparisons do not prove that every vision method fails under every object or that every wearable removes calibration.

7 Conclusion

The tested pipeline shows that wrist muscle signals can add useful contact-force supervision to human demonstrations, particularly for deliberate squeezing. The conclusion remains conditional on per-user adaptation, tracking/retargeting quality, instrumented robot feedback and the selected object tasks. A force prior can guide how hard to act without reproducing direct sensing accuracy or establishing broad deployment guarantees.

Original mathematical representation and force-fusion details

We presented ForceBand, a low-cost wrist-worn sEMG system that turns natural human manipulation into force-enriched demonstrations for robot learning. By combining muscle-aware electrode placement, an IMU, and a spectrogram-augmented EMG2Force model, ForceBand recovers per-finger contact forces that vision cannot reliably infer, especially under occlusion. After a short calibration, demonstrations are collected with only ForceBand and video, and the force-enriched demonstrations are retargeted to a robot embodiment to train a force-aware policy. Muscle-aware placement reduces force error by 18%18\% over an evenly spaced eight-channel layout, EMG2Force more than halves the error of vision-based baselines, and the resulting policy achieves object-specific squeeze control that motion-only and gripper-only baselines cannot reproduce, transferring even to OOD objects. Overall, wrist sEMG offers a scalable and unobtrusive path to adding force supervision to human demonstration data for forceful robot manipulation.

Limitations and Discussion

Absolute force is less accurate than direct sensing, and the present four-user system calibrates every user. Scaling to remove calibration or using standardized sensor-free fixtures is proposed rather than demonstrated here. Fingertip labels are still needed for current calibration. Subsequent robot interaction or reinforcement learning is suggested as a way to reduce residual error, not part of the reported human-only training result.

Acknowledgments

The acknowledgments credit Jiaxi Zheng, Eadom Dessalene and Levi Burner for support and feedback; this section adds no separate experiment or validation claim.

Original ForceBand demonstrations

These twenty-one unchanged original project videos and the hardware still preserve the authors’ visual demonstrations, including the complete presentation and human/robot paired clips. They illustrate setup, signals, calibration, task execution and selected shifts. Their visual labels do not add trials to Table2, establish a causal estimator latency, or remove the paper’s execution-time force-feedback hardware.

Human toothpaste squeeze demonstration
Robot toothpaste squeeze rollout
Human chips pick-and-place demonstration
Robot chips pick-and-place rollout
Human BBQ-sauce squeeze demonstration
Robot BBQ-sauce squeeze rollout
Original project teaser and human-to-robot overview
Original ForceBand system pipeline
Complete original project presentation
Original eight-channel hardware demonstration
Original EMG2Force model demonstration
Original EMG2Force pretraining example 1
Original EMG2Force pretraining example 2
Original EMG2Force pretraining example 3
Original EMG2Force pretraining example 4
Original multimodal pretraining dataset overview
Original calibration, collection and robot policy deployment workflow
Original contact and force-estimation results
Original electrode-placement comparison
Original force-aware policy success comparison
Original background, object, lighting and distractor examples
Original 16-channel expansion hardware photograph
Original 16-channel expansion hardware photograph

Appendix

Appendices A–H retain anatomical electrode sites, modular expansion, exact embodiment/force-token construction, model ablations, calibration workflow, robot sensors/controllers, qualitative traces and distribution-shift examples. These details are necessary to interpret the main paper’s transfer and sensing claims.

Appendix A Electrode Placement Details

The eight-site layout spans thumb, multi-finger and wrist flexor/extensor regions with a ulnar-styloid reference, skin cleaning and gesture verification. It is anatomy-guided placement with correlated muscle fields, not a one-electrode-per-independent-finger sensor. The complete original site diagram and placement definitions remain; reproducing the layout and calibration is part of the method.

Complete original Figure 7: Electrode placement details.
Figure 7: Electrode placement details.

Appendix B Hardware Extensibility

A second daisy-chained acquisition module can extend the band to sixteen channels. That is a hardware extensibility demonstration; the main accuracy and robot results use eight channels. A larger layout would require corresponding training/calibration, so no unreported sixteen-channel accuracy improvement is inferred.

Complete original Figure 8: Hardware extensibility through daisy chaining. The 8-channel acquisition configuration used in this work can be extended by cascading a second acquisition module through a daisy-chain interface, forming a 16-channel configuration. This modular design supports denser and task-specific electrode layouts without redesigning the overall wearable platform.
Figure 8: Hardware extensibility through daisy chaining. The 8-channel acquisition configuration used in this work can be extended by cascading a second acquisition module through a daisy-chain interface, forming a 16-channel configuration. This modular design supports denser and task-specific electrode layouts without redesigning the overall wearable platform.

Appendix C Details of Spatial-Relation Tokens and Embodiment Transfer

Aria hand keypoints define thumb/index midpoint position, a Gram–Schmidt orientation and aperture. Grounding DINO, SAM2, CoTracker and Orient-Anything supply object tracks/poses; LaMa inpaints the arm. Entity tokens express hand pose relative to each object and attach the thumb/index force average to the hand token. Tracking errors, geometry, camera frames and finger-to-gripper mapping can affect the learned force/action relationship. This is a structured reconstruction pipeline, not raw-video-only learning.

Original mathematical representation and force-fusion details

From video to embodied observations. Per-frame 3-D hand keypoints from Aria MPS [9] are retargeted to a parallel-jaw end-effector. The midpoint of the thumb and index fingertips defines the gripper position; the metacarpophalangeal (MCP) joints—which remain well-separated through the full pinch cycle—define a Gram–Schmidt orientation frame; and the normalized thumb–index distance gives a 1-DoF aperture. Each frame thus produces an SE⁡(3)\mathrm{SE}(3) end-effector pose Te​eT_{ee} and a grasp scalar gg that match what a parallel-jaw robot can execute. Each tracked object is detected with Grounding DINO [18], segmented with SAM2 [26], tracked across frames with CoTracker [14], and lifted to a 6-DoF pose using Orient-Anything [38]. On the visual side, we segment the human arm with SAM2 and inpaint it with LaMa [31], then overlay the virtual gripper together with the tracked object keypoints into the inpainted image, producing an embodiment-agnostic RGB observation that hides the human appearance while preserving scene layout.

Original mathematical representation and force-fusion details

Spatial-relation tokens with force fusion. We encode the scene as a set of per-entity tokens, one for each hand and each tracked object. For entity kk the token concatenates (i) an entity-type indicator, (ii) the entity’s 6-DoF pose in a shared scene frame, (iii) the hand pose expressed in the entity’s own local frame—so the sub-vector evolves with the manipulation phase as the hand approaches, contacts, or transports the entity—and (iv) a per-entity scalar (grasp state for hands, a sentinel for objects). This construction makes every token explicitly carry the spatial relation between an entity and the manipulator rather than treating them as independent points. We attach an extra force channel to the hand token only: the 1-D grip force ftf_{t}, taken as the average of the thumb and index forces in FftpF_{\text{ftp}} and time-aligned with the visual frame. Force is bound to the hand because it is physically a quantity the hand exerts on objects; placing it on the same token that carries the gripper pose lets the policy condition every action on the current contact state.

Appendix D Ablation Study on EMG2Force Model

The full time/frequency/IMU estimator has 0.92 N MAE in this ablation; removing spectrogram features raises it to 1.14 N and removing inertial inputs to 1.02 N. The variants share the stated evaluation protocol, but no confidence intervals or repeated-seed spread are given. These numbers belong to this ablation and should not silently replace the main force-estimation values from a different comparison.

Complete original Figure 9: We compare the full model against variants that remove either the spectrogram branch or IMU input. The full model achieves the lowest MAE, while removing the spectrogram branch or IMU input increases force prediction error. This shows that frequency-domain sEMG features and motion cues provide complementary information for fingertip force estimation.
Figure 9: We compare the full model against variants that remove either the spectrogram branch or IMU input. The full model achieves the lowest MAE, while removing the spectrogram branch or IMU input increases force prediction error. This shows that frequency-domain sEMG features and motion cues provide complementary information for fingertip force estimation.

Appendix E Details of the Three-Step Deployment

About fifteen minutes of paired EMG/IMU/fingertip-force data adapts the estimator to a user. Target-task demonstrations then remove human fingertip sensors, followed by retargeting and policy training. The final robot uses its own force feedback. Fifteen minutes describes calibration, not total policy-training/deployment time, and the current recipe is not calibration-free or robot-force-sensor-free.

Complete original Figure 10: Details about the three-step deployment.
Figure 10: Details about the three-step deployment.

Appendix F Details of the Experimental Setup and Controllers

Four Paxini sensors on the Robotiq fingertips provide timely force measurements that the gripper’s built-in feedback lacks. A predicted close command pauses the policy while the controller establishes 5 N pre-grasp contact, then a proportional–derivative controller adjusts aperture to track the predicted squeeze/place force. This hybrid contact-establishment routine is part of the result. Its delay, gains, force calibration and failure tolerances are not fully benchmarked here; policy learning alone is not responsible for stable initial contact.

Complete original Figure 11: Robot fingertip force sensing and force-control procedure. Left: because the Robotiq gripper does not provide sufficiently precise and timely fingertip force feedback, we attach four Paxini force sensors to the gripper fingertips. Right: when the policy predicts a close command, we pause execution for a short adjustment period and regulate the gripper to a 5 N pre-grasp force. After this stable contact is established, the policy resumes and a PD controller tracks the predicted force trajectory during the squeeze and place stages.
Figure 11: Robot fingertip force sensing and force-control procedure. Left: because the Robotiq gripper does not provide sufficiently precise and timely fingertip force feedback, we attach four Paxini force sensors to the gripper fingertips. Right: when the policy predicts a close command, we pause execution for a short adjustment period and regulate the gripper to a 5 N pre-grasp force. After this stable contact is established, the policy resumes and a PD controller tracks the predicted force trajectory during the squeeze and place stages.

Appendix G Qualitative EMG2Force Predictions on the Pretraining Dataset

The montage’s overlaid traces are model estimates; only the bottom comparison explicitly pairs predictions with measured ground truth. They illustrate timing and magnitude trends across actions and fingers, with some smoothing and temporal offsets on weak contacts. Selected traces do not substitute for a complete held-out error distribution or demonstrate calibrated uncertainty.

Complete original Figure 12: Qualitative EMG2Force predictions on the pretraining dataset. We show representative examples from the multimodal pretraining dataset, covering diverse action categories, gesture types, objects, and in-the-wild interactions. All force curves overlaid on the images are EMG2Force estimates from ForceBand signals , rather than direct fingertip force-sensor measurements. The bottom row shows a representative comparison between estimated force and ground-truth fingertip force, illustrating that the model captures the main timing and magnitude trends of contact, squeezing, and release across fingers.
Figure 12: Qualitative EMG2Force predictions on the pretraining dataset. We show representative examples from the multimodal pretraining dataset, covering diverse action categories, gesture types, objects, and in-the-wild interactions. All force curves overlaid on the images are EMG2Force estimates from ForceBand signals, rather than direct fingertip force-sensor measurements. The bottom row shows a representative comparison between estimated force and ground-truth fingertip force, illustrating that the model captures the main timing and magnitude trends of contact, squeezing, and release across fingers.

Appendix H Generalization Test

Background, object, lighting and distractor examples preserve the pick–squeeze–place task sequence, while force magnitude changes with visual appearance. This is evidence of qualitative task-structure transfer and remaining force sensitivity, not invariant force control under all shifts. The paper supplies examples without a complete quantitative per-condition test denominator or force-tolerance success breakdown.

Complete original Figure 13: Generalization test under visual and object-level distribution shifts. We evaluate the learned policy on novel backgrounds, novel objects, extreme lighting, and visual distractors. In all cases, the robot follows the correct pick-squeeze-place trajectory and preserves the three-stage task structure. Background and texture changes can still affect the precise force magnitude, suggesting that visual appearance contributes to force prediction.
Figure 13: Generalization test under visual and object-level distribution shifts. We evaluate the learned policy on novel backgrounds, novel objects, extreme lighting, and visual distractors. In all cases, the robot follows the correct pick-squeeze-place trajectory and preserves the three-stage task structure. Background and texture changes can still affect the precise force magnitude, suggesting that visual appearance contributes to force prediction.
Concept demonstration

Open image ↗