Literature
Display
← All papers

FACTR 2: Learning External Force Sensing for Commodity Robot Arms Improves Policy Learning

Steven Oh · Jason Jingzhou Liu · Tony Tao · Philip Han · Kenneth Shaw · Satoshi Funabashi · Ruslan Salakhutdinov · Deepak Pathak

Preprint posted · CoRL 2026 (accepted) ↗

Robotics12 min text · CondensedOriginal paper ↗
Contents

Overview

Project page. This edition follows the complete twenty-two-page, 12 August 2026 v2, including thirteen figures, eleven numbered tables, all equations and 51 references.

Complete original figure S0.F1
Figure 1: Overview of our approach. (a) Neural External Torque estimation (NEXT) produces high quality joint torque estimates using only 10 minutes of data without dedicated force sensors or explicit system-identification, enabling force-feedback teleoperation on low-cost arms, such as the Piper, YAM, and Nero. (b) Force-Informed Re-Sampling Training (FIRST) uses learned external torque estimates to segment demonstrations into free-space, pre-contact, and contact phases, then up-samples contact-relevant segments during training to improve policy performance.

At a glance

FACTR 2 combines two components. NEXT learns the motor torque needed for contact-free motion from a short proprioceptive history; subtracting this prediction from measured current-derived torque supplies an external-torque estimate. FIRST uses that estimate to identify and preferentially sample contact-relevant demonstration segments. Experiments support useful force feedback and improved contact-rich policy learning, with robot-specific modeling, torque-scale calibration and limited contact ground truth qualifying the claims.

1 Introduction

Commodity arms often expose motor current but lack dedicated force sensors. NEXT learns their free-motion behavior from ten minutes of contact-free trajectories, then trains in about one minute. The residual enables force-feedback teleoperation without adding a dedicated sensor and extends FACTR to more robot platforms.

FIRST targets a different problem: sparse alignment and interaction events can be underrepresented during behavior cloning. It changes their sampling frequency while retaining the policy architecture. The reported demonstrations on Piper and other arms support adoption beyond sensorized systems, but do not establish accurate sensing on every robot or under arbitrary hardware changes.

2 Related Works

2.1 Sensorless External Force Estimation

Analytic force observers subtract modeled dynamics from measured effort, making friction, dead zones and transmission effects important. Supervised contact models need labeled interaction data; anomaly detectors mainly identify departures from free motion. Learned actuator models and da Vinci inverse-dynamics models motivate NEXT’s history-conditioned regression on inexpensive arms.

2.2 Imitation Learning with Force Information

TA-VLA, BiACT and CompACT use effort/force information; ForceVLA changes modality routing, while FoAR adds contact-aware objectives. Other methods combine learned motion with force or impedance control. FIRST’s distinction is using inferred contact phases to alter demonstration sampling, with pre-contact alignment often more valuable than sustained contact alone.

2.3 Policy Learning with Dataset Reweighting

Data weighting can emphasize useful actions or demonstrations: AWR, CRR and AWAC use value-related signals; Re-Mix, DataMIL and DemInf optimize dataset selection. SARM, Demo-SCORE and CUPID provide other relevance signals. FIRST instead uses a lightweight physical-contact proxy to weight individual time steps.

3 Background

Motor torque is approximated by measured current times a torque constant. Subtracting free-space dynamics gives the paper’s external joint-torque convention. A rigid-body model must account for inertia, velocity effects, gravity and other actuation terms.

The same residual also contains modeling error: friction, hysteresis, temperature effects, saturation and sensing noise can resemble contact. Temporal context can improve that separation but does not make it identifiable or exact under every operating condition. Joint torque is also distinct from a uniquely localized Cartesian contact force.

Equation 1
τext=τm−τf,τf=𝐌⁡(𝐪)​𝐪¨+𝐂⁡(𝐪,𝐪˙)​𝐪˙+𝐠⁡(𝐪)+τother\tau_{\mathrm{ext}}=\tau_{m}-\tau_{f},\qquad\tau_{f}=\mathbf{M}(\mathbf{q})\ddot{\mathbf{q}}+\mathbf{C}(\mathbf{q},\dot{\mathbf{q}})\dot{\mathbf{q}}+\mathbf{g}(\mathbf{q})+\tau_{\mathrm{other}} (1)

4 Learning External Force Estimation

NEXT regresses free-space motor torque from recent joint positions, velocities and commanded-position tracking errors. Contact-free measured torque supplies its own supervision; no external force label is needed for training. At deployment, subtracting the predicted torque from current-derived torque gives the external estimate.

A stateless LSTM processes fifty samples, roughly half a second at 100 Hz, with its hidden state reset for each window. History captures local actuator and friction effects without explicitly estimating acceleration. Training uses about ten minutes of free motion and about one minute on an RTX 3090. This replaces manual analytic identification with learned, robot-specific calibration; it does not remove dependence on the arm, controller or current-to-torque scale.

Equations 2–3
τ^f=fθ​(𝐱),τ^ext=τm−τ^f.\hat{\tau}_{f}=f_{\theta}(\mathbf{x}),\quad\hat{\tau}_{\mathrm{ext}}=\tau_{m}-\hat{\tau}_{f}. (2)
𝐱i=[𝐪i−H:i,𝐪˙i−H:i,Δ𝐪d,i−H:i],\mathbf{x}_{i}=\left[\mathbf{q}_{i-H:i},\dot{\mathbf{q}}_{i-H:i},\Delta\mathbf{q}_{d,i-H:i}\right], (3)
Complete original figure S4.F2
Figure 2: External Force Estimation Deployment. At deployment time, we first obtain the measured joint torque from multiplying each joint’s measured current by its torque constant KK. We then use an LSTM trained on free-space data to estimate free-space torque, which is then subtracted from the measured joint torque to obtain external joint torque.
Equation 4
ℒ⁡(θ)=1N​∑i=1N‖fθ​(𝐱i)−τm,i‖22\mathcal{L}(\theta)=\frac{1}{N}\sum_{i=1}^{N}\left\|f_{\theta}(\mathbf{x}_{i})-\tau_{m,i}\right\|_{2}^{2} (4)
Complete original figure S4.F3
Figure 3: FIRST is evaluated on five long-horizon, contact-rich tasks. Each task comprises multiple stages, many requiring precise alignment or fine-grained, force-sensitive adjustments. Please see videos of these tasks at https://jasonjzliu.com/factr2
Equation
𝒟free={(𝐱i,τm,i)}i=1N,\mathcal{D}_{\mathrm{free}}=\{(\mathbf{x}_{i},\tau_{m,i})\}_{i=1}^{N},

5 Force-Informed Re-Sampling Training

The policy receives camera images, joint state and NEXT torque, then predicts a chunk of future joint targets through flow matching. Equation 5 retains the velocity-regression objective; Appendix B also reports an ACT implementation.

FIRST uses the L1 norm of estimated external torque as a contact score. Two hysteresis thresholds suppress rapid switching. Contact takes priority; the one-second interval before each contact onset becomes pre-contact, and remaining samples are free space. These are offline labels derived from demonstrations, so future contact onset is available during labeling without requiring future information at deployment.

Sampling probability is proportional to a phase weight. The selected recipe is usually 1:5:1 for free/pre-contact/contact, emphasizing pre-contact alone; cap screwing uses 1:3:3. Thus the actual experiments do not always up-sample both phases despite the broad method description. Numeric hysteresis thresholds are not specified in the paper’s settings tables. The weights are selected using task progress rather than established as a universal default.

Equations 5–7
ℒFM​(θ)=𝔼⁡[‖vθ​(𝐱α,α,𝐨t)−(𝐱1−𝐱0)‖22].\mathcal{L}_{\mathrm{FM}}(\theta)=\mathbb{E}\left[\left\|v_{\theta}(\mathbf{x}_{\alpha},\alpha,\mathbf{o}_{t})-(\mathbf{x}_{1}-\mathbf{x}_{0})\right\|_{2}^{2}\right]. (5)
𝒯onset={t∣ct−1=0,ct=1}\mathcal{T}_{\mathrm{onset}}=\left\{t\mid c_{t-1}=0,\;c_{t}=1\right\} (6)
st={Contact (C),ct=1,Pre-Contact (PC),∃tonset∈𝒯onset:tonset−F≤t<tonset,Free Space (F),otherwise.s_{t}=\begin{cases}\texttt{Contact (C)},&c_{t}=1,\\ \texttt{Pre-Contact (PC)},&\exists\,t_{\mathrm{onset}}\in\mathcal{T}_{\mathrm{onset}}:t_{\mathrm{onset}}-F\leq t<t_{\mathrm{onset}},\\ \texttt{Free Space (F)},&\mathrm{otherwise}.\end{cases} (7)
𝐨t=(𝐈t1:M,𝐪t,τ^ext,t)\mathbf{o}_{t}=(\mathbf{I}^{1:M}_{t},\mathbf{q}_{t},\hat{\tau}_{\mathrm{ext},t})
𝐚^t:t+k∼πθ(⋅∣𝐨t)\hat{\mathbf{a}}_{t:t+k}\sim\pi_{\theta}(\cdot\mid\mathbf{o}_{t})
𝐱0∼𝒩⁡(𝟎,𝐈)\mathbf{x}_{0}\sim\mathcal{N}(\mathbf{0},\mathbf{I})
𝐱α=(1−α)​𝐱0+α​𝐱1\mathbf{x}_{\alpha}=(1-\alpha)\mathbf{x}_{0}+\alpha\mathbf{x}_{1}
ft=‖τ^ext,t‖1f_{t}=\|\hat{\tau}_{\mathrm{ext},t}\|_{1}
pt=w⁡(st)/∑j=1Nw⁡(sj)p_{t}=w(s_{t})/\sum_{j=1}^{N}w(s_{j})

6 Experiments

6.1 External Force Estimation and Force-Feedback Teleoperation

On Franka, five held-out minutes of human-applied contact are compared with its factory-calibrated external-torque estimate. A separate five-minute free-motion test uses zero external torque as reference. FILIC-style inverse dynamics and a momentum disturbance observer are baselines. Franka’s built-in estimate is a useful reference, not an independent exact ground-truth measurement.

Piper supplies no contact torque reference, so quantitative accuracy is evaluated only in free space. A twenty-participant wiping study tests whether feedback is useful: no feedback, disturbance observer, position-position feedback, sensor-based FACTR and NEXT-based FACTR on Franka; Piper omits the sensor-based condition. Ratings use a five-point ease scale. Measured applied torque and subjective comfort are complementary evidence, not a calibrated contact-force accuracy test.

Complete original figure S6.F4
Figure 4: User Study. Our external force estimate (using NEXT) improves teleoperation over baselines and remains comparable to FACTR Teleop, which relies on dedicated external force sensors. Lower joint torque applied corresponds to less unnecessary exertion.
Equation 8
τf​e​e​d​b​a​c​k=𝐊f,p​τe​x​t\mathbf{\tau}_{feedback}=\mathbf{K}_{f,p}\mathbf{\tau}_{ext} (8)

6.2 Imitation Learning with Force Input

Five long-horizon tasks use 250 demonstrations each on a bimanual Piper setup with four RGB cameras. Each policy receives twenty rollouts per task; the main metric is stage-based task progress, not simply complete-episode success. Baselines hold the policy architecture and cloning objective fixed while changing force input, visual curriculum or an auxiliary torque target.

The flow policy adapts Multimodal Diffusion Transformer; experiments also use ACT. Action chunks contain thirty targets and action heads start from scratch. Base Policy omits force; Base+Torque adds NEXT; FACTR applies a blur curriculum; TA-VLA adds torque reconstruction; FIRST adds phase sampling. These are controlled adaptations within this implementation, not comparisons of every original published system.

Complete original PDF block S6.T1
Table 1: Average L1L_{1} joint torque error on the Franka in free-space and contact settings. Free-space errors are measured against zero external torque, while contact errors are measured against the Franka built-in external torque estimate. NEXT achieves the lowest error in both settings.
Original selectable table
Motion Type External Torque Method L1L_{1} Error (Nm)
Contact External Sensor (GT) 0.000±0.0000.000\pm 0.000
FILIC [22] 4.395±1.5314.395\pm 1.531
Disturbance Observer [43] 1.471±0.7611.471\pm 0.761
NEXT (Ours) 0.547±0.348\mathbf{0.547\pm 0.348}
Free Space Free Space (GT) 0.000±0.0000.000\pm 0.000
FILIC [22] 2.460±1.3582.460\pm 1.358
Disturbance Observer [43] 2.429±1.2982.429\pm 1.298
External Sensor 0.449±0.2080.449\pm 0.208
NEXT (Ours) 0.414±0.278\mathbf{0.414\pm 0.278}
Complete original figure S6.F5
Figure 5: Left: In free space, the external joint torque should remain zero. Our method produces an estimate that is less noisy than the external sensor and remains near zero, while FILIC and the Disturbance Observer drift away. Right: During contact, our estimate closely tracks the external sensor, whereas FILIC and the Disturbance Observer deviate substantially.

7 Results

NEXT’s Franka contact error is 0.547±0.348 Nm, versus FILIC 4.395±1.531 and disturbance observer 1.471±0.761: reported reductions of 87.6% and 62.8%. In free motion it gives 0.414±0.278 Nm versus the built-in estimate’s 0.449±0.208, a modest difference against the zero reference rather than proof of universally better sensing. Piper free-space error is 0.018±0.012 Nm, with no corresponding instrumented contact test.

The user-study plots favor NEXT for ease and reduced applied torque, with Franka behavior similar to sensor-based FACTR. Those results demonstrate useful feedback on the tested wiping setup; the paper does not provide a formal equivalence test against the sensor-based system.

FIRST has the highest reported flow-policy progress on all five tasks. The phase ablation gives average progress 0.670 for contact-only, 0.818 for pre-contact-only and 0.811 for both; cap screwing favors both at 0.825, while pre-contact-only gives 0.673. Gains concern this progress metric; the broad “over 17%” claim should not be read as seventeen additional successful episodes per hundred on every task.

For NIST Belt, pre-contact validation loss is much larger under ordinary sampling and drops under FIRST. This is consistent with focusing learning on alignment, but the loss curves and selected recovery examples do not independently prove a causal mechanism. Raw current helps some tasks; the dedicated residual gives stronger scores on both tested input-ablation tasks.

Complete original PDF block S7.F6
Figure 6: Task progress across five contact-rich manipulation tasks using flow matching policy.
Complete original PDF block S7.F7
Figure 7: Default sampling yields higher validation loss on pre-contact and contact phases. By upsampling these phases during training, FIRST reduces their validation losses.
Complete original PDF block S7.T2
Table 2: Upsampling ablation. C: contact, PC: pre-contact, PC+C: pre-contact + contact.
Original selectable table
Upsampling Phase
Task C PC PC+C
LEGO Assembly 0.697 0.913 0.875
NIST Belt 0.494 0.767 0.649
NIST Insertion 0.800 0.933 0.933
Tool Clean Up 0.650 0.805 0.775
Cap Screwing 0.707 0.673 0.825
Average 0.670 0.818 0.811

8 Conclusion

Learning free-space effort makes force-related information usable on sensorless commodity arms, and resampling the moments before contact improves the tested policies. The practical contribution combines a small estimator, force-feedback data collection and a simple training distribution change; accuracy and transfer remain dependent on operating conditions and evaluation scope.

9 Limitations

Absolute torque scale depends on the motor constant K. If it is wrong, physical calibration may still require an external sensor; input normalization or comfortable feedback scaling does not certify accurate Newton-meter measurements. NEXT is robot-specific and may need retraining between arms. The paper does not establish robustness to arbitrary payload, temperature or controller changes, and no safety guarantee follows from a low average estimation error.

Acknowledgments

The authors acknowledge discussion and equipment support, including access to YAM arms, and funding from AFOSR and ONR MURI programs.

Appendix

Appendix A NEXT Implementation Details

A.1 Free-Space Inverse Dynamics Model Architecture

The estimator uses a two-layer LSTM of width 128 and a two-layer prediction MLP of width 256, dropout 0.1 and squared-error supervision. AdamW uses learning rate 10⁻³, weight decay 10⁻⁶ and gradient clipping 1.0; validation-based early stopping controls training.

A forward pass takes about 1.76 ms on one i9-10920X CPU thread, equivalent to 568 evaluations per second, but the deployed loop runs at 100 Hz. That microbenchmark is not end-to-end sensing/communication latency. Resetting recurrent state per rolling window avoids accumulation over a full trajectory.

Complete original PDF block A1.T3
Table 3: Network architecture and training hyperparameters for the free-space inverse dynamics neural network used in NEXT.
Original selectable table
Hyperparameter Value
Recurrent backbone 2-layer LSTM with hidden size 128
Prediction head 2-layer MLP with hidden size 256
Dropout 0.1
Loss function Mean squared error (MSE)
Optimizer AdamW
Base learning rate 1×10−31\times 10^{-3}
Weight decay 1×10−61\times 10^{-6}
Gradient clipping 1.0
Early stopping Enabled; validation loss, patience 20, warmup 10, min⁡Δ=10−5\min\Delta=10^{-5}
Hardware NVIDIA RTX 3090, 24 GB
Training time Approximately 1 minute with early stopping
Data frequency 100Hz

A.2 Free-Space Inverse Dynamics Network Ablations

Table 4 varies MLP/GRU/LSTM, inputs and histories of ten, twenty-five and fifty samples. The chosen full-input fifty-step LSTM gives the best reported free-space/contact pair, 0.414/0.547 Nm.

Adding tracking error or lengthening history is not uniformly better in every configuration: the full-input GRU’s contact errors exceed its position/velocity variant, for example. The table supports the selected configuration rather than the stronger claim that each added modality consistently improves every model. Contact reference and free-space zero targets retain the distinction used in the main evaluation.

Complete original PDF block A1.T4
Table 4: Ablation of model family, input modality, and temporal history length for external torque estimation. Values corresponding to the average joint L1L_{1} validation error between predicted free-space torque and ground-truth free-space torque on the held-out set, reported in Nm.
Original selectable table
Input 𝐇\mathbf{H} MLP GRU LSTM (Ours)
Free Motion ↓\downarrow Contact ↓\downarrow Free Motion ↓\downarrow Contact ↓\downarrow Free Motion ↓\downarrow Contact ↓\downarrow
qq 10 2.35±2.342.35{\pm}2.34 1.78±1.641.78{\pm}1.64 2.25±1.792.25{\pm}1.79 1.12±1.091.12{\pm}1.09 2.53±2.522.53{\pm}2.52 2.61±2.292.61{\pm}2.29
25 2.02±2.012.02{\pm}2.01 1.34±1.321.34{\pm}1.32 1.68±1.301.68{\pm}1.30 0.665±0.4780.665{\pm}0.478 2.21±2.202.21{\pm}2.20 1.45±1.391.45{\pm}1.39
50 1.76±1.761.76{\pm}1.76 1.13±1.061.13{\pm}1.06 1.62±1.241.62{\pm}1.24 0.843±0.8350.843{\pm}0.835 2.15±2.152.15{\pm}2.15 1.09±0.9081.09{\pm}0.908
q,q˙q,\dot{q} 10 1.04±1.031.04{\pm}1.03 0.681±0.6470.681{\pm}0.647 0.863±0.8620.863{\pm}0.862 0.580±0.5350.580{\pm}0.535 1.15±1.151.15{\pm}1.15 0.715±0.5530.715{\pm}0.553
25 0.980±0.9760.980{\pm}0.976 0.908±0.8020.908{\pm}0.802 0.814±0.8140.814{\pm}0.814 0.596±0.5060.596{\pm}0.506 1.11±1.111.11{\pm}1.11 0.551±0.5110.551{\pm}0.511
50 1.02±1.021.02{\pm}1.02 0.905±0.8920.905{\pm}0.892 0.795±0.7940.795{\pm}0.794 0.653±0.5490.653{\pm}0.549 1.12±1.121.12{\pm}1.12 0.578±0.5260.578{\pm}0.526
q,q˙,Δ​qdq,\dot{q},\Delta q_{d} 10 0.706±0.7020.706{\pm}0.702 1.89±1.711.89{\pm}1.71 0.423±0.4220.423{\pm}0.422 1.94±1.751.94{\pm}1.75 0.504±0.5030.504{\pm}0.503 0.876±0.2440.876{\pm}0.244
25 0.755±0.7470.755{\pm}0.747 1.56±1.451.56{\pm}1.45 0.473±0.4720.473{\pm}0.472 1.47±1.291.47{\pm}1.29 0.490±0.4890.490{\pm}0.489 0.789±0.4530.789{\pm}0.453
50 0.813±0.8110.813{\pm}0.811 1.47±1.331.47{\pm}1.33 0.429±0.4290.429{\pm}0.429 1.62±1.451.62{\pm}1.45 0.414±0.278\mathbf{0.414{\pm}0.278} 0.547±0.348\mathbf{0.547{\pm}0.348}

A.3 Free-Space Data Collection Guidelines for NEXT

The ten-minute collection procedure first exercises joints independently, then uses coordinated Cartesian-like motion, and repeats both slowly and quickly. Figure 8 preserves all four six-joint distributions: tracking error, position, velocity and free-space torque. Recorded motions can be replayed on another setup to collect new calibration data; this is not reuse of the same estimator without retraining.

Complete original figure A1.F8
Figure 8: Distribution of the 10-minute free-motion dataset used to train NEXT on the Piper arm. The plots show the coverage of joint-position error Δ​q\Delta q, joint position qq, joint velocity q˙\dot{q}, and free-space torque τf\tau_{f} across all six joints.

A.4 External Joint Torque Estimation Baseline Implementation

Franka baselines use identified dynamics; Piper uses the manufacturer URDF. The FILIC-style residual subtracts modeled inverse dynamics, while the momentum observer avoids explicit acceleration estimation and smooths encoder noise. These baseline model choices affect the comparison with a model fit directly to each arm’s data.

The printed observer equations use a different torque sign from Equation 1’s residual convention: with the stated momentum error and positive observer gain, a constant external residual produces its negative at steady state. The original equations remain unchanged. This is a source notation discrepancy, not evidence that the uninspected baseline implementation used the wrong sign.

Equations 9–11
τ^extFILIC=τm−τf,τf=𝐌⁡(𝐪)​𝐪¨+𝐂⁡(𝐪,𝐪˙)​𝐪˙+𝐠⁡(𝐪)\hat{\tau}_{\mathrm{ext}}^{\mathrm{FILIC}}=\tau_{m}-\tau_{f},\qquad\tau_{f}=\mathbf{M}(\mathbf{q})\ddot{\mathbf{q}}+\mathbf{C}(\mathbf{q},\dot{\mathbf{q}})\dot{\mathbf{q}}+\mathbf{g}(\mathbf{q}) (9)
𝐩=𝐌⁡(𝐪)​𝐪˙.\mathbf{p}=\mathbf{M}(\mathbf{q})\dot{\mathbf{q}}. (10)
𝐩^˙=τm−𝐂⁡(𝐪,𝐪˙)​𝐪˙−𝐠⁡(𝐪)+𝐌˙​(𝐪)​𝐪˙+τ^extMDO,τ^extMDO=𝐊o​(𝐩−𝐩^).\dot{\hat{\mathbf{p}}}=\tau_{m}-\mathbf{C}(\mathbf{q},\dot{\mathbf{q}})\dot{\mathbf{q}}-\mathbf{g}(\mathbf{q})+\dot{\mathbf{M}}(\mathbf{q})\dot{\mathbf{q}}+\hat{\tau}_{\mathrm{ext}}^{\mathrm{MDO}},\qquad\hat{\tau}_{\mathrm{ext}}^{\mathrm{MDO}}=\mathbf{K}_{o}(\mathbf{p}-\hat{\mathbf{p}}). (11)

A.5 Force-Feedback Teleoperation Baseline Implementation

The no-feedback condition uses GELLO. Disturbance-observer feedback sends estimated follower torque to the leader; position-position feedback scales leader/follower tracking error. Sensor-based and NEXT-based FACTR otherwise share gravity/friction compensation and null-space regulation. Piper has four conditions because it lacks dedicated joint-torque sensing.

Complete original PDF block A1.T5
Table 5: Average L1L_{1} joint torque error on the AgileX Piper in free-space settings. Free-space errors are measured against zero external torque. NEXT achieves the lowest error.
Original selectable table
External Torque Method Error (Nm) ↓\downarrow
Free Motion (GT) 0.000±0.0000.000\pm 0.000
FILIC 0.435±0.2250.435\pm 0.225
Disturbance Observer 0.248±0.1320.248\pm 0.132
Ours 0.018±0.012\mathbf{0.018\pm 0.012}

Complete original figure A1.F9
Figure 9: User study results comparing external torque feedback methods on the Piper.

A.6 AgileX Piper NEXT Evaluations

Piper’s free-space error is 0.018±0.012 Nm, versus FILIC 0.435±0.225 and disturbance observer 0.248±0.132. The separate ease-of-use plot favors NEXT feedback in contact. Near-zero output during free motion is necessary but insufficient to establish its absolute contact accuracy; that remains unmeasured on this platform.

Appendix B Behavior Cloning Policy Architecture and Training Hyperparameters

DINOv3-Base encodes images; MLPs embed joint/gripper state and force. Policies use the latest observation, while NEXT separately supplies its fifty-step history-derived estimate. ACT has four encoder and six decoder layers; the flow model has six decoder layers. Both use width 768, eight attention heads, feed-forward width 3,072 and thirty-action chunks.

Table 6 retains all training settings: batch 128, AdamW at 3×10⁻⁴, cosine decay, 500 warmup steps, random resized crops and 12–24 hours on an A6000. It lists 15–20 epochs while the main comparison states twenty. The main formulation samples flow time uniformly, whereas the implementation table specifies Beta(1.0,1.5), with ten Euler inference steps; both source specifications are preserved.

Complete original PDF block A2.T6
Table 6: Policy Architecture and Training Hyperparameters
Original selectable table
Hyperparameter Value
Behavior Policy Training
Optimizer AdamW
Base Learning Rate 3×10−43\times 10^{-4}
Weight Decay 0.050.05
Optimizer Momentum β1,β2=0.9,0.95\beta_{1},\beta_{2}=0.9,0.95
Batch Size 128
Learning Rate Schedule Cosine Decay
Total Epochs 15–20
Warmup Steps 500
Augmentation RandomResizeCrop
GPU A6000 (48 GB)
Wall-Clock Time 12–24 hours
Action Chunking Transformer Details
# Encoder Layers 4
# Decoder Layers 6
# MHSA Heads 8
Hidden Dim 768
Feed-Forward Dim 3072
Dropout 0.1
Positional Encoding Sinusoidal
Action Chunk 30
Flow Matching Details
# Decoder Layers 6
# MHSA Heads 8
Hidden Dim 768
Feed-Forward Dim 3072
Dropout 0.1
Positional Encoding Sinusoidal
Action Chunk 30
Eval Steps 10
Time Sampling τ∼Beta⁡(1.0,1.5)\tau\sim\mathrm{Beta}(1.0,1.5)
ODE Solver Euler
Noise Distribution 𝒩⁡(0,I)\mathcal{N}(0,I)

Appendix C Behavior Cloning Evaluation Tasks

C.1 Task Description

LEGO uses LEAP Hand V2 to assemble two small blocks onto a larger block. Gripper tasks wrap belts around pulley sets, insert NIST parts, put away a screwdriver and assemble its case, or pick/alignment/screw a cap until tight. The last task also requires stopping instead of continuing to rotate.

C.2 Task Progress Rate Calculation

Progress is the fraction of listed stages completed across rollouts. Table 7 preserves six LEGO stages, seven belt stages and four each for NIST Insertion, Tool Clean Up and Cap Screwing. The insertion score ends with picking up the aluminum part; it does not list its subsequent alignment/insertion. Therefore a high progress score does not by itself certify completion of both inserted assemblies described at task level.

Cap screwing subtracts 0.02 per excess turn after tightening. This modified score differs from an unpenalized fraction of successful stages and from binary success. “Case Assembly” elsewhere refers to the Tool Clean Up task.

Complete original PDF block A3.T7
Table 7: Stage decomposition used for task progress calculation.
Original selectable table
Task Stages
LEGO red grasp, red align, red insert, green grasp, green align, green insert
NIST Belt pick orange, align big, align small, push, pick grey, align grey, push
NIST Insertion pick black, align black, insert black, pick aluminum
Tool Clean Up pick driver, align driver, insert driver, assemble box
Cap Screwing pick cap, align cap, screw on, end properly

C.3 Imitation Learning with Force Information Baseline Implementations

The base visual/proprioceptive policy receives no torque; Base+Torque adds a projected NEXT token. FACTR applies Gaussian image blur that decreases linearly through training. TA-VLA adds a torque/contact prediction target to the action-related representation; only robot-action coordinates execute at inference. The full visual curriculum formula is retained.

Equation
σn=σ0​(1−nN),\sigma_{n}=\sigma_{0}\left(1-\frac{n}{N}\right),

Appendix D Additional Results on FIRST

D.1 Segmenting Trajectories into Contact Phases Visualization

The segmentation example overlays normalized torque, phase bands and images. It illustrates low free-motion residuals and increasing signal around interaction, including the interval immediately before contact. It is a qualitative example rather than a labeled-contact precision/recall study, and segmentation inherits estimator and threshold errors.

Complete original figure A4.F10
Figure 10: Contact phase segmentation from learned external torque. The normalized τe​x​t\tau_{ext} from NEXT is used to segment demonstrations into free motion, pre-contact, and contact phases. The image overlays show that the predicted labels align with the robot’s interaction state during the task.

D.2 Task Progress Evaluation on ACT

The ACT comparison also favors FIRST across the five plotted task-progress outcomes. This extends the result beyond the flow head within the shared visual encoder/data setup; it does not establish architecture-independent gains under arbitrary policies or larger datasets.

Complete original PDF block A4.F11
Figure 11: Task progress across five contact-rich manipulation tasks using ACT policy.

D.3 Selected Re-Sampling Ratios for Each Task

Table 8 selects free/pre-contact/contact weights 1:5:1 for LEGO, NIST Belt, NIST Insertion and Tool Clean Up, and 1:3:3 for Cap Screwing. The weights are chosen by task progress. They change sampling probabilities relative to the dataset’s original phase counts, not the final batch fractions directly.

Complete original PDF block A4.T8
Table 8: Best re-sampling ratios used by FIRST for each task. Ratios are reported as wF:wPC:wCw_{\mathrm{F}}:w_{\mathrm{PC}}:w_{\mathrm{C}}, corresponding to free-motion, pre-contact, and contact sampling weights.
Original selectable table
Task Best Re-Sampling Ratio
LEGO Assembly 1:5:11:5:1
NIST Belt Assembly 1:5:11:5:1
NIST Insertion 1:5:11:5:1
Tool Clean Up 1:5:11:5:1
Cap Screwing 1:3:31:3:3

D.4 Up-Sampling Magnitude Ablation

ACT ablations cover NIST Belt, NIST Insertion and Tool Clean Up; the flow ablation covers only Tool Clean Up. Moderate weighting often helps and aggressive weighting can lose full-trajectory coverage. Around five is a useful reported trend rather than a universal optimum: ACT’s pre-contact-only NIST Belt curve peaks at three, while its combined-phase curve peaks at five.

Equations
wF:wPC:wC=1:m:1,w_{\mathrm{F}}:w_{\mathrm{PC}}:w_{\mathrm{C}}=1:m:1,
wF:wPC:wC=1:m:m.w_{\mathrm{F}}:w_{\mathrm{PC}}:w_{\mathrm{C}}=1:m:m.
Complete original figure A4.F12
Figure 12: Moderate up-sampling generally improves performance, with best results around m=5m=5, while larger values lead to saturation or degradation.

D.5 Up-Sampling Policies Without Force Input

NEXT can label demonstrations offline while the deployed policy receives only images and proprioception. Phase sampling still helps in the two-task ablation, so not every use of FIRST requires online force input. Contact information at inference generally adds further gains; some individual sampling comparisons tie rather than improve.

For flow matching on NIST Belt, pre-contact-only sampling rises from 0.150 to 0.213 without torque and from 0.431 to 0.767 with torque. On Tool Clean Up it rises from 0.463 to 0.575 without torque and from 0.525 to 0.805 with torque. Table 9 calls these success rates, whereas the main figures use stage progress; its values and label are retained without treating the two metrics as interchangeable.

Complete original PDF block A4.T9
Table 9: Effect of phase-based upsampling on policies trained with and without external joint torque. Results are reported as task success rates.
Original selectable table
Policy Up-sampling Strategy w/o τe​x​t\tau_{ext} w/ τe​x​t\tau_{ext}
Belt Assembly Tool Clean Up Belt Assembly Tool Clean Up
ACT None 0.220 0.600 0.347 0.650
PC 0.360 0.600 0.687 0.650
PC + C 0.283 0.763 0.770 0.788
Flow Matching None 0.150 0.463 0.431 0.525
PC 0.213 0.575 0.767 0.805
PC + C 0.241 0.525 0.649 0.775

D.6 Inputting Joint Current vs. External Joint Torque on Policy Performance

ACT’s reported NIST Insertion score is 0.220 without force, 0.431 with raw current and 0.567 with NEXT. Cap Screwing is 0.440/0.440/0.525. A specialized history-based inverse-dynamics model can separate free-motion effort from contact more effectively than supplying only instantaneous current to the policy under this comparison.

Figure 13 plots only first-decoder-layer attention from action tokens to force-related tokens. Torque-conditioned attention increases around tightening, while the current-conditioned policy continues rotating. This supports a diagnostic interpretation; attention alone is not causal attribution or evidence that longer-history current-conditioned policies could not match the result.

Complete original figure A4.F13
Figure 13: Attention values during screw-cap manipulation. Learned external torque produces clearer attention around key contact events than raw joint effort.
Complete original PDF block A4.T10
Table 10: Policy performance when conditioning on raw joint current versus learned external joint torque. Learned external joint torque provides the best performance on both contact-sensitive tasks. Higher is better. Bold indicates the best result within each task.
Original selectable table
Method NIST Insertion ↑\uparrow Cap Screwing ↑\uparrow
ACT 0.2200.220 0.4400.440
ACT w/ current 0.4310.431 0.4400.440
ACT w/ τe​x​t\tau_{ext} 0.567\mathbf{0.567} 0.525\mathbf{0.525}

D.7 Per-batch Up-sampling vs. Weighted Regression

The separate ACT LEGO comparison gives progress 0.453 with no weighting, 0.345/0.478 with pre-contact/combined-phase weighted regression, and 0.573/0.501 with weighted sampling. Sampling performs better in this experiment; proposed instability/overfitting explanations are not directly established by that table.

For fixed phase weights, sampling proportional to weight and normalized weighted regression have the same expected weighted empirical objective, up to a constant scale. Their minibatch composition, gradient variance, normalization and optimizer dynamics can differ. The result therefore supports the reported sampling implementation, not a fundamental distinction between their expected objectives.

Complete original PDF block A4.T11
Table 11: Comparison of phase emphasis mechanisms for LEGO assembly using ACT.
Original selectable table
Phase Emphasis Mechanism Emphasized Phase Task Progress
None – 0.453
Weighted Regression PC 0.345
PC + C 0.478
FIRST (Ours) PC 0.573
PC + C 0.501
Concept demonstration

Open image ↗