Literature
Display
← All papers

FingerEye: Learning Dexterous Manipulation with Continuous Vision-Tactile Sensing

Zhixuan Xu · Yichen Li · Xuanye Wu · Tianyu Qiu · Lin Shao

Preprint posted · CoRL 2026 (accepted) ↗

Robotics13 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers all twenty-three pages of arXiv v4, dated 6 October 2026, including the complete sensing, calibration, policy and experiment appendices. Zhixuan Xu and Yichen Li contributed equally; Lin Shao is the corresponding author. Author names are normalized to the paper’s printed order and spelling.

All figures, tables and formulas are retained. The HTML incorrectly labels the modality-results table as a figure; this edition follows the PDF’s thirteen figures and nine tables while preserving reference IDs.

At a glance

FingerEye combines two fingertip RGB cameras with a clear acrylic contact plate suspended by a compliant silicone ring. Cameras see local geometry before touch; contact moves the plate, whose AprilTag array provides a six-dimensional deformation proxy afterward.

A learned action-chunking policy separately encodes observation groups and gives each an independent decoder attention path. Across three simulation and four real task settings, binocular FingerEye raises mean success by 39.1 and 33.8 percentage points over wrist vision. Main policies use RGB and proprioception, not explicit estimated force or plate pose.

1 Introduction

Approach, first contact and subsequent force regulation require different information. External vision can miss occluded fingertip geometry, while conventional tactile sensing becomes informative only after touch. The sensor aims to observe that transition continuously.

Relative to existing see-through designs, FingerEye emphasizes clear passive RGB, robust identity-based marker tracking, a wedge that reaches thin edges, matched real/simulation infrastructure and a policy interface that keeps local feedback accessible. Its sensing stream is continuous, although learned actions are executed in chunks between replanning steps.

The paper reports hardware/software resources at its project site. This edition reviews the paper’s evidence and does not independently audit a complete external release.

Complete original figure with every panel, label and legend.
Figure 1: FingerEye overview. FingerEye provides continuous fingertip feedback for contact-rich dexterous manipulation. Left: FingerEye tracks local interaction cues throughout approach, contact initiation, contact refinement, and release. Right: we evaluate FingerEye across seven real-world and simulation tasks spanning precise approach, edge/contact engagement, and post-contact stabilization, with objects varying in size, rigidity, delicacy, and geometry.

2 Related work

GelSight and related vision-based tactile sensors provide rich contact information. FingerVision, Finger-STS and other see-through sensors combine proximity and touch, with differing optical, illumination and geometry trade-offs.

Multisensory policies and Sparsh-style representation learning address complementary learning problems. FingerEye instead studies how encoded fingertip, wrist and proprioceptive tokens reach action queries. Its comparisons do not establish superiority to every optimized tactile representation or matched see-through sensor.

Complete original figure with every panel, label and legend.
Figure 2: FingerEye Sensing Interface Design. (a) Dimensions. (b) Cross-sectional view: fields of view of the root and tip cameras, and the contact sensing region. (c) Exploded view.
Complete original figure with every panel, label and legend.
Figure 3: Robustness of Tag Array Pose Estimation. Top: stable detection under frontal and peripheral force perturbations. Bottom: stable detection under lighting variation.

3 FingerEye Sensing Interface

The sensing design keeps a clear visual path while transmitting contact-induced motion to a trackable rigid plate. Five design choices and four capability tests separate construction, pose sensing, wrench calibration and task utility.

3.1 Sensing Interface Design Highlights

A tip camera focuses near the plate at about 10 mm; a tilted root camera preserves broader approach context. Their complementary RGB views supply implicit stereo cues without active illumination or mode switching. The system does not explicitly reconstruct dense stereo depth.

A compliant silicone ring surrounds a clear acrylic center. Frontal and peripheral loads change plate pose; thirty-five tags provide redundant corners for one joint PnP estimate. The wedge-shaped module is about 41.7×30.0×25.0 mm and costs approximately $60 in the reported bill of materials.

The layout keeps the center visually useful but does not eliminate occlusion or the friction/conformity trade-off of rigid acrylic. Figure 2 retains all dimensions and the exploded design; Figure 3 retains force/lighting examples.

Complete original figure with every panel, label and legend.
Figure 4: Visualization of delicate grasping and fingertip normal deformation. Each example shows the lifted object and temporal curves of normal displacement Δ​z\Delta z measured at the thumb and index fingertips.

3.2 Sensing Interface Capability Highlights

Multiple visible tags stabilize pose estimation when some disappear. A tag-count ablation measures jitter and agreement with the full layout, while 10,000 normal-compression cycles test one limited durability mode.

Axis-specific calibration relates plate motion to wrench. The quoted minimum detectable forces 4.30/4.22/9.93 mN and torques 0.32/0.13/8.55 mN·m are analytical estimates based on assumed subpixel localization, not directly measured detection thresholds.

A separate controller stops predefined finger motions at object-specific normal-displacement thresholds and lifts nine delicate objects. This demonstrates contact-onset utility with selected thresholds and support fixtures, not a learned universal force-control policy.

Equations
𝐅min=[ 4.30, 4.22, 9.93]​mN\mathbf{F}_{\min}=[\,4.30,\,4.22,\,9.93\,]~\mathrm{mN}
𝝉min=[ 0.32, 0.13, 8.55]​mN⋅m\bm{\tau}_{\min}=[\,0.32,\,0.13,\,8.55\,]~\mathrm{mN\cdot m}

4 FingerEye Learning Interface

The learning interface combines matched teleoperation, a compliant-plate simulation proxy and an action-query transformer. The key ablation is the scope of observation encoding and decoder attention, with sensing input held fixed.

4.1 Real-and-Sim Learning Infrastructure

Use a fixed-base uFactory xArm7 and LEAP Hand, with wrist RGB and fingertip modules. A passive leader with matched hand kinematics commands follower joints through PD control, avoiding the human-to-robot retargeting mismatch encountered in preliminary trials.

Record synchronized RGB, plate pose, robot state and actions at 10 Hz. In simulation, attach each plate by six virtual joints whose gains approximate measured pose-to-wrench behavior. Geometry, cameras and tags are matched; nonlinear silicone mechanics and deformable interactions are neither modeled nor validated.

Complete original figure with every panel, label and legend.
Figure 5: Data collection interface. A leader–follower setup enables intuitive and precise data collection of contact-sensitive tasks in both real and sim.

4.2 Policy Interface Design Highlights

Main inputs are wrist RGB, proprioception and fingertip RGB. The detailed implementation groups all FingerEye camera tokens together, separate from wrist and proprioception. Plate pose supports sensor characterization and delicate-grasp control; explicit wrench input is only an additional ablation.

NoEnc passes tokens directly; FlatEnc mixes all tokens; GroupEnc first attends within each group. FlatDec then attends to a single mixed context, whereas GroupDec gives each group a separate cross-attention path and averages their residual updates. This avoids one shared softmax competition, but equal residual weighting does not guarantee equal causal contribution.

Freeze RADIO and cache each image’s summary for policy training. Predict sixteen 10-Hz actions; most tasks execute eight before replanning, while simulated coin and peg tasks execute all sixteen. Continuous sensing therefore coexists with 0.8- or 1.6-second action execution intervals.

Equations
NoEnc:𝒁tnoenc=𝑼t,FlatEnc:𝒁tflat=Eflat​(𝑼t),GroupEnc:𝒁tgroup=[E1​(𝑼t1),…,EM​(𝑼tM)].\begin{array}[]{@{}r@{\;}c@{\quad}r@{\;}c@{\;}l@{}}\text{NoEnc}&:&\bm{Z}_{t}^{\mathrm{noenc}}&=&\bm{U}_{t},\\ \text{FlatEnc}&:&\bm{Z}_{t}^{\mathrm{flat}}&=&E_{\mathrm{flat}}(\bm{U}_{t}),\\ \text{GroupEnc}&:&\bm{Z}_{t}^{\mathrm{group}}&=&[E_{1}(\bm{U}_{t}^{1}),\ldots,E_{M}(\bm{U}_{t}^{M})].\end{array}
FlatDec CrossAttn:𝒉~tl=𝒉tl+CrossAttnl​(𝒉tl,𝒁t),GroupDec CrossAttn:𝒉~tl=𝒉tl+1M​∑i=1MCrossAttnil​(𝒉tl,𝒛ti).\begin{array}[]{@{}r@{\;}c@{\quad}r@{\;}c@{\;}l@{}}\text{FlatDec CrossAttn}&:&\tilde{\bm{h}}_{t}^{l}&=&\bm{h}_{t}^{l}+\mathrm{CrossAttn}^{l}(\bm{h}_{t}^{l},\bm{Z}_{t}),\\ \text{GroupDec CrossAttn}&:&\tilde{\bm{h}}_{t}^{l}&=&\bm{h}_{t}^{l}+\frac{1}{M}\sum_{i=1}^{M}\mathrm{CrossAttn}_{i}^{l}(\bm{h}_{t}^{l},\bm{z}_{t}^{i}).\end{array}
𝑶t=(𝒐t1,…,𝒐tM)\bm{O}_{t}=(\bm{o}_{t}^{1},\ldots,\bm{o}_{t}^{M})
𝑨t=πθ​(𝑶t)=(𝒂t,…,𝒂t+Tp−1)\bm{A}_{t}=\pi_{\theta}(\bm{O}_{t})=(\bm{a}_{t},\ldots,\bm{a}_{t+T_{p}-1})
𝑼ti=τi​(𝒐ti)\bm{U}_{t}^{i}=\tau_{i}(\bm{o}_{t}^{i})
𝑼t=[𝑼t1,…,𝑼tM]\bm{U}_{t}=[\bm{U}_{t}^{1},\ldots,\bm{U}_{t}^{M}]
𝒒=(𝒒1,…,𝒒Tp)\bm{q}=(\bm{q}^{1},\ldots,\bm{q}^{T_{p}})
𝒁t=[𝒛t1,…,𝒛tM]\bm{Z}_{t}=[\bm{z}_{t}^{1},\ldots,\bm{z}_{t}^{M}]

5 Experiments

Evaluate three simulated tasks—coin standing, nut picking and peg-in-hole pushing—and four real tasks—coin standing, chip picking, letter retrieval and syringe manipulation. These are seven settings, including coin standing in both domains.

Simulation uses three training seeds and three evaluation seeds with 64 rollouts per evaluation seed. Real evaluation uses twenty trials per method/task. Mean success averages tasks; the real percentages have coarse five-point increments and no reported confidence intervals. Architecture variants are size-matched by adjusting widths when needed.

Complete original figure with every panel, label and legend.
Figure 6: FingerEye task overview. We evaluate FingerEye across seven real-world and simulation tasks spanning precise approach, edge or contact engagement, and post-contact reorientation or stabilization, with objects that vary in size, rigidity, delicacy, and geometry. The red bounding box marks the coin initialization region.
Table 1: Modality comparison across simulation and real-world tasks. W = wrist camera, M = tip-only monocular FingerEye camera, and FE = binocular FingerEye cameras; proprioception is included by default. SR in %; simulation uses mean±\pmstd and real-world tasks use 2020 trials. Green values are percentage-point gains over W.
Modalities Simulation Real-world
Coin Nut Pick Peg-in-Hole Mean Coin Chip Letter Syringe Mean
W 25.5±8.325.5{\pm}8.3 38.7±23.238.7{\pm}23.2 16.0±6.616.0{\pm}6.6 26.726.7 2525 3535 3030 6060 37.537.5
W+M 56.4±3.756.4{\pm}3.7 84.4±5.284.4{\pm}5.2 36.5±5.536.5{\pm}5.5 59.159.1 (+32.4) 5050 5555 4545 7575 56.356.3 (+18.8)
W+FE 65.6±4.8\mathbf{65.6{\pm}4.8} 88.2±4.2\mathbf{88.2{\pm}4.2} 43.8±4.2\mathbf{43.8{\pm}4.2} 65.9\mathbf{65.9} (+39.1) 𝟔𝟓\mathbf{65} 𝟕𝟎\mathbf{70} 𝟔𝟓\mathbf{65} 𝟖𝟓\mathbf{85} 71.3\mathbf{71.3} (+33.8)

5.1 Q1–Q2: Sensing Modality Comparison

With the same GroupEnc+GroupDec interface and proprioception, wrist-only W achieves mean success 26.7% in simulation and 37.5% in reality. Tip-camera-only W+M reaches 59.1%/56.3%; binocular W+FE reaches 65.9%/71.3%. Table 1 preserves all task values and simulation variability.

The binocular gain over monocular input is 6.8 and 15.0 percentage points. Selected failures suggest better edge-side and approach coverage, but the comparison does not independently separate stereo depth, field of view, extra tokens and additional visual information. Nut picking remains highly variable with wrist-only input.

Complete original figure with every panel, label and legend.
Figure 7: Modality failure modes. The panels summarize local contact-geometry errors for W and residual monocular coverage errors for W+M; full binocular FingerEye reduces these failures with complementary views.

5.2 Q3: Continuous Sensing vs. Post-Contact Signals

Simulation W+T uses proximity/contact-depth maps, not physical force measurements or a deformable tactile simulator. Its mean success is 24.1% versus wrist-only 26.7% and W+FE 65.9%. The result concerns this intentionally near-contact-only proxy encoded by the frozen RGB backbone.

Real GelSight reaches 45% on chip picking and 65% on syringe manipulation, versus FingerEye’s 70%/85%, with matched demonstrations and policy architecture. Coin and letter demonstrations were infeasible with the tested GelSight geometry. Sensor shape and sensing modality are therefore confounded; a matched see-through-policy comparison remains absent.

Table 2: Continuous vs. post-contact sensing. All methods use GEnc+GDec and proprioception. SR in %; simulation reports mean±\pmstd, and real-world tasks use 2020 trials per method and task. T denotes simulated contact-depth maps in simulation and GelSight observations in the real world.
Modalities Simulation Real-world
Coin Nut Pick Peg-in-Hole Mean Chip Syringe
W 25.5±8.325.5{\pm}8.3 38.7±23.238.7{\pm}23.2 16.0±6.616.0{\pm}6.6 26.726.7 3535 6060
W+T 27.3±6.827.3{\pm}6.8 28.6±5.128.6{\pm}5.1 16.5±7.416.5{\pm}7.4 24.124.1 4545 6565
W+FE 65.6±4.8\mathbf{65.6{\pm}4.8} 88.2±4.2\mathbf{88.2{\pm}4.2} 43.8±4.2\mathbf{43.8{\pm}4.2} 65.9\mathbf{65.9} 𝟕𝟎\mathbf{70} 𝟖𝟓\mathbf{85}

5.3 Q4: Policy Interface for Distributed FingerEye Feedback

Table 3 gives mean simulated success 48.5% for FEnc+FDec, 52.0% for NoEnc+FDec, 59.1% for NoEnc+GDec, 59.8% for GEnc+FDec and 65.9% for the full grouped model. The full model wins the mean but not every task: NoEnc+GDec slightly leads nut picking.

Real coin/letter mean success is 65.0% versus 52.5% for NoEnc+FDec. Attention diagnostics are consistent with preserving fingertip access, but are correlational and do not causally prove shortcut removal. Size matching helps control capacity while changing some widths across variants.

Table 3: Policy-interface comparison. W+FE input; simulation SR in %, mean±\pmstd.
Policy Coin Nut Pick Peg-in-Hole Mean
FEnc+FDec 43.6±8.343.6{\pm}8.3 83.7±6.183.7{\pm}6.1 18.2±7.518.2{\pm}7.5 48.548.5
NoEnc+FDec 48.6±8.348.6{\pm}8.3 80.7±11.380.7{\pm}11.3 26.6±7.626.6{\pm}7.6 52.052.0
NoEnc+GDec 54.9±7.554.9{\pm}7.5 88.9±4.1\mathbf{88.9{\pm}4.1} 33.5±8.033.5{\pm}8.0 59.159.1
GEnc+FDec 60.2±9.1¯\underline{60.2{\pm}9.1} 85.4±5.585.4{\pm}5.5 33.9±5.3¯\underline{33.9{\pm}5.3} 59.8¯\underline{59.8}
GEnc+GDec 65.6±4.8\mathbf{65.6{\pm}4.8} 88.2±4.2¯\underline{88.2{\pm}4.2} 43.8±4.2\mathbf{43.8{\pm}4.2} 65.9\mathbf{65.9}

6 Limitations and Conclusions

The exposed acrylic remains low-friction despite added gripping material in camera-hidden regions. Long-term and multi-axis durability are untested, and manual camera/tag design leaves coverage and occlusion trade-offs.

Evidence is limited to fixed-base, single-arm tasks. Simulation is a calibrated plate proxy rather than a validated nonlinear contact model; policies learn from demonstrations without tested recovery beyond those behaviors. The study supports the combined sensing/interface design on these tasks, not general force regulation or bimanual/mobile robustness.

Acknowledgement

The authors acknowledge assistance with simulation, hardware, discussions and rebuttal experiments. Funding includes Singapore Ministry of Education Academic Research Fund Tier 1 (FY2024), the President’s Graduate Fellowship and RoboScience; the source states that conclusions do not imply sponsor endorsement.

A.1 Additional Sensing Interface Design Details

The hardware appendix specifies optics, compliance, fabrication, marker estimation and the tracking/visual-coverage trade-off.

A.1.1 Hardware Design Details

Two HBVCAM-2307-FPC82 V11 USB cameras measure about 8.6×8.6×5.4 mm each and provide 120° fields of view. The tip works at roughly 10 mm; the root is tilted 21.48° with about 80-mm working distance.

A 1-mm acrylic plate carries thirty-five 2-mm tag25h9 markers with 0.2-mm white borders. A molded silicone ring with triangular internal elements provides directional compliance around the plate. Printed fixtures and M2×4 screws maintain alignment; cameras connect through USB-C.

The wedge affects physical accessibility to coin/letter edges, not just sensor packaging. It is part of the task advantage and must remain distinct from visual feedback alone.

A.1.2 Fabrication Details and Bill of Materials

Print base, camera wrappers and mold components. Mix silicone base and curing agent at mass ratio 100:(1–3), mold and cure for more than four hours, then demold. Apply the tag sticker inside the acrylic plate, bond the ring with JL-401 adhesive and let it cure; mount cameras and connect wiring.

Table 4 retains every material cost and the $60.19 total. The two cameras account for $56.49. These are the authors’ reported component costs, not a current vendor quotation or a full robot-system cost.

Complete original figure with every panel, label and legend.
Figure 8: Materials used for FingerEye fabrication. The module uses silicone elastomer and curing agent, camera modules, an acrylic plate with AprilTag sticker, silicone adhesive, and 3D-printed mold and fixture components.
Table 4: Bill of materials for one FingerEye module.
Item Cost (USD)
Acrylic plate 0.28
AprilTag sticker 0.42
Silicone elastomer & curing agent (20 g) 0.10
Silicone adhesive (JL-401, ∼\sim0.5 ml) 0.17
Soft silicone ring 0.02
Sensor base (3D-printed) 0.44
Camera wrappers (3D-printed) 0.45
Soft-ring mold (3D-printed) 1.78
M2×\times4 screws (8 pcs) 0.04
Camera modules (×\times2) 56.49
Total 60.19

A.1.3 Multi-Tag Deformation Estimation

Aggregate every visible tag corner into one 3D-to-2D correspondence set and solve equation 1 with EPnP plus Levenberg–Marquardt refinement. Known tag identities and a shared rigid-plate transform reduce instability from noisy single-tag estimates.

Subtract a no-contact reference pose to form six deformation coordinates. This estimates global plate motion, not a dense contact shape/location map; calibration then maps motion to wrench.

Equations 1
{(𝐗i,jref,𝐮i,j)},\big\{(\mathbf{X}_{i,j}^{\mathrm{ref}},\mathbf{u}_{i,j})\big\},
𝐓cam←ref=arg⁡min⁡∑i,j𝐓∈S​E​(3)⁡‖𝐮i,j−π⁡(𝐓𝐗i,jref)‖22,\mathbf{T}_{\mathrm{cam}\leftarrow\mathrm{ref}}=\arg\min_{\mathbf{T}\in SE(3)}\sum_{i,j}\left\|\mathbf{u}_{i,j}-\pi\!\left(\mathbf{T}\mathbf{X}_{i,j}^{\mathrm{ref}}\right)\right\|_{2}^{2}, (1)

A.1.4 Comparison with GelSight and Keyline Tracking

In coin standing, the tested GelSight struggled to sense side/table-edge interaction and its square geometry could not wedge under the coin reliably. Figure 9 preserves that failed configuration.

A keyline pipeline inspired by TacThru uses blob detection, nearest-neighbor association and temporal filtering. Qualitative tests show brittleness under large inter-frame motion and lighting change compared with identified AprilTags. No matched quantitative tracking benchmark is given here.

Complete original figure with every panel, label and legend.
Figure 9: GelSight failure in coin standing. The square sensor geometry makes edge wedging difficult in this setup.
Complete original figure with every panel, label and legend.
Figure 10: Qualitative comparison of marker-tracking pipelines. Keyline tracking is brittle under deformation and lighting, while AprilTags preserve tag identity and multi-corner pose constraints.

A.1.5 Design Trade-offs

Binocular hardware adds wiring/calibration complexity but separates near-contact and wider-view optics. Clear acrylic improves RGB continuity while reducing conformity and exposed-surface friction; gripping material is restricted to regions hidden from cameras.

Passive lighting avoids mode-dependent colors, with contrast enhancement for tag detection. Peripheral compliance supports edge and lateral loading. The deliberate output is RGB plus global plate pose, trading explicit contact maps for continuous approach information; neither darkness invariance nor complete contact reconstruction is established.

A.1.6 Tag-Count Ablation

Table 5 compares nested 1/5/20/35-tag layouts. Thirty-five tags reduce static jitter to 0.012 mm/0.109°, with 19.82 detected on average; twenty tags give 0.015 mm/0.120° and 11.86 detected.

Deviation is measured against the simultaneous full-layout estimate, not external pose ground truth. Twenty tags achieve 68.3% simulated mean success versus 65.9% for thirty-five, so better tracking redundancy does not monotonically improve the policy: tags also occupy visual area.

Table 5: Pose robustness versus tag count. Lower jitter and deviation are better.
Tags Jitter (mm/∘) Mean detected tags Deviation (mm/∘)
1 0.426 / 6.969 0.85 0.363 / 6.823
5 0.207 / 1.473 3.65 0.237 / 2.203
20 0.015 / 0.120 11.86 0.068 / 0.326
35 0.012 / 0.109 19.82 Reference

A.2 Sensing Interface Capability and Experiment Details

Calibration, analytical sensitivity, delicate-grasp control and compression cycling test different properties. Their results should not be collapsed into one general accuracy or durability guarantee.

A.2.1 Wrench–Deformation Calibration Setup

Apply one force/torque axis at a time using a digital scale and fixtures; derive torque from measured force and a known lever arm. Collect over 1,000 synchronized pose–wrench samples and split each axis 80:20 for fitting/testing.

Force held-out R² is at least 0.999, with RMSE 0.013/0.009/0.047 N for x/y/z. Torque R² is 0.999/0.994/0.999. This supports the calibrated proxy under axis-wise loading; it does not validate arbitrary combined loads, hysteresis compensation or unseen operating conditions.

Complete original figure with every panel, label and legend.
Figure 11: Calibration setup. A scale and 3D-printed fixtures provide axis-specific force and torque loading while the tag-array pose is recorded.
Equations
𝚫​𝑳=[δ​lx,δ​ly,δ​lz,δ​θx,δ​θy,δ​θz]𝖳,\bm{\Delta L}=[\delta l_{x},\delta l_{y},\delta l_{z},\delta\theta_{x},\delta\theta_{y},\delta\theta_{z}]^{\mathsf{T}},
𝐅=[fx,fy,fz,τx,τy,τz]𝖳.\mathbf{F}=[f_{x},f_{y},f_{z},\tau_{x},\tau_{y},\tau_{z}]^{\mathsf{T}}.
Complete original figure with every panel, label and legend.
Figure 12: Wrench–deformation calibration. Predicted forces and torques closely match held-out measurements.
Equation
𝐅=g⁡(𝚫​𝑳)\mathbf{F}=g(\bm{\Delta L})

A.2.2 Sensitivity Analysis Details

Assume tag localization accuracy of 0.25 pixel. For a 2-mm tag spanning 37 pixels, equation 2 gives translation sensitivity 0.0135 mm. Equation 3 uses θ=π/12 and image radius 18.5 pixels to obtain 0.0136 rad rotational sensitivity.

Propagate the six-component pose threshold through the calibrated linear mapping to obtain equation 5’s force/torque estimates. These assumption-based sensitivities are different from empirical prediction RMSE, and do not establish an observed false-alarm-controlled detection limit.

Equations 2–5
Δ​lmin=wtagwimg​dR.\Delta l_{\min}=\frac{w_{\text{tag}}}{w_{\text{img}}}\,d_{R}. (2)
Δ​lmin≈0.0135​mm.\Delta l_{\min}\approx 0.0135~\mathrm{mm}.
Δ​θmin=θ2​r​sin⁡(θ/2)​dR.\Delta\theta_{\min}=\frac{\theta}{2r\sin(\theta/2)}\,d_{R}. (3)
Δ​θmin≈0.0136​rad.\Delta\theta_{\min}\approx 0.0136~\mathrm{rad}.
𝚫​𝑳min=[ 0.0135, 0.0135, 0.0135, 0.0136, 0.0136, 0.0136]𝖳,\bm{\Delta L}_{\min}=[\,0.0135,\;0.0135,\;0.0135,\;0.0136,\;0.0136,\;0.0136\,]^{\mathsf{T}}, (4)
𝐅min=[ 4.30, 4.22, 9.93, 0.32, 0.13, 8.55]𝖳,\mathbf{F}_{\min}=[\,4.30,\;4.22,\;9.93,\;0.32,\;0.13,\;8.55\,]^{\mathsf{T}}, (5)

A.2.3 Delicate Grasping Setup and Hyperparameters

Following the contact-sensitive motivation of SpikeATac, interpolate predefined joint trajectories at 0.02-s intervals. Stop approach when normal plate displacement crosses a threshold selected once per object, then lift. Most objects use a custom support to avoid table scraping.

Table 6 retains all nine thresholds and trajectory lengths, including 0.01 mm for seaweed and 0.02 mm for 0.7-mm pencil lead. This 50-Hz scripted controller is separate from the 10-Hz learned policy. Object-specific thresholds and supports limit claims of autonomous general contact detection.

Table 6: Hyperparameters for delicate grasping experiments.
Object Contact threshold (mm) Total frames
0.7 mm pencil lead 0.02 15
Chip 0.10 30
Dessert cup 0.05 10
Seaweed 0.01 150
Grape 0.06 10
Cone 0.10 20
Eggshell 0.05 20
Wafer 0.05 5
Balloon 0.05 3

A.2.4 Normal-Compression Durability

After 10,000 cycles of 3-mm normal compression at 1 Hz, no cracking/delamination is observed. Maximum displacement hysteresis increases 0.39→0.46 mm, about 13→15% of the tested range; before/after responses differ by 0.13 mm RMS.

In twenty trials each, coin success is unchanged and chip picking loses one success. This is useful normal-compression evidence, with no long-term multi-axis or statistical equivalence conclusion.

A.3 Learning Interface Details

The architecture appendix makes the current implementation explicit: cached global visual summaries, one-step observation context, grouped transformer processing and clean learned action queries.

A.3.1 Policy Architecture Details

Use one observation step. Resize RGB to 192×256; frozen RADIO returns one 2,304-dimensional summary per camera. A camera-specific two-layer GELU MLP maps it to 512 dimensions and adds camera identity. Vector groups use separate MLP projections. No dense image-token grid is retained by this interface.

All FingerEye camera tokens form one group. GroupEnc applies self-attention only in groups with at least two tokens; singleton wrist/proprioception groups pass through unchanged. Defaults are four encoder/decoder layers, sixteen heads, width 512 and feed-forward width 2,048, with adjusted widths for size-matched ablations.

Initialize sixteen clean learnable action queries. Each decoder layer applies query self-attention, either flat cross-attention or the mean of separate group residuals, then a feed-forward residual. An MLP maps decoded queries to task-specific joint/arm commands. Equations 6–17 retain every tensor definition and update.

Main learned policies omit explicit plate pose/wrench. Optional plate histories are ordinary tokens when used, not query modulation. This architecture predicts chunks directly; the description does not specify a diffusion-noise process or a separate explicit-force controller.

Complete original figure with every panel, label and legend.
Figure 13: Policy architecture comparison. The ablation varies how observation tokens are encoded and how action-query tokens attend to modality groups; FEnc/GEnc and FDec/GDec abbreviate the FlatEnc/GroupEnc and FlatDec/GroupDec labels shown in the diagram. Left: NoEnc+FDec exposes tokenized observations directly to a flat decoder. Middle: FEnc+FDec applies flat self-attention across all observation tokens before flat decoding. Right: GEnc+GDec encodes proprioception, wrist, and FingerEye groups separately, then gives each group an independent decoder cross-attention path.
Equations 6–17
𝑨t=πθ​(𝑶t)=(𝒂t,𝒂t+1,…,𝒂t+Tp−1),\bm{A}_{t}=\pi_{\theta}(\bm{O}_{t})=(\bm{a}_{t},\bm{a}_{t+1},\ldots,\bm{a}_{t+T_{p}-1}), (6)
𝑶t=(𝒐t1,…,𝒐tM),\bm{O}_{t}=(\bm{o}_{t}^{1},\ldots,\bm{o}_{t}^{M}), (7)
𝒔t(i)∈ℝ2304.\bm{s}_{t}^{(i)}\in\mathbb{R}^{2304}. (8)
ϕi:ℝ2304→ℝ512,𝒗t(i)=ϕi​(𝒔t(i)).\phi_{i}:\mathbb{R}^{2304}\rightarrow\mathbb{R}^{512},\qquad\bm{v}_{t}^{(i)}=\phi_{i}(\bm{s}_{t}^{(i)}). (9)
𝒖t(i)=𝒗t(i)+𝒆cam(i).\bm{u}_{t}^{(i)}=\bm{v}_{t}^{(i)}+\bm{e}_{\mathrm{cam}}^{(i)}. (10)
𝑼tm=τm​(𝒐tm),𝑼t=[𝑼t1,…,𝑼tM],\bm{U}_{t}^{m}=\tau_{m}(\bm{o}_{t}^{m}),\qquad\bm{U}_{t}=[\bm{U}_{t}^{1},\ldots,\bm{U}_{t}^{M}], (11)
NoEnc:𝒁tnoenc=𝑼t,FlatEnc:𝒁tflat=Eflat​(𝑼t),GroupEnc:𝒁tgroup=[E1​(𝑼t1),…,EM​(𝑼tM)].\begin{array}[]{@{}r@{\;}c@{\quad}r@{\;}c@{\;}l@{}}\text{NoEnc}&:&\bm{Z}_{t}^{\mathrm{noenc}}&=&\bm{U}_{t},\\ \text{FlatEnc}&:&\bm{Z}_{t}^{\mathrm{flat}}&=&E_{\mathrm{flat}}(\bm{U}_{t}),\\ \text{GroupEnc}&:&\bm{Z}_{t}^{\mathrm{group}}&=&[E_{1}(\bm{U}_{t}^{1}),\ldots,E_{M}(\bm{U}_{t}^{M})].\end{array} (12)
𝒉t,k0=𝒒k,𝒒k∈ℝ512,k=1,…,Tp.\bm{h}_{t,k}^{0}=\bm{q}_{k},\qquad\bm{q}_{k}\in\mathbb{R}^{512},\qquad k=1,\ldots,T_{p}. (13)
𝒉¯tl=𝒉tl+SelfAttnl​(𝒉tl).\bar{\bm{h}}_{t}^{l}=\bm{h}_{t}^{l}+\mathrm{SelfAttn}^{l}(\bm{h}_{t}^{l}). (14)
FlatDec:𝒉~tl=𝒉¯tl+CrossAttnl​(𝒉¯tl,𝒁t),GroupDec:𝒉~tl=𝒉¯tl+1M​∑m=1MCrossAttnml​(𝒉¯tl,𝒛tm).\begin{array}[]{@{}r@{\;}c@{\quad}r@{\;}c@{\;}l@{}}\text{FlatDec}&:&\tilde{\bm{h}}_{t}^{l}&=&\bar{\bm{h}}_{t}^{l}+\mathrm{CrossAttn}^{l}(\bar{\bm{h}}_{t}^{l},\bm{Z}_{t}),\\ \text{GroupDec}&:&\tilde{\bm{h}}_{t}^{l}&=&\bar{\bm{h}}_{t}^{l}+\frac{1}{M}\sum_{m=1}^{M}\mathrm{CrossAttn}_{m}^{l}(\bar{\bm{h}}_{t}^{l},\bm{z}_{t}^{m}).\end{array} (15)
𝒉tl+1=𝒉~tl+FFNl​(𝒉~tl).\bm{h}_{t}^{l+1}=\tilde{\bm{h}}_{t}^{l}+\mathrm{FFN}^{l}(\tilde{\bm{h}}_{t}^{l}). (16)
𝒂^t+k−1=fact(𝒉t,kL),k=1,…,Tp.\hat{\bm{a}}_{t+k-1}=f_{\mathrm{act}}(\bm{h}_{t,k}^{L}),\qquad k=1,\ldots,T_{p}. (17)
𝒂t∈ℝda\bm{a}_{t}\in\mathbb{R}^{d_{a}}
𝑱t∈ℝds\bm{J}_{t}\in\mathbb{R}^{d_{s}}
i∈{1,…,Ncam}i\in\{1,\ldots,N_{\mathrm{cam}}\}
𝒆cam(i)∈ℝ512\bm{e}_{\mathrm{cam}}^{(i)}\in\mathbb{R}^{512}
𝒁t=[𝒛t1,…,𝒛tM]\bm{Z}_{t}=[\bm{z}_{t}^{1},\ldots,\bm{z}_{t}^{M}]

A.4 Decoder Attention Diagnostics

Table 7 measures flat-decoder attention for training seed 42 across three evaluation seeds, alongside success statistics from the main multi-training-seed comparison. The single proprioception token receives 35.6–59.0% attention versus a token-uniform reference of 16.7% among six tokens.

For each of three tasks, the stronger flat variant also has more total FingerEye attention. Wrist mass stays 7.8–12.9%. This is consistent with competition at the decoder, but attention is not a causal importance measure; encoded tokens can already contain cross-modal information and group sizes differ.

Table 7: Decoder cross-attention diagnostic for flat decoders. We aggregate test-time action-query cross-attention over context tokens and map tokens to FingerEye (FE), Wrist, and Proprio groups. Prop. win rate denotes the fraction of action-query samples for which the Proprio group receives the largest group-level attention mass. SR is reported for context from the corresponding main-paper policy-interface evaluation; attention values are computed for train seed 4242 and averaged over three evaluation seeds. All entries are percentages.
Task Policy SR FE mass Wrist mass Prop. mass Prop. win rate
Coin NoEnc+FDec 48.6±8.348.6{\pm}8.3 51.6±0.451.6{\pm}0.4 12.9±0.112.9{\pm}0.1 35.6±0.435.6{\pm}0.4 18.9±0.818.9{\pm}0.8
Coin FEnc+FDec 43.6±8.343.6{\pm}8.3 46.0±0.146.0{\pm}0.1 10.5±0.010.5{\pm}0.0 43.4±0.143.4{\pm}0.1 37.6±0.437.6{\pm}0.4
Nut Pick NoEnc+FDec 80.7±11.380.7{\pm}11.3 31.3±0.231.3{\pm}0.2 9.8±0.29.8{\pm}0.2 59.0±0.159.0{\pm}0.1 77.2±0.477.2{\pm}0.4
Nut Pick FEnc+FDec 83.7±6.183.7{\pm}6.1 48.8±0.048.8{\pm}0.0 11.8±0.011.8{\pm}0.0 39.5±0.039.5{\pm}0.0 34.1±0.134.1{\pm}0.1
Peg-in-Hole NoEnc+FDec 26.6±7.626.6{\pm}7.6 52.0±0.352.0{\pm}0.3 10.1±0.110.1{\pm}0.1 38.0±0.338.0{\pm}0.3 22.8±0.722.8{\pm}0.7
Peg-in-Hole FEnc+FDec 18.2±7.518.2{\pm}7.5 42.2±0.042.2{\pm}0.0 7.8±0.07.8{\pm}0.0 50.0±0.150.0{\pm}0.1 61.0±1.161.0{\pm}1.1

A.5 Experiment Details

The final appendix preserves shared optimizer settings, task-specific demonstrations/horizons and the precise tactile/wrench baselines.

A.5.1 Default Policy-Training Settings

Table 8 retains AdamW learning rate 10⁻⁴, β=(0.95,0.999), ε=10⁻⁸ and weight decay 10⁻⁶. Use cosine scheduling, 500 warmup steps, batch 256, zero dropout and EMA with maximum decay 0.9999; evaluate the final checkpoint. Epoch counts and action horizons vary by task.

Table 8: Default policy-training settings.
Category Default setting
Temporal context To=1T_{o}=1 observation step; task-specific Tp/TaT_{p}/T_{a} in Table 9.
Policy backbone 44 encoder layers, 44 decoder layers, hidden dimension 512512, 1616 attention heads, feed-forward dimension 20482048, GELU activation, dropout 0.00.0.
Modality fusion Group encoding and group decoding enabled by default, corresponding to GEnc+GDec.
Optimizer AdamW with learning rate 1.0×10−41.0\times 10^{-4}, β=(0.95,0.999)\beta=(0.95,0.999), ϵ=1.0×10−8\epsilon=1.0\times 10^{-8}, and weight decay 1.0×10−61.0\times 10^{-6}.
Schedule and batch Cosine learning-rate schedule, 500500 warmup steps, batch size 256256.
EMA and evaluation Exponential moving average enabled with maximum decay 0.99990.9999; evaluation uses the final checkpoint.

A.5.2 Task-Specific Training Settings

Simulation uses 50 coin, 50 nut and 200 peg demonstrations; real tasks use 60 coin, 30 chip, 60 letter and forty syringe demonstrations. Train 500 epochs except simulated peg at 300. All methods within a task share data and splits.

Simulated train/test initial states come from the same randomized ranges. Real tests interpolate/extrapolate within listed configurations: coin diameters 20/40 mm after 10/30/50-mm training, 45° letters after 30°/60°, and 20-mL syringes after 10/30 mL. Chip tests include full and irregular pieces. These are limited configuration generalization tests, not arbitrary unseen object categories.

Four FE cameras mean two binocular fingertip modules, not four modules. Syringe uses six cameras/three modules. Table 9 retains every action dimension, horizon, object condition and camera count; simulated coin/peg execute full sixteen-action chunks.

Table 9: Task-specific policy and dataset settings.
Setting Task Cams in use Demos Epochs Tp/TaT_{p}/T_{a} Action space Train config. Test config.
Simulation Coin standing 4FE+1W 50 500 16/1616/16 Abs. hand joints (8) Randomized coin pose Same randomized range
Simulation Nut picking 4FE+1W 50 500 16/816/8 Abs. wrist position (3) + abs. hand joints (8) Randomized nut/support + wrist init Same randomized range
Simulation Peg-in-Hole pushing 4FE+1W 200 300 16/1616/16 Abs. hand joints (8) Randomized peg/hole offsets Same randomized range
Real-world Coin standing 4FE+1W 60 500 16/816/8 Abs. hand joints (8) Rand. pose; coin D=10/30/50D=10/30/50 mm Rand. pose; coin D=20/40D=20/40 mm
Real-world Chip picking 4FE+1W 30 500 16/816/8 Abs. hand joints (8) + rel. zz (1) Rand. pose; chip full/half/quarter Rand. pose; chip full/irregular
Real-world Letter retrieving 4FE+1W 60 500 16/816/8 Abs. hand joints (8) + arm joints (7) Rand. pose; letter A=60∘/30∘A=60^{\circ}/30^{\circ} Rand. pose; letter A=45∘A=45^{\circ}
Real-world Syringe manipulation 6FE+1W 40 500 16/816/8 Abs. hand joints (12) + binary arm go-to pose command (1) Rand. pose; syringe 10/30 mL Rand. pose; syringe 20 mL

Note. Here, 4​FE4\mathrm{FE} denotes four FingerEye camera views, i.e., two FingerEye modules with two cameras each; 6​FE6\mathrm{FE} denotes three FingerEye modules with two cameras each. 1​W1\mathrm{W} denotes one wrist RGB camera.

A.5.3 Post-Contact Tactile Map Baseline

Rasterize thumb/index contact planes into 64×96 grids. Equation 18 returns max(0,1 mm−distance to nearby collision geometry), so signals begin only within 1 mm. Encode each map with the same frozen RADIO backbone used for RGB.

This simulated baseline represents local proximity geometry rather than forces, silicone deformation or a validated tactile image distribution. Its poor performance cannot by itself show that all post-contact tactile policies are ineffective.

Equation 18
cg=max⁡(0,dcontact−dg),dcontact=1​mm.c_{g}=\max(0,d_{\mathrm{contact}}-d_{g}),\qquad d_{\mathrm{contact}}=1~\mathrm{mm}. (18)

A.5.4 Explicit Wrench-Input Ablation

Adding six-dimensional simulated wrench input to wrist vision gives 32.8/43.9/23.8% success across coin/nut/peg, averaging 33.5% versus 26.7% without it. Adding wrench to W+FE raises its mean 65.9→67.1%.

The small incremental gain is consistent with redundancy on these tasks, but does not prove explicit force input is unnecessary for other force-regulation tasks. Per-seed variability for this auxiliary ablation is not reported.

Concept demonstration

Open image ↗