Literature
Display
← All papers

FlowTouch: View-Invariant Visuo-Tactile Prediction

Seongjin Bien · Carlo Kneissl · Tobias Jülg · Frank Fundel · Thomas Ressler-Antal · Florian Walter · Björn Ommer · Gitta Kutyniok · Wolfram Burgard

Preprint posted · arXiv (preprint) ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

At a glance

FlowTouch predicts static tactile images from local object geometry and a sensor’s no-contact background. A reconstructed mesh and a proposed contact pose provide points with surface normals, which condition a latent flow-matching generator. Simulation pretraining and limited real-data adaptation improve prediction, including partial transfer to unseen sensor instances. A simulated grasp-stability classifier demonstrates useful downstream information. “View-invariant” describes the geometry-based representation; reconstruction and millimeter-scale alignment remain significant dependencies.

I Introduction

Touch is unavailable before contact, so anticipating tactile patterns could help evaluate candidate interactions. Direct image-to-touch mappings entangle contact geometry with scene appearance and camera viewpoint. FlowTouch inserts a local 3D representation between vision and tactile generation. It predicts contact appearance at a supplied pose rather than selecting a grasp, inferring arbitrary interaction forces, or demonstrating a complete manipulation controller.

Refer to caption
Fig. 1: FlowTouch at a glance: The robot first looks at the object and creates a mesh with scene mesh generation foundation models. Given the particular touch point on the mesh and the static background image of the tactile sensor, FlowTouch then predicts the resulting tactile image.

II Related Work

The work combines visual-to-tactile prediction, tactile simulation, and datasets with sufficiently accurate geometry and contact poses.

II-A Vision-to-Tactile Image Prediction

Image-conditioned methods often require interaction images or camera views close to the eventual tactile contact. Object-specific neural-field approaches can provide those views but need per-object fitting. FlowTouch instead uses a scene-reconstruction model to obtain geometry from an initial observation, then queries hypothetical contacts. This shifts the burden from camera-view matching to mesh and pose accuracy.

II-B Mesh-based Tactile Image Generation in Simulation

Tactile simulators render gel deformation from geometry and contact, but their appearance differs from physical sensors. FlowTouch uses simulated data for scale and then learns adaptation from real images, rather than assuming simulator output directly matches hardware.

II-C Tactile Datasets

Geometry-conditioned prediction requires object meshes and sensor poses in addition to images. Existing real datasets provide these only for limited contacts and objects. Synthetic primitive shapes therefore supply diverse local geometries and pose variations, while real data adapt sensor appearance.

Refer to caption
Fig. 2: The FlowTouch architecture: The blue area shows the image-to-PCN sampling pipeline, while the green area shows the generative model’s structure. Snowflakes indicate components with frozen weights.

III Methodology

The model separates contact-geometry extraction from tactile image generation. The retained equations specify the latent interpolation, conditioning, and adaptation losses.

III-A Preliminaries

Each prediction is conditioned on 5,000 sampled 3D points with normals near the proposed contact and a no-contact background from the relevant tactile sensor. The target is a distribution over tactile RGB images. The background is therefore an explicit sensor-instance input even when the sensor was not used for model training.

III-B FlowTouch

The first stage creates an aligned local contact representation; the second generates a tactile image through conditional flow matching in a compressed latent space.

III-B 1 Sampling the PCN

SceneComplete [26] takes RGB-D, camera intrinsics, and a language description to segment and reconstruct the object, then aligns its mesh to the robot frame. A candidate gripper pose determines the tactile-pad poses. CoACD decomposes collision geometry for MuJoCo, while samples are taken from the original mesh. Pads move inward from open-gripper positions until they penetrate the collision model by 3 mm; points within each pad’s visible area form the conditioning input. This procedure uses a fixed contact-depth assumption and precise calibration, rather than predicting contact from an unconstrained RGB image alone.

III-B 2 Latent Encoding

A frozen tiny autoencoder reduces image resolution eightfold and maps RGB images to four-channel latents. The generator operates in this smaller space and the decoder reconstructs the final tactile image.

III-B 3 Flow Matching Objective

Conditional flow matching [25] trains a U-Net-style transformer to predict the velocity along a straight interpolation from Gaussian noise to the target latent. At inference, Euler integration follows the learned velocity field from noise to an image latent. Geometry conditions the predicted flow throughout generation.

Equations 1–2
𝐳t=(1−t)​𝐳0+t⋅𝐳1,\mathbf{z}_{t}=(1-t)\mathbf{z}_{0}+t\cdot\mathbf{z}_{1}, (1)
ℒC​F​M​(θ)=𝔼t,𝐳0,𝐳1​[|vθ​(𝐳t,t,𝐦)−(𝐳1−𝐳0)|2].\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,\mathbf{z}_{0},\mathbf{z}_{1}}\left[\left|v_{\theta}(\mathbf{z}_{t},t,\mathbf{m})-(\mathbf{z}_{1}-\mathbf{z}_{0})\right|^{2}\right]. (2)

III-B 4 Background Image as Channel-Stacked Context

The encoded no-contact background is concatenated channel-wise with the noisy tactile latent at every integration step. This gives the model spatially aligned information about sensor illumination and appearance without a separate image-encoder attention pathway.

Equation 3
𝐳~t=[𝐳t|𝐛z]∈ℝh×w×8\tilde{\mathbf{z}}_{t}=[\mathbf{z}_{t}|\mathbf{b}_{z}]\in\mathbb{R}^{h\times w\times 8} (3)

III-B 5 Cross-attention Mesh Conditioning

A linear projection maps each six-dimensional point-and-normal sample to the transformer’s hidden dimension. Image tokens query these projected points through cross-attention. The model therefore receives detailed local geometry directly, without an additional pretrained point-cloud encoder in its default configuration.

Equation 4
CrossAttn​(SelfAttn​(𝐗),𝐦𝐖mesh).\text{CrossAttn}(\text{SelfAttn}(\mathbf{X}),\mathbf{mW_{\text{mesh}}}). (4)

III-B 6 Domain Adaptation

Pretraining uses synthetic data for 30,000 steps; another 20,000 steps mix 80% synthetic and 20% real data. Tested additions are a domain token, optimizer reset with reduced learning rate, color augmentation, and a perceptual tactile loss. The latter compares generated and target images through the frozen Sparsh encoder [34], using a one-step estimate of the clean latent during training. This loss favors tactile-relevant features rather than necessarily maximizing pixel-level image metrics.

Equations 5–7
CrossAttn​(SelfAttn​(𝐗),[𝐦𝐖pcn|𝐖dom])\text{CrossAttn}(\text{SelfAttn}(\mathbf{X}),[\mathbf{mW_{\text{pcn}}}|\mathbf{W}_{\text{dom}}]) (5)
ℒSparsh=‖ℰSparsh​(𝒟⁡(𝐳^1))−ℰSparsh​(𝐱1)‖2,\mathcal{L}_{\text{Sparsh}}=\|\mathcal{E}_{\text{Sparsh}}(\mathcal{D}(\hat{\mathbf{z}}_{1}))-\mathcal{E}_{\text{Sparsh}}(\mathbf{x}_{1})\|_{2}, (6)
ℒF​T=ℒCFM+ℒSparsh.\mathcal{L}_{FT}=\mathcal{L}_{\text{CFM}}+\mathcal{L}_{\text{Sparsh}}. (7)

III-C Synthetic Data Generation

Taxim [18] is integrated with MuJoCo to generate contacts on varied primitive shapes. Sampling emphasizes edges and corners, with local translation and rotation perturbations, rather than overrepresenting flat surfaces. Multi-scale normal-direction noise approximates mesh-reconstruction errors and randomized backgrounds reduce sensor-specific bias. The paper reports over 100,000 samples in under an hour on a consumer laptop; this is synthetic generation throughput, not full reconstruction-and-prediction latency.

Refer to caption
Fig. 3: Left: A selection of primitive geometries used for generating the simulation data. Right: Taxim GelSight images rendered from a single contact point (corner of a cube), with local translations and rotations.

IV Results

Evaluation covers architecture, sim-to-real adaptation, unseen sensor instances, and a simulated grasp-stability task. Web datasets use known meshes and contact poses with object-level splits; the custom real dataset uses reconstructed meshes. The grasp experiment instead splits contacts within the same six objects. Source counts are inconsistent: Table I lists 547 custom DIGIT samples versus 537 in prose, and 13,372/3,379 grasp train/test samples versus a prose statement of 10,000 collected samples. The original table is retained without silently reconciling these differences.

TABLE I: List of datasets used for evaluation. * indicates within-object split.
Name Training # (# obj.) Validation # (# obj.)
GelSight-Web 887 (36) 135 (6)
GelSight-Sim 200K (135) 113K (60)
DIGIT-Web 2578 (7) 1084 (3)
DIGIT-Sim 200K (135) —
DIGIT-Ours — 547 (13)
GelSight-Grasp 13372 (6*) 3379 (6*)

IV-A Model Architecture Design Choices

On held-out GelSight-Web objects, background stacking improves PSNR from 24.28 to 25.53, SSIM from 0.52 to 0.59, and LPIPS from 0.40 to 0.35. It performs similarly to background encoding while avoiding the additional encoder. Adding the overall scene image does not improve results. A pretrained point-cloud encoder likewise shows little advantage in the reported test. The authors caution that tactile image metrics are sensitive to calibration-induced shifts and do not directly measure contact usefulness.

TABLE II: Ablation study on network design choices evaluated on the GelSight OFR dataset. All metrics are reported on the held-out validation set.
Variant PSNR (↑\uparrow) SSIM (↑\uparrow) LPIPS (↓\downarrow)
Base 24.28 0.52 0.40
BG-Enc 25.52 0.57 0.34
BG-Scene-Enc 24.91 0.54 0.38
BG-Stack 25.53 0.59 0.35

IV-B Training Techniques for Domain Adaptation

Domain conditioning gives the strongest image-metric improvements across all three validation datasets, while color augmentation hurts them. Combining domain conditioning, optimizer reset, and tactile loss is slightly worse on these metrics than domain conditioning alone, despite some sharper-looking contacts. Double-edge structures can be forgotten during fine-tuning, indicating imperfect geometry transfer.

Table III and its prose disagree about some tactile-loss changes. The table gives PSNR changes of +0.17 for GelSight-Web, −0.20 for DIGIT-Web, and +0.06 for unseen DIGIT-Ours. The summary follows these values rather than the prose’s claim that both DIGIT scores increase. Better downstream classification is separate evidence for the tactile loss.

TABLE III: Quantitative comparison of different sim-to-real training techniques from finetuning on GelSight-Web, DIGIT-Web, and unseen DIGIT-Ours datasets after simulation pre-training. Values for experimental methods are shown as absolute differences relative to the Naive Finetuning baseline and are calculated on the validation dataset.
Training Method GelSight-Web DIGIT-Web DIGIT-Ours (unseen)
PSNR (↑\uparrow) SSIM (↑\uparrow) LPIPS (↓\downarrow) PSNR (↑\uparrow) SSIM (↑\uparrow) LPIPS (↓\downarrow) PSNR (↑\uparrow) SSIM (↑\uparrow) LPIPS (↓\downarrow)
Naive Finetuning 24.5824.58 0.5530.553 0.4100.410 20.2320.23 0.6120.612 0.2840.284 16.0516.05 0.3550.355 0.4150.415
Color Augmentation −0.43-0.43 −0.031-0.031 +0.009+0.009 −0.56-0.56 −0.015-0.015 +0.013+0.013 −0.37-0.37 −0.030-0.030 +0.004+0.004
Tactile Loss +0.17+0.17 −0.001-0.001 −0.006-0.006 −0.20-0.20 −0.006-0.006 +0.005+0.005 +0.06+0.06 −0.000-0.000 −0.002-0.002
Optimizer Reset +0.28+0.28 −0.005-0.005 −0.004-0.004 −0.03-0.03 −0.001-0.001 +0.000+0.000 +0.07+0.07 +0.001+0.001 −0.001-0.001
Domain Conditioning +0.90\mathbf{+0.90} +0.029\mathbf{+0.029} −0.045\mathbf{-0.045} +2.18\mathbf{+2.18} +0.058\mathbf{+0.058} −0.055\mathbf{-0.055} +0.78\mathbf{+0.78} +0.032\mathbf{+0.032} −0.021\mathbf{-0.021}
Combined +0.86+0.86 +0.025+0.025 −0.041-0.041 +1.77+1.77 +0.050+0.050 −0.047-0.047 +0.23+0.23 +0.001+0.001 −0.003-0.003

IV-C Generalization Towards New Sensor Instances

The held-out custom dataset uses thirteen household objects, a Franka FR3 with DIGIT fingertips, a wrist RGB-D camera, and calibrated kinematics. Objects are fixed while contact samples are collected. Models trained on the web and simulated datasets partially transfer to these sensor instances, but absolute image quality is substantially worse: the baseline unseen-set PSNR is 16.05 versus 20.23 on DIGIT-Web, and domain conditioning improves it to 16.83. Qualitative shape recovery demonstrates some transfer, not uniform sensor-independent fidelity.

Refer to caption
Fig. 4: Tactile predictions of the model ablations listed in Tab. III. Samples are taken from the validation dataset.
Refer to caption
Fig. 5: Left: Objects used for data collection. Center: Robot setup. Right: Collected grasp poses aligned to generated mesh, with unique color for each finger pair.

IV-D Grasp Stability

A ResNet-based classifier predicts simulated grasp success from either ground-truth tactile images or FlowTouch predictions. Experiments use six objects, balanced successes/failures, a within-object split, and a 3,379-example test set; reported accuracies average 100 classifier trainings. Ground-truth tactile input reaches 85.83%, while prediction variants achieve 86.06%, 85.17%, 83.74%, and 81.35% under progressively less task-specific adaptation.

The last variant uses no grasp-dataset adaptation, and removing tactile loss lowers its accuracy to 78.59%, a 2.76-point difference. This supports retention of useful contact information across data regimes. The tiny advantage over ground-truth input is not evidence that generated touch is intrinsically superior, and the experiment is simulated classification rather than a real-robot grasp-success trial or held-out-object evaluation.

Refer to caption
Fig. 6: Meshes used for the ObjectFolder Benchmark grasp stability test.
TABLE IV: Training dataset ablations for grasp stability prediction task. Accuracies are averaged from 100 separate trainings over the test dataset (n=3379).
Classifier Image Source Pre-training Datasets Fine-tuning Datasets Acc. (%)
GT — — 85.83
Comb. A GelSight-Grasp — 86.06
Comb. B Both Sims GelSight-Grasp 85.17
Comb. C Sims, Webs 10% GelSight-Grasp 83.74
Comb. D Both Sims Both Webs 81.35
No Tactile Loss Both Sims Both Webs 78.59
Refer to caption
Fig. 7: Tactile predictions from the grasp stability task on the dataset ablations listed in Tab. IV. Samples are taken from the validation dataset.

IV-E Limitations

Prediction depends strongly on mesh quality and millimeter-level alignment. Unseen geometry remains difficult. Force is not an explicit input; its effects are represented only indirectly through sampled penetration geometry. Material properties, detailed textures, gel deformation, and dynamic contact are therefore not fully modeled.

V Conclusion and Future Work

Local geometry and sensor-background conditioning provide a useful route from visual reconstruction to anticipated tactile images. The evidence supports partial sim-to-real and sensor-instance transfer plus simulated downstream classification. Texture-aware conditioning, better geometry, calibrated force/deformation models, and real manipulation evaluation remain natural extensions.

Acknowledgment

The authors acknowledge GeniusRobot, Robotics Institute Germany, the Erlangen HPC resources, the DAAD Konrad Zuse Schools program, and the Next Generation AI Computing project and their listed funders.

Concept demonstration

Open image ↗