At a glance
FlowTouch predicts static tactile images from local object geometry and a sensor’s no-contact background. A reconstructed mesh and a proposed contact pose provide points with surface normals, which condition a latent flow-matching generator. Simulation pretraining and limited real-data adaptation improve prediction, including partial transfer to unseen sensor instances. A simulated grasp-stability classifier demonstrates useful downstream information. “View-invariant” describes the geometry-based representation; reconstruction and millimeter-scale alignment remain significant dependencies.
I Introduction
Touch is unavailable before contact, so anticipating tactile patterns could help evaluate candidate interactions. Direct image-to-touch mappings entangle contact geometry with scene appearance and camera viewpoint. FlowTouch inserts a local 3D representation between vision and tactile generation. It predicts contact appearance at a supplied pose rather than selecting a grasp, inferring arbitrary interaction forces, or demonstrating a complete manipulation controller.
II Related Work
The work combines visual-to-tactile prediction, tactile simulation, and datasets with sufficiently accurate geometry and contact poses.
II-A Vision-to-Tactile Image Prediction
Image-conditioned methods often require interaction images or camera views close to the eventual tactile contact. Object-specific neural-field approaches can provide those views but need per-object fitting. FlowTouch instead uses a scene-reconstruction model to obtain geometry from an initial observation, then queries hypothetical contacts. This shifts the burden from camera-view matching to mesh and pose accuracy.
II-B Mesh-based Tactile Image Generation in Simulation
Tactile simulators render gel deformation from geometry and contact, but their appearance differs from physical sensors. FlowTouch uses simulated data for scale and then learns adaptation from real images, rather than assuming simulator output directly matches hardware.
II-C Tactile Datasets
Geometry-conditioned prediction requires object meshes and sensor poses in addition to images. Existing real datasets provide these only for limited contacts and objects. Synthetic primitive shapes therefore supply diverse local geometries and pose variations, while real data adapt sensor appearance.
III Methodology
The model separates contact-geometry extraction from tactile image generation. The retained equations specify the latent interpolation, conditioning, and adaptation losses.
III-A Preliminaries
Each prediction is conditioned on 5,000 sampled 3D points with normals near the proposed contact and a no-contact background from the relevant tactile sensor. The target is a distribution over tactile RGB images. The background is therefore an explicit sensor-instance input even when the sensor was not used for model training.
III-B FlowTouch
The first stage creates an aligned local contact representation; the second generates a tactile image through conditional flow matching in a compressed latent space.
III-B 1 Sampling the PCN
SceneComplete [26] takes RGB-D, camera intrinsics, and a language description to segment and reconstruct the object, then aligns its mesh to the robot frame. A candidate gripper pose determines the tactile-pad poses. CoACD decomposes collision geometry for MuJoCo, while samples are taken from the original mesh. Pads move inward from open-gripper positions until they penetrate the collision model by 3 mm; points within each pad’s visible area form the conditioning input. This procedure uses a fixed contact-depth assumption and precise calibration, rather than predicting contact from an unconstrained RGB image alone.
III-B 2 Latent Encoding
A frozen tiny autoencoder reduces image resolution eightfold and maps RGB images to four-channel latents. The generator operates in this smaller space and the decoder reconstructs the final tactile image.
III-B 3 Flow Matching Objective
Conditional flow matching [25] trains a U-Net-style transformer to predict the velocity along a straight interpolation from Gaussian noise to the target latent. At inference, Euler integration follows the learned velocity field from noise to an image latent. Geometry conditions the predicted flow throughout generation.
Equations 1–2
| (1) |
| (2) |
III-B 4 Background Image as Channel-Stacked Context
The encoded no-contact background is concatenated channel-wise with the noisy tactile latent at every integration step. This gives the model spatially aligned information about sensor illumination and appearance without a separate image-encoder attention pathway.
Equation 3
| (3) |
III-B 5 Cross-attention Mesh Conditioning
A linear projection maps each six-dimensional point-and-normal sample to the transformer’s hidden dimension. Image tokens query these projected points through cross-attention. The model therefore receives detailed local geometry directly, without an additional pretrained point-cloud encoder in its default configuration.
Equation 4
| (4) |
III-B 6 Domain Adaptation
Pretraining uses synthetic data for 30,000 steps; another 20,000 steps mix 80% synthetic and 20% real data. Tested additions are a domain token, optimizer reset with reduced learning rate, color augmentation, and a perceptual tactile loss. The latter compares generated and target images through the frozen Sparsh encoder [34], using a one-step estimate of the clean latent during training. This loss favors tactile-relevant features rather than necessarily maximizing pixel-level image metrics.
Equations 5–7
| (5) |
| (6) |
| (7) |
III-C Synthetic Data Generation
Taxim [18] is integrated with MuJoCo to generate contacts on varied primitive shapes. Sampling emphasizes edges and corners, with local translation and rotation perturbations, rather than overrepresenting flat surfaces. Multi-scale normal-direction noise approximates mesh-reconstruction errors and randomized backgrounds reduce sensor-specific bias. The paper reports over 100,000 samples in under an hour on a consumer laptop; this is synthetic generation throughput, not full reconstruction-and-prediction latency.
IV Results
Evaluation covers architecture, sim-to-real adaptation, unseen sensor instances, and a simulated grasp-stability task. Web datasets use known meshes and contact poses with object-level splits; the custom real dataset uses reconstructed meshes. The grasp experiment instead splits contacts within the same six objects. Source counts are inconsistent: Table I lists 547 custom DIGIT samples versus 537 in prose, and 13,372/3,379 grasp train/test samples versus a prose statement of 10,000 collected samples. The original table is retained without silently reconciling these differences.
IV-A Model Architecture Design Choices
On held-out GelSight-Web objects, background stacking improves PSNR from 24.28 to 25.53, SSIM from 0.52 to 0.59, and LPIPS from 0.40 to 0.35. It performs similarly to background encoding while avoiding the additional encoder. Adding the overall scene image does not improve results. A pretrained point-cloud encoder likewise shows little advantage in the reported test. The authors caution that tactile image metrics are sensitive to calibration-induced shifts and do not directly measure contact usefulness.
IV-B Training Techniques for Domain Adaptation
Domain conditioning gives the strongest image-metric improvements across all three validation datasets, while color augmentation hurts them. Combining domain conditioning, optimizer reset, and tactile loss is slightly worse on these metrics than domain conditioning alone, despite some sharper-looking contacts. Double-edge structures can be forgotten during fine-tuning, indicating imperfect geometry transfer.
Table III and its prose disagree about some tactile-loss changes. The table gives PSNR changes of +0.17 for GelSight-Web, −0.20 for DIGIT-Web, and +0.06 for unseen DIGIT-Ours. The summary follows these values rather than the prose’s claim that both DIGIT scores increase. Better downstream classification is separate evidence for the tactile loss.
| Training Method | GelSight-Web | DIGIT-Web | DIGIT-Ours (unseen) | ||||||
| PSNR () | SSIM () | LPIPS () | PSNR () | SSIM () | LPIPS () | PSNR () | SSIM () | LPIPS () | |
| Naive Finetuning | |||||||||
| Color Augmentation | |||||||||
| Tactile Loss | |||||||||
| Optimizer Reset | |||||||||
| Domain Conditioning | |||||||||
| Combined | |||||||||
IV-C Generalization Towards New Sensor Instances
The held-out custom dataset uses thirteen household objects, a Franka FR3 with DIGIT fingertips, a wrist RGB-D camera, and calibrated kinematics. Objects are fixed while contact samples are collected. Models trained on the web and simulated datasets partially transfer to these sensor instances, but absolute image quality is substantially worse: the baseline unseen-set PSNR is 16.05 versus 20.23 on DIGIT-Web, and domain conditioning improves it to 16.83. Qualitative shape recovery demonstrates some transfer, not uniform sensor-independent fidelity.
IV-D Grasp Stability
A ResNet-based classifier predicts simulated grasp success from either ground-truth tactile images or FlowTouch predictions. Experiments use six objects, balanced successes/failures, a within-object split, and a 3,379-example test set; reported accuracies average 100 classifier trainings. Ground-truth tactile input reaches 85.83%, while prediction variants achieve 86.06%, 85.17%, 83.74%, and 81.35% under progressively less task-specific adaptation.
The last variant uses no grasp-dataset adaptation, and removing tactile loss lowers its accuracy to 78.59%, a 2.76-point difference. This supports retention of useful contact information across data regimes. The tiny advantage over ground-truth input is not evidence that generated touch is intrinsically superior, and the experiment is simulated classification rather than a real-robot grasp-success trial or held-out-object evaluation.
| Classifier Image Source | Pre-training Datasets | Fine-tuning Datasets | Acc. (%) |
| GT | — | — | 85.83 |
| Comb. A | GelSight-Grasp | — | 86.06 |
| Comb. B | Both Sims | GelSight-Grasp | 85.17 |
| Comb. C | Sims, Webs | 10% GelSight-Grasp | 83.74 |
| Comb. D | Both Sims | Both Webs | 81.35 |
| No Tactile Loss | Both Sims | Both Webs | 78.59 |
IV-E Limitations
Prediction depends strongly on mesh quality and millimeter-level alignment. Unseen geometry remains difficult. Force is not an explicit input; its effects are represented only indirectly through sampled penetration geometry. Material properties, detailed textures, gel deformation, and dynamic contact are therefore not fully modeled.
V Conclusion and Future Work
Local geometry and sensor-background conditioning provide a useful route from visual reconstruction to anticipated tactile images. The evidence supports partial sim-to-real and sensor-instance transfer plus simulated downstream classification. Texture-aware conditioning, better geometry, calibrated force/deformation models, and real manipulation evaluation remain natural extensions.
Acknowledgment
The authors acknowledge GeniusRobot, Robotics Institute Germany, the Erlangen HPC resources, the DAAD Konrad Zuse Schools program, and the Next Generation AI Computing project and their listed funders.