At a glance
Norm2Tex adds fine material texture to simulated vision-based tactile images. It converts a UV-matched normal map into a pseudo-height map, samples the region touching the sensor, and adds that texture to the simulator’s depth image before rendering.
The paper demonstrates integration with Taxim and TACTO, approximately 72% material classification on real tactile images, and material-dependent behavior after policy transfer. It changes rendered tactile appearance rather than the physical contact/friction model. The robot results illustrate transferable cues rather than reliable optimal gripping.
I Introduction
Vision-based tactile sensors observe a deformable surface through an internal camera. Contact shape produces broad deformations, while material texture contributes finer patterns. Simulators often omit those fine details, making real tactile data difficult to replace.
Explicitly modeling every ridge in a mesh is expensive. Norm2Tex instead injects spatially anchored depth detail between contact/deformation simulation and image rendering. It can reuse an existing renderer while preserving object-relative texture as contact moves.
The pipeline assumes a suitable UV normal map and contact-to-mesh mapping. A procedural Blender workflow generates those assets; adapting a normal map to a new object still requires material choice and parameter preparation.
II Related Work
The related work distinguishes contact/deformation modeling from tactile image rendering, and compares analytic texture injection with learned image translation or generation.
II-A Simulators for Vision-Based Tactile Sensors
Existing tactile simulators obtain contact geometry through clipping, empirical mappings, depth rendering or physical deformation models. Norm2Tex operates after a depth representation is available and before optical rendering.
The experiments integrate the depth-based Taxim and TACTO paths. Compatibility with more detailed finite-element or material-point simulators is proposed future work, rather than an evaluated integration.
The evaluated integrations build on Taxim [11] and TACTO [10].
II-B Textures in Simulated Tactile Images
Learned sim-to-real image translation can model appearance but needs real observations and may change contact structure. Independent image generation can also lose object-relative texture consistency across frames.
Norm2Tex uses an object’s UV map to anchor patterns to its surface. This provides a controlled appearance mechanism, without claiming to reproduce every deformation-dependent optical effect of a real sensor.
III Method
The method has three steps: prepare a UV-matched material normal map, integrate its gradients into height, and query/add the relevant texture patch during simulation. All six numbered equations and ten additional mathematical displays are retained.
III-A Preliminaries
Normal-map RGB channels are remapped from [0,1] to [−1,1] and normalized. Under a height-field representation, the horizontal and vertical gradients are−nx/nz and−ny/nz. These recover directional surface detail, with height determined only up to a constant.
The map supplies tangent-space appearance detail rather than a denser physical collision mesh. The original mesh and contact simulation remain the geometric basis for the renderer.
Equations 1–6
| (1) |
| (2) |
| (3) |
| (4) |
| (5) |
| (6) |
III-B Generating Textures for Arbitrary Meshes
Procedural Blender materials combine noise and structural patterns for fabric, wood, rock and other surfaces. Adjustable parameters vary direction, scale and local structure. A script exports RGB textures, matching normal maps and UV-adjusted meshes.
Figure 3 retains all four fabric variants in their original arrangement. The method can apply a chosen procedural material to arbitrary meshes; it does not infer the correct real material automatically from an unannotated mesh.
Procedural asset generation uses Blender [18].
III-C Converting RGB Normal to Height Map
Each UV island is processed separately. Least-squares Poisson integration projects potentially inconsistent normal-derived gradients onto an integrable height field. A discrete Fourier transform solves the periodic five-point Laplacian; the zero-frequency value is set to zero to choose a zero-mean solution.
Fine and coarse mask-aware Gaussian filtering use σ=0.35 and 1.1. Their reconstructed heights are combined with coarse weight 0.2 and normalized using the 0.05th/99.95th percentiles. Negative-only, centered and positive-only height ranges are offered; the original negative-only notation[0,−1] is preserved.
This reconstructs pseudo-height under chosen filtering and boundary assumptions. It is not a measurement of true material roughness or a full model of texture-induced mechanics.
The integration follows the Frankot–Chellappa method [27].
Equations
III-D Querying Height Map Information During Simulation
Object pose, contact mesh faces, UV coordinates and estimated millimeters per pixel identify the part of the global texture touching the sensor. The sampled local height has the same valid pixels and dimensions as the base sensor depth.
A configurable amplitude β, for example 0.5 mm, scales the local texture before addition: Hfinal=Hsensor+βHlocal. The simulator’s normal rendering pipeline then continues. Amplitude and normalization control appearance; they do not independently identify a material’s friction coefficient.
Equation
IV Experiments
Three experiments test rendering overhead, real-image material classification and closed-loop policy behavior. These support different aspects of the method; classification accuracy should not be treated as robot-task success.
IV-A Applicability and Performance Impact
The timing test uses an eight-vertex box with fabric texture, averaged over 1,000 frames. Taxim CPU timing rises from 0.1019 to 0.1106 s, or 8.54%; CUDA timing rises from 0.0130 to 0.0132 s, or 1.54%. TACTO’s depth path rises from 0.3639 to 0.3701 s, or 1.70%.
These are specific rendering configurations, rather than a complete system-throughput study across complex scenes. The CPU overhead is attributed partly to more complex shadow post-processing. The table’s original values and “Total(s)” heading are retained alongside the stated averaging protocol.
| Condition | Total (s) | Difference (%) |
|---|---|---|
| Taxim (CPU) | 0.1019 | - |
| Taxim +Norm2Tex (CPU) | 0.1106 | 8.54% |
| Taxim (CUDA) | 0.0130 | - |
| Taxim +Norm2Tex (CUDA) | 0.0132 | 1.54% |
| TACTO (CPU) | 0.3639 | - |
| TACTO +Norm2Tex (CPU) | 0.3701 | 1.70% |
IV-B Zero-shot Sim-to-Real Material Classification
The authors report collecting 100 real examples per category and 6,000 simulated Taxim images per category across fabric, smooth/acrylic, rock, wood and3D print. A ResNet-18 with ImageNet initialization, a linear/dropout classifier head and background subtraction is trained using simulated labeled images.
Table II reports 72.8% overall accuracy and 74.4% macro precision. Its per-material “accuracy” corresponds to row-wise recall. Rock has 93.0% listed accuracy;3D print has 56.7%. Acrylic/3D-print confusion and granular fabric mistaken for rock are prominent errors. Figure 4 retains the complete confusion matrix and real tactile examples.
The real images are also used to tune procedural material parameters. Thus “zero-shot” concerns classifier training, rather than a renderer developed without target-domain observations. The paper does not specify a separate calibration/test split or explain why the confusion matrix contains 486 examples rather than the 500 reportedly collected. No textureless-classifier baseline or uncertainty interval is reported.
Figure 5 shows remaining frequency-domain mismatch: simulated wood has stronger directional/high-frequency structure than real DIGIT images. Useful recognition does not require pixel-perfect appearance, but these results do not establish broad unseen-material generalization.
The visual encoder follows ResNet [29].
| Material | Accuracy (%) | Precision (%) |
|---|---|---|
| Fabric | 72.0 | 92.3 |
| Acrylic | 68.5 | 66.3 |
| Rock | 93.0 | 72.7 |
| Wood | 73.0 | 89.0 |
| 3D Print | 56.7 | 51.9 |
| Overall | 72.8 | 74.4 |
IV-C Zero-shot Sim-to-Real Robot Policy
The robot learns to hold a cube while opening its gripper as far as possible. Opening is a proxy for lighter grip; direct force-optimality is not measured. Actions change normalized opening by+0.005,−0.005 or zero. Observations use gripper state and left/right DIGIT tactile images; the external cameras pictured in Figure 6 are unnecessary.
Experiments use wood, fabric and acrylic, selected from the more distinguishable materials. Cubes have equal weight, with a 500 ml water bottle attached in hardware to induce slip. PPO uses 128-step horizons, randomized initial opening 0.45–0.55, contact-height variation up to 2.5 cm and 24 texture variants.
Blind, textureless Taxim and Norm2Tex policies are compared. The two tactile variants share architecture and classifier-derived visual initialization. The dense reward favors larger openings, while a drop ends the episode with zero terminal reward; the source itself identifies reward design as a limitation.
Figure 8 retains all real/simulated trajectories and empirically measured release thresholds. Norm2Tex produces material-dependent responses and longer simulated holds for some materials, while blind/Taxim behavior is largely material independent. None perfectly optimizes the task, and no aggregate success, force or repeated-seed result is reported.
Hardware behavior differs from simulation, especially strong closing on acrylic. Figure 7 shows real pad-compression shadows absent from Taxim; their role is a proposed explanation rather than an isolated causal test. All policies continue opening when empty, a condition absent from nonterminal training states.
Policy training uses PPO [30] and the cited Stable-Baselines3 implementation [31].
V Conclusions
Texture injection supplies useful material cues while reusing existing contact and rendering pipelines. The evidence comprises a renderer-overhead test, simulation-trained classification and qualitative changes in closed-loop policy behavior.
Remaining gaps include deformation-dependent lighting, detailed contact mechanics, reproduction of real texture statistics and stronger robot-performance evaluation. The paper proposes extending the approach to finite-element/material-point simulators.
Acknowledgments
The work acknowledges NHR@FAU computing resources, BayernKI, the Robotics Institute Germany and GeniusRobot support. Seongjin Bien and Débora Oliveira Makowski contributed equally.
The authors disclose ChatGPT assistance for portions of code creation, with generated code reviewed, adapted and validated by them. They state that methods, evaluations and interpretation were their own work.
Original project policy rollouts
The authors’ project page provides nine policy rollouts: blind, Taxim and Norm2Tex policies on wood, acrylic and fabric. All nine original video files are retained unchanged with locally generated preview frames.
The overview video and code repository remain linked. These selected rollouts illustrate behavior and do not add an independent quantitative success estimate.