Literature
Display
← All papers

Norm2Tex: Augmenting Visuo-Tactile Simulations with Texture

Seongjin Bien · Débora Oliveira Makowski · Roberto Calandra · Florian Walter · Wolfram Burgard

Preprint posted · arXiv (preprint) ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

At a glance

Norm2Tex adds fine material texture to simulated vision-based tactile images. It converts a UV-matched normal map into a pseudo-height map, samples the region touching the sensor, and adds that texture to the simulator’s depth image before rendering.

The paper demonstrates integration with Taxim and TACTO, approximately 72% material classification on real tactile images, and material-dependent behavior after policy transfer. It changes rendered tactile appearance rather than the physical contact/friction model. The robot results illustrate transferable cues rather than reliable optimal gripping.

I Introduction

Vision-based tactile sensors observe a deformable surface through an internal camera. Contact shape produces broad deformations, while material texture contributes finer patterns. Simulators often omit those fine details, making real tactile data difficult to replace.

Explicitly modeling every ridge in a mesh is expensive. Norm2Tex instead injects spatially anchored depth detail between contact/deformation simulation and image rendering. It can reuse an existing renderer while preserving object-relative texture as contact moves.

The pipeline assumes a suitable UV normal map and contact-to-mesh mapping. A procedural Blender workflow generates those assets; adapting a normal map to a new object still requires material choice and parameter preparation.

Complete original Figure 1, retaining all original panels and layout
Fig. 1: Comparison of real-world visuo-tactile data from DIGIT (top) and the commonly simulated rendering on Taxim (center). Independent of the texture, the Taxim rendering looks the same. Norm2Tex (bottom) fills this sim-to-real gap by augmenting surface details (left to right: wood, fabric, 3D print, and rock) to the rendering.

II Related Work

The related work distinguishes contact/deformation modeling from tactile image rendering, and compares analytic texture injection with learned image translation or generation.

II-A Simulators for Vision-Based Tactile Sensors

Existing tactile simulators obtain contact geometry through clipping, empirical mappings, depth rendering or physical deformation models. Norm2Tex operates after a depth representation is available and before optical rendering.

The experiments integrate the depth-based Taxim and TACTO paths. Compatibility with more detailed finite-element or material-point simulators is proposed future work, rather than an evaluated integration.

The evaluated integrations build on Taxim [11] and TACTO [10].

II-B Textures in Simulated Tactile Images

Learned sim-to-real image translation can model appearance but needs real observations and may change contact structure. Independent image generation can also lose object-relative texture consistency across frames.

Norm2Tex uses an object’s UV map to anchor patterns to its surface. This provides a controlled appearance mechanism, without claiming to reproduce every deformation-dependent optical effect of a real sensor.

III Method

The method has three steps: prepare a UV-matched material normal map, integrate its gradients into height, and query/add the relevant texture patch during simulation. All six numbered equations and ten additional mathematical displays are retained.

Complete original Figure 2, retaining all original panels and layout
Fig. 2: The Norm2Tex architecture. First, a procedural texture is created for a given mesh using Blender. At initialization, the global texture height map HglobalH_{\text{global}} is computed, and a copy of the mesh is created internally for reference. During simulation, given the relative transform of the mesh to the sensor at the contact point, the corresponding UV coordinates are used to fetch the pixels from HglobalH_{\text{global}} to create HlocalH_{\text{local}}. Finally, HsensorH_{\text{sensor}} and HlocalH_{\text{local}} are added together to create the final depth image HfinalH_{\text{final}}, which is then used for rendering the tactile image by the tactile simulator.

III-A Preliminaries

Normal-map RGB channels are remapped from [0,1] to [−1,1] and normalized. Under a height-field representation, the horizontal and vertical gradients are−nx/nz and−ny/nz. These recover directional surface detail, with height determined only up to a constant.

The map supplies tangent-space appearance detail rather than a denser physical collision mesh. The original mesh and contact simulation remain the geometric basis for the renderer.

Equations 1–6
𝐜⁡(x,y)=(cx,cy,cz)∈[0,1]3\mathbf{c}(x,y)=(c_{x},c_{y},c_{z})\in[0,1]^{3} (1)
𝐧~​(x,y)=2​𝐜​(x,y)−1,\widetilde{\mathbf{n}}(x,y)=2\mathbf{c}(x,y)-1, (2)
𝐧⁡(x,y)=𝐧~​(x,y)‖𝐧~​(x,y)‖=(nx,ny,nz)∈[−1,1]3.\mathbf{n}(x,y)=\frac{\widetilde{\mathbf{n}}(x,y)}{\left\|\widetilde{\mathbf{n}}(x,y)\right\|}=(n_{x},n_{y},n_{z})\in[-1,1]^{3}. (3)
tx=(1,0,hx),ty=(0,1,hy),t_{x}=(1,0,h_{x}),t_{y}=(0,1,h_{y}), (4)
tx×ty=(−∂h∂x,−∂h∂y,1)∝𝐧,t_{x}\times t_{y}=(-\frac{\partial h}{\partial x},-\frac{\partial h}{\partial y},1)\propto\mathbf{n}, (5)
dx=−nxnz,dy=−nynz.d_{x}=-\frac{n_{x}}{n_{z}},\qquad d_{y}=-\frac{n_{y}}{n_{z}}. (6)

III-B Generating Textures for Arbitrary Meshes

Procedural Blender materials combine noise and structural patterns for fabric, wood, rock and other surfaces. Adjustable parameters vary direction, scale and local structure. A script exports RGB textures, matching normal maps and UV-adjusted meshes.

Figure 3 retains all four fabric variants in their original arrangement. The method can apply a chosen procedural material to arbitrary meshes; it does not infer the correct real material automatically from an unannotated mesh.

Procedural asset generation uses Blender [18].

Complete original Figure 3, retaining all original panels and layout
Fig. 3: Examples of different fabric renderings generated from a single procedural texture (fabric) by adjusting its parameters.

III-C Converting RGB Normal to Height Map

Each UV island is processed separately. Least-squares Poisson integration projects potentially inconsistent normal-derived gradients onto an integrable height field. A discrete Fourier transform solves the periodic five-point Laplacian; the zero-frequency value is set to zero to choose a zero-mean solution.

Fine and coarse mask-aware Gaussian filtering use σ=0.35 and 1.1. Their reconstructed heights are combined with coarse weight 0.2 and normalized using the 0.05th/99.95th percentiles. Negative-only, centered and positive-only height ranges are offered; the original negative-only notation[0,−1] is preserved.

This reconstructs pseudo-height under chosen filtering and boundary assumptions. It is not a measurement of true material roughness or a full model of texture-induced mechanics.

The integration follows the Frankot–Chellappa method [27].

Equations
𝐝p​(x,y)=[dp,x​(x,y)dp,y​(x,y)],\mathbf{d}_{p}(x,y)=\begin{bmatrix}d_{p,x}(x,y)\\ d_{p,y}(x,y)\end{bmatrix},
E(H)=∑i,j[\displaystyle E(H)=\sum_{i,j}\Big[ (Dx​(hp)−dp,x​(i,j))2\displaystyle\left(D_{x}(h_{p})-d_{p,x}(i,j)\right)^{2}
+\displaystyle+{} (Dy(hp)−dp,y(i,j))2]\displaystyle\left(D_{y}(h_{p})-d_{p,y}(i,j)\right)^{2}\Big]
Δd​Hp=∇d⋅𝐝p,\Delta_{d}H_{p}=\nabla_{d}\cdot\mathbf{d}_{p},
D^​(u,v)=ℱ⁡{∇d⋅𝐝p}​(u,v).\widehat{D}(u,v)=\mathcal{F}\left\{\nabla_{d}\cdot\mathbf{d}_{p}\right\}(u,v).
λp​(u,v)=2​cos⁡(2​π​uWp)+2​cos⁡(2​π​vHp)−4.\lambda_{p}(u,v)=2\cos\left(\frac{2\pi u}{W_{p}}\right)+2\cos\left(\frac{2\pi v}{H_{p}}\right)-4.
Hp^​(u,v)=D^p​(u,v)λp​(u,v),(u,v)≠(0,0).\widehat{H_{p}}(u,v)=\frac{\widehat{D}_{p}(u,v)}{\lambda_{p}(u,v)},\qquad(u,v)\neq(0,0).
H^p​(0,0)=0,\widehat{H}_{p}(0,0)=0,
Hp=ℱ−1​{H^p}.H_{p}=\mathcal{F}^{-1}\left\{\widehat{H}_{p}\right\}.
Hglobal=normalize​((1−α)​Hfine+α​Hcoarse)H_{\text{global}}=\text{normalize}\big((1-\alpha)H_{\text{fine}}+\alpha H_{\text{coarse}}\big)

III-D Querying Height Map Information During Simulation

Object pose, contact mesh faces, UV coordinates and estimated millimeters per pixel identify the part of the global texture touching the sensor. The sampled local height has the same valid pixels and dimensions as the base sensor depth.

A configurable amplitude β, for example 0.5 mm, scales the local texture before addition: Hfinal=Hsensor+βHlocal. The simulator’s normal rendering pipeline then continues. Amplitude and normalization control appearance; they do not independently identify a material’s friction coefficient.

Equation
Hfinal=Hsensor+β​Hlocal.H_{\text{final}}=H_{\text{sensor}}+\beta H_{\text{local}}.

IV Experiments

Three experiments test rendering overhead, real-image material classification and closed-loop policy behavior. These support different aspects of the method; classification accuracy should not be treated as robot-task success.

IV-A Applicability and Performance Impact

The timing test uses an eight-vertex box with fabric texture, averaged over 1,000 frames. Taxim CPU timing rises from 0.1019 to 0.1106 s, or 8.54%; CUDA timing rises from 0.0130 to 0.0132 s, or 1.54%. TACTO’s depth path rises from 0.3639 to 0.3701 s, or 1.70%.

These are specific rendering configurations, rather than a complete system-throughput study across complex scenes. The CPU overhead is attributed partly to more complex shadow post-processing. The table’s original values and “Total(s)” heading are retained alongside the stated averaging protocol.

TABLE I: Comparison of render time for TACTO and Taxim with and without Norm2Tex, averaged over 1000 frames.
Condition Total (s) Difference (%)
Taxim (CPU) 0.1019 -
Taxim +Norm2Tex (CPU) 0.1106 8.54%
Taxim (CUDA) 0.0130 -
Taxim +Norm2Tex (CUDA) 0.0132 1.54%
TACTO (CPU) 0.3639 -
TACTO +Norm2Tex (CPU) 0.3701 1.70%

IV-B Zero-shot Sim-to-Real Material Classification

The authors report collecting 100 real examples per category and 6,000 simulated Taxim images per category across fabric, smooth/acrylic, rock, wood and3D print. A ResNet-18 with ImageNet initialization, a linear/dropout classifier head and background subtraction is trained using simulated labeled images.

Table II reports 72.8% overall accuracy and 74.4% macro precision. Its per-material “accuracy” corresponds to row-wise recall. Rock has 93.0% listed accuracy;3D print has 56.7%. Acrylic/3D-print confusion and granular fabric mistaken for rock are prominent errors. Figure 4 retains the complete confusion matrix and real tactile examples.

The real images are also used to tune procedural material parameters. Thus “zero-shot” concerns classifier training, rather than a renderer developed without target-domain observations. The paper does not specify a separate calibration/test split or explain why the confusion matrix contains 486 examples rather than the 500 reportedly collected. No textureless-classifier baseline or uncertainty interval is reported.

Figure 5 shows remaining frequency-domain mismatch: simulated wood has stronger directional/high-frequency structure than real DIGIT images. Useful recognition does not require pixel-perfect appearance, but these results do not establish broad unseen-material generalization.

The visual encoder follows ResNet [29].

Complete original Figure 4, retaining all original panels and layout
Fig. 4: (Left) Confusion matrix for the classifier trained on simulated Norm2Tex data and deployed zero-shot in the real world. (Right) Real-world input image to the classifier, i.e., DIGIT image minus initial blank image. Note that acrylic and 3D-printed smooth textures are hard to distinguish. When the fabric texture is granular, it can be mistaken for rock.
TABLE II: Per-material classification accuracy and precision.
Material Accuracy (%) Precision (%)
Fabric 72.0 92.3
Acrylic 68.5 66.3
Rock 93.0 72.7
Wood 73.0 89.0
3D Print 56.7 51.9
Overall 72.8 74.4
Complete original Figure 5, retaining all original panels and layout
Fig. 5: Norm2Tex(top) and real-world (bottom) tactile images and 2D Fourier transform for wood. Norm2Tex retains directional/high-frequency spectral structure corresponding to the wood grain, while the real tactile capture is dominated by low frequencies when the texture is faint.

IV-C Zero-shot Sim-to-Real Robot Policy

The robot learns to hold a cube while opening its gripper as far as possible. Opening is a proxy for lighter grip; direct force-optimality is not measured. Actions change normalized opening by+0.005,−0.005 or zero. Observations use gripper state and left/right DIGIT tactile images; the external cameras pictured in Figure 6 are unnecessary.

Experiments use wood, fabric and acrylic, selected from the more distinguishable materials. Cubes have equal weight, with a 500 ml water bottle attached in hardware to induce slip. PPO uses 128-step horizons, randomized initial opening 0.45–0.55, contact-height variation up to 2.5 cm and 24 texture variants.

Blind, textureless Taxim and Norm2Tex policies are compared. The two tactile variants share architecture and classifier-derived visual initialization. The dense reward favors larger openings, while a drop ends the episode with zero terminal reward; the source itself identifies reward design as a limitation.

Figure 8 retains all real/simulated trajectories and empirically measured release thresholds. Norm2Tex produces material-dependent responses and longer simulated holds for some materials, while blind/Taxim behavior is largely material independent. None perfectly optimizes the task, and no aggregate success, force or repeated-seed result is reported.

Hardware behavior differs from simulation, especially strong closing on acrylic. Figure 7 shows real pad-compression shadows absent from Taxim; their role is a proposed explanation rather than an isolated causal test. All policies continue opening when empty, a condition absent from nonterminal training states.

Policy training uses PPO [30] and the cited Stable-Baselines3 implementation [31].

Refer to caption
Fig. 6: (Left) Simulation setup for the holding cube experiment inside MuJoCo simulation. (Center) Cubes with wood, acrylic, and fabric wrapping materials used for real-world grasping. (Right) Real-world robot setup. Note that neither the side nor the wrist camera is needed in the experiments.
Complete original Figure 7, retaining all original panels and layout
Fig. 7: (Left) Real DIGIT images of the acryl box’s surface and edge. (Right) Simulated textureless tactile images of an equal-sized box’s surface and edge. Note that the shadow and lighting caused by the compression of the silicon pad are not equally reproduced on Taxim.
Refer to caption
Fig. 8: Gripper state for real-world and simulation rollouts of the cube-holding experiment for blind, Taxim, and Norm2Tex policies using wood, acrylic, and fabric materials. In the empty condition, all policies continue opening because no object-induced termination occurs. The blind and Taxim policies exhibit similar opening behavior across materials, whereas Norm2Tex produces distinct material-dependent responses in the real world. These results indicate that Norm2Tex observes tactile distinctions between grasp conditions. The black lines and values represent the maximum gripper opening before dropping. Those values were determined empirically.

V Conclusions

Texture injection supplies useful material cues while reusing existing contact and rendering pipelines. The evidence comprises a renderer-overhead test, simulation-trained classification and qualitative changes in closed-loop policy behavior.

Remaining gaps include deformation-dependent lighting, detailed contact mechanics, reproduction of real texture statistics and stronger robot-performance evaluation. The paper proposes extending the approach to finite-element/material-point simulators.

Acknowledgments

The work acknowledges NHR@FAU computing resources, BayernKI, the Robotics Institute Germany and GeniusRobot support. Seongjin Bien and Débora Oliveira Makowski contributed equally.

The authors disclose ChatGPT assistance for portions of code creation, with generated code reviewed, adapted and validated by them. They state that methods, evaluations and interpretation were their own work.

Original project policy rollouts

The authors’ project page provides nine policy rollouts: blind, Taxim and Norm2Tex policies on wood, acrylic and fabric. All nine original video files are retained unchanged with locally generated preview frames.

The overview video and code repository remain linked. These selected rollouts illustrate behavior and do not add an independent quantitative success estimate.

Blind policy · Wood
Blind policy · Acrylic
Blind policy · Fabric
Taxim policy · Wood
Taxim policy · Acrylic
Taxim policy · Fabric
Norm2Tex policy · Wood
Norm2Tex policy · Acrylic
Norm2Tex policy · Fabric
Concept demonstration

Open image ↗