Literature
Display
← All papers

DAVATaR: Depth-Aware Visually Augmented Tactile Learning Across the Air-Water Interface

Bowen Jiang · Siqi Shang · Rahul Ajay Iyer · Ashwin Prakash · Lillian Chin · Roberto Martín-Martín

Conference session · CoRL 2026 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

At a glance

DAVATaR learns manipulation policies for a robot whose end-effector repeatedly crosses the air–water interface. It combines a fluidically sensorized gripper, tactile pressure and immersion-depth inputs, and visual training augmentations that approximate glare and apparent scale changes.

Across four evaluated tasks, mean success is 71%, compared with 41% for the augmented vision-only variant and 55% for tactile policies without the submerged-domain augmentations. These are averages over normal and adversarial settings. The evidence supports complementary benefits from sensing and augmentation in this laboratory setup.

figure 1
Figure 1: DAVATaR. Contact-rich submerged tasks are difficult as water introduces reflection and refraction artifacts that make visual observations unreliable, while water risks damaging electronics. We combine visual augmentation with waterproof tactile sensing to train visuo-tactile policies for submerged manipulation tasks.

1. Introduction

A camera above water can see objects at misleading positions or scales, and foam or glare can hide the surface and contact region. Tasks such as sampling at a valid depth, grasping a spoon, or wiping a plate therefore require physical information that vision alone may not reliably supply.

The system treats changing hydrostatic pressure as useful depth information while placing sensing electronics above the wet region. Its scope is submerged end-effector interaction from a dry robot, rather than a fully underwater robot.

3. System design

Three components work together: long waterproof sensing fingers, training-time visual perturbations, and a Transformer behavior-cloning policy that fuses recent images with tactile and depth channels. Only the end-effector enters the water.

3.1. Submerged gripper

Each finger combines a 305 mm aluminum beam with a 90 mm compliant tip. Five pairs of L-shaped air channels per fingertip change pressure under deformation. Differential transducers near the finger roots reduce common-mode disturbances while keeping readout electronics away from the submerged region.

An open/sealed tube pair supplies hydrostatic depth information. The reported policy sensing vector has eleven channels: ten contact-sensitive pressure channels plus one immersion-depth channel, sampled at 20 Hz. Figure 2 preserves the full construction and dimensions.

figure 2
Figure 2: DAVATaR gripper design. The long-finger gripper uses fluidic tactile sensing to keep sensing electronics away from the water-exposed region. Embedded fingertip air channels provide contact-force sensing, while a rear open/sealed tube pair enables immersiondepth sensing through hydrostatic pressure.

3.2. Visual augmentation

Standard translation and color jitter are followed by two submerged-domain perturbations. Reflection-like augmentation adds bright translucent patches whose brightness and spatial scattering vary. Refraction-like augmentation resizes the image and crops or pads it back to its original size.

These perturbations mimic occlusion and apparent scale changes. They do not reproduce complete refractive geometry, dynamic water motion, or optical defocus. Images alone are augmented; tactile signals, depth readings, and demonstrated actions remain paired with their original values.

figure 3
Figure 3: Visualization of submerged-domain visual augmentations. Distortions at the air–water interface affect raw wrist-camera images, adding occlusions and zoom effects. We augment images used in training by mimicking those distortions, such as overlaying light patches on the image and changing perceived image scale.

3.3. Multimodal policy

The policy receives five synchronized observations. Each includes a 256 × 256 wrist image and, for tactile variants, all eleven local sensing channels. ResNet-18 with spatial softmax encodes images; a multilayer perceptron encodes local sensing. A Transformer fuses both modalities with learned readout tokens.

The readout parameterizes a mixture-density action distribution for operational-space motion and gripper commands. Training minimizes negative log likelihood of demonstrated actions. Augmentation is disabled during validation and robot execution. Figure 4 retains the complete training and architecture diagram.

figure 4
Figure 4: Policy training pipeline and architecture. At timestep t, each demonstration sample contains a wrist RGB image It, contact-force readings τt, immersion-depth reading dt, and action label at. During training, the visual augmentation operator A produces augmented images ̃Ij t = Aj(It) while leaving dt, τt, and at unchanged. The policy consumes a synchronized history of H observations ending at t: wrist images, contact-force readings, and immersion-depth readings. Image histories are encoded with a ResNet-18 encoder with spatial softmax, local sensing histories are encoded with an MLP, and the resulting tokens are fused by a multimodal transformer with a learned readout token to predict the operational-space action.

4. Evaluation

The evaluation first characterizes contact-force and depth sensing, then compares task performance with and without tactile inputs and visual augmentation. The two ablations remove either all local tactile/depth sensing or the submerged-domain visual augmentations.

Force and immersion-depth characterization

Force calibration uses repeated one-minute pressing trials at dry, 5 cm, 10 cm, and 20 cm immersion. A wrist force-torque sensor supplies reference measurements. Support vector regression with a radial basis function kernel maps tactile features to force under five-fold cross-validation.

Figure 5 retains every calibration row. Grip-force root mean squared error (RMSE) rises from 0.11 N dry to 0.85 N at 20 cm; push-force RMSE is 0.46 N dry and 0.80 N at 20 cm. Overall reported values are 0.37 N for grip and 0.66 N for push. These characterize the tested 0–20 N range; performance varies with depth and force direction.

Depth characterization uses five two-minute trials with randomized motion at 1 cm/s in a 25 cm tank. Robot kinematics relative to the water surface provide reference depth. Linear calibration gives R² = 0.9996 and 1.6 mm RMSE, with repeatability of 0.30% of full-scale range. This is calibration evidence in the tested setup, not a guarantee under arbitrary temperature or water conditions.

figure 5
Figure 5: DAVATaR gripper characterization. Characterization results show stable force estimation across water depths and near-linear immersion-depth sensing and Red arrows represent gripper motion directions.

Task setup

A Franka Emika Panda performs water sampling, spoon pickup, sink-plug removal, and plate wiping. Each task has forty demonstrations collected under normal conditions. All policy variants use the same behavior-cloning loss.

Each evaluation condition uses three trained seeds and six rollouts per seed, for eighteen rollouts. Adversarial conditions add thicker foam, bubbles, reflection, or deeper water. Robot evaluation runs at 5 Hz with a 300-step cap, distinct from the gripper’s 20 Hz sensing rate.

figure 6
Figure 6: Submerged manipulation task suite. We evaluate DAVATaR across four contact-rich tasks, focusing on tasks that involve sustained surface contact, fixture interaction, grasping, and sampling in water.

Task-level performance

Table 1 retains all task means. DAVATaR scores 0.67 for water sampling, 0.75 for spoon pickup, 0.61 for sink-plug removal, and 0.81 for plate wiping. The overall mean is 0.71 versus 0.28 for unaugmented vision-only, 0.41 for augmented vision-only, and 0.55 for tactile without submerged-domain augmentation.

The 71% versus 41% difference is thirty percentage points. Each task mean gives equal weight to normal success and its averaged adversarial success; it is not a single pooled success fraction over all rollouts.

table 1
Table 1: Task-level success rates. Entries report success rates averaged over normal and adversarial visual settings for each task. DAVATaR achieves the highest success across all evaluated tasks, showing that multimodal tactile sensing and submerged-domain visual augmentation improve submerged manipulation performance.

Contribution of tactile sensing

Adding contact and immersion-depth channels helps infer hidden physical state, such as insertion depth beneath foam or whether fingers have reached the tank bottom. The task comparison supports a useful contribution from the combined tactile/depth stream.

Figure 7 shows attention changes during representative approach, contact, and completion phases. Its curves are normalized separately within each modality and cannot be interpreted as percentages of total modality importance. They also do not separate contact-force information from depth information or establish that attention caused a particular action.

figure 7
Figure 7: Modality attention during representative rollouts. We visualize normalized readout-token attention to visual and tactile tokens for plate wiping and spoon pickup. The marked frames correspond to key task phases, including approach, contact/alignment, wiping or grasping, and completion. The attention patterns show increased tactile attention during contact-sensitive phases.

Contribution of visual augmentation

Augmentation raises overall success from 28% to 41% without tactile sensing and from 55% to 71% with it. Under adversarial settings, the corresponding changes are 9% to 25% and 37% to 61%. For adversarial sink-plug removal, success rises from 0% to 25% without tactile and from 11% to 50% with tactile.

The single-augmentation spoon tests distinguish the two effects. In reflective conditions, reflection-only augmentation raises vision-only success from 0% to 54%, and tactile success from 46% to 71%. In deep water, refraction-only augmentation changes vision-only success from 38% to 39%, but tactile success from 37% to 67%.

The suffix names indicate what is removed: “-RefrA” retains reflection-like augmentation, while “-ReflA” retains refraction-like augmentation. Figure 8 preserves every panel and the original explanatory caption.

figure 8
Figure 8: Success rates under adversarial visual conditions. We evaluate all policies under adversarial versions of the tasks (see Appendix). Left: Success rates averaged across all evaluated tasks under Normal and Adversarial settings; DAVATaR achieves the highest success in both settings. Middle and right: Single-augmentation ablations on spoon pickup. The middle panel evaluates the Reflective setting, where DAVATaR-T-RefrA and DAVATaR-RefrA use only reflection-like augmentation, i.e., the refraction-like augmentation is removed from the full augmentation set. The right panel evaluates the Deep-water setting, where DAVATaR-T-ReflA and DAVATaR-ReflA use only refraction-like augmentation, i.e., the reflectionlike augmentation is removed from the full augmentation set. VisionOnly and DAVATaR-VA are the corresponding no-augmentation baselines without and with tactile sensing, respectively. The ablations show that targeted visual augmentation improves performance in the corresponding visual setting, with refraction-like augmentation helping most when paired with tactile sensing.

5. Conclusion and limitations

The combination of local physical sensing and visually perturbed imitation training performs best in the reported submerged tasks. The sensor contributes information hidden by water, while augmentation improves tolerance to corrupted images.

Repeated large forces can deform the compliant tips and peel channel sealant. Long air pathways are susceptible to temperature-induced pressure drift. The patch and zoom perturbations omit dynamic lensing, changing water motion, and refractive defocus. The experiments do not establish long-term hardware durability or generalization to every submerged environment.

A.1. Normal and adversarial conditions

Demonstrations use normal conditions; stronger visual disturbances are reserved for evaluation. Figure 9 and Table 2 retain all settings, including normal depth ranges, pose and plate-angle variation, 3–15 cm thick foam, glare, bubbles, and the 17–20 cm deep-water spoon condition.

figure 9
Figure 9: Normal and adversarial conditions. We evaluate policies under normal settings and adversarial settings that introduce stronger water-induced visual distortions. The adversarial conditions include increased foam thickness, bubbles and surface foam, deep-water refraction, and light reflection/glare. These variations distort or corrupt the wrist-camera observation while preserving the underlying task objective, allowing us to evaluate policy robustness to air–water interface artifacts. Scale bars indicate 10 cm for reference.
table 2
Table 2: Task-specific evaluation conditions. The top block lists task variables used in the normal setting. The bottom block lists additional adversarial visual conditions used during evaluation.

Water sampling

A syringe must enter through surface foam, fill within the valid depth interval, and return without touching bottom sand. No water collection, sand contact, or timeout counts as failure. Foam hides the surface while sand limits how deep insertion may go.

Spoon pickup

The policy must grasp a spoon from the tank bottom and lift it out. Early closure, missing or dropping the spoon, and timeout are failures. Changing depth and refraction make the apparent gripper-to-spoon geometry unreliable.

Sink plug removal

A submerged plug must be aligned, rotated into its unlocked configuration, then removed. Premature lifting, over-rotation or misalignment, lost contact, and timeout are failures. Visual appearance alone does not reliably reveal the fixture’s mechanical state.

Plate wiping

A sponge must maintain contact with a tilted submerged plate and remove the marked residue. Missing the plate, losing contact, incomplete residue removal, and timeout are failures. The task tests sustained contact and surface alignment.

A.2. Complete result breakdown

Table 3 preserves every normal, adversarial, and averaged result. The full model is strongest on the reported task means, but does not strictly beat every ablation in every condition: normal spoon pickup ties the tactile/no-augmentation model at 0.81, and adversarial water sampling ties it at 0.56.

Where a task has multiple adversarial conditions, their results are averaged before normal/adversarial averaging. The table reports aggregate rates rather than per-seed variability or confidence intervals.

table 3
Table 3: Full per-task success rates. Entries report success rates for each method under normal and adversarial settings. For tasks with multiple adversarial settings, Adv. averages over those settings; Avg. is the mean of Normal and Adv.

A.3. Attention visualization

Attention rollout averages heads, adds the residual identity, row-normalizes, and multiplies layer matrices. The current-output readout token supplies scores for visual and tactile token groups; all three original equation displays are retained.

The tactile token contains both contact and immersion-depth channels. Each modality’s trajectory is independently min–max normalized and smoothed over five frames. These plots are qualitative attribution proxies, as the appendix explicitly states, and provide neither causal evidence nor a calibrated comparison of absolute reliance between modalities.

Equations
equation attention normalization
equation attention rollout
equation attention modalities

B.1. Sensor construction

Tips use EPU40 elastomer, approximately Shore A 72, printed on a Carbon M2 and sealed with Sil-Poxy. Air channels are 1 mm in diameter and lattice structures 2 mm. DLHR-L05D-E1BD-C-NAV8 differential transducers sit near the finger roots.

Tactile tubing has 1.59 mm inner and 3.18 mm outer diameter. Depth tubing uses larger 3.18 mm inner and 6.35 mm outer diameters to reduce flow resistance along the sensing pathway.

B.2. Augmentation implementation

The retained equation gives the order: translation, color jitter, reflection-like patches, then refraction-like zoom. Table 4 retains all parameter ranges. Translation is up to fifteen pixels with a common crop across temporal history; color jitter probability is 0.9.

Each submerged-domain augmentation has probability 0.3. Patches target about 6% of image area, with brightness sampled from 0.3–0.8 and scattering from 0.2–0.8. Zoom is sampled from 0.95–1.05. One stochastic augmented sequence is generated per forward pass; the multiple variants in Figure 4 represent possible outcomes rather than a fixed expanded dataset.

Equation
equation augmentation
table 4
Table 4: Visual augmentation parameters.

B.3. Network architecture

The ImageNet-pretrained ResNet-18 uses spatial softmax with 32 keypoints. Inputs and actions are normalized from dataset statistics. Table 5 retains the architecture as reported: five-step history, 64-dimensional tokens, four Transformer layers, six attention heads, and five mixture-density components.

Transformer dropout is 0.1. Tactile policies use modality dropout of 0.2, while vision-only variants use zero modality dropout. The reported token and head dimensions are preserved without inferring an unreported attention projection implementation.

table 5
Table 5: Policy architecture hyperparameters.

Training details

Forty demonstrations per task are collected with SpaceMouse teleoperation and split 90%/10% at sequence level. AdamW uses learning rate 0.0001, weight decay 0.01, batch size 32, and 101 epochs with 300 gradient steps each. Cosine annealing reaches 0.00001; validation occurs every five epochs and checkpoints every twenty.

The public project hosts an anonymous submission-format PDF; author names and the University of Texas at Austin affiliation are verified from the project page. The edition preserves that PDF’s complete figures, tables, equations, and appendix.

table 6
Table 6: Training hyperparameters.
Concept demonstration

Open image ↗