Overview
This edition covers the full 22-page paper and Appendices A–J. Every figure/table panel, equation, caption, reference and original paired RGB/depth demonstration remains. Native PDF layouts restore fine chart labels and table formatting while retaining selectable original values/notation. Haiyue Zhu and Kezhi Mao are corresponding authors. The clean-depth training target used for single-view MetaWorld is distinguished from the noisy cross-view targets used for RLBench and the real-robot motivation.
At a glance
Depth errors become incorrect token positions in 3D manipulation policies. NISE combines training-only Geometric Cross-View Self-Supervision (GCVSS) with deployed Gaussian Marginalized 3D Attention (GM3A). On the reported noisy-trained RLBench setting it scores 89.0/85.7/84.1% for clean/mild/saturated depth; MetaWorld scores 79.0/78.2/77.4%. A strong geometry-pretrained backbone supplies part of the advantage, so within-stack ablations matter. Five real tasks use thirty trials per method. ‘Noise invariant’ means lower measured representation drift, not mathematically identical features or immunity to systematic depth error.
1 Introduction
A depth-lifted policy uses metric distances and reachability, but missing values, edge artifacts and wrong surfaces corrupt those coordinates before action prediction. Matched noisy training helps without fully removing the error. NISE targets the learned spatial interface: cross-view consistency shapes features, and position-uncertainty attenuation changes how attention uses them. Figure 1’s saturated task-mean gaps over the strongest listed baseline are about 33.2 RLBench and 40.2 MetaWorld percentage points.
2 Related Work
Point-cloud, voxel, virtual-view and depth-lifted-token policies all depend on geometry. Depth repair and geometry-pretrained backbones improve inputs/features, while related representation methods also address uncertainty or consistency. NISE’s claim is their specific combination of cross-view feature supervision and adaptive rotary-position attenuation. The experiments explicitly include strong-backbone and repair controls, rather than attributing every cross-architecture gap to these two mechanisms.
3 NISE : Noise-Invariant Spatial Encoding
The design has a shared trainable spatial feature map above a frozen visual backbone. During training a geometry decoder predicts a calibrated, visibility-filtered cross-view target. At deployment that decoder disappears; adaptive positional attenuation stays inside the action policy.
3.1 From Depth Noise to Action Error
Back-projecting noisy depth through camera calibration turns each depth perturbation into a coordinate error, mainly along its camera ray. Rotary phases and cross-camera spatial relations then inherit that error. The complete observation, corruption, lifting and feature definitions remain. This is a failure-channel argument, rather than a universal quantitative bound from depth noise to action failure.
Equations
3.2 Geometric Cross-View Self-Supervision
For a target view, source depth is forward-warped with calibrated intrinsics/extrinsics, z-buffered and filtered for valid visible splats. A masked L1 loss trains a geometry head on the shared policy features. The target is still noisy; the Noise2Noise analogy does not impose independent unbiased noise across camera views. Multiple views that agree on the same background through a transparent object can supply a consistent but wrong target. The geometry head supervises representation learning rather than preprocessing deployed depth.
Equations
3.3 Gaussian Marginalized 3D Attention
For locally Gaussian coordinate uncertainty, the exact sinusoidal moment is the mean phase times exp(−variance·frequency²/2). Larger uncertainty attenuates high frequencies more. Per-token/per-axis/per-band residuals then adapt that prior under an L2 anchor. The analytic moment concerns positional features, not an exact expectation of the full nonlinear attention output. Once learned residuals are added, coefficients need not equal a Gaussian marginal; the manuscript does not specify a clamp guaranteeing every learned coefficient stays in [0,1]. Biased or multimodal wrong-surface errors also exceed the local Gaussian approximation.
Equations
3.4 Training and Deployment
Total training loss combines the base action objective, geometry loss weighted .1 and prior regularizer weighted 1.0. Deployment removes the geometry decoder, keeping the frozen backbone, shared feature aggregator, GM3A and native action expert. No additional cross-view reconstruction branch or clean-depth input is required at inference.
Equation
4 Experiments
Experiments test noise robustness, clean-depth retention, physical manipulation, component attribution, repair complementarity and representation diagnostics. Baselines retain their own architectures; same-stack controls provide the more direct mechanism comparison.
4.1 Setup
A common dataloader wrapper adds range/disparity noise, edge dropout, holes and displaced lookup. Clean, mild and saturated levels are sampled equally in the controlled noisy-training comparison; Table 2 averages three independent training runs, with 25 episodes per condition. Real methods share demonstrations/raw streams but use their own tuned protocols. Crucially, four-view RLBench uses noisy cross-view supervision, while single-view MetaWorld uses clean simulator depth as its auxiliary training target. Calibration is described for D405/Azure Kinect; the visible sensor names do not establish exhaustive calibration of every commodity camera.
Accessible table, original values and citation links
| Method | Close Jar | Drag Stick | Insert Peg | Meat off Grill | Open Drawer | Place Cups | Place Wine | Push Buttons | Put Cupboard | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| PerAct [2] | ||||||||||
| RVT [36] | ||||||||||
| 3D Diffuser Actor [8] | ||||||||||
| RVT-2 [6] | ||||||||||
| ARP [37] | ||||||||||
| TVVE [38] | ||||||||||
| NISE (ours) | ||||||||||
| Method | Put Drawer | Put Safe | Screw Bulb | Slide Block | Sort Shape | Stack Blocks | Stack Cups | Sweep Dustpan | Turn Tap | Avg. |
| PerAct [2] | ||||||||||
| RVT [36] | ||||||||||
| 3D Diffuser Actor [8] | ||||||||||
| RVT-2 [6] | ||||||||||
| ARP [37] | ||||||||||
| TVVE [38] | ||||||||||
| NISE (ours) |
4.2 Results
Table 1 compares NISE clean-input performance with published clean-trained baselines; its NISE deviations reflect repeated evaluations of a fixed checkpoint. Table 2 instead compares mixed-noise-trained methods across three training runs. The 85.7% mild-noise NISE mean exceeds the strongest listed cross-architecture baseline’s 63.7%, but its same-stack baseline already reaches 75.5%. Real tasks give NISE 25/30 gear assembly, 20/30 lid removal, 22/30 pipe insertion, 13/30 tube-to-mortar and 16/30 gear sorting, with full Wilson intervals retained. Success for the tube task means reaching the specified pouring pose, not a quantified fluid-transfer outcome.
4.3 Ablations
With the frozen-backbone/DPT/action stack fixed, mild-noise success is 75.5% baseline, 78.2% with GM3A, 80.5% with GCVSS and 85.7% combined. Thus the direct combined gain is 10.2 points in this setting. Clean-trained controls are lower under mild corruption. Native-policy insertions, feature drift and target-construction ablations add supporting evidence, with different task subsets/protocols and generally no uncertainty intervals for every appendix comparison.
5 Limitations and Conclusion
Cross-view supervision needs calibrated overlapping views and can reinforce shared wrong-background surfaces behind transparent objects. Local Gaussian attenuation only partially models biased/multimodal depth corruption. The paper supports improved empirical robustness on tested settings, with substantial real-task failures remaining. Its proposed future uncertainty models do not constitute completed solutions to those limits.
Acknowledgments
Support comes from Singapore’s National Robotics Programme 2.0 initiative on domain-specific robotics foundation models for manufacturing. The complete 42-reference bibliography remains.
Original paired RGB and depth demonstrations
The primary NISE project supplies six paired RGB/depth scenes: twelve unchanged original WebM clips with original source stills are retained here. They include transparent tube/lid observations and the five physical task demonstrations, with the repeated lid example deduplicated. RGB/depth views are labeled separately. The project’s interactive 3D diagnostics remain available through that link; selected clips are illustrations rather than additional quantitative trials.
Appendix A Depth Corruption Model
The full corruption model preserves all seven equations and preset values. Axial disparity noise grows approximately with depth squared; lateral jitter changes which surface is looked up; multiplicative masks introduce invalid values. Wrong-surface returns are produced by displaced lookup, not a separate independent Bernoulli variable. Mild/saturated presets use pixel jitter .30/.50, disparity standard deviation .150/.250, edge dropout .30/.50 and hole probability .002/.005. ‘Saturated’ is called ‘moderate’ in the implementation. The calibration protocol is described, but the manuscript does not enumerate every camera parameter, edge threshold or measured residual distribution needed to exactly reproduce calibration from text alone.
Equations 1–7
| (1) |
| (2) |
| (3) |
| (4) |
| (5) |
| (6) |
| (7) |
Accessible table, original values and citation links
| Preset | (px) | (px) | ||
|---|---|---|---|---|
| Clean | 0.00 | 0.000 | 0.000 | 0.0000 |
| Mild | 0.30 | 0.150 | 0.300 | 0.0020 |
| Saturated | 0.50 | 0.250 | 0.500 | 0.0050 |
Appendix B Visual Backbone and Policy Ablations
Backbone controls separate a strong pretrained spatial representation from the proposed learning/attention changes. Their absolute scores use distinct evaluation/noise protocols, so native-stack and full-stack numbers should not be pooled as one benchmark row.
B.1 Visual Backbone
The public VGGT-Omega checkpoint is frozen. A trainable DPT aggregator and DDPM action expert consume its features. This large geometry-pretrained base is shared by the same-stack controls.
B.2 Same-Stack Base-Model Control
Under clean-input evaluation with the same stack/training recipe, disabling both mechanisms gives 79.0±5.0%, while NISE gives 89.0±4.3%. This is the reported within-stack clean-input benefit. It complements, rather than replaces, the mild-noise component comparison and published-baseline table.
B.3 Native-Policy Ablations on 3DDA and RVT-2
Three-task native 3DDA and RVT-2 controls retain each architecture and fine-tune all variants on saturated-noise demonstrations. Combined modules raise saturated means 56.7→70.3% and 24.0→44.0%, respectively. 3DDA’s higher noisy than clean score reflects specialization to that training distribution. These are three-task means, not the full eighteen-task Table 2 protocol.
Accessible table, original values and citation links
| Policy stack | Ablation row | GCVSS | GM3A | Clean | Mild | Saturated |
| Native 3DDA insertion | ||||||
| 3DDA | Plain noise-aware | 41.0 | 54.0 | 56.7 | ||
| 3DDA | + GCVSS | 45.0 | 62.0 | 63.5 | ||
| 3DDA | + GM3A | 43.0 | 59.0 | 61.0 | ||
| 3DDA | + GCVSS + GM3A | 47.0 | 68.0 | 70.3 | ||
| Native RVT-2 insertion | ||||||
| RVT-2 | Plain noise-aware | 52.0 | 34.0 | 24.0 | ||
| RVT-2 | + GCVSS | 55.0 | 42.0 | 35.0 | ||
| RVT-2 | + GM3A | 54.0 | 40.0 | 34.0 | ||
| RVT-2 | + GCVSS + GM3A | 57.0 | 48.0 | 44.0 | ||
Appendix C Depth-Repair Front-End Comparisons
Adding D3RoMa or Marigold-DC repair raises saturated NISE mean 84.1→84.8/85.2%, with similar clean scores. The changes are modest relative to the listed deviations; no significance test is supplied. Main results use raw depth. Repair and representation learning can be complementary without repair being required for NISE inference.
C.1 CDM and NISE on Real-Robot
On thirty lid-removal trials per configuration, same-stack baseline gives 15/30 raw and 18/30 CDM; NISE gives 20/30 raw and 23/30 CDM. Repair adds three successes in both policies, while NISE adds five with either depth source. This is a controlled single-task complementarity result, rather than a general additive law across all materials/tasks.
Appendix D GCVSS Target-Construction Ablation
On saturated Stack Cups, raw cross-camera supervision scores 38.2%; full calibrated/filtered GCVSS reaches 61.6%, near clean-target references 64.0/62.4%. Removing z-buffer visibility or validity masking gives 55.2/58.4%. Depth correlation rises .14→.367 and logged gradient cosine −.050→+.027. Those small cosine values indicate weak alignment, not strong proof of identical optimization signals. The complete three-panel warp/mask diagnostic remains.
Accessible table, original values and citation links
| Target construction | Success (%) |
|---|---|
| Clean-depth references | |
| Per-camera clean target | 64.0 |
| Masked clean consensus | 62.4 |
| Noisy-depth targets | |
| Raw cross-camera target | 38.2 |
| Warped target without z-buffer visibility | 55.2 |
| Warped target without validity mask | 58.4 |
| Full GCVSS | 61.6 |
D.1 Multi-View Agreement on Transparent Objects
Visibility agreement is not ground-truth correctness. Cameras looking through transparent material can agree on the background rather than the hidden object surface. RGB context and adaptive attention remain available but do not guarantee recovering that geometry.
Appendix E GM3A Ablations
The appendix separates fixed frequency attenuation from adaptive residuals, then holds GCVSS fixed to compare task sensitivity. The learned coefficients are anchored to a Gaussian prior through the policy objective.
E.1 Frequency-Only versus Adaptive Attenuation
On three tasks, adaptive attenuation gives 85.3/77.3/73.3% for clean/mild/saturated depth, versus 79.3/70.7/66.7% with a token-independent σ=1 frequency profile. This comparison changes both adaptation and its relationship to the sensor prior, so it is evidence for this adaptive design rather than an isolated proof that all fixed profiles are inferior.
Accessible table, original values and citation links
| GM3A variant | Clean | Mild | Saturated |
|---|---|---|---|
| Frequency-only | 79.3 | 70.7 | 66.7 |
| Adaptive residual | 85.3 | 77.3 | 73.3 |
E.2 Precision-Task GM3A Analysis
With GCVSS fixed, GM3A leaves push-buttons/open-drawer/turn-tap means unchanged and raises insertion/stacking means by ten points under mild noise. The tested precision-sensitive tasks benefit more. These five cases do not establish a universal taxonomy of which tasks need each frequency band.
Appendix F Feature-Invariance and Dampening Diagnostics
Same-scene tests vary only depth while holding RGB, proprioception, calibration and action labels fixed. NISE lowers mild/saturated cosine drift to .12/.17 and raises CKA to .89/.84, compared with .31/.48 and .68/.54 for the noisy-trained same-stack baseline. Region diagnostics show stronger high-frequency suppression at depth boundaries, transparent/specular proxies and precision contacts. Contact masks use ground-truth interaction neighborhoods for analysis; they are not additional deployed sensors. ‘Invariant’ here means reduced measured sensitivity.
Accessible table, original values and citation links
| Setting | Mild drift | Mild CKA | Saturated drift | Saturated CKA | Mild SR (%) |
|---|---|---|---|---|---|
| Clean-trained same-stack baseline | 0.44 | 0.55 | 0.63 | 0.39 | 52.1 |
| Noisy-trained same-stack baseline | 0.31 | 0.68 | 0.48 | 0.54 | 75.5 |
| + GM3A | 0.28 | 0.71 | 0.42 | 0.59 | 78.2 |
| + GCVSS | 0.18 | 0.82 | 0.26 | 0.74 | 80.5 |
| NISE | 0.12 | 0.89 | 0.17 | 0.84 | 85.7 |
Accessible table, original values and citation links
| Task type | Task | Reliable surface | Depth boundary | Transparent/specular proxy | Contact region |
|---|---|---|---|---|---|
| Coarse | push_buttons | 0.06 | 0.15 | 0.18 | 0.10 |
| Coarse | open_drawer | 0.07 | 0.16 | 0.19 | 0.12 |
| Precision | insert_peg | 0.08 | 0.27 | 0.31 | 0.29 |
| Precision | stack_cups | 0.09 | 0.25 | 0.34 | 0.28 |
| Real transparent | remove_petri_lid | 0.10 | 0.31 | 0.39 | 0.32 |
| Real transparent | pour_test_tube | 0.11 | 0.33 | 0.42 | 0.35 |
Appendix G Training, Implementation, and Runtime
Training and runtime details distinguish frozen, trainable, training-only and deployed components. Inference overhead is measured against the same large base model, with rendering and robot I/O excluded.
G.1 Training Configuration
Training uses 32-bit LAMB, cosine decay, 300k updates and global batch 64, with head/policy learning rates 5e-5/1e-4. The DPT head fuses ViT layers 4/11/17/23, projects 2048→120 channels and has 2.283M trainable parameters. Geometry decoding is 120→64→32→16→1; views pair round-robin [1,2,3,0]. Two GM3A MLPs each map XYZ+range through 4→64→60. RLBench geometry targets are cross-view; single-view MetaWorld targets are clean simulator depth. Real baselines share data but retain their tuned training recipes.
G.2 Clean-Only and Mixed-Noise Training on RLBench
Mixed-noise NISE gives 89.0/85.7/84.1%, versus clean-only NISE 89.7/68.6/63.0%. Clean-only 3DDA gives 81.3/55.0/15.0%. Noise exposure remains a major part of robustness; the architecture alone does not remove the need for varied training reliability. One trained policy serves all three evaluation levels.
G.3 Implementation and Runtime Cost
The deployed policy has 912.68M total parameters, mostly a 907.92M frozen backbone; 4.76M belong to trainable modules by the stated counting convention. GM3A adds 8440 deployed parameters and .8 GFLOPs; the 164321-parameter geometry head is training-only. RTX 5080 four-view batch-one policy forward time is 37.5±.6 ms versus baseline 36.1±.4, about 26.6 versus 27.7 Hz. This excludes rendering/I/O and precomputed instruction encoding, so it is not a full robot control-loop rate.
Appendix H MetaWorld Per-Task Results
The complete clean-depth tables report 46 named MetaWorld tasks in seven panels; their NISE means average exactly 79.0%. Deviations here are repeated-evaluation variation, distinct from Table 2’s three-training-run variability. NISE is not best on every task: strong baselines retain advantages on several easy tasks and difficult stick/wall variants. All task names, ties and emphasis remain with their own protocols.
Accessible table, original values and citation links
| MetaWorld easy tasks | |||||||
|---|---|---|---|---|---|---|---|
| Method | Btn Press | Btn Topdown | Btn TD Wall | Btn Wall | Coffee Btn | Dial Turn | Door Close |
| 3D Diffusion Policy | |||||||
| 3D Flow Matching | |||||||
| 3D ManiFlow | |||||||
| NISE | |||||||
| MetaWorld easy tasks | |||||||
| Method | Door Lock | Door Open | Door Unlock | Drawer Close | Drawer Open | Faucet Close | Faucet Open |
| 3D Diffusion Policy | |||||||
| 3D Flow Matching | |||||||
| 3D ManiFlow | |||||||
| NISE | |||||||
| MetaWorld easy tasks | |||||||
| Method | Handle Press | Handle Pull | Handle Pull Side | Lever Pull | Plate Slide | Plate Back | Plate Back Side |
| 3D Diffusion Policy | |||||||
| 3D Flow Matching | |||||||
| 3D ManiFlow | |||||||
| NISE | |||||||
| MetaWorld easy / medium tasks | |||||||
| Method | Plate Side | Reach | Reach Wall | Window Close | Window Open | Basketball | Bin Picking |
| 3D Diffusion Policy | |||||||
| 3D Flow Matching | |||||||
| 3D ManiFlow | |||||||
| NISE | |||||||
Accessible table, original values and citation links
| MetaWorld medium tasks | |||||||
|---|---|---|---|---|---|---|---|
| Method | Box Close | Coffee Pull | Coffee Push | Hammer | Peg Insert Side | Push Wall | Soccer |
| 3D Diffusion Policy | |||||||
| 3D Flow Matching | |||||||
| 3D ManiFlow | |||||||
| NISE | |||||||
| MetaWorld medium / hard tasks | |||||||
| Method | Sweep | Sweep Into | Assembly | Hand Insert | Pick Out Hole | Pick Place | Push |
| 3D Diffusion Policy | |||||||
| 3D Flow Matching | |||||||
| 3D ManiFlow | |||||||
| NISE | |||||||
Appendix I Real-Robot Protocol and Confidence Intervals
Each of five real tasks has thirty rollouts per method, varied placements and manual resets. Binary terminal/video criteria are explicitly task-specific; tube success is a grasp moved over the mortar into the pouring pose. Table 16 retains all twenty counts and Wilson 95% intervals from the exact formula. The small binomial samples quantify substantial uncertainty and are not independent training-seed or cross-site studies. The manuscript does not supply full demonstration counts or a detailed placement distribution.
Equation 8
| (8) |
Accessible table, original values and citation links
| Task | DP3 | 3DDA | RVT-2 | NISE |
|---|---|---|---|---|
| Assemble the Gear | 4/30, 13.3 [5.3, 29.7] | 7/30, 23.3 [11.8, 40.9] | 11/30, 36.7 [21.9, 54.5] | 25/30, 83.3 [66.4, 92.7] |
| Remove the Petri Lid | 6/30, 20.0 [9.5, 37.3] | 10/30, 33.3 [19.2, 51.2] | 5/30, 16.7 [7.3, 33.6] | 20/30, 66.7 [48.8, 80.8] |
| Insert the Metal Pipe | 3/30, 10.0 [3.5, 25.6] | 8/30, 26.7 [14.2, 44.4] | 12/30, 40.0 [24.6, 57.7] | 22/30, 73.3 [55.6, 85.8] |
| Pour Test Tube into Mortar | 2/30, 6.7 [1.8, 21.3] | 6/30, 20.0 [9.5, 37.3] | 4/30, 13.3 [5.3, 29.7] | 13/30, 43.3 [27.4, 60.8] |
| Vision-Sorted Gear Collection | 4/30, 13.3 [5.3, 29.7] | 3/30, 10.0 [3.5, 25.6] | 7/30, 23.3 [11.8, 40.9] | 16/30, 53.3 [36.1, 69.8] |
Appendix J Hyperparameter Sweep
One-at-a-time validation sweeps select geometry weight .1 and regularizer weight 1.0 while keeping the other fixed. Too little auxiliary supervision or overly stiff anchoring lowers the listed mean. These are validation choices, not a full joint hyperparameter search or independent final-test significance result. The note confirms the unweighted regularizer is multiplied by its coefficient once.
Accessible table, original values and citation links
| Sweep | Value | Val SR (%) | Selected | Readout |
| GCVSS weight | ||||
| GCVSS | 0.00 | 78.2 | – | GM3A-only control; no cross-view geometry shaping |
| GCVSS | 0.03 | 82.4 | – | useful signal, but still underweighted |
| GCVSS | 0.10 | 85.7 | best balance between geometry shaping and policy supervision | |
| GCVSS | 0.30 | 84.8 | – | similar but begins to overemphasize the auxiliary target |
| GCVSS | 1.00 | 81.6 | – | geometry loss competes too strongly with action learning |
| GM3A regularizer | ||||
| GM3A | 0.00 | 84.1 | – | dampening is flexible but less tied to the sensor prior |
| GM3A | 0.10 | 85.0 | – | stable but slightly weaker than the selected anchor |
| GM3A | 1.00 | 85.7 | selected sensor-prior anchor | |
| GM3A | 3.00 | 84.9 | – | stronger anchoring remains competitive |
| GM3A | 10.00 | 82.8 | – | prior becomes too stiff and limits adaptation |
Note: The listed coefficients are the applied loss weights; multiplies the unweighted regularizer once in the total training loss.