VR-Robo trains a legged navigation policy in reconstructed real environments, using Gaussian splats for photorealistic observations and a corresponding mesh for physical interaction. A hierarchical controller transfers to a real quadruped without real-world policy fine-tuning. The central task is reaching a specified colored cone across ground, platforms, slopes, and stairs. Quantitative evidence comes mainly from one reconstructed room; broader scenes, lighting conditions, and robot types are demonstrated qualitatively.
I Introduction
Visual locomotion requires both recognizing a goal and choosing terrain that the body can traverse. VR-Robo combines real-scene reconstruction, interactive rendering, and reinforcement learning to train these decisions together. Its digital twin retains the appearance of the captured environment while supporting randomized robot and object placements. The transfer result depends on reconstruction, calibrated cameras, a pretrained locomotion controller, and domain randomization; zero-shot deployment refers to policy transfer after this preparation.
Fig. 1: Our VR-Robo introduces a unified real-to-sim-to-real framework. We generate photorealistic and physically interactive “digital twin” simulation environments for visual policy training, and enable zero-shot real-world deployment.Fig. 2: Diverse experiment settings including scenes, conditions, and robot types (better viewed when zoomed in).
II Related Works
The approach connects simulation-based legged control with appearance reconstruction. Its aim is to reduce visual mismatch while retaining collision geometry and sufficiently fast simulation for reinforcement learning.
II-A Sim-to-Real Policy Transfer
Proprioceptive and geometry-based locomotion policies already transfer from simulation, but high-level visual navigation adds semantic and appearance sensitivity. VR-Robo combines a low-level locomotion policy with a separate visual command policy. This separates fast joint control from slower goal-directed planning while allowing the latter to learn feasible approaches to elevated targets.
II-B Real-to-Sim Scene Transfer
Neural scene representations reproduce real appearance but require a physical geometry representation for interactive robot learning. VR-Robo builds on 3D Gaussian splatting [12] and planar Gaussian reconstruction [30], pairing their renderings with an extracted mesh. Joint rendering of the robot, target objects, and environment preserves occlusion as their positions change.
III Task definition
The robot must reach a cone of the commanded color using an egocentric RGB image and proprioception. Targets can lie on terrain that requires an indirect route: the low-level policy can negotiate approximately 15 cm stairs but cannot directly climb a 30 cm platform. The high-level policy must therefore choose a traversable approach. RGB-only describes the exteroceptive modality, not the absence of joint or body-state observations.
Fig. 3: Task definition. The robot aims to reach the target cone of a specified color in a specified spatial location, relying solely on ego-viewRGB observation and proprioception.Fig. 4: VR-Robo real-to-sim-to-real framework. We build a realistic and interactive simulation environment with GS-mesh hybrid representation and occlusion-aware composition & randomization for policy training. Finally, we zero-shot transfer the RL policy trained in simulation into the real robot for ego-centric navigation and visual locomotion.
IV A Real-to-Sim-to-Real System for Visual Locomotion
The pipeline reconstructs real scenes, aligns visual and physical coordinates, and trains a visual policy inside the resulting simulator. Successful transfer requires consistency between the rendered view, collision mesh, randomized object poses, and robot camera.
IV-A Geometry-Consistent Reconstruction
RGB captures and COLMAP camera estimates initialize planar Gaussian reconstruction. Flattened Gaussians and plane-intersection depth encourage usable surfaces; monocular depth is aligned to sparse structure-from-motion depths, while normal priors and multiview normalized cross-correlation regularize geometry. A truncated signed distance field integrates the recovered depths into a collision mesh. These losses improve geometry without replacing the Gaussian representation used for appearance.
Visual and simulation coordinates are aligned through a regular reference object and four manually selected noncoplanar points. The transformation is then applied to the scene and irregular objects; spherical-harmonic coefficients are rotated consistently. Manual cropping and mesh postprocessing remain part of preparing the digital twin. Reconstruction is therefore a practical assisted workflow, rather than an entirely automatic RGB-to-simulator conversion.
Equations 1–6
(1)
(2)
(3)
(4)
(5)
(6)
IV-B Building Realistic and Interactive Simulation
Each episode randomizes robot and cone placement in the physics simulator and applies the same transforms to their Gaussian models. Joint rendering and depth ordering preserve occlusions between the scene, robot, and objects. This creates task variation without replacing the captured environment with an unrelated background. Camera pose, image appearance, noise, blur, and observation delay are also randomized to reduce deployment mismatch.
Fig. 5: Coordinate alignment. We first compute the transformation for a regular reference object and then apply this transformation to the irregular object of interest. Equations 7–11
(7)
(8)
(9)
(10)
(11)
Fig. 6: RL policy training in the reconstructed simulation environment. The agent leverages the ego-view GS photorealistic rendering as visual observations and interacts with the GS-extracted mesh in the Issac Sim environment.Equations 12–13
(12)
(13)
Fig. 7: Agent-object Randomization and Scene Composition. At the beginning of each episode, we randomly sample the positions for the robot and three cones in the Isaac Sim environment (upper row). We synchronously merge the agent and object Gaussians into the environment and compose for joint rendering (lower row).
IV-C Reinforcement Learning in Reconstructed Simulation
A low-level recurrent controller runs at 50 Hz from proprioception and velocity commands. A high-level policy runs at 5 Hz from pretrained visual features, target color, the previous action, and proprioception, producing bounded velocity commands. Its privileged training critic additionally observes world-coordinate information unavailable to the deployed actor. Rewards combine reaching, distance and height progress, heading, command tracking, stopping near the target, and action regularization.
Low-level training uses 4,096 parallel agents for 80,000 iterations; high-level training uses 64 agents for 8,000 iterations and takes roughly three days on one RTX 4090D. The real Go2 runs both policies onboard a Jetson Orin Nano, with calibrated RGB observations and joint-level proportional–derivative control. These implementation costs are material to reproducing the transfer result.
Equations 14–19
(14)
(15)
(16)
(17)
(18)
(19)
V Experiments
Evaluation separates simulation comparisons, real deployment, and ablations of appearance representation, visual encoder, and randomization. Success and time are defined differently across the simulation and physical experiments, so their percentages should be read within each protocol.
V-A Experiment Setup
Six indoor scenes are reconstructed from mobile-device images. Simulation evaluation uses 150 fixed robot/target configurations in Scene 1 with a red target, a 15-second horizon, and a 0.25 m reaching threshold. Failed trials receive the full horizon when average reaching time is computed. Baselines include imitation learning from 60 real teleoperated trajectories and a random ImageNet-background approximation inspired by prior generative-background work; the latter is not a faithful implementation of the unavailable original system.
TABLE I: Comparison and ablation experimental results in the real-world setting.
In simulation, VR-Robo reaches every evaluated target with a mean time of 4.94 seconds, compared with 8.67% success for imitation learning and 43.33% for random backgrounds. Replacing Gaussian appearance with a textured mesh reduces success to 22%, and replacing the pretrained vision transformer with a CNN reduces it to 54.67%. Reconstruction PSNR is 29.73 for planar Gaussians versus 27.73 for the textured mesh. These ablations support the full representation and encoder in the tested environment, without isolating every source of visual or geometric error.
Fig. 8: Training reward comparison. Pretrained CNN [41] encoder struggles to extract precise semantic features, leading to unstable and worse rewards compared to ViT [38] encoder.Fig. 9: Illustration of textured mesh and random background.
TABLE II: Comparison results in the simulation setting.
Real quantitative tests use three target configurations at each of three difficulty levels, repeated five times: 45 trials per method in Scene 1. Physical success requires contacting the cone. VR-Robo succeeds in all easy and hard trials and 14 of 15 medium trials, with average times of 4.96, 6.28, and 9.09 seconds. Removing domain randomization, changing the encoder, or using a textured mesh substantially reduces success, especially on the hard routes.
SARO is allowed 60 seconds because of its language-model latency, whereas the learned policies use 15 seconds; reaching-time comparisons therefore include different failure penalties and horizons. Qualitative demonstrations span additional rooms, lighting conditions, and robot embodiments, but should not be interpreted as the same controlled quantitative generalization study.
Fig. 10: Tasks of different difficulties (easy, medium, and hard) and different target positions (left, middle, and right).
VI Limitations and Conclusion
The demonstrated benefit is a visually realistic, interactive training environment that preserves terrain geometry well enough for goal-directed locomotion transfer. Remaining bottlenecks concern scene scope, reconstruction reliability, and training cost.
VI-A Limitations
Experiments focus on static indoor terrain. The high-level policy requires about three days of training, limiting rapid task adaptation. RGB reconstruction can introduce structural mesh errors and still needs manual postprocessing. Dynamic environments, broader outdoor conditions, and automatic reconstruction repair remain open directions.
VI-B Conclusion
VR-Robo links photorealistic rendering and physical simulation through a shared reconstructed scene, then trains hierarchical navigation that transfers to the real robot. The results support this combination for the evaluated cone-reaching tasks; they do not establish unrestricted navigation in unseen or dynamic environments.
42 references
[1]
E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5026–5033.
[2]
V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al., “Isaac gym: High performance gpu-based physics simulation for robot learning,” arXiv preprint arXiv:2108.10470, 2021.
[3]
A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al., “Habitat 2.0: Training home assistants to rearrange their habitat,” Advances in neural information processing systems, vol. 34, pp. 251–266, 2021.
[4]
S. Tao, F. Xiang, A. Shukla, Y. Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y. Liu, T.-k. Chan, et al., “Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai,” arXiv preprint arXiv:2410.00425, 2024.
[6]
X. Cheng, K. Shi, A. Agarwal, and D. Pathak, “Extreme parkour with legged robots,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 11 443–11 450.
[7]
D. Hoeller, N. Rudin, D. Sako, and M. Hutter, “Anymal parkour: Learning agile navigation for quadrupedal robots,” Science Robotics, vol. 9, no. 88, p. eadi7566, 2024.
[8]
J. Wu, G. Xin, C. Qi, and Y. Xue, “Learning robust and agile legged locomotion using adversarial motion priors,” IEEE Robotics and Automation Letters, 2023.
[10]
A. Yu, G. Yang, R. Choi, Y. Ravan, J. Leonard, and P. Isola, “Learning visual parkour from generated images,” in 8th Annual Conference on Robot Learning, 2024.
[11]
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021.
[12]
B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering.” ACM Trans. Graph., vol. 42, no. 4, pp. 139–1, 2023.
[13]
A. Byravan, J. Humplik, L. Hasenclever, A. Brussee, F. Nori, T. Haarnoja, B. Moran, S. Bohez, F. Sadeghi, B. Vujatovic, et al., “Nerf2real: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 9362–9369.
[14]
H. Lou, Y. Liu, Y. Pan, Y. Geng, J. Chen, W. Ma, C. Li, L. Wang, H. Feng, L. Shi, et al., “Robo-gs: A physics consistent spatial-temporal model for robotic arm with hybrid representation,” arXiv preprint arXiv:2408.14873, 2024.
[15]
X. Li, J. Li, Z. Zhang, R. Zhang, F. Jia, T. Wang, H. Fan, K.-K. Tseng, and R. Wang, “Robogsim: A real2sim2real robotic gaussian splatting simulator,” arXiv preprint arXiv:2411.11839, 2024.
[16]
Y. Wu, L. Pan, W. Wu, G. Wang, Y. Miao, and H. Wang, “Rl-gsbridge: 3d gaussian splatting based real2sim2real method for robotic manipulation learning,” arXiv preprint arXiv:2409.20291, 2024.
[17]
Z. Xie, Z. Liu, Z. Peng, W. Wu, and B. Zhou, “Vid2sim: Realistic and interactive simulation from video for urban navigation,” arXiv preprint arXiv:2501.06693, 2025.
[19]
L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang, “Gensim: Generating robotic simulation tasks via large language models,” arXiv preprint arXiv:2310.01361, 2023.
[20]
X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018, pp. 3803–3810.
[21]
J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, 2017, pp. 23–30.
[22]
J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” Science Robotics, vol. 4, no. 26, p. eaau5872, 2019.
[23]
J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, “Sim-to-real: Learning agile locomotion for quadruped robots,” arXiv preprint arXiv:1804.10332, 2018.
[24]
X. Li, Y. Zhang, and X. Y. Drivingdiffusion, “Layout-guided multi-view driving scene video generation with latent diffusion model,” arXiv preprint arXiv:2310.07771, vol. 2, no. 3, 2023.
[25]
Z. Chen, Z. Mandi, H. Bharadhwaj, M. Sharma, S. Song, A. Gupta, and V. Kumar, “Semantically controllable augmentations for generalizable robot learning,” The International Journal of Robotics Research, p. 02783649241273686, 2024.
[26]
M. Torne, A. Simeonov, Z. Li, A. Chan, T. Chen, A. Gupta, and P. Agrawal, “Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation,” arXiv preprint arXiv:2403.03949, 2024.
[27]
A. Quach, M. Chahine, A. Amini, R. Hasani, and D. Rus, “Gaussian splatting to real world flight navigation transfer with liquid networks,” arXiv preprint arXiv:2406.15149, 2024.
[28]
J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4104–4113.
[29]
B. Huang, Z. Yu, A. Chen, A. Geiger, and S. Gao, “2d gaussian splatting for geometrically accurate radiance fields,” arXiv preprint arXiv:2403.17888, 2024.
[30]
D. Chen, H. Li, W. Ye, Y. Wang, W. Xie, S. Zhai, N. Wang, H. Liu, H. Bao, and G. Zhang, “Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction,” arXiv preprint arXiv:2406.06521, 2024.
[31]
M. Turkulainen, X. Ren, I. Melekhov, O. Seiskari, E. Rahtu, and J. Kannala, “Dn-splatter: Depth and normal priors for gaussian splatting and meshing,” arXiv preprint arXiv:2403.17822, 2024.
[32]
G. Bae and A. J. Davison, “Rethinking inductive biases for surface normal estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9535–9545.
[33]
S. Galliani, K. Lasinger, and K. Schindler, “Massively parallel multiview stereopsis by surface normal diffusion,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 873–881.
[34]
Q. Fu, Q. Xu, Y. S. Ong, and W. Tao, “Geo-neus: Geometry-consistent neural implicit surfaces learning for multi-view reconstruction,” Advances in Neural Information Processing Systems, vol. 35, pp. 3403–3416, 2022.
[35]
B. Curless and M. Levoy, “A volumetric method for building complex models from range images,” in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 1996, pp. 303–312.
[39]
S. Macenski, T. Foote, B. Gerkey, C. Lalancette, and W. Woodall, “Robot operating system 2: Design, architecture, and uses in the wild,” Science robotics, vol. 7, no. 66, p. eabm6074, 2022.
[40]
S. Zhu, D. Li, L. Mou, Y. Liu, N. Xu, and H. Zhao, “Saro: Space-aware robot system for terrain crossing via vision-language model,” arXiv preprint arXiv:2407.16412, 2024.
[41]
A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, et al., “Searching for mobilenetv3,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1314–1324.
[42]
A. Guédon and V. Lepetit, “Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 5354–5363.