Literature
Display
← All papers

VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotion

Shaoting Zhu · Linzhan Mou · Derun Li · Baijun Ye · Runhan Huang · Hang Zhao

· IEEE Robotics and Automation Letters ↗

Robotics6 min text · CondensedOriginal paper ↗
Contents

At a glance

VR-Robo trains a legged navigation policy in reconstructed real environments, using Gaussian splats for photorealistic observations and a corresponding mesh for physical interaction. A hierarchical controller transfers to a real quadruped without real-world policy fine-tuning. The central task is reaching a specified colored cone across ground, platforms, slopes, and stairs. Quantitative evidence comes mainly from one reconstructed room; broader scenes, lighting conditions, and robot types are demonstrated qualitatively.

I Introduction

Visual locomotion requires both recognizing a goal and choosing terrain that the body can traverse. VR-Robo combines real-scene reconstruction, interactive rendering, and reinforcement learning to train these decisions together. Its digital twin retains the appearance of the captured environment while supporting randomized robot and object placements. The transfer result depends on reconstruction, calibrated cameras, a pretrained locomotion controller, and domain randomization; zero-shot deployment refers to policy transfer after this preparation.

Refer to caption
Fig. 1: Our VR-Robo introduces a unified real-to-sim-to-real framework. We generate photorealistic and physically interactive “digital twin” simulation environments for visual policy training, and enable zero-shot real-world deployment.
Refer to caption
Fig. 2: Diverse experiment settings including scenes, conditions, and robot types (better viewed when zoomed in).

II Related Works

The approach connects simulation-based legged control with appearance reconstruction. Its aim is to reduce visual mismatch while retaining collision geometry and sufficiently fast simulation for reinforcement learning.

II-A Sim-to-Real Policy Transfer

Proprioceptive and geometry-based locomotion policies already transfer from simulation, but high-level visual navigation adds semantic and appearance sensitivity. VR-Robo combines a low-level locomotion policy with a separate visual command policy. This separates fast joint control from slower goal-directed planning while allowing the latter to learn feasible approaches to elevated targets.

II-B Real-to-Sim Scene Transfer

Neural scene representations reproduce real appearance but require a physical geometry representation for interactive robot learning. VR-Robo builds on 3D Gaussian splatting [12] and planar Gaussian reconstruction [30], pairing their renderings with an extracted mesh. Joint rendering of the robot, target objects, and environment preserves occlusion as their positions change.

III Task definition

The robot must reach a cone of the commanded color using an egocentric RGB image and proprioception. Targets can lie on terrain that requires an indirect route: the low-level policy can negotiate approximately 15 cm stairs but cannot directly climb a 30 cm platform. The high-level policy must therefore choose a traversable approach. RGB-only describes the exteroceptive modality, not the absence of joint or body-state observations.

Refer to caption
Fig. 3: Task definition. The robot aims to reach the target cone of a specified color in a specified spatial location, relying solely on ego-view RGB observation and proprioception.
Refer to caption
Fig. 4: VR-Robo real-to-sim-to-real framework. We build a realistic and interactive simulation environment with GS-mesh hybrid representation and occlusion-aware composition & randomization for policy training. Finally, we zero-shot transfer the RL policy trained in simulation into the real robot for ego-centric navigation and visual locomotion.

IV A Real-to-Sim-to-Real System for Visual Locomotion

The pipeline reconstructs real scenes, aligns visual and physical coordinates, and trains a visual policy inside the resulting simulator. Successful transfer requires consistency between the rendered view, collision mesh, randomized object poses, and robot camera.

IV-A Geometry-Consistent Reconstruction

RGB captures and COLMAP camera estimates initialize planar Gaussian reconstruction. Flattened Gaussians and plane-intersection depth encourage usable surfaces; monocular depth is aligned to sparse structure-from-motion depths, while normal priors and multiview normalized cross-correlation regularize geometry. A truncated signed distance field integrates the recovered depths into a collision mesh. These losses improve geometry without replacing the Gaussian representation used for appearance.

Visual and simulation coordinates are aligned through a regular reference object and four manually selected noncoplanar points. The transformation is then applied to the scene and irregular objects; spherical-harmonic coefficients are rotated consistently. Manual cropping and mesh postprocessing remain part of preparing the digital twin. Reconstruction is therefore a practical assisted workflow, rather than an entirely automatic RGB-to-simulator conversion.

Equations 1–6
𝒢i​(𝐱)=exp⁡(−12​(𝐱−μi)T​Σi−1​(𝐱−μi)),\mathcal{G}_{i}(\mathbf{x})=\exp\left(-\frac{1}{2}\left(\mathbf{x}-\mu_{i}\right)^{T}\Sigma_{i}^{-1}\left(\mathbf{x}-\mu_{i}\right)\right), (1)
𝐜⁡(x)=∑i∈NTi​ci​αi​(𝐱),Ti=∏i=1i−1(1−αi​(𝐱)).\mathbf{c}(x)=\sum_{i\in N}T_{i}c_{i}\alpha_{i}(\mathbf{x}),\quad T_{i}=\prod_{i=1}^{i-1}\left(1-\alpha_{i}(\mathbf{x})\right).\vskip-2.84526pt (2)
ℒ𝚜𝚌𝚊𝚕𝚎=1N​∑i∈N‖min⁡(s1,s2,s3)‖,\mathcal{L}_{\mathtt{scale}}=\frac{1}{N}\sum_{i\in N}\left\|\min\left(s_{1},s_{2},s_{3}\right)\right\|,\vskip-2.84526pt (3)
𝑫​(p)𝚛𝚎𝚗𝚍𝚎𝚛=𝒟𝒩⁡(p)​K−1​p~.\boldsymbol{D}(p)_{\mathtt{render}}=\frac{\mathcal{D}}{\mathcal{N}(p)K^{-1}\tilde{p}}.\vskip-2.84526pt (4)
𝒔^,𝒕^=arg⁡min𝒔,𝒕​∑p∈𝑫𝚜𝚏𝚖‖(𝒔∗𝑫​(p)𝚖𝚘𝚗𝚘+𝒕)−𝑫​(p)𝚜𝚏𝚖‖22.\hat{\boldsymbol{s}},\hat{\boldsymbol{t}}=\underset{\boldsymbol{s},\boldsymbol{t}}{\arg\min}\sum_{p\in\boldsymbol{D}_{\mathtt{sfm}}}\left\|\left(\boldsymbol{s}*\boldsymbol{D}(p)_{\mathtt{mono}}+\boldsymbol{t}\right)-\boldsymbol{D}(p)_{\mathtt{sfm}}\right\|_{2}^{2}.\vskip-2.84526pt (5)
ℒ𝚖𝚟=1‖𝒫‖​∑𝐩∈𝒫∑p∈𝐩(1−N​C​C​(𝐈^​(ℋ​p),𝐈⁡(p))).\mathcal{L}_{\mathtt{mv}}=\frac{1}{\|\mathcal{P}\|}\sum_{\mathbf{p}\in\mathcal{P}}\sum_{p\in\mathbf{p}}(1-NCC(\hat{\mathbf{I}}(\mathcal{H}p),\mathbf{I}(p))).\vskip-2.84526pt (6)

IV-B Building Realistic and Interactive Simulation

Each episode randomizes robot and cone placement in the physics simulator and applies the same transforms to their Gaussian models. Joint rendering and depth ordering preserve occlusions between the scene, robot, and objects. This creates task variation without replacing the captured environment with an unrelated background. Camera pose, image appearance, noise, blur, and observation delay are also randomized to reduce deployment mismatch.

Refer to caption
Fig. 5: Coordinate alignment. We first compute the transformation for a regular reference object and then apply this transformation to the irregular object of interest.
Equations 7–11
Tblock=arg⁡min⁡∑i=14T⁡‖T⋅𝐩iℬ−𝐩iCOLMAP‖2,T_{\text{block}}=\arg\min_{T}\sum_{i=1}^{4}\|T\cdot\mathbf{p}_{i}^{\mathcal{B}}-\mathbf{p}_{i}^{\text{COLMAP}}\|^{2},\vskip-5.69054pt (7)
Tcone=Tblock⋅Δ​𝐩cone.T_{\text{cone}}=T_{\text{block}}\cdot\Delta\mathbf{p}_{\text{cone}}.\vskip-2.84526pt (8)
μ′=R𝚑𝚘𝚖𝚘​μ+t𝚑𝚘𝚖𝚘,\mu^{\prime}=R_{\mathtt{homo}}\mu+t_{\mathtt{homo}},\vskip-2.84526pt (9)
S′=S+l​o​g​(s𝚑𝚘𝚖𝚘),S^{\prime}=S+log(s_{\mathtt{homo}}),\vskip-2.84526pt (10)
R𝚗𝚘𝚛𝚖=R𝚑𝚘𝚖𝚘s𝚑𝚘𝚖𝚘Σ′=R𝚗𝚘𝚛𝚖​Σ​R𝚗𝚘𝚛𝚖T,R_{\mathtt{norm}}=\frac{R_{\mathtt{homo}}}{s_{\mathtt{homo}}}\quad\Sigma^{\prime}=R_{\mathtt{norm}}\Sigma R_{\mathtt{norm}}^{T},\vskip-2.84526pt (11)
Refer to caption
Fig. 6: RL policy training in the reconstructed simulation environment. The agent leverages the ego-view GS photorealistic rendering as visual observations and interacts with the GS-extracted mesh in the Issac Sim environment.
Equations 12–13
𝐂(ℓ)′=Dℓ​(α,−β,γ)​𝐂(ℓ).\mathbf{C}^{(\ell)^{\prime}}=D_{\ell}(\alpha,-\beta,\gamma)\mathbf{C}^{(\ell)}.\vskip-5.69054pt (12)
T𝚘𝚋𝚓𝙲𝙾𝙻𝙼𝙰𝙿−𝚎𝚗𝚟=T𝚜𝚒𝚖𝙲𝙾𝙻𝙼𝙰𝙿−𝚎𝚗𝚟⋅T𝙲𝙾𝙻𝙼𝙰𝙿−𝚘𝚋𝚓𝚜𝚒𝚖⋅T𝚋𝚋𝚘𝚡.T_{\mathtt{obj}}^{\mathtt{COLMAP-env}}=T_{\mathtt{sim}}^{\mathtt{COLMAP-env}}\cdot T_{\mathtt{COLMAP-obj}}^{\mathtt{sim}}\cdot T_{\mathtt{bbox}}.\vskip-2.84526pt (13)
Refer to caption
Fig. 7: Agent-object Randomization and Scene Composition. At the beginning of each episode, we randomly sample the positions for the robot and three cones in the Isaac Sim environment (upper row). We synchronously merge the agent and object Gaussians into the environment and compose for joint rendering (lower row).

IV-C Reinforcement Learning in Reconstructed Simulation

A low-level recurrent controller runs at 50 Hz from proprioception and velocity commands. A high-level policy runs at 5 Hz from pretrained visual features, target color, the previous action, and proprioception, producing bounded velocity commands. Its privileged training critic additionally observes world-coordinate information unavailable to the deployed actor. Rewards combine reaching, distance and height progress, heading, command tracking, stopping near the target, and action regularization.

Low-level training uses 4,096 parallel agents for 80,000 iterations; high-level training uses 64 agents for 8,000 iterations and takes roughly three days on one RTX 4090D. The real Go2 runs both policies onboard a Jetson Orin Nano, with calibrated RGB observations and joint-level proportional–derivative control. These implementation costs are material to reproducing the transfer result.

Equations 14–19
rreach_goal={Rmax,if ​‖𝐩robot−𝐩goal‖2≤ϵ,0,otherwise.r_{\text{reach\_goal}}=\begin{cases}R_{\text{max}},&\text{if }\|\mathbf{p}_{\text{robot}}-\mathbf{p}_{\text{goal}}\|_{2}\leq\epsilon,\\ 0,&\text{otherwise}.\end{cases}\vskip-2.84526pt (14)
rgoal_dis\displaystyle r_{\text{goal\_dis}} =dprev−dcurrent,\displaystyle=d_{\text{prev}}-d_{\text{current}}, (15)
d\displaystyle d =‖𝐩robot−𝐩goal‖2.\displaystyle=\|\mathbf{p}_{\text{robot}}-\mathbf{p}_{\text{goal}}\|_{2}.\vskip-8.53581pt (16)
rgoal_dis_z\displaystyle r_{\text{goal\_dis\_z}} =dz,prev−dz,current,\displaystyle=d_{z,\text{prev}}-d_{z,\text{current}}, (17)
dz\displaystyle d_{z} =|zrobot−zgoal|.\displaystyle=|z_{\text{robot}}-z_{\text{goal}}|.\vskip-8.53581pt (18)
rgoal_heading=−|ψrobot−ψgoal|,r_{\text{goal\_heading}}=-|\psi_{\text{robot}}-\psi_{\text{goal}}|,\vskip-2.84526pt (19)

V Experiments

Evaluation separates simulation comparisons, real deployment, and ablations of appearance representation, visual encoder, and randomization. Success and time are defined differently across the simulation and physical experiments, so their percentages should be read within each protocol.

V-A Experiment Setup

Six indoor scenes are reconstructed from mobile-device images. Simulation evaluation uses 150 fixed robot/target configurations in Scene 1 with a red target, a 15-second horizon, and a 0.25 m reaching threshold. Failed trials receive the full horizon when average reaching time is computed. Baselines include imitation learning from 60 real teleoperated trajectories and a random ImageNet-background approximation inspired by prior generative-background work; the latter is not a faithful implementation of the unavailable original system.

TABLE I: Comparison and ablation experimental results in the real-world setting.
Method Exteroception Success Rate ↑\uparrow Average Reaching Time (s) ↓\downarrow
Easy Medium Hard Easy Medium Hard
Ours RGB 100.00% 93.33% 100.00% 4.96 6.28 9.09
Imitation Learning (IL) RGB 0.00% 0.00% 0.00% 15.00 15.00 15.00
SARO [40] RGB 66.67% 26.67% 0.00% 46.49 57.24 60.00
Textured Mesh RGB 20.00% 6.67% 0.00% 12.90 14.90 15.00
CNN Encoder RGB 73.33% 66.67% 6.67% 9.10 11.41 14.90
w/o Domain Randomization RGB 53.33% 6.67% 0.00% 10.04 14.76 15.00

V-B Simulation Experiments

In simulation, VR-Robo reaches every evaluated target with a mean time of 4.94 seconds, compared with 8.67% success for imitation learning and 43.33% for random backgrounds. Replacing Gaussian appearance with a textured mesh reduces success to 22%, and replacing the pretrained vision transformer with a CNN reduces it to 54.67%. Reconstruction PSNR is 29.73 for planar Gaussians versus 27.73 for the textured mesh. These ablations support the full representation and encoder in the tested environment, without isolating every source of visual or geometric error.

Refer to caption
Fig. 8: Training reward comparison. Pretrained CNN [41] encoder struggles to extract precise semantic features, leading to unstable and worse rewards compared to ViT [38] encoder.
Refer to caption
Fig. 9: Illustration of textured mesh and random background.
TABLE II: Comparison results in the simulation setting.
Method SR ↑\uparrow ART (s) ↓\downarrow
Ours 100.00% 4.94
Imitation Learning (IL) 8.67% 14.01
Random Background 43.33% 11.75
TABLE III: Ablation results in the simulation setting.
Method SR ↑\uparrow ART (s) ↓\downarrow
Ours 100.00% 4.94
Textured Mesh 22.00% 12.73
CNN Encoder 54.67% 10.04
TABLE IV: Reconstruction quality comparison.
Method Planar 3DGS Textured Mesh
PSNR ↑\uparrow 29.73 27.73

V-C Real-World Experiments

Real quantitative tests use three target configurations at each of three difficulty levels, repeated five times: 45 trials per method in Scene 1. Physical success requires contacting the cone. VR-Robo succeeds in all easy and hard trials and 14 of 15 medium trials, with average times of 4.96, 6.28, and 9.09 seconds. Removing domain randomization, changing the encoder, or using a textured mesh substantially reduces success, especially on the hard routes.

SARO is allowed 60 seconds because of its language-model latency, whereas the learned policies use 15 seconds; reaching-time comparisons therefore include different failure penalties and horizons. Qualitative demonstrations span additional rooms, lighting conditions, and robot embodiments, but should not be interpreted as the same controlled quantitative generalization study.

Refer to caption
Fig. 10: Tasks of different difficulties (easy, medium, and hard) and different target positions (left, middle, and right).

VI Limitations and Conclusion

The demonstrated benefit is a visually realistic, interactive training environment that preserves terrain geometry well enough for goal-directed locomotion transfer. Remaining bottlenecks concern scene scope, reconstruction reliability, and training cost.

VI-A Limitations

Experiments focus on static indoor terrain. The high-level policy requires about three days of training, limiting rapid task adaptation. RGB reconstruction can introduce structural mesh errors and still needs manual postprocessing. Dynamic environments, broader outdoor conditions, and automatic reconstruction repair remain open directions.

VI-B Conclusion

VR-Robo links photorealistic rendering and physical simulation through a shared reconstructed scene, then trains hierarchical navigation that transfers to the real robot. The results support this combination for the evaluated cone-reaching tasks; they do not establish unrestricted navigation in unseen or dynamic environments.

Concept demonstration

Open image ↗