Literature
Display
← All papers

SkyJEPA: Learning Long-Horizon World Models for Zero-Shot Sim-to-Real Control of Quadrotors

Pratyaksh Rao · Wancong Zhang · Randall Balestriero · Yann LeCun · Giuseppe Loianno

Preprint posted · arXiv (preprint; under review) ↗

World models8 min text · CondensedOriginal paper ↗
Contents

At a glance

SkyJEPA learns drone dynamics from simulated state–action histories, predicts their future in a compact latent space, then converts frozen latent predictions into physical states using a structured prober. A sampling-based controller uses those states to choose actions. Outdoor flights test simulation-only training without real-flight fine-tuning. This is a low-dimensional state-based control system, not a camera-based visual world model, and its measured robustness is not a general flight-safety guarantee.

I Introduction

A useful controller needs more than a low one-step prediction loss: it needs repeated forecasts that stay useful, physically meaningful outputs and enough speed to evaluate many possible actions. SkyJEPA addresses those requirements with three connected pieces: multi-step latent learning, a physics-inspired state readout and diverse simulation data. The evidence comes from open-loop predictions and real closed-loop tracking, each of which tests a different part of the system.

Refer to caption
Figure 1: Desirable properties of a quadrotor world model: accurate long-horizon prediction, interpretable, real-time inference for closed-loop control, and zero-shot task generalization.

II Related Works

The comparisons distinguish the space in which dynamics are learned, the information used to train them and whether the model is specialized to a reward. These choices can be combined in several ways. The paper’s ablations are therefore more informative than treating all prior predictive or reinforcement-learning models as one uniform baseline.

II-A Predictive Modeling

Direct state predictors repeatedly consume their own imperfect outputs, so small errors can shift later inputs away from the training distribution. Residual models, physical penalties and online adaptation address parts of this problem. SkyJEPA instead rolls out learned representations, but it still uses a recurrent predictor. Its experiments show reduced error growth, rather than a theorem eliminating recursive prediction error.

II-B Joint Embedding Predictive Architectures

JEPA methods predict representations instead of reconstructing every observation. Here the emphasis is fast dynamics forecasting for a physical aircraft, using estimated states rather than image patches. The paper should be compared with other state-based controllers under matched sensing and hardware assumptions; success does not establish the proposed future camera-navigation extension.

II-C Model-Based Reinforcement Learning

A reward-conditioned model may learn only the details useful for one policy or objective. SkyJEPA trains its latent dynamics without a task reward, then supplies a tracking cost at planning time. That separates model training from the chosen path. The final controller is still task-dependent because its reference trajectory, cost weights and action limits specify what it should do.

III Background

The background defines the physical state and explains why recursive forecasts are harder than isolated one-step predictions. Keep the distinction between a latent coordinate and a measured physical quantity: tracking costs need position and attitude, so an additional state-recovery stage remains necessary even when prediction happens in a latent space.

III-A Learning System Dynamics

The state includes position, velocity, a 3×3 orientation matrix and body angular velocity. Four motor forces form the modeled action. Representing orientation with a rotation matrix avoids treating three-dimensional rotations as ordinary independent numbers. These quantities are supplied by the state-estimation/control stack; the learned model does not infer them from raw camera images.

Equations 1–2
𝐱t+1=f⁡(𝐱t,𝐚t).\mathbf{x}_{t+1}=f(\mathbf{x}_{t},\mathbf{a}_{t}). (1)
𝐱t=[𝐩t⊤𝐯t⊤𝐫x,t⊤𝐫y,t⊤𝐫z,t⊤𝝎t⊤]⊤.\mathbf{x}_{t}=\begin{bmatrix}\mathbf{p}_{t}^{\top}&\mathbf{v}_{t}^{\top}&\mathbf{r}_{x,t}^{\top}&\mathbf{r}_{y,t}^{\top}&\mathbf{r}_{z,t}^{\top}&\bm{\omega}_{t}^{\top}\end{bmatrix}^{\top}. (2)

III-B Predictive Modeling

A one-step model can look accurate when each input is a true recorded state. During an open-loop rollout, its input is instead the previous prediction, so an early error changes every later calculation. This is the motivation for training and evaluating multi-step predictions. It does not mean a different representation automatically becomes immune to accumulated error.

Equations 3–6
𝐱~i+1=h𝜽​(𝐱i,𝐚i).\tilde{\mathbf{x}}_{i+1}=h_{\bm{\theta}}(\mathbf{x}_{i},\mathbf{a}_{i}). (3)
min𝜽⁡1N​∑i=1N‖𝐱i+1−h𝜽​(𝐱i,𝐚i)‖22.\min_{\bm{\theta}}\frac{1}{N}\sum_{i=1}^{N}\left\|\mathbf{x}_{i+1}-h_{\bm{\theta}}(\mathbf{x}_{i},\mathbf{a}_{i})\right\|_{2}^{2}. (4)
𝐱^t+T=h𝜽​(…​h𝜽​(h𝜽​(𝐱t,𝐚t),𝐚t+1)​…,𝐚t+T).\hat{\mathbf{x}}_{t+T}=h_{\bm{\theta}}(\dots h_{\bm{\theta}}(h_{\bm{\theta}}(\mathbf{x}_{t},\mathbf{a}_{t}),\mathbf{a}_{t+1})\dots,\mathbf{a}_{t+T}). (5)
𝐱~t+1\displaystyle\tilde{\mathbf{x}}_{t+1} =h𝜽​(𝐱t,𝐚t),\displaystyle=h_{\bm{\theta}}(\mathbf{x}_{t},\mathbf{a}_{t}), (6)
𝐱~t+2\displaystyle\tilde{\mathbf{x}}_{t+2} =h𝜽​(𝐱t+1−ϵt+1,𝐚t+1),\displaystyle=h_{\bm{\theta}}(\mathbf{x}_{t+1}-\bm{\epsilon}_{t+1},\mathbf{a}_{t+1}),
𝐱~t+T\displaystyle\tilde{\mathbf{x}}_{t+T} =h𝜽​(𝐱t+T−1−ϵt+T−1,𝐚t+T−1).\displaystyle=h_{\bm{\theta}}(\mathbf{x}_{t+T-1}-\bm{\epsilon}_{t+T-1},\mathbf{a}_{t+T-1}).
Refer to caption
Figure 2: Overview of the proposed framework. We learn a JEPA-style latent dynamics model with a physics-inspired prober that maps abstract embeddings to physically meaningful states, enabling stable long-horizon quadrotor prediction. Trained entirely on domain-randomized simulation data and deployed within a sampling-based controller, the model achieves real-time execution and robust zero-shot sim-to-real transfer validated in outdoor flight experiments.

IV Methodology

The training and deployment pipeline has four stages: learn action-conditioned latent dynamics, fit a physical-state prober with the latent model frozen, optimize candidate controls using the combined forecast, and generate training trajectories in randomized simulators. Separating these stages makes it possible to see which information and supervision each component uses.

IV-A Problem Formulation

Temporal-convolution encoders summarize recent states and actions, and a recurrent predictor advances the state embedding under future actions. The history provides evidence about delayed actuators and recent motion. Prediction remains recursive in latent space, while training asks the full predicted sequence to agree with encoded future histories. The intended benefit is a representation that is easier to roll out, not the removal of recurrence.

Equations 7–8
𝐗t=[𝐱t−H⊤⋯𝐱t⊤]⊤,𝐀t=[𝐚t−H⊤⋯𝐚t⊤]⊤.\mathbf{X}_{t}=\begin{bmatrix}\mathbf{x}_{t-H}^{\top}&\cdots&\mathbf{x}_{t}^{\top}\end{bmatrix}^{\top},\qquad\mathbf{A}_{t}=\begin{bmatrix}\mathbf{a}_{t-H}^{\top}&\cdots&\mathbf{a}_{t}^{\top}\end{bmatrix}^{\top}. (7)
𝐬~t+T=Predφ​(…​Predφ​(Predφ​(𝐬t,𝐳t),𝐳t+1),…,𝐳t+T−1).\tilde{\mathbf{s}}_{t+T}=\text{Pred}_{\varphi}(\dots\text{Pred}_{\varphi}(\text{Pred}_{\varphi}(\mathbf{s}_{t},\mathbf{z}_{t}),\mathbf{z}_{t+1}),\dots,\mathbf{z}_{t+T-1}). (8)

IV-B Training Objective

The multi-step loss alone could be minimized by mapping every trajectory to a constant. SIGReg opposes that shortcut by checking whether random one-dimensional projections of the predicted embedding batch resemble a Gaussian. The ideal all-projection characterization motivates the method; finite directions, batches and quadrature approximate it. Anti-collapse regularization does not itself guarantee preservation of every unstable or safety-relevant state direction.

Equations 9–13
ℒpred=1T​∑k=1T‖𝐬~t+k−𝐬t+k‖22.\mathcal{L}_{\mathrm{pred}}=\frac{1}{T}\sum_{k=1}^{T}\|\tilde{\mathbf{s}}_{t+k}-\mathbf{s}_{t+k}\|_{2}^{2}. (9)
ϕN​(t,𝐡(m))=1B​∑b=1Bei​t​hb(m),\phi_{N}(t;\mathbf{h}^{(m)})=\frac{1}{B}\sum_{b=1}^{B}e^{ith_{b}^{(m)}}, (10)
T(m)=∫−∞∞w⁡(t)​|ϕB​(t,𝐡(m))−ϕ0​(t)|2​𝑑t,T^{(m)}=\int_{-\infty}^{\infty}w(t)\left|\phi_{B}(t;\mathbf{h}^{(m)})-\phi_{0}(t)\right|^{2}dt, (11)
ℒSIGReg=1M​∑m=1MT(m).\mathcal{L}_{\mathrm{SIGReg}}=\frac{1}{M}\sum_{m=1}^{M}T^{(m)}. (12)
ℒtotal=ℒpred+λsig​ℒSIGReg.\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{pred}}+\lambda_{\mathrm{sig}}\mathcal{L}_{\mathrm{SIGReg}}. (13)
Refer to caption
Figure 3: Two-stage training pipeline of our proposed approach. In the first stage, an encoder maps a history of past states into a latent representation, and a predictor propagates this latent forward in time conditioned on a sequence of control actions; in the second stage, a physics-inspired prober is trained on frozen latent embeddings to map them to physically grounded states.

IV-C Physics-Inspired Probing Mechanism

After freezing the encoders and predictor, train a small network to produce acceleration corrections from their latent rollouts. A structured integrator combines these corrections with thrust, gravity and rotation updates to recover physical trajectories. This second stage uses supervised state targets. The rotation exponential keeps attitude updates on the space of valid rotations; that geometric property does not guarantee a correct forecast under arbitrary disturbances.

Equations 14–17
{Δ​𝐯˙t+k,𝐊t+k}=ψ⁡(𝐬~t+k).\{\Delta\mathbf{\dot{v}}_{t+k},\mathbf{K}_{t+k}\}=\psi(\tilde{\mathbf{s}}_{t+k}). (14)
𝐯˙t\displaystyle\mathbf{\dot{v}}_{t} =∑i=03fi,tm​𝐑t​𝐞3−𝐠+Δ​𝐯˙t,\displaystyle=\frac{\sum_{i=0}^{3}f_{i,t}}{m}\mathbf{R}_{t}\mathbf{e}_{3}-\mathbf{g}+\Delta\mathbf{\dot{v}}_{t}, (15)
Δ​𝝉t\displaystyle\Delta\bm{\tau}_{t} =𝐊t​𝐚t.\displaystyle=\mathbf{K}_{t}\mathbf{a}_{t}.
𝐩t+1\displaystyle\mathbf{p}_{t+1} =𝐩t+𝐯t​Δ​t,\displaystyle=\mathbf{p}_{t}+\mathbf{v}_{t}\Delta t, (16)
𝐯t+1\displaystyle\mathbf{v}_{t+1} =𝐯t+𝐯˙t​Δ​t,\displaystyle=\mathbf{v}_{t}+\mathbf{\dot{v}}_{t}\Delta t,
𝐑t+1\displaystyle\mathbf{R}_{t+1} =𝐑t​exp⁡([𝝎t]×​Δ​t),\displaystyle=\mathbf{R}_{t}\exp\!\left([\bm{\omega}_{t}]_{\times}\Delta t\right),
𝝎t+1\displaystyle\bm{\omega}_{t+1} =𝝎t+Δ​𝝉t​Δ​t.\displaystyle=\bm{\omega}_{t}+\Delta\bm{\tau}_{t}\Delta t.
ℒprober=1T​∑k=1T‖𝐱~t+k−𝐱t+k‖22.\mathcal{L}_{\text{prober}}=\frac{1}{T}\sum_{k=1}^{T}\|\tilde{\mathbf{x}}_{t+k}-\mathbf{x}_{t+k}\|_{2}^{2}. (17)

IV-D Sampling-Based Control with Learned Dynamics

MPPI proposes many noisy variations of a nominal action sequence, clips them to allowed limits and rolls out each candidate through the learned dynamics and prober. Candidates with lower tracking cost receive larger softmax weights. Execute only the first updated action, obtain a new state estimate and plan again. Replanning helps correct errors, but action clamping alone does not enforce obstacle avoidance or guarantee safe states.

Equations 18–24
𝐚k(s)=𝐚knom+ϵk(s),ϵk(s)∼𝒩⁡(0,𝚺),\mathbf{a}^{(s)}_{k}=\mathbf{a}^{\mathrm{nom}}_{k}+\bm{\epsilon}^{(s)}_{k},\qquad\bm{\epsilon}^{(s)}_{k}\sim\mathcal{N}(0,\bm{\Sigma}), (18)
𝒜={𝐚∈ℝ4|𝐚min≤𝐚≤𝐚max},\mathcal{A}=\left\{\mathbf{a}\in\mathbb{R}^{4}\;\middle|\;\mathbf{a}_{\min}\leq\mathbf{a}\leq\mathbf{a}_{\max}\right\}, (19)
𝐚k(s)←Π𝒜​(𝐚k(s)),\mathbf{a}^{(s)}_{k}\leftarrow\Pi_{\mathcal{A}}\!\left(\mathbf{a}^{(s)}_{k}\right), (20)
𝒥(s)=1T​∑k=1Tℓ⁡(𝐱~t+k(s),𝐱t+kref,𝐚t+k−1(s)),\mathcal{J}^{(s)}=\frac{1}{T}\sum_{k=1}^{T}\ell\!\left(\tilde{\mathbf{x}}^{(s)}_{t+k},\mathbf{x}^{\mathrm{ref}}_{t+k},\mathbf{a}^{(s)}_{t+k-1}\right), (21)
ℓ(.)\displaystyle\ell(.) =‖𝐱~(s)−𝐱ref‖𝐐x2+‖𝐚(s)−𝐚ref‖𝐐a2,\displaystyle=\|\tilde{\mathbf{x}}^{(s)}-\mathbf{x}^{\mathrm{ref}}\|_{\mathbf{Q}_{\text{x}}}^{2}+\|\mathbf{a}^{(s)}-\mathbf{a}^{\mathrm{ref}}\|_{\mathbf{Q}_{\text{a}}}^{2}, (22)
w(s)=exp⁡(−1λ​(𝒥(s)−𝒥min))∑r=1Sexp⁡(−1λ​(𝒥(r)−𝒥min)),w^{(s)}=\frac{\exp\!\left(-\frac{1}{\lambda}(\mathcal{J}^{(s)}-\mathcal{J}_{\min})\right)}{\sum_{r=1}^{S}\exp\!\left(-\frac{1}{\lambda}(\mathcal{J}^{(r)}-\mathcal{J}_{\min})\right)}, (23)
𝐚knom←𝐚knom+∑s=1Sw(s)​δ​𝐚k(s),δ​𝐚k(s)=𝐚k(s)−𝐚knom.\mathbf{a}^{\mathrm{nom}}_{k}\leftarrow\mathbf{a}^{\mathrm{nom}}_{k}+\sum_{s=1}^{S}w^{(s)}\,\delta\mathbf{a}^{(s)}_{k},\qquad\delta\mathbf{a}^{(s)}_{k}=\mathbf{a}^{(s)}_{k}-\mathbf{a}^{\mathrm{nom}}_{k}. (24)

IV-E Automated Data Synthesis

Generate smooth random reference paths, track them with established controllers inside simulation, and record the resulting feasible state–action sequences. Randomizing mass, inertia, drag and actuator properties broadens the dynamics the learner encounters. This is more structured than independently sampling arbitrary states. Its transfer benefit still depends on whether the simulated family and sensing assumptions adequately represent the real aircraft.

Equations 25–27
pj​(t)∼𝒢​𝒫​(0,kj​(t,t′)),j∈{x,y,z}.p_{j}(t)\sim\mathcal{GP}(0,k_{j}(t,t^{\prime})),\qquad j\in\{x,y,z\}. (25)
𝜼={m,𝐃,𝐉,α,kf,kτ,l},\bm{\eta}=\left\{m,\;\mathbf{D},\;\mathbf{J},\;\alpha,\;k_{f},\;k_{\tau},\;l\right\}, (26)
𝜼r∼𝒰⁡(𝜼¯−Δ​𝜼,𝜼¯+Δ​𝜼),\bm{\eta}_{r}\sim\mathcal{U}\left(\bar{\bm{\eta}}-\Delta\bm{\eta},\bar{\bm{\eta}}+\Delta\bm{\eta}\right), (27)
Refer to caption
Figure 4: Closed-loop real-world evaluation settings under trajectory tracking. We test the proposed framework under three scenarios: (a) nominal trajectory tracking, (b) payload variation, and (c) propeller switching.

V Implementation Details

Implementation details matter here because three different clocks appear: the data/model sampling step, the learned rollout length and the frequency of replanning on the aircraft. They are not interchangeable. The hardware, state estimator and low-level flight controller are also part of the demonstrated system, rather than capabilities supplied by the neural model alone.

V-A Platform Setup

Experiments use a 1.3 kg quadrotor with onboard Orin NX computation, a PX4 flight controller and GPS-based state estimation. A ground laptop sends high-level commands, while the aircraft runs the control computation. Trials cover nominal tracking, a payload and changed propellers outdoors. These conditions define the demonstrated sensing and platform scope.

V-B Dataset Generation

The simulation pipeline samples 500 domains and tracks 20,000 ten-second reference trajectories. Data are resampled at 0.05 seconds, or 20 Hz, and split 80/10/10 for training, validation and testing. A held-out trajectory split evaluates new rollouts from this generated distribution; it should not automatically be interpreted as a split that holds out entire simulator parameter domains.

V-C Network Architecture and Training

Small temporal-convolution encoders feed a single-layer GRU predictor. Training uses twenty rollout steps, a one-second horizon at the 20 Hz data rate, with roughly half a second of history. The objective combines multi-step consistency and SIGReg. The compact architecture, about nine thousand parameters in the reported runtime discussion, is central to evaluating hundreds of control candidates onboard.

Table I: Domain randomization parameters used for simulation data collection.
Parameter Randomization
Mass mm (kg) ±50%\pm 50\% nominal
Inertia 𝐉\mathbf{J} (kg m2) ±30%\pm 30\% nominal
Motor time constant α\alpha (s) [0.01, 0.1][0.01,\,0.1]
Drag coefficients 𝐃\mathbf{D} [0.1, 0.5][0.1,\,0.5]
Thrust coefficient kfk_{f} ±50%\pm 50\% nominal
Torque coefficient kmk_{m} ±50%\pm 50\% nominal
No. of domains 500
Total trajectories 20000
Trajectory duration (s) 10
Table II: MPPI Parameters
Parameter Value
Prediction timestep Δ​t\Delta t 0.05 s
Horizon TT 15
Number of samples SS 512
Temperature λ\lambda 10−410^{-4}
Action noise Σ\Sigma diag⁡(0.60,0.15,0.15,0.05)\mathrm{diag}(0.60,0.15,0.15,0.05)
State cost 𝐐\mathbf{Q} diag⁡(400,40,20,20)\mathrm{diag}(400,40,20,20)
Control cost 𝐑\mathbf{R} diag⁡(0.01,0.05,0.05,0.10)\mathrm{diag}(0.01,0.05,0.05,0.10)

V-D MPPI Controller Parameters

The controller is implemented in C++ and uses TensorRT for neural inference on the Jetson. The timing experiment asks which sample counts and rollout lengths fit a ten-millisecond budget. Table II lists a fifteen-step control horizon, while the runtime discussion identifies twenty steps with 512 candidates near the boundary. Keep those reported settings distinct, along with the 0.05-second prediction step and approximately 100 Hz replanning target.

VI Experiments

Open-loop tests ask whether forecasts remain accurate without measurement corrections. Closed-loop flights test whether those forecasts help a controller track paths when it repeatedly observes and replans. The study also ablates physical probing and dataset quality. Strong tracking can coexist with imperfect open-loop models, so neither metric should be used as a substitute for the other.

Original figure
Figure 5: Inference speed on NVIDIA Orin NX. Inference time increases with unroll length and number of MPPI rollouts SS; the selected horizon is chosen to remain below the 10 ms real-time control budget.

VI-A Latent Dynamics Modeling

The compounding ratio compares recursive rollout error with teacher-forced error, where true past states are available; a small denominator can make early ratios unstable. At longer horizons, the latent model’s ratio and incremental error growth are lower than the direct predictor’s. Latent paths are also smoother, although the reconstruction-based latent baseline has the highest straightening score. Added Gaussian observation noise favors the full method in the tested range. These diagnostics support reduced error accumulation without establishing a universal stability bound.

Equations 28–30
ek𝐱=1D𝐬​‖𝐱~k−𝐱k‖22,e_{k}^{\mathbf{x}}=\sqrt{\frac{1}{D_{\mathbf{s}}}\left\|\tilde{\mathbf{x}}_{k}-\mathbf{x}_{k}\right\|_{2}^{2}}, (28)
CRk=ek,rollout𝐱ek,TF𝐱.\mathrm{CR}_{k}=\frac{e^{\mathbf{x}}_{k,\mathrm{rollout}}}{e^{\mathbf{x}}_{k,\mathrm{TF}}}. (29)
ERk=𝔼⁡[ek𝐱−ek−1𝐱].\mathrm{ER}_{k}=\mathbb{E}\left[e_{k}^{\mathbf{x}}-e_{k-1}^{\mathbf{x}}\right]. (30)
Original figure
Figure 6: Recursive rollout error analysis. The compounding ratio (left) compares open-loop prediction to teacher-forced prediction; values above 11 indicate error accumulation caused by recursion. The error growth rate (right) measures the additional pose error introduced at each rollout step. Our method stays closer to teacher-forced behavior and has lower error growth than the predictive baseline, showing that latent-space dynamics modeling mitigates long-horizon compounding error.
Original figure
Figure 7: Temporal straightening analysis of latent rollouts. (a) Example Cartesian trajectory, with shading indicating progression from earlier to later states. (b) PCA projection of the corresponding latent trajectories, showing how each model evolves in representation space over time. (c) Temporal straightening score; higher values indicate smoother and more directionally consistent latent evolution. Temporally straighter latent trajectories suggest that complex system dynamics are represented in a simpler geometry, making recursive prediction easier and reducing directional drift during rollout.
Takeaway: Latent-space dynamics modeling reduces compounding error and has a much smaller gap between teacher-forced and recursive rollout predictions.
Equation
Sstraight(i)=1T−2​∑t=1T−2⟨s~˙t(i),s~˙t+1(i)⟩‖s~˙t(i)‖​‖s~˙t+1(i)‖.S_{\text{straight}}^{(i)}=\frac{1}{T-2}\sum_{t=1}^{T-2}\frac{\left\langle\dot{\tilde{s}}^{(i)}_{t},\,\dot{\tilde{s}}^{(i)}_{t+1}\right\rangle}{\|\dot{\tilde{s}}^{(i)}_{t}\|\,\|\dot{\tilde{s}}^{(i)}_{t+1}\|}.
Takeaway: Both latent models learn temporally smoother latent trajectories than direct predictive modeling, suggesting that temporal smoothness is an important emergent property.
Original figure
Figure 8: Open-loop rollout fidelity across dynamics models. Position, velocity, and attitude RMSE are shown versus unroll length, where lower error indicates more accurate long-horizon prediction. Direct predictive models accumulate error rapidly, especially in attitude, while latent dynamics models grow more slowly. The proposed physics-inspired prober substantially improves metric-state recovery, showing that structured physical decoding is critical for accurate state prediction.
Original figure
Figure 9: Robustness to observation corruption. Pose RMSE under increasing i.i.d. Gaussian observation noise, showing that the JEPA model consistently outperforms the predictive baseline and remains more robust as corruption increases.
Takeaway: Our JEPA-style approach remains more accurate under input corruption, indicating stronger robustness to corrupted inputs.

VI-B Physics Interpretability

The strongest result is the physical prober ablation: replacing a generic decoder with the structured readout cuts the JEPA model’s mean position error from 5.56 m to 1.43 m in the reported open-loop test. A reconstruction-trained latent model also improves sharply with that prober. Therefore much of the gain comes from state-recovery structure; it should not all be attributed to the JEPA objective. Attitude error measures the angle of the relative rotation, not a raw difference between matrix entries.

Table III: Open loop prediction analysis of different neural dynamics frameworks.
Approach Pos. RMSE [m] ↓\downarrow Att. Err. [∘] ↓\downarrow
Mean Var. Mean Var.
Predictive 8.808.80 2.32.3 53.453.4 14.914.9
Predictive + Physics Reg. [8] 7.127.12 2.12.1 49.149.1 13.213.2
Reconstruction + Prober 6.826.82 1.61.6 45.245.2 9.79.7
Reconstruction + PI Prober 1.531.53 0.130.13 5.285.28 0.700.70
Ours + Prober 5.565.56 1.311.31 40.2040.20 9.309.30
Ours + PI Prober 1.43\mathbf{1.43} 0.10\mathbf{0.10} 4.71\mathbf{4.71} 0.50\mathbf{0.50}
Equations 31–34
ek𝐩=‖𝐩~k−𝐩k‖2.e_{k}^{\mathbf{p}}=\left\|\tilde{\mathbf{p}}_{k}-\mathbf{p}_{k}\right\|_{2}. (31)
𝐑err,k=𝐑k⊤​𝐑~k.\mathbf{R}_{\mathrm{err},k}=\mathbf{R}_{k}^{\top}\tilde{\mathbf{R}}_{k}. (32)
ϕk=Log𝒮​𝒪​(3)⁡(𝐑k⊤​𝐑~k)∨∈ℝ3,\bm{\phi}_{k}=\operatorname{Log}_{\mathcal{SO}(3)}\left(\mathbf{R}_{k}^{\top}\tilde{\mathbf{R}}_{k}\right)^{\vee}\in\mathbb{R}^{3}, (33)
ek𝐑=180π​‖ϕk‖2.e_{k}^{\mathbf{R}}=\frac{180}{\pi}\left\|\bm{\phi}_{k}\right\|_{2}. (34)
Table IV: Quantitative results of real-world trajectory tracking across reference trajectories. Each trajectory is executed 5 times. The mean and variance are reported.
Trajectory ‖𝐯‖max\|\mathbf{v}\|_{\max} ‖𝐯˙‖max\|\dot{\mathbf{v}}\|_{\max} Position RMSE [m] Attitude Error [∘]
[m s-1] [m s-2] Ours MPPI (Pred.+Phy.) MPPI (Pred.) Ours MPPI (Pred.+Phy.) MPPI (Pred.)
Mean Var. Mean Var. Mean Var. Mean Var. Mean Var. Mean Var.
Circle 2.452.45 1.401.40 0.24\mathbf{0.24} 0.010\mathbf{0.010} 0.360.36 0.0220.022 0.390.39 0.0240.024 7.87\mathbf{7.87} 0.82\mathbf{0.82} 10.9910.99 3.663.66 11.9511.95 3.813.81
Oval 4.504.50 3.783.78 0.33\mathbf{0.33} 0.018\mathbf{0.018} 0.440.44 0.0830.083 0.480.48 0.0960.096 9.11\mathbf{9.11} 1.06\mathbf{1.06} 15.2015.20 5.465.46 16.5316.53 5.685.68
Figure 8 5.205.20 5.445.44 0.35\mathbf{0.35} 0.022\mathbf{0.022} 0.470.47 0.0410.041 0.510.51 0.0440.044 9.25\mathbf{9.25} 1.18\mathbf{1.18} 17.7517.75 5.885.88 20.2020.20 6.106.10
Fish 5.705.70 7.687.68 0.40\mathbf{0.40} 0.031\mathbf{0.031} 0.540.54 0.0560.056 0.590.59 0.0610.061 10.78\mathbf{10.78} 1.42\mathbf{1.42} 20.9520.95 6.856.85 22.7822.78 7.197.19
Lemniscate 7.207.20 12.512.5 0.45\mathbf{0.45} 0.047\mathbf{0.047} 0.560.56 0.0760.076 0.610.61 0.0830.083 19.43\mathbf{19.43} 2.65\mathbf{2.65} 26.8326.83 7.517.51 29.1629.16 7.907.90
Original figure
Figure 10: Real-world zero-shot trajectory tracking using the proposed controller. The plots show executed flight trajectories overlaid on the desired references. Color indicates tracking error in centimeters, with darker regions corresponding to lower error. The controller closely follows diverse real-world trajectories using a dynamics model trained only in simulation.
Takeaway: Embedding kinematic structure enables high-fidelity metric-state recovery, showing that the latent rollouts retain sufficient information to reconstruct the full dynamics.

VI-C Zero-Shot Sim-to-Real Transfer

Across five reference-path families, each repeated five times, the simulation-trained full method has lower mean position and attitude errors than the two compared predictive-model controllers. For example, circle position error is 0.24 m versus 0.39 m for the unregularized predictive baseline. Here zero-shot means no real-flight model fine-tuning. It does not mean no engineered state estimator, tracking objective, simulator design or controller tuning.

Table V: Quantitative results of closed-loop tracking under propeller switching and payload scenarios. All trajectories were executed at an average velocity of 2 ms-1 and executed 5 times. The table reports the mean and variance.
Scenario Trajectory Position RMSE [m] Attitude Error [∘]
Ours MPPI (Pred. + Phy.) MPPI (Pred.) Ours MPPI (Pred. + Phy.) MPPI (Pred.)
Mean Var. Mean Var. Mean Var. Mean Var. Mean Var. Mean Var.
Propeller Switching Circle 0.33\mathbf{0.33} 0.02\mathbf{0.02} 0.430.43 0.030.03 0.450.45 0.030.03 9.54\mathbf{9.54} 1.04\mathbf{1.04} 12.4012.40 1.351.35 12.8812.88 1.401.40
Figure 8 0.35\mathbf{0.35} 0.01\mathbf{0.01} 0.460.46 0.010.01 0.470.47 0.010.01 10.21\mathbf{10.21} 0.98\mathbf{0.98} 13.2713.27 1.271.27 13.7813.78 1.321.32
Fish 0.39\mathbf{0.39} 0.02\mathbf{0.02} 0.510.51 0.030.03 0.530.53 0.030.03 10.89\mathbf{10.89} 0.88\mathbf{0.88} 14.1614.16 1.141.14 14.7014.70 1.191.19
Payload Transportation Circle 0.46\mathbf{0.46} 0.08\mathbf{0.08} 0.600.60 0.100.10 0.620.62 0.110.11 10.11\mathbf{10.11} 1.43\mathbf{1.43} 13.1413.14 1.861.86 13.6513.65 1.931.93
Figure 8 0.49\mathbf{0.49} 0.07\mathbf{0.07} 0.640.64 0.090.09 0.660.66 0.090.09 9.44\mathbf{9.44} 1.37\mathbf{1.37} 12.2712.27 1.781.78 12.7412.74 1.851.85
Fish 0.53\mathbf{0.53} 0.08\mathbf{0.08} 0.690.69 0.100.10 0.720.72 0.110.11 11.87\mathbf{11.87} 1.92\mathbf{1.92} 15.4315.43 2.502.50 16.0216.02 2.592.59
Refer to caption
Figure 11: Real-world tracking under platform changes without retraining. (a) Payload transportation modifies the mass and inertial properties of the quadrotor. (b) Propeller switching changes the actuation characteristics. The executed trajectories remain close to the reference paths, with color indicating tracking error in centimeters. These rollouts show that the proposed framework remains effective under non-nominal deployment conditions.

VI-D Robustness

Changed propellers and a 300 g payload test two specific deployment variations without retraining. The full method tracks better than the compared controllers in the reported repeats. Those scenarios were absent as explicit data-collection configurations, but simulation already randomized mass and actuation. The results support robustness to these tests, rather than proof of transfer to every aircraft or a completely disjoint physical-parameter distribution.

Takeaway: A well covered diverse domain-randomized simulation data is sufficient for zero-shot sim2real deployment.

VI-E Data Quality

TDQ scores how broadly data occupy clusters of state–action features, local transitions and simulator parameters. Each component combines occupancy entropy with sufficient cluster coverage; a harmonic mean prevents one weak component being hidden by the others. Larger datasets have higher TDQ and lower test error here. Because size and coverage change together, the plot is a correlation, not an isolated causal test or a certified out-of-distribution coverage measure.

Equations 35–36
𝐲i=[𝐯i⊤𝐫x,i⊤𝐫y,i⊤𝐫z,i⊤𝝎i⊤𝐚i⊤]⊤,\mathbf{y}_{i}=\begin{bmatrix}\mathbf{v}_{i}^{\top}&\mathbf{r}_{x,i}^{\top}&\mathbf{r}_{y,i}^{\top}&\mathbf{r}_{z,i}^{\top}&\bm{\omega}_{i}^{\top}&\mathbf{a}_{i}^{\top}\end{bmatrix}^{\top}, (35)
𝐠i=[𝐲i⊤Δ​𝐲i⊤]⊤,Δ​𝐲i=𝐲i+1−𝐲i.\mathbf{g}_{i}=\begin{bmatrix}\mathbf{y}_{i}^{\top}&\Delta\mathbf{y}_{i}^{\top}\end{bmatrix}^{\top},\qquad\Delta\mathbf{y}_{i}=\mathbf{y}_{i+1}-\mathbf{y}_{i}. (36)
Original figure
Figure 12: As dataset size increases, TDQ rises from 0.010.01 to 0.940.94, while state RMSE e𝐱e^{\mathbf{x}} decreases from 5.45.4 to 1.41.4, showing that higher trajectory distribution quality correlates with better learned dynamics prediction.
Equations 37–38
S=(−1log⁡K∑k=1Kqklog(qk))(1K∑k=1K𝕀[nk≥nmin]),S=\left(-\frac{1}{\log K}\sum_{k=1}^{K}q_{k}\log(q_{k})\right)\left(\frac{1}{K}\sum_{k=1}^{K}\mathbb{I}[n_{k}\geq n_{\min}]\right), (37)
TDQ⁡(𝒟)=31Scov+1Sdyn+1Sη.\mathrm{TDQ}(\mathcal{D})=\frac{3}{\frac{1}{S_{\mathrm{cov}}}+\frac{1}{S_{\mathrm{dyn}}}+\frac{1}{S_{\eta}}}. (38)
Takeaway: Higher TDQ leads to lower state prediction error, indicating that broader state–action coverage, richer transitions, and better parameter diversity improve learned dynamics prediction.

VII Discussion

The results support a useful combined design: simulation diversity, recurrent latent dynamics and a structured supervised state readout. The prober ablation shows that physical integration is a major contributor. Remaining questions include sensitivity to the simulated parameter family, estimator failures and disturbances beyond the tests. Neither low prediction error nor Gaussian latent regularization provides a formal control-stability or flight-safety guarantee.

VIII Conclusion and Future Works

The demonstrated contribution is state-based quadrotor dynamics and tracking with simulation-only model training. Learning from onboard RGB or depth images, navigating clutter, representing uncertainty and incorporating explicit safety constraints are proposed future directions. They should remain separate from the present outdoor path-tracking results.

Concept demonstration

Open image ↗