Literature
Display
← All papers

OMG: Omni-Modal Motion Generation for Generalist Humanoid Control

Siqiao Huang · Kun-Ying Lee · Dongming Qiao · Guanqi He · Zhenyu Wang · Yitang Li · Shaoting Zhu · Hang Zhao

Preprint posted · arXiv (preprint) ↗

Robotics16 min text · CondensedOriginal paper ↗
Contents

Overview

OMG generates future Unitree G1 motion from language, audio, human movement and combinations of these signals, then delegates execution to an existing tracker. The overview also introduces scaling and adaptation experiments.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 1: Overview. OMG decomposes humanoid whole-body control into a scalable motion generation brain and a reactive motion tracking cerebellum. Built on OMG-Data, a curation of 1000+ hours omni-modal humanoid motion data, OMG-DiT maps language, audio, human reference, and their compositions into robot-executable future motions, which are deployed on a Unitree G1 in real time, paired with a pretrained motion tracker [4]. This unified generator-tracker hierarchy enables general-purpose omni-modal control and sample-efficient adaptation to new tasks and modalities.

At a glance

OMG combines a curated motion corpus with a diffusion transformer that predicts robot motion directly. Modality-specific encoders steer a shared motion prior, and a separate tracker executes generated trajectories. Results cover motion quality, execution fidelity, transfer to held-out motions and sparse-keypoint or visual interfaces, plus qualitative physical demonstrations. The evidence supports a reusable motion generator for this robot and data regime; it does not establish an unrestricted general-purpose humanoid controller.

1 Introduction

Motion tracking scales beyond individually engineered skills but still needs reference trajectories. OMG supplies those references from heterogeneous human commands. Its contributions are a G1-aligned multimodal corpus, a shared conditional diffusion backbone, and demonstrations of scaling, fine-tuning and compositional guidance. The generator–tracker hierarchy reuses HoloMotion for low-level execution; the new contribution lies mainly in the upstream motion prior and its conditioning interfaces.

2 Related Works

Related work separates the physical execution layer from the high-level generation layer that OMG aims to unify across modalities.

Humanoid Whole-Body Control and Motion Tracking.

Broad motion trackers follow many references but generally assume a motion or low-level command is already available. OMG builds on such a tracker rather than retraining the entire control hierarchy jointly.

Interactive and Multi-Modal Motion Generation.

Graphics models generate expressive human motion from text, audio or poses, often offline; previous robot generator–tracker systems usually support narrower inputs. OMG targets interactive multimodal generation directly in G1 space, reducing the inference-time need to retarget human output.

3 OMG-Data : Scaling Multi-Modal Data for Humanoid Motion Generation

OMG-Data merges heterogeneous sources into G1 trajectories at 30 Hz. The reported 1,174.66-hour total includes the 38-hour CMU/Weizmann holdout row; the appendix states those subsets are excluded from pretraining. Summing the other processed-hour rows gives 1,136.66 hours. These reported dataset totals should therefore not be confused with the effective pretraining subset.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 2: Dataset Statistics of OMG-Data. We curate a large-scale omni-modal humanoid motion corpus by aggregating heterogeneous datasets and unifying them into the Unitree G1 motion space. Left: processed data statistics across conditioning modalities and source datasets. Right: representative conditioning modalities, including language, audio, and human reference motions.

Data Curation and Preprocessing.

Corrupted files, broken links, severe frame gaps and invalid joints are removed. Audio waveforms and features are synchronized to motion frames so temporally aligned conditioning can be learned across dance and gesture datasets.

Motion Retargeting and Annotation.

GMR retargets heterogeneous skeletons into G1 space. Seed-1.8 labels rendered views or keyframes where language is absent or insufficiently granular. Training clips follow language boundaries, audio phrases or fixed sliding windows.

Simulation-In-The-Loop Filtering.

Motions are executed through the tracker in MuJoCo and rejected for persistent low root height or excessive tilt. The appendix also rejects non-finite states and, for stricter subsets, joint-limit violations. This screens feasibility under the chosen tracker and simulator; it is not a guarantee of collision freedom or safe physical deployment.

4 OMG-DiT : Unified Diffusion Backbone for Motion Generation

A shared denoising model learns the motion distribution while separate encoders translate commands into conditioning signals. This separates reuse of the motion prior from extension of the input interface.

Original figure
Figure 3: OMG-DiT learns a shared diffusion backbone while enabling conditioning with modality-specific encoders. History motion and language are injected as global context tokens via cross-attention, whereas frame-aligned signals (i.e., audio and human reference motions) are injected through FiLM [39] adapters. New modalities are attached non-invasively through zero-initialized adapters, and multiple conditions can be composed at inference via classifier-free guidance.

4.1 Problem Formulation

The high-level policy predicts future reference states from conditions and recent motion history. The low-level HoloMotion policy maps those references and observed state into actions. OMG trains the generator and reuses the tracker, so generation and execution errors are evaluated separately.

Equation
π(𝐚t:t+H,𝐨t+1:t+H+1∣𝐨≤t,𝒞)≈πϕ(𝐨t+1:t+H+1∣𝐨t−L:t,𝒞)⏟Motion Generation/Planning⋅πψ(𝐚t:t+H∣𝐨t−L:t,𝐨t+1:t+H+1)⏟Motion Tracking/IDM.{\pi(\mathbf{a}_{t:t+H},\mathbf{o}_{t+1:t+H+1}\mid\mathbf{o}_{\leq t},\mathcal{C})\approx\underbrace{\pi_{\phi}(\mathbf{o}_{t+1:t+H+1}\mid\mathbf{o}_{t-L:t},\mathcal{C})}_{\text{Motion Generation/Planning}}\cdot\underbrace{\pi_{\psi}(\mathbf{a}_{t:t+H}\mid\mathbf{o}_{t-L:t},\mathbf{o}_{t+1:t+H+1})}_{\text{Motion Tracking/IDM}}.}

4.2 Diffusion Transformer Backbone

The generator denoises motion features directly, with no learned motion tokenizer or latent autoencoder.

Motion Representation.

Each frame contains 125 features: local root position and 6D orientation, 29 joint angles, and 29 non-root body-link positions. Anchoring translation and heading at the last observed state removes arbitrary world placement while retaining local whole-body motion.

Model Architecture.

A diffusion transformer predicts clean motion. Recent motion enters through cross-attention; timestep features are concatenated with noisy input features. Global and frame-aligned command streams use different conditioning paths.

Training Objective.

Gaussian noise corrupts the future segment; squared-error training reconstructs its clean features. Independently dropping available modalities teaches conditional and null branches for classifier-free guidance and allows one model to use partially labelled datasets.

Equations 1–2
𝐱1:Hτ=α¯τ𝐱1:H+1−α¯τϵ,ϵ∼𝒩(𝟎,𝐈).\mathbf{x}^{\tau}_{1:H}=\sqrt{\bar{\alpha}_{\tau}}\,\mathbf{x}_{1:H}+\sqrt{1-\bar{\alpha}_{\tau}}\,\boldsymbol{\epsilon},\qquad\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}). (1)
ℒ=𝔼𝐱,𝒞s,τ,ϵ[‖𝐱^t+1:t+H−𝐱t+1:t+H‖22],where 𝐱^t+1:t+H=𝐱θ(𝐱t+1:t+Hτ,τ,𝐱t−L:t,𝒞s).\mathcal{L}=\mathbb{E}_{\mathbf{x},\mathcal{C}_{s},\tau,\boldsymbol{\epsilon}}\left[\left\|\hat{\mathbf{x}}_{t+1:t+H}-\mathbf{x}_{t+1:t+H}\right\|_{2}^{2}\right],\text{where }\hat{\mathbf{x}}_{t+1:t+H}=\mathbf{x}_{\theta}\!\left(\mathbf{x}^{\tau}_{t+1:t+H},\tau,\mathbf{x}_{t-L:t},\mathcal{C}_{s}\right). (2)

4.3 Omni-Modal Condition Encoding

Language supplies global context, whereas audio and human movement provide frame-aligned conditions.

Pretraining Condition Encoders.

Frozen T5 language features join motion-history tokens in cross-attention. Audio and human-reference features pass through MLPs and modulate corresponding future frames with FiLM at each transformer layer.

Task-specific Finetuning Encoders.

New interfaces use lightweight encoders and initially zero-output adapters, such as 18D Pico keypoints for teleoperation. The appendix clarifies that the reported adaptation experiments fine-tune all backbone parameters, rather than demonstrating encoder-only training.

5 Experiments

Experiments test pretrained generation, transfer versus matched training from scratch, and qualitative command composition and model-size scaling. A separate appendix evaluates a small visual locomotion task.

Refer to caption
Figure 4: Real-World Omni-Modal Control. OMG generates diverse Unitree G1 motions across various conditioning modalities in real time, executable in the real world.

5.1 Experiment Setup

Generated G1 references are assessed both before and after tracker execution using held-out validation motions.

Evaluation Protocol.

Pretraining evaluation covers text, audio and human references. Adaptation covers unseen motion distributions and Pico keypoints, with the same architecture and data budgets for pretrained and scratch models. HoloMotion executes generated trajectories unless stated otherwise.

Metrics.

Generation metrics measure text alignment, distributional similarity, beat alignment or reference pose error. Execution metrics measure tracker deviation, sliding, jerk, falls and joint-limit violations. Stronger distributional scores need not imply better rhythm alignment or safer execution.

5.2 Pretraining Experiments

The three benchmark tables retain every generation, physical-quality and tracker-fidelity metric, including tradeoffs hidden by the broad claims in the prose.

Table 1: Text-conditioned motion benchmark. For better display, R@K, Fall, and J-Limit are reported in percentage. FID and C-Slide are scaled by 10−210^{-2} and 10−110^{-1}, respectively. Lower is better for metrics marked with ↓\downarrow, and higher is better for metrics marked with ↑\uparrow.
Methods Motion Generation Quality Tracking Fidelity and Execution Failures
FID ↓\downarrow Matching ↓\downarrow R@1 ↑\uparrow R@2 ↑\uparrow R@3 ↑\uparrow Diversity ↑\uparrow C-Slide ↓\downarrow Jerk ↓\downarrow MPJPE ↓\downarrow g-MPJPE ↓\downarrow E_vel ↓\downarrow E_acc ↓\downarrow Fall ↓\downarrow J-Limit ↓\downarrow
GENMO 43.78 1.33 12.99 23.83 30.86 1.28 2.66 173.54 50.35 150.71 6.91 2.51 1.46 29.10
HYMotion 24.68 1.30 31.84 48.73 57.71 1.29 12.20 112.56 69.05 287.71 11.94 3.24 4.98 55.57
Kimodo-SMPLX-RP 38.44 1.31 26.27 39.36 46.48 1.25 4.52 189.39 43.51 198.96 7.80 2.64 2.64 23.63
Kimodo-G1-RP 46.36 1.32 25.59 36.13 44.63 1.21 3.92 81.44 37.25 125.04 5.72 1.68 1.46 19.14
Kimodo-G1-SEED 50.22 1.33 22.46 33.69 41.11 1.20 4.30 99.47 37.26 161.63 6.38 1.84 1.56 20.70
OMG-L 9.55 1.14 58.50 74.90 81.45 1.39 3.68 56.03 39.06 98.37 5.11 1.47 0.29 14.94
OMG-XL 6.03 1.11 65.43 80.37 86.91 1.39 3.98 52.31 42.68 111.41 5.48 1.53 0.78 19.34
Table 2: Audio-conditioned motion benchmark. For better display, Fall and J-Limit are reported in percentage. Lower is better for metrics with ↓\downarrow, and higher is better for metrics with ↑\uparrow.
Methods Audio Alignment and Target Motion Quality Target Physical Quality and Tracker Fidelity
BeatAlign ↑\uparrow FIDk ↓\downarrow FIDg ↓\downarrow PFC ↓\downarrow C-Slide ↓\downarrow Jerk ↓\downarrow MPJPE ↓\downarrow g-MPJPE ↓\downarrow E_vel ↓\downarrow E_acc ↓\downarrow Fall ↓\downarrow J-Limit ↓\downarrow
GENMO 0.57 506.07 9.42 0.02 0.03 72.56 35.73 111.69 3.87 1.22 0.39 12.50
LODGE 0.58 390.06 11.98 1.41 0.65 471.45 149.19 363.24 18.89 6.96 39.45 66.80
Bailando 0.56 134.90 15.71 0.39 0.25 117.45 38.30 106.39 5.55 1.99 0.00 1.76
OMG-L 0.49 59.20 0.74 0.57 0.41 95.73 41.07 95.10 6.10 1.90 0.00 14.26
OMG-XL 0.51 40.46 0.63 0.64 0.43 106.06 41.47 97.60 6.03 1.94 0.00 15.43
Table 3: Human-reference-conditioned motion benchmark. For better display, Fall and J-Limit are reported in percentage. Lower / higher is better for metrics marked with ↓\downarrow / ↑\uparrow.
Methods Target Motion and Physical Quality Tracker-Executed Target Fidelity and Failures
MPJPE ↓\downarrow g-MPJPE ↓\downarrow EE Error ↓\downarrow E_vel ↓\downarrow E_acc ↓\downarrow C-Slide ↓\downarrow Jerk ↓\downarrow Target MPJPE ↓\downarrow Target g-MPJPE ↓\downarrow Target E_vel ↓\downarrow Target E_acc ↓\downarrow Fall ↓\downarrow J-Limit ↓\downarrow
GMR 81.95 223.69 219.23 11.72 10.41 0.45 404.80 90.54 249.47 12.52 5.22 11.33 68.75
NMR 173.77 390.82 477.08 14.87 7.43 0.77 97.75 102.28 299.27 11.26 3.32 14.84 87.89
PHC 51.44 153.44 152.18 5.06 3.97 0.45 131.38 47.80 115.92 5.86 1.99 3.71 39.45
OmniRetarget 56.90 119.71 131.14 5.55 4.87 0.31 188.92 44.34 111.58 5.94 2.35 0.98 24.22
OMG-L 26.40 64.64 72.39 6.16 3.91 0.35 61.01 42.81 110.66 5.63 1.62 0.20 22.66
OMG-XL 18.84 47.83 52.26 5.03 3.69 0.32 64.04 43.07 112.60 5.55 1.63 0.39 23.05

Text to Motion.

OMG-XL achieves the best displayed text FID and retrieval scores: FID 6.03 in the table’s ×10⁻² scale and R@1 65.43%. OMG-L has lower fall and joint-limit rates than XL (0.29%/14.94% versus 0.78%/19.34%). GENMO has less contact sliding and some Kimodo variants lower tracker MPJPE, so generation-quality gains are not uniform physical-metric wins.

Audio to Motion.

OMG-XL has the lowest audio distribution distances, FID-k 40.46 and FID-g 0.63. However, its BeatAlign 0.51 is below GENMO 0.57, LODGE 0.58 and Bailando 0.56, contrary to the source’s blanket claim of superior audio alignment. OMG has zero reported falls here but higher joint-limit violation than Bailando (15.43% versus 1.76% for XL); the full table shows the tradeoff.

Human Reference to Motion.

With human-reference conditioning, OMG-XL reduces target MPJPE to 18.84 mm and global error to 47.83 mm, versus 51.44/153.44 for PHC and 56.90/119.71 for OmniRetarget. Tracker-executed errors are larger: XL reaches 43.07/112.60 mm, with 0.39% falls and 23.05% joint-limit violations. Better generated references therefore do not remove the execution bottleneck.

5.3 Finetuning Experiments

Fine-tuning comparisons hold the model size, selected data and split fixed, changing pretrained versus random initialization.

Table 4: Text-to-Motion Finetuning.
Finetuning Data % FID ↓\downarrow R@1 ↑\uparrow R@2 ↑\uparrow R@3 ↑\uparrow
From Scratch
1% 0.6615 0.0430 0.0762 0.1025
10% 0.1763 0.1162 0.2080 0.2998
100% 0.1001 0.1689 0.2764 0.3779
OMG Finetuned
1% 0.1378 0.1533 0.2637 0.3428
10% 0.0886 0.1816 0.3096 0.3994
100% 0.0827 0.2021 0.3115 0.4141
Table 5: Pico Keypoint-Based Teleoperation.
Finetuning Data % MPJPE ↓\downarrow g-MPJPE ↓\downarrow E_vel ↓\downarrow E_acc ↓\downarrow
From Scratch
1% 128.24 448.09 24.34 17.13
10% 82.46 279.21 14.90 3.92
100% 39.42 134.85 8.03 2.95
OMG Finetuned
1% 93.61 286.84 16.11 3.74
10% 52.99 164.91 9.91 3.09
100% 25.73 77.01 5.70 2.62

Few-shot Text-to-Motion Finetuning.

At 1% of held-out target slices, text-motion fine-tuning lowers FID from 0.6615 to 0.1378; full-data scratch reaches 0.1001. Thus the small-data model approaches, but does not match, full-data scratch on FID or retrieval. At 100% data, pretrained FID is 0.0827. The results establish useful transfer; “1% equals full data” would overstate the table.

Pico Keypoint-Based Teleoperation.

Pretraining improves all displayed Pico metrics at 1%, 10% and 100% target data. With full data, MPJPE falls from 39.42 to 25.73 mm and global MPJPE from 134.85 to 77.01 mm. This extends the prior to a new sparse-keypoint interface with additional supervised adaptation.

5.4 Analysis

The analysis studies composition and three model sizes rather than establishing a universal scaling law.

Zero-shot Compositionality.

Separate text and audio guidance directions are combined at inference. Physical examples show raising or waving hands while following a music condition. These demonstrations illustrate unseen signal combinations qualitatively; no comprehensive compositional success-rate benchmark is reported.

Original figure
Figure 5: Scaling OMG-DiT.

Scaling Behavior.

Increasing the backbone from 50M to 300M to 500M parameters improves the plotted human-reference MPJPE and several generation metrics. Some tracker metrics worsen from L to XL, so capacity improves the motion prior more consistently than execution fidelity. The study does not vary data scale independently or fit a predictive scaling law.

6 Conclusion

OMG shows that a shared motion prior can support several command types and transfer to new interfaces while using an existing tracker. Its strongest evidence is the benchmark and matched fine-tuning results; open-world generality remains beyond the reported experiments.

7 Limitations

Training mainly covers flat ground. Uneven terrain and in-the-wild conditions remain challenging, and the separately trained generator and tracker are not jointly optimized. Execution feedback or joint adaptation is proposed for future work. Simulator filtering and selected physical demonstrations do not resolve these limits.

Acknowledgments

The authors acknowledge discussions and feedback from Ziwen Zhuang, Zekun Qi, Haoyang Weng and Yue Chen.

Appendix A Extended Related Work

The extended context distinguishes low-level reusable behavior priors from the high-level multimodal interface studied here.

Behavior Foundation Models for Humanoid Robots.

Forward–backward and latent-conditioned behavior models support multiple execution objectives, while generative middleware can rewrite references for recovery. OMG instead concentrates on translating language, audio and human movement into robot references, complementing these low-level approaches.

Appendix B Details on OMG -Data

The data appendix specifies source composition, skeleton conversion, annotation, segmentation and simulator filtering. The complete original annotation prompt is preserved.

B.1 Dataset Composition

All sequences use 30 Hz. Table 6 lists original and processed samples, lengths, hours and available modalities. Its 1,174.66-hour total includes 38 hours of CMU/Weizmann held out from pretraining; MotionGV is the largest processed row at 637.14 hours. The total is a corpus accounting figure, not a claim that all listed rows were used to pretrain.

Table 6: Statistics of the Pretraining Dataset by Conditioning Modality. We curate publicly available motion datasets from graphics and humanoid domains and then apply careful filtering, annotation, and augmentation, yielding 1174.66 hours of data in total. , , and denote text descriptions, paired audio, and human reference motion, respectively.
Dataset Label Original Processed
# Samples Avg. Length Total Hours # Samples Avg. Length Total Hours
AMASS [33] , 17.9K 1.4K 94.8 51.1K 111 52.7
AMASS (holdout CMU & WEIZMANN) [33] , 13.65K 1.4K 44.9 39.0K 105 38
LAFAN1 [10] 40 6.6K 2.5 977 240 2.2
OMOMO [16] , 5.9K 179 10 15K 67 9.4
100style [34] , 0.8K 2.9K 22.12 10.7K 223 22.10
HumanML [8] , 26.8K 219 54.5 25.7K 218 52.04
Kungfu [8] 1.0K 453 4.3 – – –
MotionGV [8] , 56K 140 727 53.8K 128 637.14
fitness [8] , 262 610 1.48 337 244 0.76
MotionLLAMA (subset) [8] , 4.5K 286 12 4.9K 236 10.7
SnapMoGen [9] , 50K 407 188.7 41.0K 222 84.01
FineDance [19] , , 0.2K 4.0K 14.6 6.0K 240 13.26
BEAT2 [27] , 1.7K 3.4K 60 417 2.0K 8.06
AIST++ [20] , , 1.4K 400 5.2 2.2K 240 4.98
IDEA400 [69, 25] , 12.5K 176 20 10K 177 17.24
PerMo [14] , 6.6K 140 8.6 6.5K 139 8.38
BONES-SEED [42] 71K 219 144 79K 182 134.35
ChoreoMaster [3] , 2.7K 46.7 1.2 2.5K 47 1.11
CoMPAS3D [2] , 4.8K 138 6.2 4.66K 129 5.57
AIOZ-GDANCE [15] , 6K 1.1K 60.8 5.8K 1091 59.05
OpenDance [66] , 5.0K 300 14.0 4.9K 300 13.61
Total , , 288.8K 411 1496.9 364.5K 177 1174.66

B.2 Details on Retargeting

SMPL/SMPL-X bodies are geometrically retargeted with GMR. Video-derived motions first use GENMO to recover human meshes. FBX skeletons use standardized joint mapping or GMR depending on their structure.

B.3 Details on Labeling

Seed-1.8 annotates changes in atomic action or motion style as consecutive, non-overlapping segments. The original prompt requires English style/action labels, action descriptions of at most 25 words, timestamps to one decimal place and a structured JSON response. Its full wording and indented example are retained below.

Prompt B.1: Prompt used for VLM-based temporal motion annotation.
You are a humanoid robot motion labeling model (VLM). Your task is to perform structured motion labeling for the input video, rather than generating descriptive text.

[Most Important Rule]
You must segment when there is a change in "motion atomic action". If the specific action changes, even if the overall style remains consistent, you still need to create a new segment.

[Core Requirement]
Please divide the input video into multiple consecutive time segments, and generate a structured action label for each segment.

The output of each segment must include:
1. start time
2. end time
3. style
4. action

[Segmentation Criteria]
You should decide whether to split mainly based on:
- whether the motion style changes
- whether a finer motion style / type can be judged to have changed
- whether the main action changes

If the motion remains continuous and no obvious change occurs, do not over-segment.

[Action Requirements]
1. action must be written in English
2. action should summarize the concrete motion performed in this segment
3. action should describe the motion itself as much as possible, and should not write overly abstract content
4. action should be concise, and no more than 25 words
5. action should be helpful for training a text-to-motion generation model

[Style Requirements]
1. style must be written in English
2. style should summarize the motion style of this segment, such as smooth, intense, rhythmic, explosive, etc.
3. style should be concise

[Output Format]
Please output in JSON format:
{
  "video_summary": "overall summary of the video",
  "segments": [
    {
      "start_time": 0.0,
      "end_time": 1.5,
      "style": "smooth",
      "action": "raise both arms and step slowly to the left"
    },
    {
      "start_time": 1.5,
      "end_time": 3.0,
      "style": "rhythmic",
      "action": "turn torso and swing arms in alternating beats"
    }
  ]
}

[Output Constraints]
1. Output valid JSON only
2. Do not output any explanatory text
3. Time must be in seconds
4. Keep all start_time and end_time values to one decimal place
5. Segments must be consecutive and non-overlapping
6. start_time must be less than end_time
7. video_summary should be a brief summary of the whole video

[Goal]
Your output should help train a motion generation model, so the labels should preserve motion content, motion style, and temporal changes as much as possible.

B.4 Segmentation Details

Text clips follow annotation boundaries; audio clips follow phrases or uniform windows. Long episodes are split with sliding windows. Thus data-fraction experiments later select motion slices rather than independent people or episodes.

B.5 Simulation-in-the-Loop Filtering Details

The deployment tracker produces actions clipped at 10.0, which become desired joint positions for a PD executor over ten simulation substeps per control step. Filtering examines executed trajectories, rejecting non-finite states and sustained fall conditions; stricter subsets also reject joint-limit violations. The appendix uses at least ten fall frames, whereas the main text says over ten. Retained training trajectories are tracker executions, not merely original kinematic references.

Equations 3–4
τ=Kp​(qdes−q)−Kd​q˙,\tau=K_{p}(q_{\mathrm{des}}-q)-K_{d}\dot{q}, (3)
ht​<0.20orθt>​85∘or(ht<0.35∧θt>60∘).h_{t}<0.20\quad\mathrm{or}\quad\theta_{t}>85^{\circ}\quad\mathrm{or}\quad(h_{t}<0.35\land\theta_{t}>60^{\circ}). (4)

Appendix C Details of OMG-DiT

The model appendix specifies representation, denoising, conditioning, guidance, adaptation and deployment details needed to interpret the experiments.

C.1 Motion Representation and Canonicalization

Canonicalization and normalization make trajectories comparable across world positions while retaining the robot’s full pose.

Motion Representation.

The 125D representation decomposes as 3 root-position values, 6 rotation values, 29 joint angles and 87 non-root link coordinates. Generation occurs directly in this feature space without a learned latent code.

Equation 5
𝐱i=[𝐩~i,𝐫~i,𝜽i,𝐣~i],𝐱i∈ℝ125.\mathbf{x}_{i}=[\tilde{\mathbf{p}}_{i},\tilde{\mathbf{r}}_{i},\boldsymbol{\theta}_{i},\tilde{\mathbf{j}}_{i}],\qquad\mathbf{x}_{i}\in\mathbb{R}^{125}. (5)

Root Rotation Parameterization.

The first two rotation-matrix columns form a continuous 6D representation; orthonormalization recovers a valid orientation. This avoids quaternion sign ambiguity and output normalization constraints and was empirically more stable than the other parameterizations considered.

Canonicalization.

A ten-frame history predicts sixty future frames at 30 Hz. The last history root defines the anchor; subtracting its position and undoing its yaw canonicalizes root and link trajectories. Outputs are transformed back with the current anchor. Figure 6 illustrates removal of arbitrary translation and heading, not elimination of meaningful motion diversity.

Equations 6–7
𝐩~i=𝐡a−1⊗(𝐩i−𝐩a),𝐪~i=𝐡a−1⊗𝐪i,\tilde{\mathbf{p}}_{i}=\mathbf{h}_{a}^{-1}\otimes(\mathbf{p}_{i}-\mathbf{p}_{a}),\qquad\tilde{\mathbf{q}}_{i}=\mathbf{h}_{a}^{-1}\otimes\mathbf{q}_{i}, (6)
𝐣~i,ℓ=𝐡a−1⊗(𝐣i,ℓ−𝐩a).\tilde{\mathbf{j}}_{i,\ell}=\mathbf{h}_{a}^{-1}\otimes(\mathbf{j}_{i,\ell}-\mathbf{p}_{a}). (7)
Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 6: Canonicalization.

Normalization.

Each canonical feature is standardized with training-split mean and variance, with a minimum standard deviation for stability. Samples are denormalized before conversion back to world-frame G1 state.

Equation 8
𝐱¯i=𝐱i−𝝁𝝈,\bar{\mathbf{x}}_{i}=\frac{\mathbf{x}_{i}-\boldsymbol{\mu}}{\boldsymbol{\sigma}}, (8)

C.2 Denoising Backbone

The denoiser predicts the normalized clean future segment from noisy motion, timestep, recent history and available commands.

Equation 9
𝐱^t+1:t+H=𝐱θ(𝐱t+1:t+Hτ,τ,𝐱t−L:t,𝒞s),𝐱^t+1:t+H∈ℝH×125.\hat{\mathbf{x}}_{t+1:t+H}=\mathbf{x}_{\theta}\left(\mathbf{x}_{t+1:t+H}^{\tau},\tau,\mathbf{x}_{t-L:t},\mathcal{C}_{s}\right),\qquad\hat{\mathbf{x}}_{t+1:t+H}\in\mathbb{R}^{H\times 125}. (9)

Architecture.

Bidirectional temporal self-attention spans sixty future frames; cross-attention supplies history and global conditions, while aligned signals modulate frames. RoPE encodes time and sinusoidal embeddings encode diffusion steps. B/L/XL use 50M/300M/500M parameters, widths 512/1152/1536, depths 10/14/13 and 8/18/24 attention heads.

Table 7: Architecture variants of OMG-DiT. To understand the scalability of motion generation, we introduce three variants of the denoising backbone randing from 50M to 500M parameters (B / L / XL), differing in transformer width, depth, and number of attention heads.
Model Params Hidden Dim. Layers Heads
OMG-DiT-B 50M 512 10 8
OMG-DiT-L 300M 1152 14 18
OMG-DiT-XL 500M 1536 13 24

Denoising Objective and Sampling.

Training uses a cosine noise schedule and clean-sample prediction. Default inference applies deterministic DDIM with fifty denoising steps. Single-condition guidance defaults to 2.5, while composed inputs can use separate scales.

Equation 10
𝐱τ=α¯τ​𝐱0+1−α¯τ​ϵ.\mathbf{x}_{\tau}=\sqrt{\bar{\alpha}_{\tau}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{\tau}}\boldsymbol{\epsilon}. (10)

C.3 Omni-Modal Condition Encoders

History, language, audio and human-reference channels use shared interfaces but different input encoders and dropout rates.

Motion History.

A two-layer MLP maps normalized ten-frame history into context tokens, accessed by cross-attention in every denoising block.

Language.

Frozen T5-Base encodes at most fifty text tokens; a projection maps them to the backbone width. Language is dropped with probability 0.3 during training.

Audio.

Audio is converted to mono, peak normalized and analyzed with a hop matched to 30 Hz and a Hann window of at least 50 ms. Equation 13 concatenates 32 normalized log-frequency-band energies, RMS energy, zero-crossing rate, and the sum of spectral flux and centroid, yielding 35 components. Features are padded/trimmed with a validity mask, projected and injected through FiLM; dropout is 0.1.

Equations 11–13
h=round⁡(s/f),h=\mathrm{round}(s/f), (11)
w=max⁡{h,round⁡(0.05​s)},w=\max\{h,\mathrm{round}(0.05s)\}, (12)
𝐚i=[𝐞i1:32,rmsi,zcri,fluxi+centroidi]∈ℝ35.\mathbf{a}_{i}=\left[\mathbf{e}^{1:32}_{i},\;\mathrm{rms}_{i},\;\mathrm{zcr}_{i},\;\mathrm{flux}_{i}+\mathrm{centroid}_{i}\right]\in\mathbb{R}^{35}. (13)

Human Reference Motion.

Human references contain 22 three-dimensional joint positions (66D), projected into frame-wise FiLM conditions. Dropout is 0.5. Availability masks disable missing modalities so datasets with different label subsets can train the same generator.

C.4 Adapting to New Modalities

Global new inputs join the cross-attention context; aligned inputs use FiLM or adaptive-normalization adapters. Zero initialization preserves the pretrained function at the start. Despite describing a lightweight default interface, the reported experiments use full backbone fine-tuning. Parameter-efficient adaptation is suggested as a possibility, not established by these results.

C.5 Classifier-Free Guidance and Composition

Guidance adds weighted differences between condition-specific and null predictions, allowing separately learned command effects to be combined.

Classifier-Free Guidance and Multi-Modal Composition.

For text, audio and human motion, separate conditional branches share a null branch. Their weighted prediction differences are added to the null prediction. Modality dropout makes these branches available; this additive composition does not guarantee compatibility for arbitrary conflicting commands.

Equation 14
𝐱^0=𝐱^0∅+ωtext​(𝐱^0text−𝐱^0∅)+ωaudio​(𝐱^0audio−𝐱^0∅)+ωhuman​(𝐱^0human−𝐱^0∅),\hat{\mathbf{x}}_{0}=\hat{\mathbf{x}}_{0}^{\varnothing}+\omega_{\mathrm{text}}(\hat{\mathbf{x}}_{0}^{\mathrm{text}}-\hat{\mathbf{x}}_{0}^{\varnothing})+\omega_{\mathrm{audio}}(\hat{\mathbf{x}}_{0}^{\mathrm{audio}}-\hat{\mathbf{x}}_{0}^{\varnothing})+\omega_{\mathrm{human}}(\hat{\mathbf{x}}_{0}^{\mathrm{human}}-\hat{\mathbf{x}}_{0}^{\varnothing}), (14)

Practical Implementation.

A single global scale compares a full-condition branch with null; separate scales instead combine separate modality branches. Defaults are 2.5 for a global scale, text/audio 3.0/1.5 for composition, and human-reference 2.0. Increasing a scale biases generation toward that signal but can hurt other metrics.

Equation 15
𝐱^0=𝐱^0∅+ω⁡(𝐱^0all−𝐱^0∅),\hat{\mathbf{x}}_{0}=\hat{\mathbf{x}}_{0}^{\varnothing}+\omega(\hat{\mathbf{x}}_{0}^{\mathrm{all}}-\hat{\mathbf{x}}_{0}^{\varnothing}), (15)

Extending Classifier-Free Guidance to New Modalities.

A new modality receives its own conditional-minus-null guidance direction, which can be combined with existing signals after adaptation.

Equation 16
𝐱^0=𝐱^0∅+ωnew​(𝐱^0new−𝐱^0∅),\hat{\mathbf{x}}_{0}=\hat{\mathbf{x}}_{0}^{\varnothing}+\omega_{\mathrm{new}}(\hat{\mathbf{x}}_{0}^{\mathrm{new}}-\hat{\mathbf{x}}_{0}^{\varnothing}), (16)

C.6 Real-Time Deployment

Deployment runs the tracker onboard NVIDIA Orin and the generator offboard on an RTX 4090 linked by wired Ethernet. ONNX/TensorRT, FP16 and denoising caching reduce computation. This is a networked generator–tracker deployment, not evidence that the full generator runs onboard.

C.7 Training Hyperparameters

Pretraining uses AdamW at 6×10⁻⁵, weight decay 0.01, 2,000 warmup steps, cosine decay, gradient norm clipping at 1.0 and mixed bfloat16. Table 8 gives a maximum 100,000 steps and global batch 1,024 across eight A800 GPUs. The authors report completion in under ten hours; these settings describe this training run rather than a measured scaling study.

Table 8: Hyperparameters for OMG-DiT Pretraining. Unless otherwise specified, evaluation is conducted on the -L (300M) version of the model.
Hyperparameter Value
Motion Setup
Prediction / history length 60 / 10 frames
Motion frame rate 30 FPS
Motion representation 125D G1 features with 6D root rotation
Optimization
Optimizer AdamW
Learning rate 6.0×10−56.0\times 10^{-5}
Weight decay 0.010.01
LR schedule Linear warmup, cosine decay
Warmup steps 2,0002{,}000
Minimum LR 1.0×10−61.0\times 10^{-6}
Gradient clipping Global norm 1.01.0
Training
Max training steps 100,000100{,}000
Precision bf16 mixed precision
GPUs 8
Per-GPU batch size 128
Global batch size 1,024
Inference
Method DDIM
NFE calls 50

Appendix D Experiment Details

The evaluation appendix standardizes robot output format, splits, metrics, evaluator training and baseline conversions.

D.1 Evaluation Protocols

All task comparisons ultimately use G1 outputs and a shared evaluation pipeline, including baselines that originally generate human skeletons.

D.1.1 Evaluated Tasks and Motion Format

The common representation separates stored robot configuration from the richer features used by the generator.

Evaluated Tasks.

Six tasks are evaluated: text, audio and human-reference generation, Pico teleoperation, held-out text-motion fine-tuning, and visual locomotion.

Motion Format.

Stored G1 qpos has 36 values: 3 root positions, 4 quaternion components and 29 joint angles. Forward kinematics yields 90 coordinates for thirty body points. Non-G1 baseline outputs are converted or retargeted before this shared evaluation.

Equations 17–18
𝐪t=[𝐱troot,𝐫troot,𝐪tjoint]∈ℝ36,\mathbf{q}_{t}=\left[\mathbf{x}^{\text{root}}_{t},\ \mathbf{r}^{\text{root}}_{t},\ \mathbf{q}^{\text{joint}}_{t}\right]\in\mathbb{R}^{36}, (17)
𝐩=FK⁡(𝐪)∈ℝT×90,\mathbf{p}=\operatorname{FK}(\mathbf{q})\in\mathbb{R}^{T\times 90}, (18)

D.1.2 Evaluation Data Settings

Validation conditions are sampled separately from generator training; held-out subsets are reserved for adaptation comparisons.

Evaluation Data Split.

Text generation and text fine-tuning each use 1,024 validation conditions; audio, human-reference and Pico evaluations use 512. Default clips contain sixty frames. Text retrieval uses batches of 32 with dataset-stratified reordering, and FID references match sample count unless otherwise set. CMU and Weizmann are excluded from normal pretraining.

Table 9: Evaluation data and sampling settings.
Task Evaluation data # samples
Text-to-motion Caption-bearing validation samples 1024
Audio-to-motion Validation samples with audio/music features 512
Human reference-to-motion Validation samples with human reference motion 512
Pico teleoperation AMASS (CMU + WEIZMANN), with Pico keypoints 512
Text-to-motion finetuning AMASS (CMU + WEIZMANN), with captions 1024

D.2 Evaluation Metrics

Metrics distinguish target similarity, distributional quality, physical plausibility and actual tracker failures; no single score measures all of them.

D.2.1 Shared Evaluation Metrics

MPJPE compares root-relative body positions; global MPJPE retains translation. Velocity and acceleration errors use frame differences, whereas jerk and contact sliding include frame-rate factors. Fall and joint-limit rates are sequence-level indicators; a limit violation above 10⁻⁴ anywhere in a sequence counts. Original formulas and units remain in Table 10.

Equation 19
𝐯b,t,j=𝐩b,t+1,j−𝐩b,t,j,𝐚b,t,j=𝐩b,t+2,j−2​𝐩b,t+1,j+𝐩b,t,j.\mathbf{v}_{b,t,j}=\mathbf{p}_{b,t+1,j}-\mathbf{p}_{b,t,j},\quad\mathbf{a}_{b,t,j}=\mathbf{p}_{b,t+2,j}-2\mathbf{p}_{b,t+1,j}+\mathbf{p}_{b,t,j}. (19)
Table 10: Shared Evaluation Metrics.
Metric Formula Description
c-slide 1|𝒞|​∑(t,f)∈𝒞maxp∈𝒮f⁡∥𝐬t+1,px​y−𝐬t,px​y∥2⋅ffps\frac{1}{|\mathcal{C}|}\sum_{(t,f)\in\mathcal{C}}\max_{p\in\mathcal{S}_{f}}\lVert\mathbf{s}^{xy}_{t+1,p}-\mathbf{s}^{xy}_{t,p}\rVert_{2}\cdot f_{\mathrm{fps}} Horizontal foot sliding speed during contact
Jerk 1|𝒱|​∑(t,j)∈𝒱∥𝐩t+3,j−3​𝐩t+2,j+3​𝐩t+1,j−𝐩t,j∥2⋅ffps3\frac{1}{|\mathcal{V}|}\sum_{(t,j)\in\mathcal{V}}\lVert\mathbf{p}_{t+3,j}-3\mathbf{p}_{t+2,j}+3\mathbf{p}_{t+1,j}-\mathbf{p}_{t,j}\rVert_{2}\cdot f_{\mathrm{fps}}^{3} Third-order motion variation
g-MPJPE 1000⋅1B​T​J∑b,t,j∥𝐩b,t,jp​r​e​d−𝐩b,t,jr​e​f∥21000\cdot\frac{1}{BTJ}\sum_{b,t,j}\lVert\mathbf{p}^{pred}_{b,t,j}-\mathbf{p}^{ref}_{b,t,j}\rVert_{2} Global body position error in mm
MPJPE 1000⋅1B​T​J∑b,t,j∥𝐩~b,t,jp​r​e​d−𝐩~b,t,jr​e​f∥21000\cdot\frac{1}{BTJ}\sum_{b,t,j}\lVert\tilde{\mathbf{p}}^{pred}_{b,t,j}-\tilde{\mathbf{p}}^{ref}_{b,t,j}\rVert_{2} Root-relative body position error in mm
e-vel 1000⋅1B⁡(T−1)​J∑b,t,j∥𝐯b,t,jp​r​e​d−𝐯b,t,jr​e​f∥21000\cdot\frac{1}{B(T-1)J}\sum_{b,t,j}\lVert\mathbf{v}^{pred}_{b,t,j}-\mathbf{v}^{ref}_{b,t,j}\rVert_{2} Velocity error in mm/frame
e-acc 1000⋅1B⁡(T−2)​J∑b,t,j∥𝐚b,t,jp​r​e​d−𝐚b,t,jr​e​f∥21000\cdot\frac{1}{B(T-2)J}\sum_{b,t,j}\lVert\mathbf{a}^{pred}_{b,t,j}-\mathbf{a}^{ref}_{b,t,j}\rVert_{2} Acceleration error in mm/frame2
Fall Rate 1N​∑i=1N𝕀⁡[sequencei​fallen]\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}[\mathrm{sequence}_{i}\ \mathrm{fallen}] Sequence-level fall rate
J-Limit 1N∑i=1N𝕀[maxt,jei,t,j>ϵ]\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}[\max_{t,j}e_{i,t,j}>\epsilon] Sequence-level joint-limit violation rate

D.2.2 Text-to-Motion Metrics

A separately learned embedding space supplies text-motion distance, within-batch top-K retrieval, Gaussian Fréchet distance and pairwise diversity. Diversity uses 300 random pairs by default. These are evaluator-dependent proxies for semantic and distributional quality, not direct physical execution measures.

Table 11: Text-to-motion metrics.
Metric Formula Description
Matching Score 1N​∑i=1N∥𝐦i−𝐭i∥2\frac{1}{N}\sum_{i=1}^{N}\lVert\mathbf{m}_{i}-\mathbf{t}_{i}\rVert_{2} Average distance between paired text-motion embeddings
R-precision@K 1N∑i=1N𝕀[ranki(i)≤K]\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}[\operatorname{rank}_{i}(i)\leq K] Whether the ground-truth text is in the top-KK retrieval results
FID ∥𝝁r−𝝁g∥22+Tr⁡(𝚺r+𝚺g−2​(𝚺r​𝚺g)1/2)\lVert\boldsymbol{\mu}_{r}-\boldsymbol{\mu}_{g}\rVert_{2}^{2}+\operatorname{Tr}(\boldsymbol{\Sigma}_{r}+\boldsymbol{\Sigma}_{g}-2(\boldsymbol{\Sigma}_{r}\boldsymbol{\Sigma}_{g})^{1/2}) Fréchet distance between real and generated motion embeddings
Diversity 1|𝒫|​∑(a,b)∈𝒫∥𝐦a−𝐦b∥2\frac{1}{|\mathcal{P}|}\sum_{(a,b)\in\mathcal{P}}\lVert\mathbf{m}_{a}-\mathbf{m}_{b}\rVert_{2} Average embedding distance between generated samples
Equation 20
Di​j=∥𝐦i−𝐭j∥22,D_{ij}=\lVert\mathbf{m}_{i}-\mathbf{t}_{j}\rVert_{2}^{2}, (20)

D.2.3 Audio-to-Motion Metrics

Beat alignment scores the temporal distance between music and nearest motion beats, with Gaussian width 3/30 seconds. Kinetic and geometric FID compare motion statistics; PFC penalizes joint foot sliding during upward root acceleration. Their different targets explain why a method can improve FID while worsening rhythm alignment.

Table 12: Audio-to-motion metrics.
Metric Formula Description
Beat Align 1|ℬa|∑iexp(−di2/(2σ2))\frac{1}{|\mathcal{B}^{a}|}\sum_{i}\exp(-d_{i}^{2}/(2\sigma^{2})) Temporal alignment between music and motion beats
FID-k ∥𝝁k,r−𝝁k,g∥22+Tr⁡(𝚺k,r+𝚺k,g−2​(𝚺k,r​𝚺k,g)1/2)\lVert\boldsymbol{\mu}_{k,r}-\boldsymbol{\mu}_{k,g}\rVert_{2}^{2}+\operatorname{Tr}(\boldsymbol{\Sigma}_{k,r}+\boldsymbol{\Sigma}_{k,g}-2(\boldsymbol{\Sigma}_{k,r}\boldsymbol{\Sigma}_{k,g})^{1/2}) Fréchet distance over kinetic features
FID-g ∥𝝁g​e​o,r−𝝁g​e​o,g∥22+Tr⁡(𝚺g​e​o,r+𝚺g​e​o,g−2​(𝚺g​e​o,r​𝚺g​e​o,g)1/2)\lVert\boldsymbol{\mu}_{geo,r}-\boldsymbol{\mu}_{geo,g}\rVert_{2}^{2}+\operatorname{Tr}(\boldsymbol{\Sigma}_{geo,r}+\boldsymbol{\Sigma}_{geo,g}-2(\boldsymbol{\Sigma}_{geo,r}\boldsymbol{\Sigma}_{geo,g})^{1/2}) Fréchet distance over geometric features
PFC 10000⋅1T−2∑tLtRtA^t10000\cdot\frac{1}{T-2}\sum_{t}L_{t}R_{t}\hat{A}_{t} Penalizes foot sliding during root upward acceleration

D.2.4 Details on Evaluator Training for Text-to-Motion Evaluation

A motion encoder and frozen T5-3B plus trainable projection learn a shared normalized embedding using symmetric InfoNCE. The motion encoder uses four transformer layers, eight heads and 512D features. Evaluation training uses the corresponding training split; adaptation evaluators are restricted to CMU/Weizmann. This encoder is an evaluation instrument, separate from the generator’s T5-Base conditioning encoder.

Table 13: Motion Evaluator architecture used for evaluation.
Stage Architecture Output
Movement encoder Linear + LayerNorm + GELU + Dropout Per-frame feature
Hidden projection Linear or Identity Hidden sequence
Temporal encoder RoPE Transformer Temporal feature sequence
Pooling/projection Mean pooling + MLP projection Motion embedding
Normalization L2 normalization Unit-norm embedding
Table 14: Hyperparameters for Text-to-Motion Evaluator Training.
Hyperparameter Value
Motion input dim 90
Movement dim 512
Hidden dim 512
Output dim 512
Transformer layers 4
Attention heads 8
MLP ratio 4
Equations 21–23
𝐦i=Emotion​(𝐩i),𝐭i=Etext​(𝐜i),\mathbf{m}_{i}=E_{\mathrm{motion}}(\mathbf{p}_{i}),\quad\mathbf{t}_{i}=E_{\mathrm{text}}(\mathbf{c}_{i}), (21)
ℒ=12​(ℒm→t+ℒt→m),\mathcal{L}=\frac{1}{2}\left(\mathcal{L}_{m\rightarrow t}+\mathcal{L}_{t\rightarrow m}\right), (22)
ℒm→t=−1B∑i=1Blogexp⁡(sim⁡(𝐦i,𝐭i)/τ)∑j=1Bexp⁡(sim⁡(𝐦i,𝐭j)/τ).\mathcal{L}_{m\rightarrow t}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp(\operatorname{sim}(\mathbf{m}_{i},\mathbf{t}_{i})/\tau)}{\sum_{j=1}^{B}\exp(\operatorname{sim}(\mathbf{m}_{i},\mathbf{t}_{j})/\tau)}. (23)

D.3 Details of Evaluation Baselines

Baseline output spaces and conversion routes are explicitly retained, since retargeting can affect the final robot metrics.

D.3.1 Text-to-Motion Baselines

GENMO and HYMotion outputs pass through human-body conversion and GMR. Kimodo variants either generate SMPL-X for retargeting or produce G1 directly. The compared graphics generators produce full sequences in advance rather than the same interactive replanning interface.

Table 15: Text-to-motion Baselines.
Baseline Original output space Conversion to G1 Final artifact
GENMO [17] + GMR SMPL-X motion GMR retargeting G1 qpos_36
HYMotion [55] + GMR SMPL/SMPL-H style motion Convert to SMPL-X, then GMR G1 qpos_36
Kimodo-SMPLX-RP-v1 [42] + GMR SMPL-X motion GMR retargeting G1 qpos_36
Kimodo-G1-RP-v1 [42] G1 qpos Direct use G1 qpos_36
Kimodo-G1-SEED-v1 [42, 44] G1 qpos Direct use G1 qpos_36

D.3.2 Audio-to-Motion Baselines

GENMO, LODGE and Bailando++ use their respective audio interfaces and human-motion formats, then convert through GMR to G1. The main table labels the last baseline Bailando while the appendix calls it Bailando++; both source labels are retained.

Table 16: Audio-to-motion Baselines.
Baseline Condition input Original output Conversion to G1 Final artifact
GENMO [17] + GMR Raw audio/music embedding SMPL-X motion GMR retargeting G1 qpos_36
LODGE [18] + GMR Music feature .npy Dance motion array Convert to SMPL/SMPL-X, then GMR G1 qpos_36
Bailando++ [46] + GMR Raw audio/extracted feature SMPL pickle Convert to SMPL-X, then GMR G1 qpos_36

D.3.3 Human Reference-to-Motion Baselines

GMR, PHC and OmniRetarget use geometric or physics-based retargeting, while NMR learns a mapping. All are compared after conversion to G1, with the same 22-joint human input interpretation.

Equation 24
𝐡∈ℝT×22×3.\mathbf{h}\in\mathbb{R}^{T\times 22\times 3}. (24)
Table 17: Human reference-to-motion Baselines.
Baseline Type Intermediate representation Final artifact
GMR [1] Geometric/rule-based retargeting Compact SMPL-X motion G1 qpos_36
PHC [31] Physics/fitting-based retargeting AMASS-style SMPL motion G1 qpos_36
OmniRetarget [59] Optimization-based retargeting Holosoma-compatible SMPL-X motion G1 qpos_36
NMR [72] Learned retargeting model Standard SMPL-X motion G1 qpos_36

D.4 Details on Finetuning and Scaling Experiments

Fine-tuning fractions and model-size experiments control different variables and should not be conflated.

Datasets and Evaluation Splits.

Held-out annotated segments are exhaustively sliced into two-second windows, shuffled with a fixed seed, and truncated to the requested percentage. These percentages measure slices, not independent episodes. Scaling experiments reuse the same pretraining corpus.

Setup.

Adaptation compares the pretrained 300M L model with the same randomly initialized architecture. Parameter-scaling comparisons vary size while keeping the other settings fixed. The visual locomotion appendix adds a distinct adaptation test.

Appendix E Extended Experiments and Visualizations

Additional material contains timing measurements, physical qualitative sequences, temporal composition, guidance sweeps and visual locomotion adaptation.

E.1 Runtime Acceleration Analysis

For the 50M B model on one A800, sampling a two-second text-conditioned chunk falls from 1,297.9±12.5 ms in eager PyTorch to 51.7±1.5 ms with TensorRT, FP16 and caching (25.1×). The final denoiser time is 31.2 ms with 38 conditional/null predictions in nineteen backend calls. Timings average five steady-state runs and begin with cached condition tensors; they exclude the full sensing/network/control loop and do not directly measure the deployed RTX 4090 configuration.

Table 18: Runtime Acceleration Analysis. Inference time is measured while generating a 60-frame future chunk with text conditioning, running OMG-DiT-B on a single NVIDIA A800 GPU. We report the mean and standard deviation over five steady-state runs.
Optimization Sampling (ms) Denoiser infer (ms) NFE (calls) Speedup
PyTorch eager 1297.9 ±\pm 12.5 1265.0 ±\pm 12.4 100 (100) 1.0×\times
+ torch.compile 467.9 ±\pm 2.9 435.6 ±\pm 2.8 100 (100) 2.8×\times
+ ONNX Runtime CUDA 358.2 ±\pm 24.2 337.0 ±\pm 22.7 100 (50) 3.6×\times
+ TensorRT engine 127.3 ±\pm 7.3 100.9 ±\pm 4.3 100 (50) 10.2×\times
+ TensorRT FP16 99.8 ±\pm 5.2 73.3 ±\pm 5.2 100 (50) 13.0×\times
+ DiT cache 51.7 ±\pm 1.5 31.2 ±\pm 1.3 38 (19) 25.1×\times

E.2 Extended Visualizations on Real-Time Omni-Modal Control

Figure 7 preserves the physical sequences and embedded commands for text, music, human motion and composed text/audio. These illustrate diversity and executability under selected conditions, without a trial-level success denominator.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 7: Qualitative visualization. Unitree G1 execution sequences produced by OMG under text, audio, human-reference, and composed text-audio conditions. Frames are uniformly sampled within each sequence, and embedded prompts are preserved.

E.3 Interactive Control with Temporal Composition

Figure 8 changes commands over time from text through audio and human reference to combined text/audio. The displayed rollout illustrates transitions while retaining local conditioning; smoothness and generality are assessed qualitatively here.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 8: Interactive Control: Composition in the Temporal Horizon.

E.4 Analysis on Classifier-Free Guidance

Guidance sweeps examine alignment and the competing influence of command modalities. They show useful steering with metric-dependent tradeoffs.

Single-Modality Classifier-Free Guidance.

Increasing human guidance from 0.5 to 1.5 improves aggregate MPJPE/global MPJPE from 190.3/557.0 to 74.0/218.9 mm. At 2.0, MPJPE improves further to 71.0 but global error worsens to 343.9 mm; the time-specific columns are also non-monotone. Moderate guidance helps overall, but no single scale dominates every metric and horizon.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 9: Human-reference CFG sweep. The translucent reference overlay shows the target motion, and the opaque robot shows the generated motion.
Table 19: Human-reference CFG sweep. We report MPJPE and g-MPJPE under varying classifier-free guidance scales for human-reference motion across different rollout lengths. MPJPE and g-MPJPE are reported in millimeters; lower is better.
Human CFG MPJPE g-MPJPE 1s 3s 5s 7s 9s
0.5 190.3 557.0 62.7 407.2 236.1 213.2 91.5
1.0 96.6 261.8 42.9 59.3 117.3 144.0 216.6
1.5 74.0 218.9 38.5 51.6 108.9 93.2 139.2
2.0 71.0 343.9 38.3 60.4 109.5 82.6 73.4
Original PDF visual retaining all panels, vector labels and native embedded image detail.
Figure 10: Text-audio CFG sweep. Columns show snapshots over time and rows vary the text guidance scale while keeping the audio condition fixed. Larger text guidance improves adherence to the language instruction in the composed audio-language setting.

Multi-Modality Classifier-Free Guidance.

With a fixed music condition, stronger text guidance makes the robot raise its hands more clearly. The snapshots show semantic steering, while preservation of rhythm is qualitative rather than accompanied by a beat-score sweep.

E.5 Adaptation to New Modalities: Perceptive Locomotion

A small simulated visual task tests adaptation of the prior to an input modality absent from the original conditioning set.

Task and Environment.

The robot sees an egocentric camera frame and a categorical instruction to reach one of three adjacent colored squares. This isolates simple visual grounding rather than general obstacle-aware navigation.

Original PDF visual retaining all panels, vector labels and native embedded image detail.
(a) Dataset.
Refer to caption
(b) Egocentric RGB input.
Refer to caption
(c) Third-person layout.
Figure 11: Perceptive Locomotion setup. The dataset contains 300 Kimodo-generated demonstrations per target color. The policy observes only low-resolution egocentric RGB and a discrete color command. The goal is to follow the command and locomote to the corresponding color.

Training Data.

Kimodo generates 300 demonstrations per color, each with 210 frames and a terminal hold. Training samples two-second windows and a 64×64 RGB image. A learned yellow/blue/red embedding replaces language to keep the comparison controlled.

Model and Training Details.

Scratch and pretrained 300M models share the new RGB patch encoder, visual cross-attention and color embedding. A visual gate starts at 0.05, rather than zero. Both use one GPU, batch size sixteen and learning rate 6×10⁻⁵.

Evaluation and Results.

Replanning samples two seconds, executes the first half-second, and repeats for a 210-frame budget. Success requires the pelvis to enter the instructed square at any time, not remain there or meet a stopping criterion. After 500 updates, pretraining succeeds in 53/90 trials (58.9%) versus 44/90 (48.9%) from scratch: a ten-point improvement on this limited simulated task.

Table 20: Success Rate on Perceptive Locomotion. We report success rate over 90 rollouts.
Initialization Step Success Rate
Scratch 500 44/90 (48.9%)
Pretrained 500 53/90 (58.9%)
Refer to caption
Figure 12: Third-person timelapse of a successful rollout by the pretrained checkpoint. The robot enters the commanded blue target.
Concept demonstration

Open image ↗