Overview
GRAIL generates robot-oriented human interaction videos from known 3D scenes, reconstructs their motion, and converts selected task families into simulated demonstrations and visual robot policies.
At a glance
The pipeline fixes scene geometry, camera, scale, and a robot-proportioned human before generating video, then jointly refines human/object trajectories in that known frame. Two adaptations of a pretrained whole-body controller turn the references into executable demonstrations. More than twenty thousand sequences cover manipulation, sitting, and terrain traversal. Separate visual policies achieve 84% pick-up on five seen object types, 80% on five unseen types, and a reported 90% stair-climbing success. “Generated data only” describes downstream task demonstrations, not an absence of pretrained models or prior controller training.
1 Introduction
Collecting robot demonstrations requires physical setups and operation; reconstructing uncontrolled monocular video instead leaves scale, geometry, morphology, and contacts ambiguous. GRAIL specifies those variables before video generation and reuses them during reconstruction. The approach shifts work into assets, simulation, model inference, and optimization. It avoids new physical demonstration collection for these tasks while depending on pretrained video, vision, motion, and robot-control models.
2 Related Work
Prior motion capture, video reconstruction, and generative interaction methods trade collection cost against geometric/physical ambiguity. GRAIL’s contribution is upstream scene control plus downstream task-family tracking. It does not eliminate reconstruction errors or turn unconstrained real-world video into robot-ready motion; the known assets and camera are central inputs.
3 Method
The pipeline renders a known scene, generates a human interaction video, reconstructs metric human/object motion, retargets it to G1, and learns physics-based trackers. Tracker rollouts then train separate egocentric visual policies for physical pick-up and stairs. Kinematic reconstruction, simulated executability, and hardware success are evaluated at distinct stages.
3.1 Robot-Centric Human Video Generation
Infinigen environments contain a human character fitted to G1 proportions and an object settled by rigid-body simulation. Blender renders the initial frame with known intrinsics, extrinsics, scale, and background depth. A vision-language model proposes an interaction prompt and Kling generates a static-camera video. Human generation is chosen because video and reconstruction models have stronger human priors; consistency of the requested static camera and object appearance remains an assumption to check.
3.2 Interaction-Aware HOI Reconstruction
Independent body/hand/object estimates initialize a joint optimization that uses the original scene to resolve scale and contact ambiguity. Known geometry constrains recovery but does not guarantee accurate tracking through occlusion or generated appearance changes.
3.2.1 Initial Motion Estimation
GENMO estimates body pose with the prefitted shape held fixed. WiLoR estimates hand poses; missing detections are interpolated and temporally smoothed before wrist inverse-kinematic alignment. FoundationPose is finetuned for five epochs with depth channels zeroed and initialized from the known first object pose. Its RGB-only tracking uses known geometry/texture/camera, and mask consistency filtering rejects some failures. Interpolation cannot recover all unobserved fast hand behavior.
3.2.2 Joint Optimization
Joint refinement optimizes residual human/object translations and rotations while keeping hand poses fixed. Projected keypoint and object-vertex losses preserve image alignment. MoGe-2 depth is scaled against rendered background depth; segmented human/object point clouds then supply metric Chamfer targets.
A vision-language model labels contact on sampled frames, with labels propagated over neighboring intervals. Contact refinement restricts candidate object vertices by image overlap and penalizes depth-direction mismatch, rather than solving full contact dynamics. Terrain-only sequences omit hand-object contact loss. Foot-contact regularization suppresses skating; pelvis-velocity and first/second temporal differences reduce depth oscillation and jitter. The resulting trajectories are physically motivated references, not guaranteed dynamically feasible motions.
Equations 1–5
| (1) |
| (2) |
| (3) |
| (4) |
| (5) |
3.3 Task-General Loco-Manipulation Tracking
GMR retargets reconstructed motion to G1. For manipulation, an adaptor adds a scaled latent residual and two hand primitives to the frozen SONIC controller. It sees privileged object geometry/state, future-reference deltas, proprioception, and contact cues. Each hand primitive selects predefined open/close configurations for seven finger joints; this is not independent continuous control of every finger.
The scene tracker instead finetunes SONIC with a local height-map encoder on reconstructed scene interactions plus original flat-ground data. An auxiliary kinematic decoder regularizes latent fidelity. Both trackers use motion tracking and regularization; manipulation adds object/contact rewards. Training pools motions within each family, not all behaviors in one universal policy. Each stage uses thirty thousand PPO iterations on sixty-four L40 GPUs with 1,024 environments per GPU.
Equations 6–9
| (6) |
| (7) |
| (8) |
| (9) |
3.4 Sim-to-Real Deployment
Object and scene trackers are distilled into separate head-camera RGB policies for pick-up and stairs, using visual randomization and camera alignment. Deployment streams visual/proprioceptive input to a desktop RTX 5090 and returns actions to G1 at ten hertz using a Luxonis OAK-D W camera. This is offboard inference; the privileged tracking observations are training interfaces rather than direct physical-policy inputs.
4 Results
Experiments separately measure reconstructed interaction quality, simulated robot tracking, and physical visual-policy transfer. Their success rates use different definitions and should not be combined into one metric.
4.1 Human-Object Interaction Generation
On twenty shared everyday objects, GRAIL achieves contact distance 0.008, penetration 0.90%, VLM interaction score 3.58/5, and physics-tracking success 88.9%, versus 24.0% for DAViD. Human smoothness is not best: DAViD’s 0.0024 is lower than GRAIL’s 0.0033. Executability is evaluated with capsule SMPL-X humanoids through InterMimic, without robot retargeting; success is the fraction of frames where both body and object deviations, normalized by object size, are below 0.25. It is not an 88.9% physical G1 task-success rate.
| Geometric | Perceptual | Smoothness | Physical Executability | |||||
| Methods | Contact | Pen. | Inter. Score | Human Smo. | Obj Smo. | SR | Body Dev. | Obj Dev. |
| HOIDiff (Peng et al., 2025) | 0.012 | 2.07% | 1.79 | 0.0043 | 0.0118 | 15.8% | 0.2120 | 0.3352 |
| CHOIS (Li et al., 2024) | 0.034 | 3.74% | 2.47 | 0.0055 | 0.0062 | 10.5% | 0.2564 | 0.3642 |
| DAViD (Kim et al., 2025a) | 0.246 | 1.46% | 2.74 | 0.0024 | 0.0605 | 24.0% | 0.4723 | 0.5826 |
| Ours | 0.008 | 0.90% | 3.58 | 0.0033 | 0.0022 | 88.9% | 0.0913 | 0.0851 |
| Method | SR | ObjPos | MPJPE-L |
| HDMI (Weng et al., 2025) | 48.5% | 0.283 | 122.3 |
| ResMimic (Zhao et al., 2025) | 49.2% | 0.393 | 80.9 |
| Ours w/o SONIC | 45.0% | 0.395 | 243.5 |
| Ours w/o | 39.7% | 0.303 | 37.1 |
| Ours w/o Rel. Obs. | 57.9% | 0.257 | 43.0 |
| Ours (Full) | 81.4% | 0.135 | 41.8 |
| Seen Objects | Unseen Objects | |||||||||||
| Cube | Apple | Tea Box | Carrot | Wet Wipes | Avg. | Spray Can | Lint Roller | Peach | Flashlight | Medicine Bottle | Avg. | |
| SR | 100% | 60% | 100% | 70% | 90% | 84% | 100% | 50% | 90% | 80% | 80% | 80% |
4.2 Task-General Loco-Manipulation Tracking
The reported dataset spans over twenty thousand sequences, one thousand object assets and one thousand procedural terrains, covering pick-up, whole-body object movement, sitting, and traversal. Task-family trackers amortize learning across related references, with finetuning when families change.
On 124 motions across forty-three objects, full GRAIL reaches 81.4% episode success, defined by average object-position error below twenty centimetres, versus HDMI 48.5% and ResMimic 49.2%. The baselines use per-task policies and no per-finger actuation, so the comparison includes different control capabilities and training organization. Removing SONIC gives 45.0%; removing adaptation gives 39.7% despite the best body tracking, and absolute instead of relative observations gives 57.9%. Body imitation alone is insufficient for the evaluated object interactions.
4.3 Sim-to-Real Deployment
Seen-object pick-up uses two hundred generated approach/pick-up sequences per object and ten physical trials per category. Results are cube 10/10, apple 6/10, tea box 10/10, carrot 7/10, and wipes 9/10: 42/50 overall. Five unseen categories total 40/50, including lint roller 5/10. Stair climbing is reported at 90%, but the paper does not specify its trial denominator alongside that claim. Hardware validation covers these two skills, not every sitting or whole-body manipulation behavior in the generated dataset.
5 Conclusion
Known scenes and robot-oriented morphology make generated video a useful source of task demonstrations, while optimization and physics tracking bridge toward executable actions. The results show a promising fully digital collection route for the evaluated tasks. They depend on substantial pretrained priors, asset preparation, filtering, training compute, and task-family specialization.
6 Limitations
The pipeline requires suitable 3D assets, simulation scenes, and a video model that follows instructions. Occlusion, fast motion, and appearance inconsistency degrade reconstruction. A nontrivial but unquantified fraction is discarded, so the paper does not establish a generation yield or accepted-sequence cost. Trackers still need training/finetuning when the motion family changes substantially; physical evidence is narrower than the full generated dataset.
Appendix A Human-Object Interaction Generation Details
Generation details specify scene templates, video settings, wall-clock cost, and mask-based rejection. The PDF’s omitted scene-configuration figure is restored here; its PDF number is six, while the HTML labels the later qualitative comparison six.
A.1 Video Generation
A vision-language model chooses floor-only or furnished tabletop placement based on object affordance. The character is prefitted to G1. Kling 2.5 Turbo Pro generates five- or ten-second 1,920×1,080 videos at twenty-four frames per second. Optional ending frames control final placement, and chaining the previous final frame supports longer sequences. Such chaining remains dependent on temporal/appearance consistency rather than guaranteeing it.
A.2 Generation Runtime
A five-second, 121-frame sequence takes approximately fourteen minutes: one video generation, two human estimation, one object tracking, two depth/segmentation preprocessing, and eight joint optimization. Local stages are measured on an A100, while video generation calls the Kling API, so “single A100” does not describe all remote generation compute. The timings do not supply a filtered-data yield or total monetary cost.
| Stage | Time |
| Video Generation (Kling API) | 1 min |
| Human Motion Estimation | 2 min |
| Object Pose Tracking | 1 min |
| Optimization Preprocessing | 2 min |
| Joint Optimization | 8 min |
| Total | 14 min |
A.3 Failure Case Filtering
The filter measures the fraction of SAM2 object-mask pixels not covered by a rendered predicted silhouette, aggregated over frames, and rejects error above 0.2. This is a one-sided coverage error, not intersection-over-union: an overly large silhouette is not penalized symmetrically. It detects some tracking/appearance failures but is not a complete physical-validity test.
Equation 10
| (10) |
Appendix B Task-General Loco-Manipulation Tracking Details
Tracking details distinguish manipulation adaptation from scene-conditioned finetuning. Original observation and reward tables are retained, including differences between their implementation-level gates and the main conceptual description.
B.1 Object-Aware Adaptor
The manipulation adaptor is a 512/256/128 SiLU network producing sixty-four latent-residual values plus two sigmoid-thresholded hand primitives. Residual scale is 0.1 before finite scalar quantization, and SONIC remains frozen. A privileged critic receives full body state/contact flags. Episodes initialize within the first thirty reference frames before labeled contact and terminate on excessive object-height, root-height, or orientation error. PPO uses actor learning rate 2×10⁻⁵, critic 10⁻³, discount 0.99, GAE 0.95, and the complete retained optimization settings.
| Term | Description | Dim | Used |
| Proprioception | |||
| Joint position | robot joint angles | 43 / 29 | Both |
| Joint velocity | robot joint angular velocities | 43 / 29 | Both |
| Base angular velocity | in body frame | 3 | Both |
| Previous action | last meta-action | 66 / 29 | Both |
| Base linear velocity | in body frame | 3 | O |
| Gravity direction | in body frame | 3 | S |
| Reference motion targets | |||
| Motion anchor position | target root position in body frame | 3 | O |
| Motion anchor orientation | target root orientation 6D | 6 | O |
| Current command | ref. joint position + velocity, current frame | 58 | O |
| Multi-future command | ref. joint position + velocity, 10 future frames | 580 | O |
| Multi-future anchor ori | 10 future root orientations 6D | 60 | O |
| Object state | |||
| Object position | current object pos in body frame | 3 | Both |
| Object orientation 6D | current object ori in body frame | 6 | Both |
| Object pos delta (10 fut.) | , position | 30 | Both |
| Object ori delta (10 fut.) | relative rotation, 6D | 60 | Both |
| Target object position | reference object pos in base frame | 3 | O |
| Object BPS encoding | per-object shape descriptor | 10 | O |
| Table position | in body frame | 3 | O |
| Table orientation 6D | in body frame | 6 | O |
| Hand-object contact | |||
| Hand-to-object transform | per right hand (3 pos + 6 ori) | 9 | O |
| Fingertip contact forces | 3D force per fingertip | 12 | O |
| Scene | |||
| Local height map | terrain in body frame | 121 | S |
B.2 Scene-Aware Tracker
Scene tracking uses an eleven-by-eleven raycast grid over 1.5 metres, at 0.15-metre spacing, transformed into the yaw-aligned robot frame. The appendix prose describes an 11×11×3 position tensor, while Table 5 lists 121 height values; both source representations are retained rather than reconciled without code. A three-layer convolutional encoder and fusion network condition the finetuned controller, with kinematic reconstruction weight 0.01 and full-motion reset sampling. Manipulation-specific rewards are disabled.
B.3 Training Cost
A full task-family run takes about thirty hours on sixty-four L40 GPUs and jointly trains 2,000–4,000 motions. The reported 0.5–0.9 minutes per motion divides cluster wall time by pool size; it is not single-GPU compute or the cost to learn an isolated new motion. Warm-started in-family finetuning typically takes six thousand iterations, roughly six hours, while cost per added motion depends on how many are incorporated.
Appendix C Experiment Details
Additional comparisons document baseline initialization, a human preference study, qualitative trajectories, and reconstruction-loss ablations.
C.1 Human-Object Interaction Generation
CHOIS and HOIDiff are training-based interaction generators. DAViD uses the same Kling setting, but its initial frame is manually selected from twenty-four generated candidates per object. A thirty-person study presents three of four methods per trial; GRAIL wins 74.7% affordance and 70.9% physical-plausibility picks, close to the 75% maximum under that sampling scheme. Preferences complement rather than replace physics or hardware evaluation.
Removing projection, depth, or contact refinement lowers downstream tracking success from 81.4% to 41.6%, 42.6%, or 53.3%. Some ablations improve individual smoothness or penetration proxies, so minimizing any one reconstruction metric does not ensure executable interaction. The full model has the best downstream tracking in this comparison.
| Methods | Aff. Real. | Phys. Real. |
| HOIDiff (Peng et al., 2025) | 2.0% | 1.9% |
| CHOIS (Li et al., 2024) | 12.2% | 16.8% |
| DAViD (Kim et al., 2025a) | 11.2% | 10.4% |
| Ours | 74.7% | 70.9% |
| Reconstruction Quality | Tracking Quality | |||||||
| Methods | Contact | Pen. | MPJPE | Human Smo. | Obj Smo. | SR | ObjPos | MPJPE-L |
| w/o | 0.016 | 1.93% | 16.70 | 0.0023 | 0.0011 | 41.6% | 0.374 | 47.1 |
| w/o | 0.017 | 1.97% | 4.35 | 0.0020 | 0.0011 | 42.6% | 0.372 | 49.3 |
| w/o | 0.024 | 1.52% | 4.81 | 0.0022 | 0.0009 | 53.3% | 0.332 | 52.4 |
| Ours (Full) | 0.015 | 1.81% | 4.89 | 0.0020 | 0.0009 | 81.4% | 0.135 | 41.8 |