Literature
Display
← All papers

GRAIL: Generating Humanoid Loco-Manipulation from World Models

Tianyi Xie · Haotian Zhang · Jinhyung Park · Zi Wang · Bowen Wen · Jiefeng Li · Xueting Li · Qingwei Ben · Haoyang Weng · Yufei Ye · David Minor · Tingwu Wang · Chenfanfu Jiang · Sanja Fidler · Jan Kautz · Linxi Fan · Yuke Zhu · Zhengyi Luo · Umar Iqbal · Ye Yuan

Preprint posted · CoRL 2026 (accepted) ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

GRAIL generates robot-oriented human interaction videos from known 3D scenes, reconstructs their motion, and converts selected task families into simulated demonstrations and visual robot policies.

Refer to caption
Figure 1: From Fully Digital Data Generation to Real-World Deployment. GRAIL generates humanoid loco-manipulation data by grounding video world models in fully specified 3D scenes, without physical scene rebuilds or robot teleoperation. Top: simulated humanoids execute generated references spanning pick-up, whole-body manipulation, sitting, and terrain traversal. Bottom: egocentric visual policies trained only on GRAIL-generated data are deployed on a Unitree G1 for stair-climbing and object pick-up.

At a glance

The pipeline fixes scene geometry, camera, scale, and a robot-proportioned human before generating video, then jointly refines human/object trajectories in that known frame. Two adaptations of a pretrained whole-body controller turn the references into executable demonstrations. More than twenty thousand sequences cover manipulation, sitting, and terrain traversal. Separate visual policies achieve 84% pick-up on five seen object types, 80% on five unseen types, and a reported 90% stair-climbing success. “Generated data only” describes downstream task demonstrations, not an absence of pretrained models or prior controller training.

1 Introduction

Collecting robot demonstrations requires physical setups and operation; reconstructing uncontrolled monocular video instead leaves scale, geometry, morphology, and contacts ambiguous. GRAIL specifies those variables before video generation and reuses them during reconstruction. The approach shifts work into assets, simulation, model inference, and optimization. It avoids new physical demonstration collection for these tasks while depending on pretrained video, vision, motion, and robot-control models.

2 Related Work

Prior motion capture, video reconstruction, and generative interaction methods trade collection cost against geometric/physical ambiguity. GRAIL’s contribution is upstream scene control plus downstream task-family tracking. It does not eliminate reconstruction errors or turn unconstrained real-world video into robot-ready motion; the known assets and camera are central inputs.

3 Method

The pipeline renders a known scene, generates a human interaction video, reconstructs metric human/object motion, retargets it to G1, and learns physics-based trackers. Tracker rollouts then train separate egocentric visual policies for physical pick-up and stairs. Kinematic reconstruction, simulated executability, and hardware success are evaluated at distinct stages.

3.1 Robot-Centric Human Video Generation

Infinigen environments contain a human character fitted to G1 proportions and an object settled by rigid-body simulation. Blender renders the initial frame with known intrinsics, extrinsics, scale, and background depth. A vision-language model proposes an interaction prompt and Kling generates a static-camera video. Human generation is chosen because video and reconstruction models have stronger human priors; consistency of the requested static camera and object appearance remains an assumption to check.

Refer to caption
Figure 2: Asset-Conditioned 4D HOI Generation. Given a 3D object asset, we render a fully specified 3D scene with a character prefitted to the target humanoid and known camera parameters, synthesize a static-camera interaction video via a VFM conditioned on the rendered frame, and reconstruct metric 4D human-object motion by jointly refining initial human and object trajectory estimates with keypoint, depth, and contact losses anchored to the privileged 3D configuration.

3.2 Interaction-Aware HOI Reconstruction

Independent body/hand/object estimates initialize a joint optimization that uses the original scene to resolve scale and contact ambiguity. Known geometry constrains recovery but does not guarantee accurate tracking through occlusion or generated appearance changes.

3.2.1 Initial Motion Estimation

GENMO estimates body pose with the prefitted shape held fixed. WiLoR estimates hand poses; missing detections are interpolated and temporally smoothed before wrist inverse-kinematic alignment. FoundationPose is finetuned for five epochs with depth channels zeroed and initialized from the known first object pose. Its RGB-only tracking uses known geometry/texture/camera, and mask consistency filtering rejects some failures. Interpolation cannot recover all unobserved fast hand behavior.

3.2.2 Joint Optimization

Joint refinement optimizes residual human/object translations and rotations while keeping hand poses fixed. Projected keypoint and object-vertex losses preserve image alignment. MoGe-2 depth is scaled against rendered background depth; segmented human/object point clouds then supply metric Chamfer targets.

A vision-language model labels contact on sampled frames, with labels propagated over neighboring intervals. Contact refinement restricts candidate object vertices by image overlap and penalizes depth-direction mismatch, rather than solving full contact dynamics. Terrain-only sequences omit hand-object contact loss. Foot-contact regularization suppresses skating; pelvis-velocity and first/second temporal differences reduce depth oscillation and jitter. The resulting trajectories are physically motivated references, not guaranteed dynamically feasible motions.

Equations 1–5
L=λkp​Lkp+λproj​Lproj+λdepth​Ldepth+λcont​Lcont+λreg​Lreg,L=\lambda_{\text{kp}}L_{\text{kp}}+\lambda_{\text{proj}}L_{\text{proj}}+\lambda_{\text{depth}}L_{\text{depth}}+\lambda_{\text{cont}}L_{\text{cont}}+\lambda_{\text{reg}}L_{\text{reg}}, (1)
Lkp=1T​∑t=1T‖𝒦ℋ​(𝚯tℋ)−pt‖,L_{\text{kp}}=\frac{1}{T}\sum_{t=1}^{T}\left\|\mathcal{K}^{\mathcal{H}}(\bm{\Theta}^{\mathcal{H}}_{t})-p_{t}\right\|, (2)
Lproj=∑t=1T‖𝒫⁡(Vt𝒪)−𝒫⁡(V^t𝒪)‖,L_{\text{proj}}=\sum_{t=1}^{T}\left\|\mathcal{P}(V^{\mathcal{O}}_{t})-\mathcal{P}(\widehat{V}^{\mathcal{O}}_{t})\right\|, (3)
Ldepth=1T​∑t=1T𝒞​𝒟​(Vtℋ,vis,𝐏tℋ)+𝒞​𝒟​(Vt𝒪,vis,𝐏t𝒪),L_{\text{depth}}=\frac{1}{T}\sum_{t=1}^{T}\mathcal{CD}(V^{\mathcal{H},\mathrm{vis}}_{t},\mathbf{P}^{\mathcal{H}}_{t})+\mathcal{CD}(V^{\mathcal{O},\mathrm{vis}}_{t},\mathbf{P}^{\mathcal{O}}_{t}), (4)
Lcont=1|𝒯c|​∑t∈𝒯c𝒞​𝒟z​(Vtℋ,cont,Vt𝒪,cont),Vt𝒪,cont=ℱ⁡(Vt𝒪,Vtℋ,cont),L_{\text{cont}}=\frac{1}{|\mathcal{T}_{c}|}\sum_{t\in\mathcal{T}_{c}}\mathcal{CD}_{z}(V^{\mathcal{H},\mathrm{cont}}_{t},V^{\mathcal{O},\mathrm{cont}}_{t}),\quad V^{\mathcal{O},\mathrm{cont}}_{t}=\mathcal{F}(V^{\mathcal{O}}_{t},V^{\mathcal{H},\mathrm{cont}}_{t}), (5)

3.3 Task-General Loco-Manipulation Tracking

GMR retargets reconstructed motion to G1. For manipulation, an adaptor adds a scaled latent residual and two hand primitives to the frozen SONIC controller. It sees privileged object geometry/state, future-reference deltas, proprioception, and contact cues. Each hand primitive selects predefined open/close configurations for seven finger joints; this is not independent continuous control of every finger.

The scene tracker instead finetunes SONIC with a local height-map encoder on reconstructed scene interactions plus original flat-ground data. An auxiliary kinematic decoder regularizes latent fidelity. Both trackers use motion tracking and regularization; manipulation adds object/contact rewards. Training pools motions within each family, not all behaviors in one universal policy. Each stage uses thirty thousand PPO iterations on sixty-four L40 GPUs with 1,024 environments per GPU.

Refer to caption
Figure 3: Task-General Tracking via Complementary Controller Adaptation. Retargeted 4D HOI trajectories are converted into robot-action data by adapting different parts of a pretrained whole-body controller. Object-manipulation trajectories are used to train an object-aware adaptor πϕ\pi_{\phi} while the controller remains frozen: the adaptor observes proprioception 𝒔t\bm{s}_{t} and an object reference 𝒐t\bm{o}_{t}, injects a latent residual Δ​𝒛t\Delta\bm{z}_{t}, and emits hand actions 𝒂thand\bm{a}^{\mathrm{hand}}_{t}. Terrain traversal and chair-sitting trajectories are used to fine-tune the controller with a scene-aware height encoder ϵh\epsilon_{h}: the encoder maps the local height map hth_{t} into terrain context for scene-conditioned whole-body control.
Equations 6–9
(Δ​𝒛t,𝒂thand)=πϕ​(𝒔t,𝒐t),𝒂tbody=𝒢⁡(𝒛t+λ​Δ​𝒛t),(\Delta\bm{z}_{t},\bm{a}^{\mathrm{hand}}_{t})=\pi_{\phi}(\bm{s}_{t},\bm{o}_{t}),\quad\bm{a}^{\mathrm{body}}_{t}=\mathcal{G}(\bm{z}_{t}+\lambda\Delta\bm{z}_{t}), (6)
Rtmotion=∑iwi​exp⁡(−‖𝒙~i,t−𝒙i,t‖2σi2),R^{\mathrm{motion}}_{t}=\sum_{i}w_{i}\exp\!\left(-\frac{\|\widetilde{\bm{x}}_{i,t}-\bm{x}_{i,t}\|^{2}}{\sigma_{i}^{2}}\right), (7)
Rtobj=wp​exp⁡(−αp​‖𝒑~t𝒪−𝒑t𝒪‖)+wr​exp⁡(−αr​‖𝒓~t𝒪⊖𝒓t𝒪‖),R^{\mathrm{obj}}_{t}=w_{p}\exp\!\left(-\alpha_{p}\|\widetilde{\bm{p}}^{\mathcal{O}}_{t}-\bm{p}^{\mathcal{O}}_{t}\|\right)+w_{r}\exp\!\left(-\alpha_{r}\|\widetilde{\bm{r}}^{\mathcal{O}}_{t}\ominus\bm{r}^{\mathcal{O}}_{t}\|\right), (8)
Rtgrasp=\displaystyle R^{\mathrm{grasp}}_{t}=\; wc​min⁡(NtcontactNmin, 1)⏟contact time+wd​[−cos⁡(𝒅tthumb,𝒅tindex)]+⏟grasp pose\displaystyle w_{c}\underbrace{\min\!\left(\frac{N^{\mathrm{contact}}_{t}}{N_{\min}},\;1\right)}_{\text{contact time}}+w_{d}\underbrace{\left[-\cos(\bm{d}^{\mathrm{thumb}}_{t},\,\bm{d}^{\mathrm{index}}_{t})\right]^{+}}_{\text{grasp pose}}
+wf​exp(−γ1Nf∑j∥𝒇j,t−𝒄t∥)⏟contact proximity.\displaystyle+\;w_{f}\underbrace{\exp\!\left(-\gamma\,\tfrac{1}{N_{f}}\textstyle\sum_{j}\|\bm{f}_{j,t}-\bm{c}_{t}\|\right)}_{\text{contact proximity}}. (9)

3.4 Sim-to-Real Deployment

Object and scene trackers are distilled into separate head-camera RGB policies for pick-up and stairs, using visual randomization and camera alignment. Deployment streams visual/proprioceptive input to a desktop RTX 5090 and returns actions to G1 at ten hertz using a Luxonis OAK-D W camera. This is offboard inference; the privileged tracking observations are training interfaces rather than direct physical-policy inputs.

4 Results

Experiments separately measure reconstructed interaction quality, simulated robot tracking, and physical visual-policy transfer. Their success rates use different definitions and should not be combined into one metric.

Refer to caption
Figure 4: Generated Loco-Manipulation Data. Representative simulated Unitree G1 executions from the generated dataset span pick-up, whole-body manipulation, sitting, and terrain traversal across diverse objects and scene geometries.

4.1 Human-Object Interaction Generation

On twenty shared everyday objects, GRAIL achieves contact distance 0.008, penetration 0.90%, VLM interaction score 3.58/5, and physics-tracking success 88.9%, versus 24.0% for DAViD. Human smoothness is not best: DAViD’s 0.0024 is lower than GRAIL’s 0.0033. Executability is evaluated with capsule SMPL-X humanoids through InterMimic, without robot retargeting; success is the fraction of frames where both body and object deviations, normalized by object size, are below 0.25. It is not an 88.9% physical G1 task-success rate.

Geometric Perceptual Smoothness Physical Executability
Methods Contact ↓\downarrow Pen. ↓\downarrow Inter. Score ↑\uparrow Human Smo. ↓\downarrow Obj Smo. ↓\downarrow SR ↑\uparrow Body Dev. ↓\downarrow Obj Dev. ↓\downarrow
HOIDiff (Peng et al., 2025) 0.012 2.07% 1.79 0.0043 0.0118 15.8% 0.2120 0.3352
CHOIS (Li et al., 2024) 0.034 3.74% 2.47 0.0055 0.0062 10.5% 0.2564 0.3642
DAViD (Kim et al., 2025a) 0.246 1.46% 2.74 0.0024 0.0605 24.0% 0.4723 0.5826
Ours 0.008 0.90% 3.58 0.0033 0.0022 88.9% 0.0913 0.0851
Table 1: Comparison of HOI Generation. Geometric quality, perceptual realism (Interaction Score, 1-5 scale), motion smoothness, and physics-based tracking on the shared 20-object evaluation set.
Method SR ↑\uparrow ObjPos ↓\downarrow MPJPE-L ↓\downarrow
HDMI (Weng et al., 2025) 48.5% 0.283 122.3
ResMimic (Zhao et al., 2025) 49.2% 0.393 80.9
Ours w/o SONIC 45.0% 0.395 243.5
Ours w/o πϕ\pi_{\phi} 39.7% 0.303 37.1
Ours w/o Rel. Obs. 57.9% 0.257 43.0
Ours (Full) 81.4% 0.135 41.8
Table 2: Task-General Loco-Manipulation Tracking. Comparison against loco-manipulation baselines (top) and ablations of GRAIL’s object-aware adaptor (bottom). Metrics: success rate (SR), object position error (ObjPos), and local per-joint error (MPJPE-L).
Refer to caption
Figure 5: Sim-to-Real Deployment. Egocentric visual policies trained only on GRAIL-generated data transfer to a Unitree G1 for object pick-up and stair-climbing.
Seen Objects Unseen Objects
Cube Apple Tea Box Carrot Wet Wipes Avg. Spray Can Lint Roller Peach Flashlight Medicine Bottle Avg.
SR ↑\uparrow 100% 60% 100% 70% 90% 84% 100% 50% 90% 80% 80% 80%
Table 3: Sim-to-Real Pick-up Results for Seen and Unseen Objects. Success rates are computed over 10 trials.

4.2 Task-General Loco-Manipulation Tracking

The reported dataset spans over twenty thousand sequences, one thousand object assets and one thousand procedural terrains, covering pick-up, whole-body object movement, sitting, and traversal. Task-family trackers amortize learning across related references, with finetuning when families change.

On 124 motions across forty-three objects, full GRAIL reaches 81.4% episode success, defined by average object-position error below twenty centimetres, versus HDMI 48.5% and ResMimic 49.2%. The baselines use per-task policies and no per-finger actuation, so the comparison includes different control capabilities and training organization. Removing SONIC gives 45.0%; removing adaptation gives 39.7% despite the best body tracking, and absolute instead of relative observations gives 57.9%. Body imitation alone is insufficient for the evaluated object interactions.

4.3 Sim-to-Real Deployment

Seen-object pick-up uses two hundred generated approach/pick-up sequences per object and ten physical trials per category. Results are cube 10/10, apple 6/10, tea box 10/10, carrot 7/10, and wipes 9/10: 42/50 overall. Five unseen categories total 40/50, including lint roller 5/10. Stair climbing is reported at 90%, but the paper does not specify its trial denominator alongside that claim. Hardware validation covers these two skills, not every sitting or whole-body manipulation behavior in the generated dataset.

5 Conclusion

Known scenes and robot-oriented morphology make generated video a useful source of task demonstrations, while optimization and physics tracking bridge toward executable actions. The results show a promising fully digital collection route for the evaluated tasks. They depend on substantial pretrained priors, asset preparation, filtering, training compute, and task-family specialization.

6 Limitations

The pipeline requires suitable 3D assets, simulation scenes, and a video model that follows instructions. Occlusion, fast motion, and appearance inconsistency degrade reconstruction. A nontrivial but unquantified fraction is discarded, so the paper does not establish a generation yield or accepted-sequence cost. Trackers still need training/finetuning when the motion family changes substantially; physical evidence is narrower than the full generated dataset.

Appendix A Human-Object Interaction Generation Details

Generation details specify scene templates, video settings, wall-clock cost, and mask-based rejection. The PDF’s omitted scene-configuration figure is restored here; its PDF number is six, while the HTML labels the later qualitative comparison six.

A.1 Video Generation

A vision-language model chooses floor-only or furnished tabletop placement based on object affordance. The character is prefitted to G1. Kling 2.5 Turbo Pro generates five- or ten-second 1,920×1,080 videos at twenty-four frames per second. Optional ending frames control final placement, and chaining the previous final frame supports longer sequences. Such chaining remains dependent on temporal/appearance consistency rather than guaranteeing it.

Four original scene-configuration views: floor-only and furnished tabletop environments.
Figure 6 (PDF): Candidate Scene Configurations. Two pre-built 3D scene templates used to place objects: an indoor floor-only environment for ground-level objects (e.g., a sofa), and a furnished room with a table for tabletop objects (e.g., a frypan). A VLM selects between the two based on each object’s typical affordances.

A.2 Generation Runtime

A five-second, 121-frame sequence takes approximately fourteen minutes: one video generation, two human estimation, one object tracking, two depth/segmentation preprocessing, and eight joint optimization. Local stages are measured on an A100, while video generation calls the Kling API, so “single A100” does not describe all remote generation compute. The timings do not supply a filtered-data yield or total monetary cost.

Stage Time
Video Generation (Kling API) ∼\sim1 min
Human Motion Estimation ∼\sim2 min
Object Pose Tracking ∼\sim1 min
Optimization Preprocessing ∼\sim2 min
Joint Optimization ∼\sim8 min
Total ∼\sim14 min
Table 4: Runtime Breakdown. Wall-clock time per stage of the GRAIL 4D HOI generation pipeline, measured on a single NVIDIA A100 GPU for a 5-second, 121-frame sequence.

A.3 Failure Case Filtering

The filter measures the fraction of SAM2 object-mask pixels not covered by a rendered predicted silhouette, aggregated over frames, and rejects error above 0.2. This is a one-sided coverage error, not intersection-over-union: an overly large silhouette is not penalized symmetrically. It detects some tracking/appearance failures but is not a complete physical-validity test.

Equation 10
eℳ=∑t=1TSum​((1−ℳt)⋅ℳ^t)Sum​(ℳt),e^{\mathcal{M}}=\sum_{t=1}^{T}\frac{\text{Sum}\left((1-{\mathcal{M}}_{t})\cdot\widehat{\mathcal{M}}_{t}\right)}{\text{Sum}\left({\mathcal{M}}_{t}\right)}, (10)

Appendix B Task-General Loco-Manipulation Tracking Details

Tracking details distinguish manipulation adaptation from scene-conditioned finetuning. Original observation and reward tables are retained, including differences between their implementation-level gates and the main conceptual description.

B.1 Object-Aware Adaptor

The manipulation adaptor is a 512/256/128 SiLU network producing sixty-four latent-residual values plus two sigmoid-thresholded hand primitives. Residual scale is 0.1 before finite scalar quantization, and SONIC remains frozen. A privileged critic receives full body state/contact flags. Episodes initialize within the first thirty reference frames before labeled contact and terminate on excessive object-height, root-height, or orientation error. PPO uses actor learning rate 2×10⁻⁵, critic 10⁻³, discount 0.99, GAE 0.95, and the complete retained optimization settings.

Term Description Dim Used
Proprioception
Joint position robot joint angles 43 / 29 Both
Joint velocity robot joint angular velocities 43 / 29 Both
Base angular velocity in body frame 3 Both
Previous action last meta-action 66 / 29 Both
Base linear velocity in body frame 3 O
Gravity direction in body frame 3 S
Reference motion targets
Motion anchor position target root position in body frame 3 O
Motion anchor orientation target root orientation 6D 6 O
Current command ref. joint position + velocity, current frame 58 O
Multi-future command ref. joint position + velocity, 10 future frames 580 O
Multi-future anchor ori 10 future root orientations 6D 60 O
Object state
Object position current object pos in body frame 3 Both
Object orientation 6D current object ori in body frame 6 Both
Object pos delta (10 fut.) reffut−simcur\mathrm{ref}_{\mathrm{fut}}-\mathrm{sim}_{\mathrm{cur}}, position 30 Both
Object ori delta (10 fut.) relative rotation, 6D 60 Both
Target object position reference object pos in base frame 3 O
Object BPS encoding per-object shape descriptor 10 O
Table position in body frame 3 O
Table orientation 6D in body frame 6 O
Hand-object contact
Hand-to-object transform per right hand (3 pos + 6 ori) 9 O
Fingertip contact forces 3D force per fingertip 12 O
Scene
Local height map 11×1111{\times}11 terrain zz in body frame 121 S
Table 5: Policy Observations. Observations for the object-aware adaptor (O) and the scene-aware tracker (S), grouped by category. When two dims are listed (e.g., 43 / 29), the first applies to the object-aware policy (43-DoF G1 including fingers) and the second to the scene-aware policy (29-DoF G1, no fingers). Proprioceptive and previous-action observations are stored as 10-frame histories; instantaneous values are listed above.
Complete original reward table with all formulas, signs, weights, and tracker applicability.
Table 6: Reward Terms. Rewards for the object-aware adaptor (O) and the scene-aware tracker (S), organized by the four categories from Sec. 3.3. The rightmost column indicates which tracker uses each term. For grasp terms, the listed weight is the right-hand value; in object-aware training, the left hand uses half the listed weight. 𝟙​{Ct}\mathbb{1}\{C_{t}\} is the per-frame motion-label contact indicator that gates the grasp finger-direction and contact-center terms. The object pose-tracking reward is additionally gated by a simulated finger–object contact indicator, so it is active only while the hand is in contact with the object. For anti-shake, ℬ={\mathcal{B}=\{left wrist, right wrist, head}\} with deadzone τ=1.5\tau=1.5 rad/s. Per-term Gaussian-kernel bandwidth σ\sigma varies by term and is given in the released config. Negative weights indicate penalties.

B.2 Scene-Aware Tracker

Scene tracking uses an eleven-by-eleven raycast grid over 1.5 metres, at 0.15-metre spacing, transformed into the yaw-aligned robot frame. The appendix prose describes an 11×11×3 position tensor, while Table 5 lists 121 height values; both source representations are retained rather than reconciled without code. A three-layer convolutional encoder and fusion network condition the finetuned controller, with kinematic reconstruction weight 0.01 and full-motion reset sampling. Manipulation-specific rewards are disabled.

B.3 Training Cost

A full task-family run takes about thirty hours on sixty-four L40 GPUs and jointly trains 2,000–4,000 motions. The reported 0.5–0.9 minutes per motion divides cluster wall time by pool size; it is not single-GPU compute or the cost to learn an isolated new motion. Warm-started in-family finetuning typically takes six thousand iterations, roughly six hours, while cost per added motion depends on how many are incorporated.

Appendix C Experiment Details

Additional comparisons document baseline initialization, a human preference study, qualitative trajectories, and reconstruction-loss ablations.

C.1 Human-Object Interaction Generation

CHOIS and HOIDiff are training-based interaction generators. DAViD uses the same Kling setting, but its initial frame is manually selected from twenty-four generated candidates per object. A thirty-person study presents three of four methods per trial; GRAIL wins 74.7% affordance and 70.9% physical-plausibility picks, close to the 75% maximum under that sampling scheme. Preferences complement rather than replace physics or hardware evaluation.

Removing projection, depth, or contact refinement lowers downstream tracking success from 81.4% to 41.6%, 42.6%, or 53.3%. Some ablations improve individual smoothness or penetration proxies, so minimizing any one reconstruction metric does not ensure executable interaction. The full model has the best downstream tracking in this comparison.

Methods Aff. Real. ↑\uparrow Phys. Real. ↑\uparrow
HOIDiff (Peng et al., 2025) 2.0% 1.9%
CHOIS (Li et al., 2024) 12.2% 16.8%
DAViD (Kim et al., 2025a) 11.2% 10.4%
Ours 74.7% 70.9%
Table 7: User Study. 30-participant pick rates for the most appropriate object affordances (Aff. Real.) and the most physically plausible motion (Phys. Real.) on the shared 20-object evaluation set. Pick rates have a theoretical upper bound of 75% under 3-of-4 random sampling.
Refer to caption
Figure 6: Qualitative Comparison. Representative 4D HOI sequences from each method on the shared 20-object evaluation set. GRAIL produces more coherent motions with accurate contact and natural hand poses, while baseline methods often yield unrealistically flat or static hand configurations.
Reconstruction Quality Tracking Quality
Methods Contact ↓\downarrow Pen. ↓\downarrow MPJPE ↓\downarrow Human Smo. ↓\downarrow Obj Smo. ↓\downarrow SR ↑\uparrow ObjPos ↓\downarrow MPJPE-L ↓\downarrow
w/o LprojL_{\text{proj}} 0.016 1.93% 16.70 0.0023 0.0011 41.6% 0.374 47.1
w/o LdepthL_{\text{depth}} 0.017 1.97% 4.35 0.0020 0.0011 42.6% 0.372 49.3
w/o LcontL_{\text{cont}} 0.024 1.52% 4.81 0.0022 0.0009 53.3% 0.332 52.4
Ours (Full) 0.015 1.81% 4.89 0.0020 0.0009 81.4% 0.135 41.8
Table 8: Ablation Study on Reconstruction Losses. Reconstruction quality (Contact, Pen., MPJPE in pixel space, motion smoothness) and downstream tracking quality (SR, ObjPos, MPJPE-L) for each loss ablation. Each loss targets a different failure mode in reconstruction; the full model achieves the best downstream tracking despite not minimizing every reconstruction proxy in isolation.
Concept demonstration

Open image ↗