At a glance
NeuHMR refines a human mesh estimated from monocular video using a rendered avatar, silhouette agreement, and image-feature matches for body joints. Its aim is to avoid blindly fitting errors made by a separate 2D keypoint detector.
A pretrained generalizable human NeRF provides the appearance model; reliable anchor frames provide joint features; a small motion network predicts pose corrections. This edition covers the complete main paper and preserves all its figures, tables, and equations. The separately linked supplement remains unavailable for local review.
1. Introduction
Accurate 3D motion labels usually require costly multi-view capture. Image-based mesh estimators therefore generalize imperfectly. Test-time refinement can use 2D keypoints as extra evidence, but confidently wrong detections can pull an otherwise reasonable 3D pose in the wrong direction [4] [50].
NeuHMR instead renders a person-specific avatar and compares it with the target video. It reduces reliance on detector-generated 2D targets while still depending on initial mesh estimates, learned appearance features, and segmentation masks. Figure 1 preserves the motivating example.
3. Method
Figure 2 retains the complete pipeline and original mathematical caption: train the generalizable renderer, adapt its appearance to the target video, register anchor joint features, then optimize pose corrections under three visual constraints.
3.1 Problem setting
The target is a sequence of SMPL body meshes, represented through its 24 major joints and rotations. NeuHMR starts from an existing estimator such as HMR2.0 rather than reconstructing motion from scratch [28].
Radiance-field rendering
Equations 1–3 preserve the radiance/density mapping, transmittance-weighted ray samples, and accumulated color and occupancy. They turn a 3D appearance field into image evidence that can be compared with a video frame.
Equations
Human deformation
A canonical body is mapped into each observed pose through bone rotations, translations, and skinning weights. Equation 4 retains the weighted deformation rule. The same representation makes the appearance model animatable as pose changes.
Equation
3.2 Generalizable Human NeRF
The Generalizable Human NeRF predicts an appearance model for a new video. Its target is monocular inference without fitting a new renderer entirely from scratch. This does not imply monocular-only training: the evaluation renderer is pretrained using multi-view, accurately posed captures.
Equation
Architecture
Image features are associated with projected SMPL mesh vertices. A SparseConvNet and transformer then predict radiance and density for sampled 3D points, following GP-NeRF. Features from selected frames provide different visible body regions; the target renderer conditions on two input frames.
Two conditioning images are distinct from the larger anchor set used for joint-feature registration. Success depends on useful views and reasonably accurate initial geometry.
Equation
Training losses
Equation 7 combines RGB reconstruction with a half-weighted squared occupancy-to-mask error. The latter discourages incomplete rendered bodies across subjects. Masks can come from an external segmentation model such as SAM [19].
Equation
3.3 NeuHMR
For a new target video, the method first selects reliable anchors and registers joint features, then uses the frozen appearance model to refine poses. The renderer remains fixed during the pose-optimization phase.
3.3.1 Anchor features
The method assumes some frames have relatively accurate initial estimates. Those frames supply appearance evidence for correcting harder frames; there is no independent ground-truth accuracy test at inference.
Anchor selection
Candidate anchors cover diverse body orientations through farthest-point sampling on projected orientation vectors. Photometric and geometric reconstruction losses then rank their reliability. Broad orientation coverage helps avoid registering a joint only from one side of the body.
Joint feature registration
A rendered depth map supplies a visibility test: a joint’s depth must lie within tolerance 0.2 of the rendered surface at its projected position. Equation 8 retains that rule. Visible anchor features are averaged per joint using Equation 9.
These features become reference descriptors, rather than 2D detector labels. The construction depends on anchor geometry, visibility, and feature quality; the main paper does not specify a fallback when a joint has no visible anchor.
Equations
3.3.2 Test-time pose optimization
Pose refinement combines color, silhouette, and joint-feature agreement in image space. The learned appearance model is frozen so optimization changes the motion representation.
Motion fields
A four-layer MLP maps a radial-basis encoding of video progress to residual joint/global rotations and translation. Rotations compose multiplicatively, while translation adds, as retained in Equation 10. Although time indexes the network, the paper does not impose an explicit temporal-consistency objective.
Equation
Photometric constraint
A per-pixel squared color difference encourages the rendered avatar to match the target. It supplies low-level appearance evidence but can be misleading when rendering is blurry or clothing/illumination changes.
Geometric constraint
Equation 11 uses a soft intersection-over-union loss between rendered occupancy and the human mask. This differs from the squared occupancy error used in renderer training: the authors report that reusing the latter for pose refinement can shrink the rendered person.
Equation
Joint matching
Around each predicted 2D joint, sample Gaussian candidate locations and select the image feature most similar to the registered anchor descriptor. Equation 12 pulls the projected joint toward that selected candidate. Candidates are refreshed as optimization proceeds.
The local search assumes initial errors are moderate enough that useful joint positions lie nearby. Figure 3 preserves the feature-matching and visibility schematic.
Equation
Combined objective
Equation 13 sums the three losses. Implementation weights are 2.5 for color, 0.001 for geometry, and 0.003 for joint matching. The implementation also adds a rotation regularizer for head, hands, and feet beyond the three-term displayed objective.
Equation
4. Experiments
The main paper evaluates one held-out indoor subject and a broader in-the-wild motion benchmark. Additional experiments are referenced in the separate supplement, whose attachment has not been recovered for this edition.
4.1 Datasets
ZJU-MoCap pretraining uses all camera angles for eight subjects: 313, 315, 377, 386, 387, 390, 392, and 394. Pose evaluation uses the first camera view of held-out subject 393. The paper presents this as a proof of concept, not a broad indoor generalization study.
EMDB protocol 1 covers seventeen challenging activities by eight actors, indoors and outdoors. Some videos only partially show the body, which challenges avatar rendering and subsequent pose refinement [18].
4.2 Evaluation
The principal initializers are HMR2.0 and HybrIK; tables also retain VIBE and optimization baselines. MPJPE and PVE measure joint and vertex errors in millimeters after converting estimates to the camera frame. Lower values are better.
SLAHMR is omitted from the EMDB table because it fails on most long videos in the authors’ setup. Segmenting those videos into shorter clips would require over a thousand runs according to their footnote; that alternative was not evaluated.
4.3 Implementation
The renderer uses ConvNeXt-tiny features at one-quarter image resolution, two conditioning frames, and 32 samples per ray. AdamW trains it for 35,000 steps with weight decay 10⁻⁴ and learning rate decreasing from 10⁻⁴ to 10⁻⁵.
Target videos are reduced to one-quarter resolution for ZJU-MoCap and one-eighth for EMDB. The anchor count is T/10. The motion MLP uses 100-dimensional radial-basis features; joint search draws up to 128 candidates with σ=4. Five optimization epochs use AdamW with cosine-decayed learning rate 10⁻⁶ to 10⁻⁷.
Only sixteen of SMPL’s 24 joints contribute to matching: crowded trunk, hand, and foot joints are excluded because low-resolution features can be difficult to distinguish.
4.4 Results
On the held-out ZJU-MoCap subject, NeuHMR achieves the lowest reported joint error: 74.2 mm with HybrIK initialization. Vertex error does not uniformly improve: the table reports 86.5 mm for initial HybrIK and 86.9 mm after refinement. With HMR2.0 initialization, NeuHMR’s 89.6 mm vertex error is higher than SLAHMR’s 87.1 mm.
On EMDB, HybrIK refinement lowers joint/vertex errors from 103.7/117.2 to 95.4/109.6 mm; HMR2.0 refinement lowers them from 115.0/129.7 to 105.7/125.2 mm. NeuHMR leads the methods reported in that table. Figure 4 retains the full selected qualitative comparisons.
The original results prose contains errors: it gives initial HybrIK vertex error as 85.5 rather than Table 1’s 86.5, and says GLAMR has lower errors although the table shows higher errors. These notes follow the retained tables.
4.5 Joint-matching analysis
Figure 5 compares pelvis matches with ViTPose estimates above confidence 0.5 across EMDB. The reported mean 2D L1 errors are 0.008 versus 0.035, about a fourfold difference. These values are not the millimeter 3D errors used in the main tables; the main text does not clearly specify their normalization.
The study concerns the pelvis, chosen for visibility, and does not establish the same improvement for every joint. ViTPose can be more accurate on simple poses while making larger high-confidence mistakes on some frames.
Running time
Table 3 reports 0.8 seconds per frame for anchor registration and 13.5 for pose optimization, compared with GLAMR’s 0.1 and SLAHMR’s 31.6 seconds. These measurements exclude pretrained-model training. NeuHMR is therefore faster than the tested SLAHMR setup but much slower than GLAMR and unsuitable for a real-time interpretation of these reported timings.
Ablations
Full NeuHMR reaches 95.4/109.6 mm joint/vertex error. Removing registered anchor features gives 101.1/116.5; removing joint matching gives 100.7/116.4. Removing color or silhouette losses also degrades both metrics.
The color-free variant remains relatively competitive, consistent with the authors’ explanation that changing appearance can weaken photometric guidance. The ablation supports complementary signals rather than color matching alone.
5. Conclusion and discussion
Performance depends on avatar-rendering quality, stable appearance, useful visibility, and adequate initial pose estimates. Incomplete observations or changes in appearance can degrade the guidance. The lack of an explicit temporal-consistency objective can leave frame-to-frame instability despite the time-indexed motion field.
Conclusions
The main contribution is a rendering- and feature-guided alternative to fitting pseudo-labeled 2D joints. The results show useful refinement on the tested initializers and datasets, with substantial test-time computation and several remaining external-model dependencies.
Acknowledgments
Support includes NIH grants R01AG089169 and P41EB027060, the Gordon and Betty Moore Foundation, the Jaswa Innovator Award, Stanford HAI and its graduate fellowship, and the Stanford Wu Tsai Human Performance Alliance.
Source availability
The reviewed source is the eleven-page main PDF linked from Stanford NMBL’s publication list. It contains five figures, four tables, thirteen numbered equations, and 56 references, all retained here.
The paper’s separate supplementary material is blocked by a verification page in the available download route. Its additional figures, symbol table, and experiments have not been incorporated. This is a main-paper edition with an explicitly outstanding supplement.