At a glance
Wild2Avatar reconstructs a moving person from an occluded monocular video by representing foreground obstacles, the human, and background separately. A canonical signed-distance surface models the person; compositional rendering, a mask-guided occlusion loss, and a geometry-completeness penalty reduce obstacle textures and missing body parts in the avatar.
The evaluation uses fourteen videos: six OcMotion sequences, three real-world clips, and five videos with synthetic occluders. Each video requires its own optimization, camera/body-pose initialization, and human segmentation masks. Plausible completion of unseen body regions is demonstrated qualitatively rather than established as recovery of their true appearance.
I Introduction
A foreground object can be mistaken for part of a person when a neural renderer fits an occluded video. Masking away those pixels avoids direct contamination but also removes supervision for the hidden body. Wild2Avatar addresses both problems through explicit three-part scene representation and geometric regularization.
The method extends OccNeRF and builds on Vid2Avatar’s signed-distance human model. Its target is the person’s appearance and geometry, not a photorealistic reconstruction of every obstacle and background region. The reported benefits concern these video-specific reconstruction settings.
II Related Work
Related work spans articulated human representations, decomposition of scenes into separate fields, and methods that explicitly handle visibility or occlusion.
II-A 3D Human Modeling and Rendering
Human radiance fields use the SMPL body prior to map a moving body into a common canonical space. HumanNeRF, MonoHuman, and related systems learn appearance across poses; Gaussian-splatting methods offer faster optimization and rendering. Strong results on unobstructed footage do not by themselves resolve missing supervision under occlusion.
II-B Rendering via Scene Decomposition
Layered scene models render components separately and composite them along camera rays. NeRF++ parameterizes unbounded background through an inverted sphere; Vid2Avatar applies a related separation to dynamic humans and background. Wild2Avatar adds an explicit foreground-occlusion region. The journal version also compares with HSR, which models human–scene relationships in a shared coordinate system.
II-C Occlusion Handling
Some occlusion-aware NeRF methods need multiple views to infer visibility, which differs from a moving human filmed monocularly. HOSNeRF attaches object bones to a human skeleton and addresses interacting dynamic objects; the authors report implementation and scene-coverage limitations that prevented it from serving as a main baseline here.
II-D Connection to Our Previous Work
This journal extension revises OccNeRF’s scene and human representations, adds more challenging real-world sequences, and studies reduced visibility and joint pose refinement. Its core change is explicit occlusion–human–background composition with losses for decoupling and completeness.
III Methods
The system maps articulated body samples into canonical space, renders the human with an implicit surface model, and composites separately modeled occluder and background contributions. Figure 2 gives the full pipeline; Equations 1–8 retain the original deformation, rendering, and loss definitions.
III-A Preliminaries and Background
A neural radiance field predicts color and density at spatial samples. For a connected human surface, Wild2Avatar instead learns a signed-distance function whose zero level set represents the body. Forward and backward skinning use SMPL-derived bone transformations and weights to connect canonical and observed coordinates. The original skinning equation is retained.
Equation 1
| (1) |
III-B Occlusion-aware Scene Parameterization
An outer sphere bounds the human-centered scene. Beyond it, the background is encoded by a direction on the sphere plus inverse distance, following NeRF++. Within it, an inner sphere separates the camera-side occlusion region from the human. Its radius is chosen by binary search so the outer camera rays intersect it. This is a constrained representation of potential foreground obstacles, not a complete reconstruction of arbitrary objects.
Human samples are transformed into canonical space. Signed distances become densities through a scaled Laplace cumulative distribution function, then the field is volume rendered. A shared scene network renders background and occlusion using separate optimized per-frame latent codes. Sphere coordinates are normalized, and a negative inverse-distance coordinate distinguishes the bounded occlusion encoding from the background encoding.
The final color combines occlusion first, then human, then background, using the remaining transmittance for farther components. The original formulas and their notation are preserved. The text references a Figure 3(C) normalization panel, but the matching PDF contains only panels A and B; no additional panel is inferred.
Equations 2–4
| (2) |
| (3) |
| (4) |
Equation 5
| (5) |
III-C Optimization Objectives
Training combines RGB photometric error, an Eikonal surface regularizer, a human/background decomposition loss, an occlusion-decoupling loss, and geometry completeness. The occlusion target is the predicted human occupancy above a 0.1 threshold where the segmentation mask says the human is not visible. Its gradient is stopped; weighted binary cross-entropy gives more weight to the relatively rare occluded pixels.
The completeness penalty drives samples near the current body mesh toward the learned zero-level surface. That mesh starts from SMPL and is updated during optimization. This encourages a filled body even where image supervision is weak, but inherits the body prior and does not establish the true hidden clothing or texture.
Equations 6–8
| (6) |
| (7) |
| (8) |
IV Experiments
Experiments compare reconstruction quality, body completeness, perceived rendering quality, pose sensitivity, and reduced visibility. Most real captures use a stationary camera; additional synthetic-occlusion sequences test moving-camera cases.
IV-A Datasets
Six challenging OcMotion sequences are selected from 48, using 100 frames per sequence and the supplied cameras, poses, and SMPL parameters. Human masks come from Segment Anything. Three additional real videos comprise two Creative Commons YouTube clips and one phone recording, sampled at approximately 150 frames according to the source; these use estimated cameras and bodies.
Five NeuMan videos receive textured, static 3D box occluders placed consistently with the scene and camera trajectory. These bring the total to fourteen. The dataset section identifies SLAHMR for estimated poses, whereas the later pose ablation refers to DPMesh; this difference in the reported preprocessing is retained rather than resolved by assumption. One of the real-world clips also uses a moving camera.
IV-B Implementation Details
Optimization is per video. The human field has an eight-layer, 256-feature implicit network and a four-layer, 256-feature color network. The shared scene field uses an eight-layer implicit network and one-layer color network, also with 256 features. Occlusion and background each have 32-dimensional per-frame latent codes.
Training samples 512 rays within the 2D human bounding box, using 64 samples per ray for training and 128 for evaluation. Vid2Avatar and Wild2Avatar receive the same segmentation masks. Loss weights are 0.1 for Eikonal, 0.003 for decomposition, 0.2 for completeness, and 0.1 for occlusion. Adam starts at 5×10⁻⁴ and halves the rate at the source’s stated 200th and 500th steps.
IV-C Evaluations
Main comparisons use Vid2Avatar and OccNeRF with shared human masks. OccNeRF is shown with both 100 and 500 training frames, because its stable regime generally needs more frames. HSR is additionally compared on two synthesized videos. The authors report that camera assumptions or implementation restrictions prevented full NeuMan and HOSNeRF comparisons; these are limitations of the evaluated baseline set.
Reported Wild2Avatar training takes 48 hours on one 24 GB GPU. The paper also quotes larger multi-GPU times for NeuMan and HOSNeRF, but these are not a hardware-normalized speed benchmark.
Visible-part PSNR measures appearance only where the mask shows the person. Silhouette IoU against SMPL measures a proxy for body completeness, not detailed surface accuracy. Twenty-five non-expert raters score renderings from 0 to 10. Synthetic cases can compare against the original unoccluded input views; Table II also reports LPIPS alongside PSNR and SSIM.
IV-D Results
On OcMotion, Wild2Avatar achieves visible PSNR 13.01, silhouette IoU 0.81, and mean user score 6.6, compared with Vid2Avatar’s 12.74, 0.74, and 3.7. On the real-world group, the corresponding values are 9.68, 0.74, and 7.0 versus 8.84, 0.66, and 3.6. Table I retains the full OccNeRF comparison and reported score dispersion.
For the two synthesized-video HSR comparisons, Table II reports PSNR 22.47 versus 18.77 and SSIM 0.9624 versus 0.9005. Its LPIPS entries are 44.36 versus 68.80; the table does not specify their scaling, so those values are retained without conversion.
The images show less occluder texture on the body and more complete silhouettes, including novel views. They support improvement in these evaluated scenes, while the small selected dataset, limited baseline coverage, and proxy metrics restrict broader claims.
IV-E Geometry Reconstructions
Figure 9 overlays reconstructed surface normals on input frames. It provides qualitative geometry evidence across several occlusions; no additional quantitative 3D surface-error measure is reported here.
IV-F Ablation Studies
Replacing the explicit occlusion parameterization with visible-only fitting leaves artifacts. Removing the occlusion loss leaves hidden regions incompletely recovered; removing the completeness penalty can distort geometry and pose. Figure 11 preserves the visual examples and their numerical annotations. Its middle panel is labeled L_dec, although the surrounding text describes removing L_occ; the source does not resolve that discrepancy.
A further OcMotion experiment jointly optimizes estimated SMPL pose and the neural field. Figure 13 shows progressive correction of limb rotations, with the final predicted-pose example at visible quality 13.55 and completeness 0.80 versus 13.84 and 0.88 using ground-truth SMPL. This shows some recoverability from pose error, not immunity to inaccurate initialization.
IV-G Visualizations of Scene Decomposition
Figure 10 separately renders the obstacle, human, and background and their composition. Background and obstacle artifacts are tolerated because the intended output is the human avatar.
IV-H Generative Ability
The visibility experiment progressively removes frames showing the person’s back. Wild2Avatar still produces a plausible complete back when that region is never clearly observed, whereas Vid2Avatar deteriorates more in the displayed comparison. This is qualitative prior-driven completion; unseen texture is not uniquely determined by the input and should not be interpreted as verified reconstruction.
V Discussion and Conclusion
The decoupling is weakly supervised by human masks and initialized body poses. Poor poses can still produce incomplete or incorrect avatars, and the extra occlusion model adds optimization and inference cost. Joint pose refinement helps in the demonstrated case but does not remove this dependence.
The contribution is a three-component representation and losses that improve occluded-human reconstruction in a small set of fitted videos. Faster radiance fields or Gaussian splatting and stronger pose correction are proposed future directions. Complete, artifact-free rendering in arbitrary scenes is not established by these experiments.
Acknowledgment.
The authors acknowledge NIH grants R01AG089169 and P41EB027060, Panasonic Holdings Corporation, the Gordon and Betty Moore Foundation, the Jaswa Innovator Award, Stanford HAI, and the Stanford Wu Tsai Human Performance Alliance.
VI Biography
The original six author biographies below are retained as publication-time information.
| Tiange Xiang is a CS Ph.D. candidate at Stanford University. He is affiliated with Stanford AI Lab & Stanford Vision and Learning Lab. He is advised by Prof. Fei-Fei Li, co-advised by Prof. Scott Delp and Prof. Ehsan Adeli. His research focuses on 3D computer vision & AI for healthcare, which is partially supported by Stanford Human-centered AI fellowship. . |
| Adam Sun is a senior and 1st-year master’s student at Stanford University. He is interested in 3D computer vision for scene understanding and medical applications. Adam is a student researcher at Stanford Vision and Learning Lab (SVL), where he is advised by Professors Fei-Fei Li and Ehsan Adeli at the Partnership in AI-Assisted Care. |
| Dr. Scott Delp is the James H. Clark Professor at Stanford University, where he holds appointments in Bioengineering, Mechanical Engineering, and Orthopaedic Surgery. He is the founding chair of Stanford’s Bioengineering Department and directs major initiatives including the Wu Tsai Human Performance Alliance, the Restore Center, and the Mobilize Center. A pioneer in movement science and digital health, he developed widely-used tools like OpenSim and has authored over 325 research papers. |
| Dr. Kazuki Kozuka is a manager at Panasonic Corporation. He received his Ph.D degrees from Nagoya Institite of Technology in 2009. In 2009, he joined Advanced Technology Laboratories, Panasonic Corporation, Japan, where he was involved in research on neuroscience, image recognition. He was also a visiting scholar in Stanford University between 2016 and 2019. His research interests lie in visual understanding through machine learning, mainly for computer vision and natural language processing. |
| Dr. Fei-Fei Li is the inaugural Sequoia Professor in the Computer Science Department at Stanford University, and Co-Director of Stanford’s Human-Centered AI Institute. She served as the Director of Stanford’s AI Lab from 2013 to 2018. And during her sabbatical from Stanford from January 2017 to September 2018, Dr. Li was Vice President at Google and served as Chief Scientist of AI/ML at Google Cloud. Since then she has served as a Board member or advisor in various public or private companies. |
| Dr. Ehsan Adeli is an assistant professor at Stanford University School of Medicine, Department of Psychiatry and Behavioral Sciences, where he directs the Stanford Translational Artificial Intelligence (STAI) in Medicine and Mental Health Lab. He is also affiliated with the computer science department, Stanford Vision and Learning (SVL) lab. With a Ph.D. in computer science & artificial intelligence and postgraduate training in biomedical imaging & computational neuroscience, Ehsan is applying his expertise to solve critical problems in healthcare and neuroscience. To this end, his team focuses on developing innovative methods to enhance our understanding of digital humans within the 3D world. |