Literature
Display
← All papers

Rendering Humans Behind Occlusions

Tiange Xiang · Adam Sun · Scott Delp · Kazuki Kozuka · Li Fei-Fei · Ehsan Adeli

· IEEE Transactions on Pattern Analysis and Machine Intelligence ↗

Computer vision8 min text · CondensedOriginal paper ↗
Contents

At a glance

Wild2Avatar reconstructs a moving person from an occluded monocular video by representing foreground obstacles, the human, and background separately. A canonical signed-distance surface models the person; compositional rendering, a mask-guided occlusion loss, and a geometry-completeness penalty reduce obstacle textures and missing body parts in the avatar.

The evaluation uses fourteen videos: six OcMotion sequences, three real-world clips, and five videos with synthetic occluders. Each video requires its own optimization, camera/body-pose initialization, and human segmentation masks. Plausible completion of unseen body regions is demonstrated qualitatively rather than established as recovery of their true appearance.

I Introduction

A foreground object can be mistaken for part of a person when a neural renderer fits an occluded video. Masking away those pixels avoids direct contamination but also removes supervision for the hidden body. Wild2Avatar addresses both problems through explicit three-part scene representation and geometric regularization.

The method extends OccNeRF and builds on Vid2Avatar’s signed-distance human model. Its target is the person’s appearance and geometry, not a photorealistic reconstruction of every obstacle and background region. The reported benefits concern these video-specific reconstruction settings.

Refer to caption
Fig. 1: In this paper, we introduce Wild2Avatar, a method to render high fidelity human avatars from in-the-wild monocular videos behind occlusions. To achieve photo-realistic rendering, we parameterize the scene into three parts: occlusion→\rightarrowhuman→\rightarrowbackground and decouple the renderings through novel optimization objectives. Faces in the figure are blurred for anonymity.

II Related Work

Related work spans articulated human representations, decomposition of scenes into separate fields, and methods that explicitly handle visibility or occlusion.

II-A 3D Human Modeling and Rendering

Human radiance fields use the SMPL body prior to map a moving body into a common canonical space. HumanNeRF, MonoHuman, and related systems learn appearance across poses; Gaussian-splatting methods offer faster optimization and rendering. Strong results on unobstructed footage do not by themselves resolve missing supervision under occlusion.

II-B Rendering via Scene Decomposition

Layered scene models render components separately and composite them along camera rays. NeRF++ parameterizes unbounded background through an inverted sphere; Vid2Avatar applies a related separation to dynamic humans and background. Wild2Avatar adds an explicit foreground-occlusion region. The journal version also compares with HSR, which models human–scene relationships in a shared coordinate system.

II-C Occlusion Handling

Some occlusion-aware NeRF methods need multiple views to infer visibility, which differs from a moving human filmed monocularly. HOSNeRF attaches object bones to a human skeleton and addresses interacting dynamic objects; the authors report implementation and scene-coverage limitations that prevented it from serving as a main baseline here.

II-D Connection to Our Previous Work

This journal extension revises OccNeRF’s scene and human representations, adds more challenging real-world sequences, and studies reduced visibility and joint pose refinement. Its core change is explicit occlusion–human–background composition with losses for decoupling and completeness.

Refer to caption
Fig. 2: Wild2Avatar  renders occluded humans from a monocular video. It parameterizes the 3D scene into three parts: occlusion, human, and background, in order from closest to farthest from the camera. The human and the occlusions/background are individually modeled via separate neural radiance fields. The human is parameterized in a bounded space between an inner sphere Π\Pi with radius 𝐫\mathbf{r} and an outer sphere with radius 𝐑\mathbf{R} by first deforming ray samples into a canonical space with the help of the pre-computed body pose PP. The canonical points 𝐱\mathbf{x} are passed into a rendering network ℱf​g\mathcal{F}^{fg} to learn the radiance 𝐜\mathbf{c} and distance 𝐬\mathbf{s} to the surface of the human, which can then be rendered through SDF-based volume rendering. The unbounded background samples outside the outer sphere are represented as coordinates on the surface of the outer sphere along with their inverted distances. Another rendering network ℱs​c​e​n​e\mathcal{F}^{scene} is used to learn the radiance and density for the background ray samples (shown as green dots ). The space of occlusion is determined as the interval between the camera and the inner sphere π\pi. We parameterize ray samples as coordinates on the surface of π\pi and the negation of the inverted distances to the center of the inner sphere. We rely on the same network ℱs​c​e​n​e\mathcal{F}^{scene} to render the occlusions. The three renderings are sequentially aggregated and supervised on a combination of losses, in which we specifically encourage the decoupling of the occlusion from the human through ℒo​c​c\mathcal{L}_{occ} and penalize the incompleteness of human geometry through ℒc​o​m​p\mathcal{L}_{comp}.

III Methods

The system maps articulated body samples into canonical space, renders the human with an implicit surface model, and composites separately modeled occluder and background contributions. Figure 2 gives the full pipeline; Equations 1–8 retain the original deformation, rendering, and loss definitions.

III-A Preliminaries and Background

A neural radiance field predicts color and density at spatial samples. For a connected human surface, Wild2Avatar instead learns a signed-distance function whose zero level set represents the body. Forward and backward skinning use SMPL-derived bone transformations and weights to connect canonical and observed coordinates. The original skinning equation is retained.

Equation 1
𝐱o=∑iwci​𝐁i​𝐱c,𝐱c=(∑iwoi​𝐁i)−1​𝐱o,\mathbf{x}_{o}=\sum_{i}w^{i}_{c}\mathbf{B}^{i}\mathbf{x}_{c},\ \ \ \ \ \ \ \ \mathbf{x}_{c}=(\sum_{i}w^{i}_{o}\mathbf{B}^{i})^{-1}\mathbf{x}_{o}, (1)

III-B Occlusion-aware Scene Parameterization

An outer sphere bounds the human-centered scene. Beyond it, the background is encoded by a direction on the sphere plus inverse distance, following NeRF++. Within it, an inner sphere separates the camera-side occlusion region from the human. Its radius is chosen by binary search so the outer camera rays intersect it. This is a constrained representation of potential foreground obstacles, not a complete reconstruction of arbitrary objects.

Human samples are transformed into canonical space. Signed distances become densities through a scaled Laplace cumulative distribution function, then the field is volume rendered. A shared scene network renders background and occlusion using separate optimized per-frame latent codes. Sphere coordinates are normalized, and a negative inverse-distance coordinate distinguishes the bounded occlusion encoding from the background encoding.

The final color combines occlusion first, then human, then background, using the remaining transmittance for farther components. The original formulas and their notation are preserved. The text references a Figure 3(C) normalization panel, but the matching PDF contains only panels A and B; no additional panel is inferred.

Refer to caption
Fig. 3: Different Scene parameterizations in 2D simplifications. (A) The regular two-part paradigm [9] parameterizes the foreground within a sphere Π\Pi with radius 𝐑\mathbf{R} and background outside the sphere. (B) Our proposed occlusion-aware paradigm parameterizes the scene into three sequential parts. The occlusion is explicitly modeled within the interval space between the camera and an inner sphere π\pi with a radius 𝐫<𝐑\mathbf{r}<\mathbf{R}.
Equations 2–4
𝐂=∑i𝐜i​(1−exp⁡(−σi​δi))​∏j<iexp⁡(−σj​δj),\mathbf{C}=\sum_{i}\mathbf{c}^{i}(1-\exp(-\mathbf{\sigma}^{i}\delta^{i}))\prod_{j<i}\exp(-\mathbf{\sigma}^{j}\delta^{j}), (2)
τi=exp(−∑j<iσj(1−exp(−σj))),\tau^{i}=\exp\left(-\sum_{j<i}\sigma^{j}\left(1-\exp(-\sigma^{j})\right)\right), (3)
𝐱bg=[xbg,ybg,zbg,1/d]\mathbf{x}_{\text{bg}}=[x_{\text{bg}},y_{\text{bg}},z_{\text{bg}},1/d](𝐨⋅𝐝)2−‖𝐨2‖+r2(\mathbf{o}\cdot\mathbf{d})^{2}-||\mathbf{o}^{2}||+r^{2}
‖𝐨+t​𝐝‖=𝐫,‖𝐝‖2​t2+2​(𝐨⋅𝐝)​t+‖𝐨‖2−𝐫=0→,t=−‖𝐨⋅𝐝‖2+(‖𝐫2‖−‖𝐨2‖)​‖𝐝2‖−‖𝐨⋅𝐝‖‖𝐝2‖.\begin{gathered}||\mathbf{o}+t\mathbf{\mathbf{d}}||=\mathbf{r},\\ ||\mathbf{d}||^{2}t^{2}+2(\mathbf{o}\cdot\mathbf{d})t+||\mathbf{o}||^{2}-\mathbf{r}=\vec{0},\\ t=\frac{-\sqrt{||\mathbf{\mathbf{o}}\cdot\mathbf{d}||^{2}+\left(||\mathbf{r}^{2}||-||\mathbf{\mathbf{o}}^{2}||\right)||\mathbf{d}^{2}||}-||\mathbf{\mathbf{o}}\cdot\mathbf{d}||}{||\mathbf{d}^{2}||}.\end{gathered} (4)
{xo​c​c,yo​c​c,zo​c​c}=𝐨+t​𝐝\{x_{occ},y_{occ},z_{occ}\}=\mathbf{o}+t\mathbf{d}
Refer to caption
Fig. 4: Workflow for the occlusion decoupling loss ℒo​c​c\mathcal{L}_{occ}. ‘∼\sim’ indicates binary inversion and ‘//’ indicates gradient stopping.
Equation 5
𝐂=𝐂o​c​c+(1−αo​c​c)​𝐂f​g+(1−αo​c​c)​(1−αf​g)​𝐂b​g.\mathbf{C}=\mathbf{C}_{occ}+(1-\alpha_{occ})\mathbf{C}_{fg}+(1-\alpha_{occ})(1-\alpha_{fg})\mathbf{C}_{bg}. (5)

III-C Optimization Objectives

Training combines RGB photometric error, an Eikonal surface regularizer, a human/background decomposition loss, an occlusion-decoupling loss, and geometry completeness. The occlusion target is the predicted human occupancy above a 0.1 threshold where the segmentation mask says the human is not visible. Its gradient is stopped; weighted binary cross-entropy gives more weight to the relatively rare occluded pixels.

The completeness penalty drives samples near the current body mesh toward the learned zero-level surface. That mesh starts from SMPL and is updated during optimization. This encourages a filled body even where image supervision is weak, but inherits the body prior and does not establish the true hidden clothing or texture.

Refer to caption
Fig. 5: Qualitative comparisons with Vid2Avatar [1] on 6 occluded sequences in the OcMotion dataset [57].
Equations 6–8
ℒo​c​c=BCE​(αo​c​c,(1−𝐌)​(sg​(αf​g)>ϵ)),\mathcal{L}_{occ}=\texttt{BCE}\left(\alpha_{occ},(1-\mathbf{M})(\texttt{sg}(\alpha_{fg})>\epsilon)\right), (6)
ℒc​o​m​p=1|𝒳n​e​a​r|​∑x∈𝒳n​e​a​r|𝐬⁡(x)|,\mathcal{L}_{comp}=\frac{1}{|\mathcal{X}_{near}|}\sum_{x\in\mathcal{X}_{near}}|\mathbf{s}(x)|, (7)
ℒr​g​b+λe​i​k​ℒe​i​k+λd​e​c​ℒd​e​c+λo​c​c​ℒo​c​c+λc​o​m​p​ℒc​o​m​p.\mathcal{L}_{rgb}+\lambda_{eik}\mathcal{L}_{eik}+\lambda_{dec}\mathcal{L}_{dec}+\lambda_{occ}\mathcal{L}_{occ}+\lambda_{comp}\mathcal{L}_{comp}. (8)

IV Experiments

Experiments compare reconstruction quality, body completeness, perceived rendering quality, pose sensitivity, and reduced visibility. Most real captures use a stationary camera; additional synthetic-occlusion sequences test moving-camera cases.

Refer to caption
Fig. 6: Comparisons with Vid2Avatar [1] on 3 occluded in-the-wild monocular videos. Occluded humans from novel views are not available. We manually blurred the faces for the in-the-wild videos to maintain anonymity.
Refer to caption
Fig. 7: Comparisons with HSR [40] on 2 synthesized occluded videos. Occluded humans from novel views are not available.

IV-A Datasets

Six challenging OcMotion sequences are selected from 48, using 100 frames per sequence and the supplied cameras, poses, and SMPL parameters. Human masks come from Segment Anything. Three additional real videos comprise two Creative Commons YouTube clips and one phone recording, sampled at approximately 150 frames according to the source; these use estimated cameras and bodies.

Five NeuMan videos receive textured, static 3D box occluders placed consistently with the scene and camera trajectory. These bring the total to fourteen. The dataset section identifies SLAHMR for estimated poses, whereas the later pose ablation refers to DPMesh; this difference in the reported preprocessing is retained rather than resolved by assumption. One of the real-world clips also uses a moving camera.

IV-B Implementation Details

Optimization is per video. The human field has an eight-layer, 256-feature implicit network and a four-layer, 256-feature color network. The shared scene field uses an eight-layer implicit network and one-layer color network, also with 256 features. Occlusion and background each have 32-dimensional per-frame latent codes.

Training samples 512 rays within the 2D human bounding box, using 64 samples per ray for training and 128 for evaluation. Vid2Avatar and Wild2Avatar receive the same segmentation masks. Loss weights are 0.1 for Eikonal, 0.003 for decomposition, 0.2 for completeness, and 0.1 for occlusion. Adam starts at 5×10⁻⁴ and halves the rate at the source’s stated 200th and 500th steps.

OcMotion In-the-wild
Methods qualityv​i​s\texttt{quality}_{vis} comp. qualityu​s​e​r\texttt{quality}_{user} qualityv​i​s\texttt{quality}_{vis} comp. qualityu​s​e​r\texttt{quality}_{user}
Vid2Avatar [1] 12.74 0.74 3.7±\pm1.3 8.84 0.66 3.6±\pm1.0
OccNeRF500 [8] 12.33 0.76 4.0±\pm0.9 7.98 0.70 3.2±\pm1.1
Wild2Avatar 13.01 0.81 6.6±\pm0.3 9.68 0.74 7.0±\pm0.8
TABLE I: Quantitative comparisons on the three datasets. We color cells that have the best metric values.
Methods PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow
HSR [40] 18.77 0.9005 68.80
Wild2Avatar 22.47 0.9624 44.36
TABLE II: Quantitative comparisons on the synthesized videos. We color cells that have the best metric values.

IV-C Evaluations

Main comparisons use Vid2Avatar and OccNeRF with shared human masks. OccNeRF is shown with both 100 and 500 training frames, because its stable regime generally needs more frames. HSR is additionally compared on two synthesized videos. The authors report that camera assumptions or implementation restrictions prevented full NeuMan and HOSNeRF comparisons; these are limitations of the evaluated baseline set.

Reported Wild2Avatar training takes 48 hours on one 24 GB GPU. The paper also quotes larger multi-GPU times for NeuMan and HOSNeRF, but these are not a hardware-normalized speed benchmark.

Visible-part PSNR measures appearance only where the mask shows the person. Silhouette IoU against SMPL measures a proxy for body completeness, not detailed surface accuracy. Twenty-five non-expert raters score renderings from 0 to 10. Synthetic cases can compare against the original unoccluded input views; Table II also reports LPIPS alongside PSNR and SSIM.

Refer to caption
Fig. 8: Comparison against OccNeRF [8] trained on 100 and 500 training frames.
Refer to caption
Fig. 9: Geometry reconstructions. We manually blurred the faces for the in-the-wild videos for maintaining anonymity.
Refer to caption
Fig. 10: Visualizations of scene decomposition. Wild2Avatar separately renders occlusion, human, and background.

IV-D Results

On OcMotion, Wild2Avatar achieves visible PSNR 13.01, silhouette IoU 0.81, and mean user score 6.6, compared with Vid2Avatar’s 12.74, 0.74, and 3.7. On the real-world group, the corresponding values are 9.68, 0.74, and 7.0 versus 8.84, 0.66, and 3.6. Table I retains the full OccNeRF comparison and reported score dispersion.

For the two synthesized-video HSR comparisons, Table II reports PSNR 22.47 versus 18.77 and SSIM 0.9624 versus 0.9005. Its LPIPS entries are 44.36 versus 68.80; the table does not specify their scaling, so those values are retained without conversion.

The images show less occluder texture on the body and more complete silhouettes, including novel views. They support improvement in these evaluated scenes, while the small selected dataset, limited baseline coverage, and proxy metrics restrict broader claims.

IV-E Geometry Reconstructions

Figure 9 overlays reconstructed surface normals on input frames. It provides qualitative geometry evidence across several occlusions; no additional quantitative 3D surface-error measure is reported here.

IV-F Ablation Studies

Replacing the explicit occlusion parameterization with visible-only fitting leaves artifacts. Removing the occlusion loss leaves hidden regions incompletely recovered; removing the completeness penalty can distort geometry and pose. Figure 11 preserves the visual examples and their numerical annotations. Its middle panel is labeled L_dec, although the surrounding text describes removing L_occ; the source does not resolve that discrepancy.

A further OcMotion experiment jointly optimizes estimated SMPL pose and the neural field. Figure 13 shows progressive correction of limb rotations, with the final predicted-pose example at visible quality 13.55 and completeness 0.80 versus 13.84 and 0.88 using ground-truth SMPL. This shows some recoverability from pose error, not immunity to inaccurate initialization.

Refer to caption
Fig. 11: Ablation results. Major differences are highlighted.

IV-G Visualizations of Scene Decomposition

Figure 10 separately renders the obstacle, human, and background and their composition. Background and obstacle artifacts are tolerated because the intended output is the human avatar.

IV-H Generative Ability

The visibility experiment progressively removes frames showing the person’s back. Wild2Avatar still produces a plausible complete back when that region is never clearly observed, whereas Vid2Avatar deteriorates more in the displayed comparison. This is qualitative prior-driven completion; unseen texture is not uniquely determined by the input and should not be interpreted as verified reconstruction.

Refer to caption
Fig. 12: Study of generative ability. Qualitative comparison with Vid2Avatar [1] on amount of training images that show the full back of the human.
Refer to caption
Fig. 13: Comparison between joint training with predicted SMPL parameters (left) and GT parameters (right). Inaccurate limb rotations in the predicted SMPL can be progressively corrected through joint optimization.

V Discussion and Conclusion

The decoupling is weakly supervised by human masks and initialized body poses. Poor poses can still produce incomplete or incorrect avatars, and the extra occlusion model adds optimization and inference cost. Joint pose refinement helps in the demonstrated case but does not remove this dependence.

The contribution is a three-component representation and losses that improve occluded-human reconstruction in a small set of fitted videos. Faster radiance fields or Gaussian splatting and stronger pose correction are proposed future directions. Complete, artifact-free rendering in arbitrary scenes is not established by these experiments.

Acknowledgment.

The authors acknowledge NIH grants R01AG089169 and P41EB027060, Panasonic Holdings Corporation, the Gordon and Betty Moore Foundation, the Jaswa Innovator Award, Stanford HAI, and the Stanford Wu Tsai Human Performance Alliance.

VI Biography

The original six author biographies below are retained as publication-time information.

Tiange Xiang is a CS Ph.D. candidate at Stanford University. He is affiliated with Stanford AI Lab & Stanford Vision and Learning Lab. He is advised by Prof. Fei-Fei Li, co-advised by Prof. Scott Delp and Prof. Ehsan Adeli. His research focuses on 3D computer vision & AI for healthcare, which is partially supported by Stanford Human-centered AI fellowship. .
Adam Sun is a senior and 1st-year master’s student at Stanford University. He is interested in 3D computer vision for scene understanding and medical applications. Adam is a student researcher at Stanford Vision and Learning Lab (SVL), where he is advised by Professors Fei-Fei Li and Ehsan Adeli at the Partnership in AI-Assisted Care.
Dr. Scott Delp is the James H. Clark Professor at Stanford University, where he holds appointments in Bioengineering, Mechanical Engineering, and Orthopaedic Surgery. He is the founding chair of Stanford’s Bioengineering Department and directs major initiatives including the Wu Tsai Human Performance Alliance, the Restore Center, and the Mobilize Center. A pioneer in movement science and digital health, he developed widely-used tools like OpenSim and has authored over 325 research papers.
Dr. Kazuki Kozuka is a manager at Panasonic Corporation. He received his Ph.D degrees from Nagoya Institite of Technology in 2009. In 2009, he joined Advanced Technology Laboratories, Panasonic Corporation, Japan, where he was involved in research on neuroscience, image recognition. He was also a visiting scholar in Stanford University between 2016 and 2019. His research interests lie in visual understanding through machine learning, mainly for computer vision and natural language processing.
Dr. Fei-Fei Li is the inaugural Sequoia Professor in the Computer Science Department at Stanford University, and Co-Director of Stanford’s Human-Centered AI Institute. She served as the Director of Stanford’s AI Lab from 2013 to 2018. And during her sabbatical from Stanford from January 2017 to September 2018, Dr. Li was Vice President at Google and served as Chief Scientist of AI/ML at Google Cloud. Since then she has served as a Board member or advisor in various public or private companies.
Dr. Ehsan Adeli is an assistant professor at Stanford University School of Medicine, Department of Psychiatry and Behavioral Sciences, where he directs the Stanford Translational Artificial Intelligence (STAI) in Medicine and Mental Health Lab. He is also affiliated with the computer science department, Stanford Vision and Learning (SVL) lab. With a Ph.D. in computer science & artificial intelligence and postgraduate training in biomedical imaging & computational neuroscience, Ehsan is applying his expertise to solve critical problems in healthcare and neuroscience. To this end, his team focuses on developing innovative methods to enhance our understanding of digital humans within the 3D world.
Concept demonstration

Open image ↗