Literature
Display
← All papers

Multi-Plane Program Induction with 3D Box Priors

Yikai Li · Jiayuan Mao · Xiuming Zhang · William T. Freeman · Joshua B. Tenenbaum · Noah Snavely · Jiajun Wu

· NeurIPS 2020 ↗

Computer Vision9 min text · CondensedOriginal paper ↗
Contents

Overview

The supplementary material derives the inner-box geometry, adapts inference to outer views, and supplies implementation details, runtime comparisons, and additional visual results.

At a glance

Box Program Induction infers an editable representation of repeated image patterns on several planar surfaces from one image. A box prior constrains the scene to an interior with four modeled planes or an exterior with two side walls. Learned vanishing-point and wireframe cues guide a search over plane partitions and repeating programs. The resulting representation supports perspective-aware inpainting, extrapolation, and view changes. Its geometric reconstruction depends on camera and scale assumptions rather than recovering absolute scene dimensions.

1 Introduction

A corridor or building contains both repeated patterns and perspective relationships among multiple surfaces. Programs fitted to a single plane miss this joint structure. BPI combines a restricted box geometry with learned visual proposals and an image-based regularity score. The chosen program partitions and rectifies surfaces, then describes their repeated elements with loops. This makes structured editing possible while restricting the method to scenes whose useful regions fit the planar, repeating assumptions.

Refer to caption
Figure 1: We present Box Program Induction (BPI), which infers a program-like scene representation that simultaneously models repeated structure on multiple 2D planes, 3D positions and orientations of the planes, relative to the camera, all from a single image. The inferred program can be used to guide perspective- and regularity-aware image manipulation tasks, including image inpainting, extrapolation, and view synthesis.

2 Related Work

The related work connects visual program induction with patch-based and learned image editing.

Visual program induction.

Earlier procedural representations cover shapes, scenes, sketches, and image markup. Program-guided editing of natural images usually assumes one visible plane. BPI extends Perspective Plane Program Induction [21] to multiple surfaces by using box constraints to reduce the search. Its explicit geometric and repetition parameters are intended for editing, beyond merely reproducing an image.

Image manipulation.

Patch matching supplies local texture but can violate larger structures. Perspective-aware methods rectify surfaces, and learned inpainting supplies semantic priors. BPI instead uses the input image’s pixels together with inferred plane geometry and lattice regularity. Pretrained visual-cue networks still contribute external learned information.

3 Box Program Induction

Inference proposes a box-shaped plane partition, fits repeated programs to rectified planes, and chooses the best combined image agreement.

Refer to caption
Figure 2: Example box programs inferred by BPI. Our box program jointly models camera parameters, the 3D positions and orientations of multiple planes, as well as the regularity structure on individual planes.

3.1 Domain-Specific Language

A box program specifies camera parameters and a program for each surface. Each plane has a normal and repeated Draw operations inside up to two nested For loops. The grammar represents lattice patterns rather than arbitrary scene semantics; Table 1 retains its precise syntax and Figure 2 illustrates interior and exterior programs.

Program ⟶\longrightarrow CameraProgram; WorldProgram;
CameraProgram ⟶\longrightarrow SetCamera(pos=Vec3, point_to=Vec3);
WorldProgram ⟶\longrightarrow PlaneProgram; ∣\mid PlaneProgram; World Program;
PlaneProgram ⟶\longrightarrow SetPlane(pos=Vec3, normal=Vec3)  For1Stmt;
For1Stmt ⟶\longrightarrow For ( ii in range(Integer, Integer) ){ DrawStmt; ∣\mid For2Stmt }
For2Stmt ⟶\longrightarrow For ( jj in range(Integer, Integer) ){ DrawStmt }
DrawStmt ⟶\longrightarrow Draw (xx=Expr, yy=Expr)
Expr ⟶\longrightarrow Real×i+Real×j|Real×i\text{\tt Real}\times i+\text{\tt Real}\times j\mid\text{\tt Real}\times i
Table 1: The domain-specific language (DSL) of box programs. Language tokens For, If, Integer, Real, and arithmetic/logical operators follow the Python convention. Vec3 denotes 3D real vectors.

3.2 Box Program Fitness

Each candidate plane is rectified to remove perspective effects. A lattice program predicts centers of repeated elements; fitness compares image patches at neighboring repetitions. For lattice patterns, this can be evaluated by shifting the rectified image by one cell and measuring pixel differences. Scores sum across planes, so a candidate must explain both the partition and the visual repetition.

Equation 1
F=−∑a,b[‖Ji[c(a,b)]−Ji[c(a+1,b)]‖22+‖Ji[c(a,b)]−Ji[c(a,b+1)]‖22],F=-\sum_{a,b}\left[\left\|J_{i}[c(a,b)]-J_{i}[c(a+1,b)]\right\|_{2}^{2}+\left\|J_{i}[c(a,b)]-J_{i}[c(a,b+1)]\right\|_{2}^{2}\right], (1)
Refer to caption
Figure 3: Our Box Program Induction finds the best-fit program that describes the input image (a). It first detects the vanishing point and wireframe segments (b), followed by a filtering step (c). It then constructs a set of candidate plane segmentation maps (d). Given each plane segmentation, it rectifies each plane and infers its regularity structure (e). We rank all candidate box programs by their fitness (f); the starred candidate is the best.

3.3 3D Box Priors and the Role of Visual Cues

An interior models two walls, floor, and ceiling; the far plane is segmented but excluded from program modeling. An exterior models two side walls, leaving the roof unmodeled. These priors restrict the combinatorial search. The main procedure below concerns interiors; Appendix B handles exteriors.

Step 1: Visual cue detection.

NeurVPS [47] supplies the most confident vanishing point, and L-CNN [48] supplies line segments. Short segments and segments whose extended lines miss a small neighborhood of the vanishing point are removed. Remaining lines propose the four plane intersections.

Step 2: Plane segmentation.

Every combination of four surviving segments yields a candidate partition. Because predicted lines may not meet exactly, the procedure refits a common vanishing point by minimizing distances to their extended lines. Rays from that point through the segments partition the image into four regions.

Step 3: Plane rectification and regularity inference.

The camera is fixed at the origin with a 35 mm focal length, zero skew, and unit pixel aspect ratio. Plane geometry derived from the candidate rays enables rectification and a search over lattice programs. Appendix A additionally fixes a one-meter normalization to resolve scale ambiguity. Thus the reconstruction supplies a consistent geometry for editing under these assumptions, not a uniquely measured metric scene.

Step 4: Box program ranking.

The four plane fitness values are summed, and the highest-scoring candidate becomes the image’s box program. Regularity therefore helps select among noisy geometric proposals.

Refer to caption
Figure 4: An illustration of the proposed program-guided PatchMatch. Given the corrupted image, we first detect its box program (b) and rectify each plane (c). We use the regular structure (d) on the plane to guide the PatchMatch to inpaint the corrupted plane (e). The color in (d) visualizes the relative position of each pixel to the center of its associated lattice cell.

3.4 Program-Guided Image Manipulation

The editing stage augments PatchMatch [4] with a regularity term. After rectification, a shift map records each pixel’s location within a repeating lattice cell. Candidate patches are favored when they match both appearance and corresponding positions in the repeated pattern, including periodic wraparound. The same mechanism fills missing pixels or extends surfaces beyond the input image for extrapolation and camera changes.

Equation 2
𝑠𝑖𝑚⁡(p,q)≜𝑠𝑖𝑚pixel+𝑠𝑖𝑚reg=𝑠𝑖𝑚pixel−λreg​‖wraparound⁡(𝑠𝑚𝑎𝑝⁡[p]−𝑠𝑚𝑎𝑝⁡[q])‖22,\mathit{sim}(p,q)\triangleq\mathit{sim}_{\mathrm{pixel}}+\mathit{sim}_{\mathrm{reg}}=\mathit{sim}_{\mathrm{pixel}}-\lambda_{\mathrm{reg}}\|\mathrm{wraparound}(\mathit{smap}[p]-\mathit{smap}[q])\|^{2}_{2}, (2)

4 Experiments

Experiments evaluate segmentation, inpainting, extrapolation, and novel-view synthesis on small, specifically collected sets of box-like scenes.

Dataset.

The datasets contain 44 corridor and 42 building images from web searches. Plane boundaries are manually annotated; corridor far planes and building foreground subjects also receive masks. These annotations and scene selection define a narrower setting than unrestricted natural images.

4.1 Plane Segmentation

Segmentation is obtained by the program search rather than by directly predicting supervised plane labels. The intended advantage is that a good partition makes the repeated visual elements easier to explain.

Baselines.

The comparisons use Huang et al.’s line- and vanishing-point-based method and PlaneRCNN, trained on ScanNet. Their different learned data and geometric assumptions should be considered when interpreting the small custom datasets.

Method CrdO CrdC BldO BldC
Huang et al. [16] 0.30 0.30 0.73 0.69
PlaneRCNN 0.52 0.52 0.63 0.63
BPI (Ours) 0.84 0.84 0.86 0.86
Table 2: Plane segmentation results in IoU between detected and groundtruth planes. BPI outperforms both baselines on both original (CrdO, BldO) and corrupted images (CrdC, BldC).

Metrics.

Predicted masks are matched to annotations by intersection over union, and their mean overlap is reported. Corridor far-plane pixels are excluded. Both original and randomly corrupted images are evaluated because segmentation must also work as an input to inpainting.

Refer to caption
Figure 5: Visualization of the plane segmentation by different methods.

Results.

BPI reaches mean intersection over union of 0.84 on both original and corrupted corridors and 0.86 on both building conditions. PlaneRCNN obtains 0.52 and 0.63, respectively; Huang et al. obtains 0.30 on corridors and 0.73/0.69 on buildings. The examples show improved partitions around complex repeated structures within these datasets.

4.2 Image Inpainting

Random masks remove image regions, and the inferred programs guide their reconstruction. This tests whether the structure inferred from undamaged pixels improves completion.

Metrics.

Evaluation reports pixel L1 error, peak signal-to-noise ratio, structural similarity, and learned perceptual distance. The middle two are computed on luma; L1 and perceptual distance use RGB. The metrics assess different properties and do not have identical winners.

Baselines.

Baselines include GatedConv, ordinary PatchMatch, Image Melding with additional patch transformations, and Huang et al.’s perspective-corrected patch matching. The comparison isolates the value of structured editing only imperfectly because each full pipeline differs.

Corridors Buildings
Method ℒ1\mathcal{L}_{1} Mean ↓\downarrow PSNR ↑\uparrow SSIM ↑\uparrow LPIPS ↓\downarrow ℒ1\mathcal{L}_{1} Mean ↓\downarrow PSNR ↑\uparrow SSIM ↑\uparrow LPIPS ↓\downarrow
PatchMatch 53.12 31.55 0.9872 0.0215 34.88 32.99 0.9896 0.0072
Image Melding 58.50 30.54 0.9880 0.0174 49.10 30.42 0.9890 0.0134
Huang et al. [16] 51.88 31.41 0.9869 0.0177 26.10 34.87 0.9917 0.0049
GatedConv 44.98 32.32 0.9883 0.0153 55.31 29.54 0.9866 0.0112
BPI (Ours) 38.45 34.21 0.9892 0.0170 26.43 34.86 0.9913 0.0054
Table 3: We compare BPI-guided PatchMatch with both patch-based and learning-based methods on the task of image inpainting. ↑\uparrow indicates that the higher the number, the better. Bold indicates models that are indistinguishable with the best one under that metric with a linear mixed model. See text for details.
Refer to caption
Figure 6: Qualitative results on the task of image inpainting. Compared with the baselines, our model can better preserve the regular structures even if they are sparse or have strong perspective effects.

Results.

On corridors, BPI obtains L1 error 38.45, PSNR 34.21, and SSIM 0.9892, improving these point estimates over the listed baselines. GatedConv has the better LPIPS value, 0.0153 versus 0.0170. On buildings, BPI is close to Huang et al., whose point estimates are slightly better on all four measures. Bold entries denote the paper’s pairwise mixed-model tests, not proof that methods are equivalent. The qualitative examples highlight preservation of ceiling lights and other regular patterns.

Image extrapolation.

Extrapolation receives a foreground building mask and target fill region, then extends regular surfaces into that region. Compared methods include Photoshop content-aware scaling, texture synthesis, InGAN, and Huang et al. Fifteen people rank outputs for twelve images: 61% of 180 responses favor BPI, versus 16%, 2%, 16%, and 5% for those alternatives. This measures preference on selected outputs, not recovery of unseen real building structure.

Refer to caption
Figure 7: The “extrapolated” buildings. Content-Aware Scale, Kaspar et al. [18], and InGAN fail to preserve the building structure while extrapolating the image: they either generate irregular patterns or change the shape of the planes. Huang et al. [16] fails to preserve the regular structure when inpainting large areas.

4.3 View Synthesis

Three five-frame trajectories move forward, backward, or backward with a pan and tilt through corridor scenes. Backward motion exposes unobserved image regions, which program-guided matching fills by extending the inferred repetition. The output is a plausible structured synthesis rather than observed novel-view ground truth.

Refer to caption
Figure 8: View synthesis from a single image of a corridor. Compared with the learning-based method SynSin, our model better preserves the regular patterns on the walls and also has remarkably fewer artifacts.

Results.

Compared with SynSin [38], BPI receives 100%, 94%, and 99.5% preference for the three trajectories. The study uses twenty images and ten raters, producing 600 responses. These strong preferences support perceptual quality for this regular corridor setting, while leaving metric geometric accuracy and general-scene performance untested.

4.4 Failure Case

Failures arise from missing or incorrect line/vanishing-point cues, solid-color surfaces that provide ambiguous partitions, and irregular surfaces that violate the box prior. User-specified wireframes can help. The resulting interaction is a possible repair, not evidence that inference succeeds automatically on those cases.

Refer to caption
Figure 9: Failure cases. Our model may fail when neural networks misdetect visual cues (a). When image contains a solid color plane, our model may segment the pure white part to other planes (b). The inference might fail on irregular scenes, but could be mitigated with user interaction (c).

5 Conclusion

Joint geometry and repetition inference makes several image edits possible from one constrained representation. The principal limitation is the assumption that relevant regions are planar and regularly structured. Incorporating irregular regions remains a proposed extension.

Broader Impact

The intended beneficiaries are people making content-aware and perspective-aware image edits. Reusing source pixels reduces dependence on external image content, although the cue detectors remain pretrained. The authors also identify fabricated imagery and image-forensics misuse as possible consequences.

Acknowledgements

The acknowledgments identify support from the Center for Brains, Minds and Machines, NSF, ONR, and IBM Research, and note Jiajun Wu’s visiting appointment at Google Research.

Appendix A Plane Reconstruction from Segmentation in Inner Views

The derivation assumes a distortion-free pinhole camera. Four image rays through the vanishing point represent intersections among the interior planes. Projecting along the vanishing direction produces a cross-section whose four corners must form a rectangle: opposite edges are parallel and adjacent edges perpendicular. The retained equation system solves the relative corner positions and hence the surface normals.

Camera focal length, aspect ratio, distance, and box size are not jointly identifiable from this image. The derivation fixes focal length at 35 mm and aspect ratio at one, then manually fixes one projected distance to one meter. This final normalization removes the remaining scale ambiguity. The authors state that these choices do not alter the rectification and downstream inpainting results; they should not be read as recovered physical dimensions.

Refer to caption
Figure 10: Illustration of (a) the perspective projection in an inner view of the box (image upside down to be consistent with the projection), and (b) the orthographic projection centered at the vanishing point.
Equation
{yk=wkxk;k=1,2,3,4(Ik′′ lies on the ray V​Ik′′.)y1−y2=u⁡(x1−x2)(definition.)y2−y3=v⁡(x2−x3)(definition.)y3−y4=u⁡(x3−x4)(I1′′​I2′′ is parallel to I3′′​I4′′.)y4−y1=v⁡(x4−x1)(I2′′​I3′′ is parallel to I4′′​I1′′.)x12+y12=1(camera-to-subject distance assumption.)u​v=−1(I1′′​I2′′ and I2′′​I3′′ are perpendicular to each other.)\begin{cases}y_{k}=w_{k}x_{k};\;k=1,2,3,4&\text{($I^{\prime\prime}_{k}$ lies on the ray $VI^{\prime\prime}_{k}$.)}\\ y_{1}-y_{2}=u(x_{1}-x_{2})&\text{(definition.)}\\ y_{2}-y_{3}=v(x_{2}-x_{3})&\text{(definition.)}\\ y_{3}-y_{4}=u(x_{3}-x_{4})&\text{($I^{\prime\prime}_{1}I^{\prime\prime}_{2}$ is parallel to $I^{\prime\prime}_{3}I^{\prime\prime}_{4}$.)}\\ y_{4}-y_{1}=v(x_{4}-x_{1})&\text{($I^{\prime\prime}_{2}I^{\prime\prime}_{3}$ is parallel to $I^{\prime\prime}_{4}I^{\prime\prime}_{1}$.)}\\ x_{1}^{2}+y_{1}^{2}=1&\text{(camera-to-subject distance assumption.)}\\ uv=-1&\text{($I^{\prime\prime}_{1}I^{\prime\prime}_{2}$ and $I^{\prime\prime}_{2}I^{\prime\prime}_{3}$ are perpendicular to each other.)}\\ \end{cases}

Appendix B Box Program Induction for Outer Views of Boxes

For exterior views, only two side planes are considered. The process again proposes partitions, infers each surface’s repetition, and ranks the combined program, but it does not use a vanishing-point detector.

Step 1: Visual cue detection.

L-CNN detects candidate wall-intersection line segments. Short segments are filtered; the vanishing-point proximity filter used for interiors is omitted.

Step 2: Plane segmentation.

Each surviving segment is extended to a line that divides the image into two candidate plane regions.

Step 3: Plane rectification and regularity inference.

Two-plane segmentation alone does not determine full geometry. The P3I algorithm jointly searches for surface normal and repeating program separately on each region, using image regularity to constrain perspective.

Step 4: Box program ranking.

The two plane scores are summed, and the highest-scoring partition/program pair is retained.

Appendix C Implementation Details

The minimum line length is 10% of the shorter image dimension; the interior vanishing-point filter radius is 1%. Rectified planes are resized to 200 × 200 pixels for inference. At least three repetitions per plane are assumed, a material constraint on suitable inputs.

Appendix D Additional Results

Additional results retain runtime tables and complete visual comparisons for programs, segmentation, completion, and extrapolation.

Time complexity.

A corridor image averages 1,506 line combinations and 46 programs evaluated per plane. The box prior gives a reported 23-fold search speedup over removing that prior; it does not make BPI the fastest listed system. Segmentation takes 43.50 s for corridors and 171.20 s for buildings, versus PlaneRCNN’s 0.14 and 0.76 s. Inpainting takes 20.1 and 41.4 s, versus GatedConv’s 1.9 and 4.9 s. Huang et al. and Image Melding run on a different CPU machine from the remaining methods, so cross-method timing also reflects hardware differences.

Qualitative results.

Figures 11–14 retain the supplementary program examples and all segmentation, inpainting, and extrapolation comparisons. User-selected regions allow selected less regular examples, including a curved wall and bamboo scene. These illustrations show possible extensions with interaction; they do not constitute a separate broad benchmark beyond the two box datasets.

Huang et al. [16] PlaneRCNN [23] BPI (Ours)
Corridor Boxes 1.91s 0.14s 43.50s
Building Boxes 6.83s 0.76s 171.20s
Table 4: Runtime of different methods on the task of plane segmentation.
PatchMatch [4] Image Melding [7] Huang et al. [16] Gated Conv [44] BPI (Ours)
Corridor Boxes 61.6s 1275.1s 18.9s 1.9s 20.1s
Building Boxes 122.1s 808.3s 64.4s 4.9s 41.4s
Table 5: Runtime of different methods on the task of image inpainting.
Refer to caption
Figure 11: More image examples and the corresponding programs. We use cyan lines to visualize the lattice structure on each plane.
Refer to caption
Figure 12: Visualization of the plane segmentation by different methods.
Refer to caption
Figure 13: Qualitative results on the task of image inpainting.
Refer to caption
Figure 14: Qualitative results on the task of image extrapolation.
Concept demonstration

Open image ↗