Literature
Display
← All papers

Perspective Plane Program Induction From a Single Image

Yikai Li · Jiayuan Mao · Xiuming Zhang · William T. Freeman · Joshua B. Tenenbaum · Jiajun Wu

· CVPR 2020 ↗

Computer Vision6 min text · CondensedOriginal paper ↗
Contents

Overview

This edition includes the main paper and seven-page supplement. All eight main figures, three supplementary figures, four scientific tables, formulas, and references are retained. Supplementary Figure 1 spans three PDF pages.

[Uncaptioned image]
Figure 1: Perspective effects and scene structure regularity are ubiquitous in natural images (a). To detect such regularity, one may directly apply regularity structure detection (RPD) [21] to natural images, but this often fails due to the existence of perspective effects (d). Attempting to remedy this, one may perform perspective correction as an independent preprocessing step, but perspective correction often relies on line and/or vanishing point cues, and fails when such cues are missing (e). We observe that these two tasks are interconnected: image regularity serves as a new perspective correction cue, and regularity detection, in turn, also benefits from perspective correction. Thus, we propose to jointly solve perspective correction and regularity structure detection (b) by simultaneously seeking the program and perspective parameters that best describe the image (c). Project page: http://p3i.csail.mit.edu

At a glance

P3I jointly estimates a camera orientation, repeated-object locations, and a compact program describing their arrangement from one image. Repetition helps recover perspective; corrected perspective makes repetition easier to identify. Coarse search followed by local gradient optimization fits lattice, circular, and hybrid patterns. On a small dataset of regular scenes, the inferred programs improve pose estimation, repeated-object localization, and program-guided inpainting. The approach assumes visually similar objects arranged on a single plane.

1 Introduction

Separately estimating perspective, objects, and structure can compound errors: a distorted image hinders repetition detection, while conventional perspective correction may lack useful straight lines. P3I fits these quantities together, using global regularity as a geometric cue. Its programs describe both repeated placements and camera orientation. The evaluation uses a newly collected image dataset and tests whether those representations help reconstruct structure and missing pixels.

2 Related Works

Camera pose estimation and shape from texture.

Shape-from-texture methods recover geometry from changes in texture density or scale; other systems estimate vanishing points, lines, or planes. P3I instead couples geometric estimation to explicit repeated-object programs, using regular arrangement even when strong line cues are absent.

Refer to caption
Figure 2: (a) Our model P3I solves an inverse graphics problem. Given an input image, P3I jointly infers the camera pose, object locations, and the global scene regularity, which is an inversion of a simplified graphics pipeline. (b) We compute the fitness of a program on the image based on a shift-and-compare routine, which we illustrates on a lattice pattern case.

Program induction and inverse graphics.

Prior program-induction systems often operate on synthetic scenes, detected objects, or known camera geometry. Related neural image programs learn on single images but assume fronto-parallel views. P3I jointly estimates perspective and program structure, while keeping an interpretable description of the inferred layout.

Image manipulation.

Pixel- and patch-based inpainting can miss global structure. Learned generators can use broader context, but may not respect an image’s particular repetition and perspective. Here an inferred program organizes source patches for an image-specific painting network.

3 Perspective Plane Program Induction

The method combines a restricted graphics language, an image-program compatibility objective, and hybrid discrete/continuous optimization. Each candidate explains object placements and the perspective transformation together.

Refer to caption
Figure 3: Example programs inferred by P3I. Our model can perform joint inference of camera pose, object localizations, and global scene structures of images having different regularity patterns: (a) lattice, (b) circular, and (c) a hybrid structure composed by a circular structure and a linearly repeated one.

3.1 Perspective Plane Programs

A program places similar objects with Draw, organizes them through at most two nested repetition loops, and sets the camera pose. The tested structures are a lattice, a circle, or a hybrid of circular and linear repetition. Two levels capture the evaluated planar arrangements; richer scenes require an expanded language and corresponding inference.

Repeated patterns.

Draw specifies a two-dimensional object center. Nested linear or rotational loops generate the repeated centers, making scene structure explicit instead of predicting an unrelated set of detections.

Perspective transformations.

Because all objects lie on one plane, projection becomes a perspective transformation. The implemented model estimates two camera-tilt angles and handles in-plane rotation through the object-coordinate program.

3.2 Inversion of the Graphics Pipeline

An ImageNet-pretrained AlexNet extracts image features. For each camera hypothesis, the feature map is warped toward a fronto-parallel view; this approximates warping the image before feature extraction and avoids repeatedly running the encoder. A candidate program defines neighboring repeated locations. Its loss sums feature differences across those neighbors—a shift-and-compare operation for lattices. Feature matching supplies the fitting signal without task-specific training annotations, but uses externally pretrained visual features.

Equations 1–2
F𝑓𝑝=WarpPerspective−r​x,−r​y,−r​z​(F).F^{\mathit{fp}}=\mathrm{WarpPerspective}_{-rx,-ry,-rz}(F). (1)
WarpPerspective−r​x,−r​y,−r​z​(ℱAlexNet​(I))\displaystyle\mathrm{WarpPerspective}_{-rx,-ry,-rz}(\mathcal{F}_{\mathrm{AlexNet}}(I))
≈\displaystyle\approx ℱAlexNet​(WarpPerspective−r​x,−r​y,−r​z​(I)).\displaystyle\mathcal{F}_{\mathrm{AlexNet}}\big(\mathrm{WarpPerspective}_{-rx,-ry,-rz}(I)\big).
ℒ=\displaystyle\mathcal{L}= ∑i,j‖F𝑓𝑝​[x⁡(i,j),y⁡(i,j)]−F𝑓𝑝​[x⁡(i+1,j),y⁡(i+1,j)]‖22\displaystyle\sum_{i,j}\left\|F^{\mathit{fp}}[x(i,j),y(i,j)]-F^{\mathit{fp}}[x(i+1,j),y(i+1,j)]\right\|_{2}^{2}
+\displaystyle+ ∑i,j‖F𝑓𝑝​[x⁡(i,j),y⁡(i,j)]−F𝑓𝑝​[x⁡(i,j+1),y⁡(i,j+1)]‖22,\displaystyle\sum_{i,j}\left\|F^{\mathit{fp}}[x(i,j),y(i,j)]-F^{\mathit{fp}}[x(i,j+1),y(i,j+1)]\right\|_{2}^{2}, (2)

3.3 Grid Search and Gradient-Based Optimization

Bilinear interpolation makes both perspective warping and coordinate sampling differentiable. Direct gradient descent is unreliable because repeated patterns create many local optima. The method therefore enumerates the three structural families, searches coarse continuous initializations, and refines each initialization with local gradient descent. This reduces reliance on a fully quantized symbolic search without removing the nonconvex optimization problem.

Refer to caption
Figure 4: Loss Surface of the displacement parameter in a perspective plane program. On the left we show the input image. The regularity in the image directly leads to many cliffs on the loss surface (d​x=10,12.5,15,⋯dx=10,12.5,15,\cdots) and many local peaks.

3.4 Implementation Details

The coarse continuous grid has spacing 2. Lattice loop bounds follow image size, assuming the pattern covers the image plane. The optic axis passes through the image center; lens distortion is ignored. Focal length and aspect ratio are not uniquely recoverable from one plane, so the examples use 35 and 1 respectively. These modeling conventions limit interpretation of recovered camera parameters as a complete physical calibration.

4 Experiments

Experiments evaluate pose error, repeated-object localization, and image inpainting separately. All primary quantitative results come from the deliberately structured NRPP collection.

4.1 Dataset

NRPP contains 64 Internet images with non-fronto-parallel lattice or circular arrangements. Human annotations specify camera pose and object centers. Its small, selected regular-scene distribution limits conclusions about arbitrary natural images.

Refer to caption
Figure 5: On the left, we show that P3I estimates the camera pose based on the global regularity of images. It outperforms AutoRectify, Lightroom and PlaneNet. Results are visualized by performing an perspective correction based on the estimated parameters. On the right, we show that P3I can perform perspective-aware repeated pattern detection, while both RPD [21] and RPD+PlaneNet fail.

4.2 Camera Pose Estimation

Baselines.

Camera baselines are texture-based AutoRectify and PlaneNet pretrained on ScanNet. The largest detected PlaneNet plane supplies its orientation. Adobe Lightroom’s automatic perspective correction is compared qualitatively because it does not expose numerical camera estimates.

Metrics.

Pose error is the mean L1 distance to the annotated orientation, using Rodrigues’ rotation representation and reporting degrees. The original table retains all values.

Method Camera Pose Error
AutoRectify [1] 30.54
PlaneNet [24] 23.75
P3I (Ours) 4.54
Table 1: Camera pose estimation. P3I outperforms texture-based baseline, AutoRectify, and the neural baseline, PlaneNet, by a remarkable margin on the NRPP dataset.

Results.

P3I’s mean pose error is 4.54°, compared with 30.54° for AutoRectify and 23.75° for PlaneNet. The qualitative comparisons show benefits when regularity is informative and line cues are weak. These are results on NRPP’s single-plane scenes.

4.3 Repeated Pattern Detection

Baselines.

Repeated-pattern detection compares with RPD on the 56 lattice images. A second baseline applies PlaneNet perspective correction before RPD, testing a sequential alternative to joint fitting.

Metrics.

The evaluation measures nearest-center distance in both directions: detections to annotations and annotations to detections. Their sum is the reported Chamfer distance. Using both directions penalizes excessive detections and missed objects.

Method Detected to GT GT to Detected Chamfer Dis.
RPD 0.0971 0.0909 0.1880
RPD + PlaneNet 0.1659 0.1013 0.2672
P3I (Ours) 0.0639 0.0881 0.1520
Table 2: Repeated object detection. P3I outperforms both RPD and RPD+PlaneNet. The degradation from “RPD” to “RPD + PlaneNet” is explained by incorrect perspective corrections, which lead to larger errors than not performing the correction at all.

Results.

P3I reduces Chamfer distance to 0.1520, versus 0.1880 for RPD and 0.2672 for RPD with PlaneNet. Incorrect preliminary correction can therefore hurt the sequential baseline. Circular and hybrid examples demonstrate the language’s additional expressiveness, but the quantitative detection comparison uses only lattices.

Refer to caption
Figure 6: Besides lattice patterns, P3I is also able to detect (1) circular patterns (rainbow colors are used to visualize the inferred program), and (2) hybrid structures composed by a circular structure and a linearly repeated one.

4.4 Image Manipulation

The inferred pose rectifies the image, then an image-specific neural painting network fills missing regions using program-aligned patches. Circular layouts add patch rotation to translation. The program specifies which repeated content belongs at the missing location; the network synthesizes its pixels.

Refer to caption
Figure 7: P3I-guided NPNs perform inpainting in a perspective-aware fashion. Results generated by P3I-guided NPNs are sharp (compared with PatchMatch), consistently connect to the global structure (e.g., the lines in 1 and 4), and respect the global perspective effects.

Baselines.

Inpainting comparisons include Image Quilting and PatchMatch, which use internal image statistics, and GatedConv pretrained on Places365. P3I-guided painting trains on each image; the upstream program-fitting encoder still uses ImageNet pretraining.

Metrics.

Metrics are mean L1 error on the reconstructed pixels and Inception Score on the inpainted region. These measure different properties and do not provide a complete perceptual or semantic quality assessment.

Results.

P3I-guided painting has the lowest L1 error, 18.72, compared with 21.92 for PatchMatch, 23.20 for GatedConv, and 35.76 for Image Quilting. Image Quilting has the highest Inception Score, consistent with that metric rewarding diversity as well as quality. The retained failure examples expose two central limits: one-plane geometry cannot explain several differently oriented planes, and single-image texture statistics cannot invent missing high-level semantic content such as a person’s head.

Method ℒ1\mathcal{L}_{1} Inception Score
Image Quilting 35.76 1.16
PatchMatch 21.92 1.13
GatedConv 23.20 1.12
P3I +NPN (Ours) 18.72 1.14
Table 3: Image inpainting. P3I-Guided NPNs outperform both classic, non-learning baselines and the learning-based baseline in ℒ1\mathcal{L}_{1} loss. Image Quilting achieves better Inception Score (IS) than P3I-NPNs, because besides patch quality, patch diversity also improves IS [34], and Image Quilting tends to produce diverse inpainting across the test images.
Refer to caption
Figure 8: Left: Perspective correction in the presence of multiple planes is a future direction that P3I can take. Right: P3I inpainting learns low-level texture statistics from single images, so it is unable to inpaint the person’s head using high-level semantics.

5 Conclusion

Joint geometric and program inference can recover useful regular-scene structure and guide perspective-aware inpainting. The evidence is limited to a small regular-pattern dataset and a restricted graphics language. Multiple planes, lens distortion, ambiguous calibration, irregular objects, and semantic completion remain outside the demonstrated capability. The authors acknowledge their funding support.

Supplement A. Program grammar

The complete formal grammar is retained below. It composes camera setup with nested linear or rotational placements, affine coordinate expressions, and conditional drawing. The grammar specifies the restricted program space searched by the method.

Program→CameraProgram;WorldProgram
CameraProgram→SetCameraPose(rx=Real, ry=Real)
WorldProgram→For1Stmt | Rotate2Stmt
For1Stmt→For (i in range(Integer, Integer)) { For2Stmt | Rotate2Stmt }
For2Stmt→For (j in range(Integer, Integer)) { CondDrawStmt }
Rotate2Stmt→For (j in range(ExprI, ExprI)) { RotateDrawStmt }
CondDrawStmt→If (Expr ≥ 0) { CondDrawStmt }
CondDrawStmt→DrawStmt
DrawStmt→Draw (x=Expr, y=Expr)
RotateDrawStmt→Draw (x=RExprX, y=RExprY)
Expr→Real * i + Real * j
ExprI→Real * i
RExprX→Real * Cos(Real * j) * i + Real | Real * Cos(Real * j) + Real
RExprY→Real * Sin(Real * j) * i + Real | Real * Sin(Real * j) + Real
Supplementary Table 1. The domain-specific language (DSL) of perspective plane programs. Language tokens For, If, Integer, Real, and arithmetic/logical operators follow the Python convention.

Supplement B. Neural painting networks

The inferred centers define a Voronoi partition into source patches. Patches are translated to align their centers with the missing region; rotational patterns also require rotation. A U-Net-style encoder-decoder combines the aligned patches and corrupted input. Its set-based design accepts varying numbers of source patches and is invariant to their ordering, subject to memory limits. The supplement refers to the original neural painting work for the full architecture.

Supplement C. Additional results

The complete supplementary examples show pose correction, repeated-object localization, induced lattice and circular programs, and inpainting. Supplementary Figure 1 spans three pages and retains all comparison columns. These additional examples illustrate the same task setting rather than an independent broad-domain validation.

Supplementary Figure 1aSupplementary Figure 1bSupplementary Figure 1c
Supplementary Figure 1. Qualitative results for camera pose estimation and repeated pattern detection. On the left, P3I outperforms both Lightroom and PlaneNet on camera pose estimation. Results are visualized by performing an perspective correction based on the estimated parameters. On the right, we show that P3I can perform perspective-aware repeated pattern detection, which outperforms the baseline methods RPD and RPD + PlaneNet.
Supplementary Figure 2
Supplementary Figure 2. Examples of the induced perspective plane programs from our dataset NRPP. We overlay the centroids of the repeated objects as green squares on the original images for better visualization. We show the induced programs for both images with lattice patterns and images with circular patterns.
Supplementary Figure 3
Supplementary Figure 3. P3I-guided NPNs perform image inpainting in a perspective-aware fashion. Results produced by NPNs are sharp and consistent with the global structure. We compare our results with both non-learning-based methods (PatchMatch and Image Quilting) and a learning-based method (GatedConv).
Concept demonstration

Open image ↗