Overview
This edition includes the main paper and seven-page supplement. All eight main figures, three supplementary figures, four scientific tables, formulas, and references are retained. Supplementary Figure 1 spans three PDF pages.
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/1f/1f3a6bde5a8c52799faf6735f13a4b9c49ef29dc470a8ae0c87c669377c63741.webp)
At a glance
P3I jointly estimates a camera orientation, repeated-object locations, and a compact program describing their arrangement from one image. Repetition helps recover perspective; corrected perspective makes repetition easier to identify. Coarse search followed by local gradient optimization fits lattice, circular, and hybrid patterns. On a small dataset of regular scenes, the inferred programs improve pose estimation, repeated-object localization, and program-guided inpainting. The approach assumes visually similar objects arranged on a single plane.
1 Introduction
Separately estimating perspective, objects, and structure can compound errors: a distorted image hinders repetition detection, while conventional perspective correction may lack useful straight lines. P3I fits these quantities together, using global regularity as a geometric cue. Its programs describe both repeated placements and camera orientation. The evaluation uses a newly collected image dataset and tests whether those representations help reconstruct structure and missing pixels.
2 Related Works
Camera pose estimation and shape from texture.
Shape-from-texture methods recover geometry from changes in texture density or scale; other systems estimate vanishing points, lines, or planes. P3I instead couples geometric estimation to explicit repeated-object programs, using regular arrangement even when strong line cues are absent.
Program induction and inverse graphics.
Prior program-induction systems often operate on synthetic scenes, detected objects, or known camera geometry. Related neural image programs learn on single images but assume fronto-parallel views. P3I jointly estimates perspective and program structure, while keeping an interpretable description of the inferred layout.
Image manipulation.
Pixel- and patch-based inpainting can miss global structure. Learned generators can use broader context, but may not respect an image’s particular repetition and perspective. Here an inferred program organizes source patches for an image-specific painting network.
3 Perspective Plane Program Induction
The method combines a restricted graphics language, an image-program compatibility objective, and hybrid discrete/continuous optimization. Each candidate explains object placements and the perspective transformation together.
3.1 Perspective Plane Programs
A program places similar objects with Draw, organizes them through at most two nested repetition loops, and sets the camera pose. The tested structures are a lattice, a circle, or a hybrid of circular and linear repetition. Two levels capture the evaluated planar arrangements; richer scenes require an expanded language and corresponding inference.
Repeated patterns.
Draw specifies a two-dimensional object center. Nested linear or rotational loops generate the repeated centers, making scene structure explicit instead of predicting an unrelated set of detections.
Perspective transformations.
Because all objects lie on one plane, projection becomes a perspective transformation. The implemented model estimates two camera-tilt angles and handles in-plane rotation through the object-coordinate program.
3.2 Inversion of the Graphics Pipeline
An ImageNet-pretrained AlexNet extracts image features. For each camera hypothesis, the feature map is warped toward a fronto-parallel view; this approximates warping the image before feature extraction and avoids repeatedly running the encoder. A candidate program defines neighboring repeated locations. Its loss sums feature differences across those neighbors—a shift-and-compare operation for lattices. Feature matching supplies the fitting signal without task-specific training annotations, but uses externally pretrained visual features.
Equations 1–2
| (1) |
| (2) |
3.3 Grid Search and Gradient-Based Optimization
Bilinear interpolation makes both perspective warping and coordinate sampling differentiable. Direct gradient descent is unreliable because repeated patterns create many local optima. The method therefore enumerates the three structural families, searches coarse continuous initializations, and refines each initialization with local gradient descent. This reduces reliance on a fully quantized symbolic search without removing the nonconvex optimization problem.
3.4 Implementation Details
The coarse continuous grid has spacing 2. Lattice loop bounds follow image size, assuming the pattern covers the image plane. The optic axis passes through the image center; lens distortion is ignored. Focal length and aspect ratio are not uniquely recoverable from one plane, so the examples use 35 and 1 respectively. These modeling conventions limit interpretation of recovered camera parameters as a complete physical calibration.
4 Experiments
Experiments evaluate pose error, repeated-object localization, and image inpainting separately. All primary quantitative results come from the deliberately structured NRPP collection.
4.1 Dataset
NRPP contains 64 Internet images with non-fronto-parallel lattice or circular arrangements. Human annotations specify camera pose and object centers. Its small, selected regular-scene distribution limits conclusions about arbitrary natural images.
4.2 Camera Pose Estimation
Baselines.
Camera baselines are texture-based AutoRectify and PlaneNet pretrained on ScanNet. The largest detected PlaneNet plane supplies its orientation. Adobe Lightroom’s automatic perspective correction is compared qualitatively because it does not expose numerical camera estimates.
Metrics.
Pose error is the mean L1 distance to the annotated orientation, using Rodrigues’ rotation representation and reporting degrees. The original table retains all values.
Results.
P3I’s mean pose error is 4.54°, compared with 30.54° for AutoRectify and 23.75° for PlaneNet. The qualitative comparisons show benefits when regularity is informative and line cues are weak. These are results on NRPP’s single-plane scenes.
4.3 Repeated Pattern Detection
Baselines.
Repeated-pattern detection compares with RPD on the 56 lattice images. A second baseline applies PlaneNet perspective correction before RPD, testing a sequential alternative to joint fitting.
Metrics.
The evaluation measures nearest-center distance in both directions: detections to annotations and annotations to detections. Their sum is the reported Chamfer distance. Using both directions penalizes excessive detections and missed objects.
| Method | Detected to GT | GT to Detected | Chamfer Dis. |
|---|---|---|---|
| RPD | 0.0971 | 0.0909 | 0.1880 |
| RPD + PlaneNet | 0.1659 | 0.1013 | 0.2672 |
| P3I (Ours) | 0.0639 | 0.0881 | 0.1520 |
Results.
P3I reduces Chamfer distance to 0.1520, versus 0.1880 for RPD and 0.2672 for RPD with PlaneNet. Incorrect preliminary correction can therefore hurt the sequential baseline. Circular and hybrid examples demonstrate the language’s additional expressiveness, but the quantitative detection comparison uses only lattices.
4.4 Image Manipulation
The inferred pose rectifies the image, then an image-specific neural painting network fills missing regions using program-aligned patches. Circular layouts add patch rotation to translation. The program specifies which repeated content belongs at the missing location; the network synthesizes its pixels.
Baselines.
Inpainting comparisons include Image Quilting and PatchMatch, which use internal image statistics, and GatedConv pretrained on Places365. P3I-guided painting trains on each image; the upstream program-fitting encoder still uses ImageNet pretraining.
Metrics.
Metrics are mean L1 error on the reconstructed pixels and Inception Score on the inpainted region. These measure different properties and do not provide a complete perceptual or semantic quality assessment.
Results.
P3I-guided painting has the lowest L1 error, 18.72, compared with 21.92 for PatchMatch, 23.20 for GatedConv, and 35.76 for Image Quilting. Image Quilting has the highest Inception Score, consistent with that metric rewarding diversity as well as quality. The retained failure examples expose two central limits: one-plane geometry cannot explain several differently oriented planes, and single-image texture statistics cannot invent missing high-level semantic content such as a person’s head.
| Method | Inception Score | |
|---|---|---|
| Image Quilting | 35.76 | 1.16 |
| PatchMatch | 21.92 | 1.13 |
| GatedConv | 23.20 | 1.12 |
| P3I +NPN (Ours) | 18.72 | 1.14 |
5 Conclusion
Joint geometric and program inference can recover useful regular-scene structure and guide perspective-aware inpainting. The evidence is limited to a small regular-pattern dataset and a restricted graphics language. Multiple planes, lens distortion, ambiguous calibration, irregular objects, and semantic completion remain outside the demonstrated capability. The authors acknowledge their funding support.
Supplement A. Program grammar
The complete formal grammar is retained below. It composes camera setup with nested linear or rotational placements, affine coordinate expressions, and conditional drawing. The grammar specifies the restricted program space searched by the method.
Program | → | CameraProgram;WorldProgram |
CameraProgram | → | SetCameraPose(rx=Real, ry=Real) |
WorldProgram | → | For1Stmt | Rotate2Stmt |
For1Stmt | → | For (i in range(Integer, Integer)) { For2Stmt | Rotate2Stmt } |
For2Stmt | → | For (j in range(Integer, Integer)) { CondDrawStmt } |
Rotate2Stmt | → | For (j in range(ExprI, ExprI)) { RotateDrawStmt } |
CondDrawStmt | → | If (Expr ≥ 0) { CondDrawStmt } |
CondDrawStmt | → | DrawStmt |
DrawStmt | → | Draw (x=Expr, y=Expr) |
RotateDrawStmt | → | Draw (x=RExprX, y=RExprY) |
Expr | → | Real * i + Real * j |
ExprI | → | Real * i |
RExprX | → | Real * Cos(Real * j) * i + Real | Real * Cos(Real * j) + Real |
RExprY | → | Real * Sin(Real * j) * i + Real | Real * Sin(Real * j) + Real |
Supplement B. Neural painting networks
The inferred centers define a Voronoi partition into source patches. Patches are translated to align their centers with the missing region; rotational patterns also require rotation. A U-Net-style encoder-decoder combines the aligned patches and corrupted input. Its set-based design accepts varying numbers of source patches and is invariant to their ordering, subject to memory limits. The supplement refers to the original neural painting work for the full architecture.
Supplement C. Additional results
The complete supplementary examples show pose correction, repeated-object localization, induced lattice and circular programs, and inpainting. Supplementary Figure 1 spans three pages and retains all comparison columns. These additional examples illustrate the same task setting rather than an independent broad-domain validation.




