Literature
Display
← All papers

Program-Guided Image Manipulators

Jiayuan Mao · Xiuming Zhang · Yikai Li · William T. Freeman · Joshua B. Tenenbaum · Jiajun Wu

· ICCV 2019 ↗

Computer Vision8 min text · CondensedOriginal paper ↗
Contents

Overview

The edition includes the main paper and its complete nine-page supplement. The supplement’s eight figures, detector formulas, ablations, failure example, and full generator listing are restored from the original PDF because the HTML conversion omitted them.

[Uncaptioned image]
Figure 1: Given an input image, the Program-Guided Image Manipulator (PG-IM) detects repeated entities in the image (pieces of cereal) and then infers a program-like representation for describing the regularity of the image. The regularity representation empowers multiple downstream tasks, such as image inpainting, extrapolation, and regularity editing.

At a glance

PG-IM combines repeated-object detection, symbolic programs describing spatial and attribute patterns, and an image-specific neural painter. A program supplies aligned source patches and determines where new objects belong. The same painter, trained for inpainting on the input image, can be used for extrapolation and regularity editing. Results favor the approach over the evaluated neural baselines on nearly regular patterns, but its language and repeated-pattern assumptions limit the range of images it can represent.

1 Introduction

Object identities and pairwise relations alone do not capture global repetition, boundaries, or alternating attributes. PG-IM represents these regularities with loops and conditions, then uses the representation to guide pixel synthesis. It evaluates inpainting, extrapolation, and regularity editing on rectified pattern and facade images, with additional qualitative Internet examples.

2 Related Work

The method combines ideas from image manipulation, procedural representations, and single-image learning.

Image manipulation.

Pixel and patch methods can fill local texture while missing object-level structure. Dataset-trained neural inpainting offers semantic priors but can introduce inappropriate content for an unusual repeated pattern. PG-IM instead supplies an explicit program for the particular image’s regularity.

Program induction and procedural modeling.

Program induction provides an interpretable description of loops, boundaries, and attributes. Unlike approaches focused on constrained synthetic imagery, this work applies such structure to selected natural patterns. The painter learns from the input image, although object detection still uses pretrained visual features.

Single-image learning.

A single image contains repeated statistics that can train an image-specific generator. The program adds global structure that a local texture prior may not preserve, such as a triangular arrangement rather than an unrestricted grid.

Refer to caption
Figure 2: The three-step inference of a program describing the shown repeated pattern. Assuming the input keypoints follow a lattice pattern, we first search for parameters defining the lattice, such as the distance between nearby keypoints and the origin. Next, we fit boundary conditions for the program. Finally, we cluster objects into groups by their visual appearance, and fit an expression describing the variation.

3 Program-Guided Image Manipulator

The pipeline detects repeated centroids, searches for a program that reconstructs their arrangement and attributes, then uses that program to condition a neural painter.

3.1 Repeated Object Detection

The detector starts from pretrained AlexNet features, identifies local activation peaks, votes for repeated displacements, and fits an implicit pattern model. This stage assumes approximately repeated lattice structure. The supplement restores the complete six-step procedure and its formulas.

3.2 Program Synthesizer

Detected centroids partition pixels into Voronoi cells. A program places objects at coordinates and assigns discrete appearance attributes. The search aims for a compact explanation of those detections rather than inferring an unrestricted scene model.

Refer to caption
Figure 3: Illustrative programs inferred from (top row) the Nearly-Regular Pattern dataset [22], (middle row) the Facade dataset [39], and (bottom row) Internet images. The DSL of the inferred programs supports for-loops, conditions, and attributes.
Program ⟶\longrightarrow For1Stmt
For1Stmt
⟶\longrightarrow
For ( ii in range(Integer, Integer) )
{ For2Stmt }
For2Stmt
⟶\longrightarrow
For ( ii in range(Integer, Integer) )
{ CondDrawStmt }
CondDrawStmt ⟶\longrightarrow If (Expr ≥\geq 0) { CondDrawStmt }
CondDrawStmt ⟶\longrightarrow DrawStmt
DrawStmt
⟶\longrightarrow
Draw (x=Expr, y=Expr,
attribute=AttributeExpr )
AttributeExpr ⟶\longrightarrow Expr // Integer
AttributeExpr ⟶\longrightarrow 1 If (Expr == 0) else 0
AttributeExpr ⟶\longrightarrow 1 If (Expr == 0 and Expr == 0) else 0
AttributeExpr ⟶\longrightarrow 1 If (Expr % Integer == 0) else 0
AttributeExpr ⟶\longrightarrow 1 If (Expr % Integer == 0 and Expr % Integer == 0) else 0
Expr ⟶\longrightarrow Integer * ii + Integer * jj + Integer
Table 1: The domain-specific language (DSL) for describing image regularities. Language tokens including For, If, Integer and arithmetic/logical operators follow the convention of Python.
Refer to caption
Figure 4: A neural painting network (NPN) takes as input an image and a set of source patches, derived from the image with its program description, and outputs a manipulated image. An NPN learns from a single image, exploiting the image’s internal statistics. Trained only on inpainting, it can directly extrapolate and edit the regularity of the input image in a unified inference framework, without any finetuning.

Domain-specific language.

The domain-specific language contains nested loops, affine coordinate expressions, boundary conditions, drawing commands, and integer-valued attributes. The full grammar is retained in Table 1; patterns outside this grammar cannot be represented directly.

Program search.

Search is decomposed heuristically into lattice coordinates, boundary conditions, and appearance attributes. This reduces the combinatorial burden but is not a guarantee of a globally optimal program.

Lattice search.

A five-parameter lattice maps loop indices to object coordinates. Its objective balances distances from detected centroids to reconstructed points against a penalty for excessive points. The regularization coefficient is five. The retained equation specifies the complete cost.

Equation 1
ℒlat=∑(x,y)∈𝐂min(u,v)∈𝐏⁡[(x−u)2+(y−v)2]+λ​|𝐏|,\mathcal{L}_{\text{lat}}=\sum_{(x,y)\in\mathbf{C}}\min\limits_{(u,v)\in\mathbf{P}}\left[(x-u)^{2}+(y-v)^{2}\right]+\lambda|\mathbf{P}|, (1)

Condition search.

A minimum-cost assignment matches reconstructed and detected centroids. The convex hull of matched loop-index coordinates determines loop bounds and optional conditions. This handles rectangular and triangular boundaries while imposing a convexity assumption.

Attribute search.

Attribute expressions assign discrete labels as functions of loop indices. Search favors similar patches sharing a label and penalizes unnecessary distinct labels, with regularization strength ten. Integer and binary expression templates can represent alternating or grouped appearances; continuous attributes are not modeled.

Equation 2
ℒattr=∑p∈𝐏∑q∈𝐏(sgn​(A⁡(p),A⁡(q))⋅d​(p,q))+μ​|A⁡(𝐏)|,\mathcal{L}_{\text{attr}}=\sum_{p\in\mathbf{P}}\sum_{q\in\mathbf{P}}\big(\text{sgn}(A(p),A(q))\cdot\textit{d}(p,q)\big)+\mu|A(\mathbf{P})|, (2)

3.3 Neural Painting Networks

The neural painting network treats manipulation as filling missing pixels. Program-derived object locations make it possible to reuse an inpainting-trained model for the other tasks.

Patch aggregation.

To fill one object, the system translates other objects to align their centroids with the missing region. These aligned source images and the corrupted input form a variable-size stack from which the painter can recover suitable content.

Architecture.

A shared-weight U-Net processes every source track. Intermediate max pooling forms a global feature that is broadcast back to all tracks, making the architecture insensitive to track order and able to handle different numbers of source images. Skip connections preserve local detail. The complete layer listing is retained in the supplement.

Extrapolation and regularity editing as recurrent inpainting.

Extrapolation edits program bounds to identify new objects, then fills them one at a time, using earlier predictions as context. Regularity editing moves each Voronoi cell with its centroid and fills the resulting gaps. This reuses the trained painter but does not imply arbitrary structural edits are supported.

Training.

Each input image receives its own trained painter, using reconstruction and adversarial losses. Training samples corrupted patches from that image rather than learning the painter once for all test images. The pretrained feature detector remains an external learned component.

4 Experiments and Applications

Quantitative inpainting and qualitative extrapolation and regularity editing compare PG-IM with patch-based and neural baselines. The supplemental ablations test source patches, recurrent filling, and attributes.

Refer to caption
Figure 5: Corrupted input images and inpainting results (zoomed-in) by PG-IM and the baselines. The white pixels in the leftmost column are missing pixels to inpaint. The rightmost column shows the ground-truth patches. PG-IM inpaints realistic image patches that are consistent with the intricate global regularity and meanwhile different from the original, ground-truth patches.

4.1 Dataset

The experiments use 48 rectified Nearly-Regular Pattern images and 109 rectified facade images. These are selected structured domains, not a representative sample of unrestricted natural scenes.

4.2 Baselines

Baselines include Image Quilting, PatchMatch, pretrained GatedConv, and image-specific Non-Stationary texture synthesis. Hyperparameters are selected across test images, while image-specific networks are trained separately. Some baselines need user-specified extrapolation masks, whereas PG-IM derives positions from its program; this is a difference in task interface as well as synthesis.

Method L1 Mean (Std.) Inception Score
Nearly-Regular Patterns [22]
Image Quilting [11] 12.30 (2.903) 1.253
PatchMatch [6] 83.91 (17.62) 1.210
GatedConv [48] 50.45 (16.46) 1.196
Non-Stationary [50] 103.7 (23.87) 1.186
PG-IM (ours) 21.48 (5.375) 1.229
Facade [39]
Image Quilting [11] 13.50 (6.379) 1.217
PatchMatch [6] 81.35 (25.28) 1.219
GatedConv [48] 26.26 (133.9) 1.186
Non-Stationary [50] 133.9 (39.75) 1.199
PG-IM (ours) 14.40 (7.781) 1.218
Table 2: We compare PG-IM against off-the-shelf neural baselines for image inpainting on both datasets. Our method outperforms neural baselines with a remarkable margin across all metrics.
Refer to caption
Figure 6: Extrapolation results by PG-IM and the baselines. The white pixels in the leftmost column indicate the pixels to be extrapolated. PG-IM generates realistic images while preserving global regularity. In contrast, GatedConv fails to capture the regularity; Non-Stationary does not preserve the original image contents; PatchMatch tends to generate blurry images in smoothing the transition; Image Quilting does not guarantee the global structure gets preserved.

4.3 Inpainting

Each image is corrupted 100 times, with test patches withheld from training. PG-IM improves reconstruction and Inception scores over the reported neural baselines, but Image Quilting has lower L1 error on both datasets. Qualitative examples motivate the value of structural consistency despite that metric difference. The roughly 100 ms neural inference time excludes per-image training; it should not be read as the complete cost of processing a new image. The source’s table contains an unusually large GatedConv standard deviation, which is retained rather than corrected without evidence.

Refer to caption
Figure 7: PG-IM can reason about the attribute regularity of images, which supports object appearance–aware image extrapolation. PG-IM w/o Attributes denotes a variant of PG-IM that does not include attributes. See the main text for detailed analysis and comparison.

4.4 Extrapolation

Extending loop ranges or relaxing conditions supplies new object locations. The visual comparisons show examples where PG-IM preserves layout and boundary continuity while baselines blur or introduce inconsistent content. These selected examples support the proposed mechanism but do not provide a quantitative generalization rate.

Refer to caption
Figure 8: PG-IM enables automated and semantic-aware irregularity exaggeration. By comparing the centroids of the detected objects and the ones reconstructed by the program, we can measure and exaggerate the structural irregularity of input images.

4.5 Image Regularity Editing

Irregularity editing magnifies the difference between detected centroids and ideal program positions. The corresponding image cells move with the objects, and recurrent inpainting repairs gaps. The edits are constrained by the inferred pattern.

4.6 Attribute Regularity

Attribute-aware source selection avoids mixing patches that occupy different roles in a global color pattern. The illustrated ablations show color inconsistencies without this selection. Attributes are discrete, leaving continuous appearance variation outside the current representation.

5 Discussion

PG-IM combines explicit regularity with local image synthesis, but its representational language restricts supported patterns. Rotation patterns and richer continuous attributes are left for future work. It also cannot infer semantically new structures, such as adding a roof to repeated windows, from repetition alone. The supplement provides a failure caused by reflection and transmission.

Supplement A. Implementation details

The supplement specifies the detector, adaptive discriminator, recurrent generation, and per-image training procedure.

A.1 Repeated object detection

The detector finds feature-map peaks, votes for repeated displacements, filters inconsistent vectors, weights feature maps, optimizes lattice offset, and lets activations vote for individual centroids. The six original displayed formulas are retained below with editorial labels S1-S6; these labels are added for navigation. The final local vote allows departures from a perfectly regular lattice.

Equations S1–S6
(S1)Supplementary detector formula 1, original PDF page11
(S2)Supplementary detector formula 2, original PDF page11
(S3)Supplementary detector formula 3, original PDF page11
(S4)Supplementary detector formula 4, original PDF page11
(S5)Supplementary detector formula 5, original PDF page11
(S6)Supplementary detector formula 6, original PDF page11

A.2 Adaptive discriminator depth

AdaPatchGAN adapts discriminator depth to the inferred object-patch scale, adding layers until its receptive field covers a typical patch. This avoids using one fixed receptive field for both small repeated textures and large objects.

A.3 Recurrent extrapolation

Filling an entire extrapolation strip at once differs from the localized corruption used for training. Recurrent filling instead generates one object at a time and includes completed objects in later context. The ablation below illustrates the resulting reduction in distorted objects.

A.4 Training parameters

Training follows the pix2pix setup but does not decay the learning rate after 100 epochs. Each epoch contains 1,000 randomly cropped samples from the same image, and the authors report convergence within roughly 150 epochs. This substantial image-specific training cost is separate from the final forward-pass timing.

Supplement B. Ablations

The three ablations isolate the role of program-derived source patches, recurrent inference, and discrete attributes. They are visual comparisons rather than a broad statistical decomposition of performance.

B.1 Source patches

Without aligned source patches, the painter can restore background while omitting the object. The two examples show the program’s value in supplying an intact counterpart of a completely corrupted repeated element.

Supplementary Figure 1. Inpainting with and without program-provided source patches. Each pane compares the corrupted input, ablated network, PG-IM, and ground truth. The source patches help recover completely missing objects.
Supplementary Figure 1: Inpainting with vs. without the source patches provided by program. For each pane, the leftmost figure is the corrupted input to inpaint; to the right are results inpainted with and without source patches as well as the ground truth. With programs describing where the object centers are, our NPNs align the intact objects to the corrupted object and exploit such alignment to inpaint the missing pixels. Without such source patches, the networks have difficulty ensuring the “objectness” of the inpainted patch.

B.2 Recurrent inpainting

Recurrent extrapolation better matches the local inpainting task used in training. Filling the whole stripe in one pass produces distorted objects in the illustrated comparisons.

Supplementary Figure 2. Extrapolation by one feedforward pass versus recurrent inpainting. Filling one object at a time better matches the network’s training task and reduces distorted extrapolated objects.
Supplementary Figure 2: Extrapolation as feedforward vs. recurrent inpainting. For each pane, the leftmost figure is the partial input to be extrapolated; to the right are results extrapolated in one go and recurrently. Recall that the networks are trained only on inpainting one entity, so the networks tend to produce distorted objects if they are asked to inpaint the whole stripe spanning multiple entities. Instead, our NPNs treat extrapolation as recurrent inpainting, where they repeatedly solve the task they are good at and therefore produce better results.

B.3 Object attributes

The attribute ablation mixes nearby colors even when the global arrangement requires one particular color. Restricting source patches to the appropriate attribute preserves the repeated pattern in the tomato and candy examples.

Supplementary Figure 3. Attribute-aware inpainting. Referencing only patches with the appropriate attribute preserves the global color pattern, whereas the ablation mixes neighboring colors.
Supplementary Figure 3: Inpainting with vs. without considering object attributes. For each pane, the leftmost figure is the corrupted input to be inpainted; to the right are results inpainted with and without considering attributes as well as the ground truth. The attribute-agnostic networks tend to produce patches with mixed colors that break the global regularity of attributes. PG-IM is able to respect such regularity by referencing only patches with the same attribute as the patch to inpaint.

Supplement C. Additional experiments

The additional galleries retain every published comparison, including the failure case and the fine-tuned baseline.

C.1 Additional inpainting results

Six additional inpainting examples compare the method with all listed baselines and ground truth. The examples emphasize preserving whole repeated structures rather than only plausible local texture.

Supplementary Figure 4. Six additional inpainting comparisons. White regions are missing pixels; methods are compared with ground truth. The original figure retains all examples, method labels, and magnified results.
Supplementary Figure 4: More inpainting results by PG-IM and the baselines. The white pixels in the leftmost column are missing pixels to inpaint; the rightmost column shows the ground-truth patches. For easier comparison, we show only the close-up views of the inpainted region (and its proximity for some context). Aware of the images’ global regularity, PG-IM inpaints the missing objects that maintain this regularity. Note this is not necessarily the case for other methods: baselines miss out the pig entity of Image 3 and the yellow tag of Image 5. Although PatchMatch gets the cross structure in Image 1, its result is less realistic than ours. Another advantage of PG-IM over the baselines is the ability to generate pixels that seamlessly connect to the original image contents (compare the results for Images 1, 2, and 4).

C.2 Additional extrapolation results

Four additional extrapolation examples compare boundary continuity and pattern extension. The baselines differ in whether they modify existing content or merely fill the newly exposed region, which matters when interpreting visual differences.

Supplementary Figure 5. Four additional extrapolation comparisons, retaining every baseline and original example. PG-IM follows the inferred structure while extending the visible pattern.
Supplementary Figure 5: Extrapolation results by PG-IM and the baselines. The white pixels in the leftmost column indicate the pixels to be extrapolated, and the red dots are object centers given by the program descriptions. With such program descriptions, PG-IM knows where to extrapolate to automatically, whereas other methods require the user to specify where to extrapolate to. PG-IM generates realistic images while preserving the global regularity. In contrast, GatedConv fails to capture the regularity; Non-Stationary does not preserve the original content of images; non-learning-based baselines sometimes generate blurry images because of the repetition of similar objects in the partial image.

C.3 Regularity editing

The source doubles irregularity displacement vectors and randomly flips their signs, then shifts patches and fills gaps. All four examples remain available for inspecting the effect.

Supplementary Figure 6. Four examples of irregularity editing. The source doubles centroid displacement vectors and randomly flips their signs, then moves patches and inpaints the gaps.
Supplementary Figure 6: PG-IM enables automated and semantic-aware irregularity exaggeration. By comparing the centroids of the detected objects and the ones reconstructed by the program, we can measure and exaggerate the structural irregularity of input images. In these examples, we first multiply by 2 the displacement vectors between the object centroids provided by the programs and the detected object centroids, and then randomly flip the sign for each displacement vector. According to these new displacement vectors, we shift the patches and then let the NPN fill in the missing pixels.

C.4 Failure cases

Transmission and reflections make corresponding objects visually inconsistent. In this example, the model generates non-photorealistic content despite a plausible repeated layout, exposing the limits of copying internal statistics.

Supplementary Figure 7. Failure case: transmission and reflection make repeated objects insufficiently homogeneous, producing non-photorealistic inpainting.
Supplementary Figure 7: A failure case.

C.5 Fine-tuned baseline

Fine-tuning GatedConv on 50 facade images improves its Inception score from 1.187 to 1.191, versus 1.210 for the image-specific PG-IM comparison. Structured windows can remain problematic after fine-tuning. The training regimes differ, so this is not an equal-data comparison.

Supplementary Figure 8. GatedConv before and after fine-tuning on 50 facade images, compared with PG-IM trained on the input image and ground truth. All comparison panels are retained.
Supplementary Figure 8: GatedConv model finetuned on the Facade dataset. Note that our PG-IM is only trained on a single input image.

Appendix: Complete generator architecture

The complete commented generator listing preserves channel counts, kernel sizes, strides, padding, pooling across tracks, and skip connections. It spans the final two PDF pages and is retained as selectable code.

('1', ReppadConv2d(3, 64, 3, stride=1, padding=1)),
           ('1a', nn.ELU()),
           # 2: maxpool across tracks
           # 3: concat 2
           ('4:+pool', ReppadConv2d(128, 96, 1, stride=1)),
           ('4a', nn.ELU()),
           ('5', ReppadConv2d(96, 96, 4, stride=2, padding=1)),
           ('5a', nn.ELU()),
           # 6: maxpool across tracks
           # 7: concat 6
           ('8:+pool', ReppadConv2d(192, 128, 1, stride=1)),
           ('8a', nn.ELU()),
           ('9', ReppadConv2d(128, 128, 4, stride=2, padding=1)),
           ('9a', nn.ELU()),
           # 10: maxpool across tracks
           # 11: concat 10
           ('12:+pool', ReppadConv2d(256, 256, 1, stride=1)),
           ('12a', nn.ELU()),
           ('13', ReppadConv2d(256, 256, 4, stride=2, padding=1)),
           ('13a', nn.ELU()),
           # 14: maxpool across tracks
           # 15: concat 14
           ('16:+pool', ReppadConv2d(512, 384, 1, stride=1)),
           ('16a', nn.ELU()),
           ('17', ReppadConv2d(384, 384, 4, stride=2, padding=1)),
           ('17a', nn.ELU()),
           ('18', ReppadConv2d(384, 384, 3, stride=1, padding=1)),
           ('18a', nn.ELU()),
           # ----------------
           ('19', nn.ConvTranspose2d(384, 384, 4, stride=2, padding=1)),
           ('19a', nn.ELU()),
           # 20: maxpool across tracks
           # 21: concat 16a, 20
           ('22:+pool+16a', ReppadConv2d(1152, 384, 1, stride=1)),
           ('22a', nn.ELU()),
           ('23', ReppadConv2d(384, 384, 3, stride=1, padding=1)),
           ('23a', nn.ELU()),
           ('24', nn.ConvTranspose2d(384, 256, 4, stride=2, padding=1)),
           ('24a', nn.ELU()),
           # 25: maxpool across tracks
           # 26: concat 12a, 25
           ('27:+pool+12a', ReppadConv2d(768, 256, 1, stride=1)),
           ('27a', nn.ELU()),
           ('28', ReppadConv2d(256, 256, 3, stride=1, padding=1)),
           ('28a', nn.ELU()),
           ('29', nn.ConvTranspose2d(256, 192, 4, stride=2, padding=1)),
           ('29a', nn.ELU()),
           # 30: maxpool across tracks
           # 31: concat 8a, 30
           ('32:+pool+8a', ReppadConv2d(512, 192, 1, stride=1)),
           ('32a', nn.ELU()),
           ('33', ReppadConv2d(192, 192, 3, stride=1, padding=1)),
           ('33a', nn.ELU()),
           ('34', nn.ConvTranspose2d(192, 96, 4, stride=2, padding=1)),
           ('34a', nn.ELU()),
           # 35: maxpool across tracks
# 36: concat 1a, 35
('37:+pool+1a', ReppadConv2d(256, 96, 1, stride=1)),
('37a', nn.ELU()),
('38', ReppadConv2d(96, 96, 3, stride=1, padding=1)),
('38a', nn.ELU()),
# 39: maxpool across tracks
('40:pool', ReppadConv2d(96, 64, 3, stride=1, padding=1)),
('40a', nn.ELU()),
('41', ReppadConv2d(64, nch_out, 3, stride=1, padding=1)),
Concept demonstration

Open image ↗