Literature
Display
← All papers

Translating a Visual LEGO Manual to a Machine-Executable Plan

Ruocheng Wang · Yunzhi Zhang · Jiayuan Mao · Chin-Yi Cheng · Jiajun Wu

· ECCV 2022 ↗

Computer Vision10 min text · CondensedOriginal paper ↗
Contents

At a glance

MEPNet translates a sequence of assembly diagrams into discrete poses for known parts and previously assembled submodules. Neural networks predict image keypoints, masks, and orientations; geometric attachment constraints then recover translations, and mask-based search refines submodule rotations. The method improves pose reconstruction on synthetic LEGO manuals, adapted manuals from real LEGO sets, and rendered Minecraft construction sequences. The output is an assembly representation, not a demonstrated robot motion, grasp, or insertion controller.

1 Introduction

A manual supplies intermediate assembly states that reduce ambiguity compared with reconstructing an object from its final image. Its components can include new combinations of known bricks, making direct pose regression difficult. MEPNet combines learned image localization with exact discrete connection geometry to handle those unseen submodules.

The input already includes the current 3D assembly and the identities, quantities, and geometry of new components. The task is to place them correctly relative to the base. It does not extract an arbitrary parts inventory from an unprocessed instruction page or solve the downstream physical manipulation problem.

Refer to caption
Figure 1: We study the problem of translating a LEGO manual to a machine-executable plan that can be executed to build the target shape. (a) Screenshots from an original LEGO manual. (b) A machine-executable plan generated by our model MEPNet. (c) LEGO built by executing the generated plan.

2 Related Work

Prior work parses diagrams, reconstructs shape programs, or estimates part poses from images. The comparison to PartAssembly [23] adapts a single-image method to sequential assembly. MEPNet differs by conditioning on the partially assembled state and exploiting discrete attachments, while reusing ideas from keypoint detection [46] and associative embeddings [28].

3 Problem Formulation

Each step receives a voxelized base shape, voxel models of the components to add, and a target diagram. It predicts each component’s translation and rotation relative to the base. Subassemblies create a tree of dependencies because a component may itself require earlier assembly steps.

LEGO studs and anti-studs restrict valid placements to discrete connections. These constraints reduce the search space and rule out modeled interpenetration. The resulting geometric assembly does not itself certify grasp accessibility, insertion forces, or the success of a real robot executing the sequence.

Refer to caption
Figure 2: Our factorized representation of LEGO manuals, which is a tree-structured plan specified by manual images. Each component is either a primitive brick or a submodule, and each submodule is recursively constructed from a sequence of manual images with corresponding components.
Refer to caption
Figure 3: MEPNet consists of two stages. (a) In the first stage, the model detects the 2D center keypoints, the masks, and the rotations for all added components on the manual. (b) and (c): We use a connection-constrained inference subroutine and an inference-by-synthesis subroutine to recover 3D poses of components from 2D information predicted in the first stage.

4 Manual-to-Executable-Plan Network

The first stage estimates two-dimensional keypoints, instance masks, and discrete orientations. The second stage combines these predictions with attachment geometry and camera projection to infer precise component poses. The pipeline can be applied sequentially to the manual, feeding its own reconstructed shape into later steps.

4.1 Neural Pose Estimation

A 3D convolutional encoder projects the current voxelized assembly into the known camera frame and concatenates its features with the target image. Another encoder represents each distinct new component shape. A component-conditioned Hourglass network predicts keypoint heatmaps and offsets, masks with instance embeddings, and orientation classes.

The architecture supports at most five distinct component types per step, while multiple instances can share a type. Keypoint losses combine focal detection and offset regression; masks use semantic classification and contrastive instance grouping. Rotation uses discrete quarter-turn classes with a symmetry-aware implementation. Training has full synthetic keypoint, mask, and orientation supervision.

Equation
ℒ=α⋅ℒkeypoint+β⋅ℒmask+γ⋅ℒrotation.\mathcal{L}=\alpha\cdot\mathcal{L}_{\text{keypoint}}+\beta\cdot\mathcal{L}_{\text{mask}}+\gamma\cdot\mathcal{L}_{\text{rotation}}.

4.2 3D Pose Inference

Under scaled orthographic projection, the image displacement from a component’s keypoint to any anti-stud is known from its geometry and orientation, independently of depth. Match these inferred anti-stud positions to projected studs on the existing assembly, then vote over compatible placements to recover translation.

For submodules, test the four allowed rotations, infer connection-compatible translations for each, and compare rendered component masks with the predicted mask. Select the pose with maximum intersection over union. This uses geometric projection rather than needing the exact color-and-lighting renderer that generated the manual. An isolated later sentence says minimize IoU for the pure-search baseline; the matching criterion is otherwise described as maximizing agreement.

4.3 Implementation Details

A basic brick is represented by a 2×2×2 voxel grid to allow half-brick translations. Primitive keypoints lie at top-surface centers; a submodule uses a topmost brick, with ambiguity handled by searching possible topmost choices. Cameras use scaled orthographic projection. Camera parameters are supplied to all methods, and a separately trained estimator is also tested; the geometric inference depends on these camera estimates.

5 Experiments

Evaluation covers three LEGO datasets and a Minecraft-style construction task. Single-step metrics use correct previous states, while final-shape evaluation rolls predictions forward. These protocols measure different aspects of error accumulation.

5.1 Setup

PartAssembly is adapted with state-aware image features, the same voxel representation, coordinate features, and connection-based pose refinement. Direct3D uses the MEPNet backbone but regresses 3D poses directly, again with coordinate features and discrete refinement.

Component accuracy checks each discrete pose; step accuracy requires every added component in that step to be correct. Chamfer distance compares sampled surface points, with reported LEGO tables scaled by 100,000. Final-shape Chamfer distance uses autoregressive assembly, including predicted submodules. The normalized mistakes-to-complete score instead evaluates incorrect steps with ground-truth base shapes and submodules, so it is not the probability of an entire error-free rollout.

Refer to caption
Figure 4: Example manual images and shapes in our LEGO datasets.

5.2 Results on LEGO

The synthetic LEGO set contains 8,000 training manuals, ten validation manuals, and twenty test manuals, covering about 200,000, 300, and 600 steps. It uses 72 primitive types and includes repeated bricks and composed submodules. The transfer sets contain eleven Classics and five Architecture models with roughly 200 steps each. Unsupported bricks are removed or replaced, and the manuals are re-rendered in the factorized format. This is transfer to adapted real-set designs, not unrestricted evaluation on original commercial manual pages.

MEPNet component accuracy is 96.96% on synthetic data, 88.69% on Classics, and 83.47% on Architecture; Direct3D achieves 88.51%, 34.84%, and 13.71%. The method also leads the reported stepwise and final-shape metrics. Errors remain, especially for heavily occluded or complex submodules.

Submodule-only accuracy rises from 22.27/20.00/23.08% without rotation search to 75.83/90.00/38.46% with it across the three sets. The Architecture submodule Chamfer metric nevertheless slightly worsens, 256.55 to 271.35, so the ablation does not improve every metric.

On Classics and Architecture, MEPNet averages 0.311 and 0.406 seconds per step, versus 10.94 and 70.77 seconds for pure analysis-by-synthesis. The latter exceeds a one-minute-per-component limit in 0.9% and 11.65% of cases. Direct3D is faster, 0.233 and 0.320 seconds, but less accurate. These are inference measurements, not physical assembly cycle times.

Refer to caption
Figure 5: Qualitative Results on the LEGO datasets. Each column contains the ground truth and the predictions from models for a single step. Components added in the step are highlighted in the manual images. To have a straightforward comparison between different models, we render their predictions in the same way as rendering the target manual.
Componentwise Stepwise Setwise
Pose Acc ↑\uparrow (%) CD ↓\downarrow Pose Acc ↑\uparrow (%) CD ↓\downarrow CD ↓\downarrow MTC ↓\downarrow (%)
Synthetic PartAssembly [23] 63.7463.74 393.21393.21 47.0547.05 226.48226.48 321.91321.91 52.7252.72
Direct3D 88.5188.51 32.6032.60 77.3977.39 5.085.08 9.189.18 25.6225.62
MEPNet 96.96\bf 96.96 11.60\bf 11.60 93.41\bf 93.41 1.93\bf 1.93 4.74\bf 4.74 8.63\bf 8.63
Classics PartAssembly [23] 2.262.26 1171.571171.57 0.000.00 296.82296.82 775.29775.29 100.00100.00
Direct3D 34.8434.84 303.56303.56 24.2724.27 3.003.00 67.7567.75 80.7780.77
MEPNet 88.69\bf 88.69 72.79\bf 72.79 90.29\bf 90.29 0.10\bf 0.10 15.52\bf 15.52 13.97\bf 13.97
Architecture PartAssembly [23] 5.125.12 1964.311964.31 3.243.24 858.58858.58 719.88719.88 96.1396.13
Direct3D 13.7113.71 936.77936.77 23.1723.17 227.41227.41 191.28191.28 87.7987.79
MEPNet 83.47\bf 83.47 136.40\bf 136.40 82.23\bf 82.23 15.35\bf 15.35 107.95\bf 107.95 16.00\bf 16.00
Table 1: Quantitative results of models on the three LEGO datasets. Chamfer distance metrics are multiplied by a factor of 10510^{5}. MEPNet outperforms baselines in all metrics on the three datasets.
Refer to caption
Figure 6: Qualitative Results of MEPNet building LEGOs from scratch. We visualize several intermediate steps and final results by rendering them in the same way as the manual images. MEPNet can successfully parse manuals into executable plans with diverse target shapes.
Synthetic Classics Architecture
Pose Acc ↑\uparrow (%) CD ↓\downarrow Pose Acc ↑\uparrow (%) CD ↓\downarrow Pose Acc ↑\uparrow (%) CD ↓\downarrow
PartAssembly [23] 0.000.00 1754.641754.64 0.000.00 2107.112107.11 0.000.00 4816.214816.21
Direct3D 22.7522.75 161.59161.59 10.0010.00 180.86180.86 0.000.00 1539.041539.04
MEPNet (w.o. RS) 22.2722.27 192.93192.93 20.0020.00 25.1725.17 23.0823.08 256.55\bf 256.55
MEPNet 75.83\bf 75.83 23.27\bf 23.27 90.00\bf 90.00 0.25\bf 0.25 38.46\bf 38.46 271.35271.35
Table 2: Quantitative results on the submodules exclusively. Chamfer distance metrics are multiplied by a factor of 10510^{5}. Rotation inference by synthesis (RS) plays an important role in inferring the pose of submodules.
Classics Architecture
Pose Acc ↑\uparrow (%) Time ↓\downarrow (s) TLE ↓\downarrow (%) Pose Acc ↑\uparrow (%) Time ↓\downarrow (s) TLE ↓\downarrow (%)
Direct3D 34.8434.84 0.233{\bf 0.233} 𝟎{\bf 0} 13.7113.71 0.320{\bf 0.320} 𝟎{\bf 0}
MEPNet (a-by-s) 48.4248.42 10.9410.94 0.90.9 34.1434.14 70.7770.77 11.6511.65
MEPNet 88.69 0.311 0 83.47 0.406 0
Table 3: Comparison with an ablation model based purely on analysis-by-synthesis. We show the component pose accuracy and the average inference time per step (which may include multiple components). TLE (Time-Limit-Exceeded) measures the percentage of components whose prediction cannot terminate in 1 minute.

5.3 Results on 3D-Craft

The 3D-Craft experiment renders human-built Minecraft houses as one-brick-per-step manuals, using eighty training houses and five each for validation and testing. Five common block types are retained, and orientation prediction is removed. MEPNet projects all ground or neighbor-supported candidate positions and matches them to the detected brick center. Translation accuracy is 86.3%, compared with 75.8% for Direct3D and 49.8% for VoxelCNN. This validates another discrete attachment rule, not general continuous furniture assembly.

Refer to caption
Figure 7: Qualitative results on the 3D-Craft dataset. Ground truth and predictions of new bricks are zoomed in. Predictions of baseline models often lie in a small neighborhood of the ground truth positions, while our model is more accurate.

6 Discussion

Combining image predictions with attachment geometry substantially improves pose reconstruction and transfer to new compositions in the tested domains. The method relies on supplied component models, discrete poses, camera information, and domain-specific connection rules. Errors can accumulate through sequential assembly. Broader connections, continuous placements, and joint reasoning across steps are open directions; physical robotic execution is outside the evaluation.

Supplementary Materials: Translating Visual LEGO Manuals to a Machine-Executable Plan

The supplement documents synthetic data construction, implementation and training, projection geometry, camera estimation, ablations, limitations, asset attribution, and full assembly visualizations.

1 Synthetic Dataset Generation

Synthetic manuals are generated by building valid random assemblies forward, then removing visible components backward to create feasible intermediate instruction images.

Forward Stage

The generator samples a bounding box, partitions it, and fills regions or grows boundaries with primitive bricks, stacked repetitions, and randomly composed submodules. Rejection sampling requires attachment to another component and prevents collisions. This produces varied valid structures but also imposes a specific procedural training distribution.

Backward Stage

Starting from a finished shape, the backward stage identifies removable components visible from top and side views. Components are grouped into steps with no more than ten instances and five distinct types. Removing groups and rendering the resulting states yields the manual sequence. This visibility rule helps ensure the added parts can be inferred from the diagrams.

2 3D-Craft Dataset Generation

Minecraft manuals preserve the crowdworker construction order and add one block per step. A predefined viewpoint is selected to show the new block when possible; otherwise one is chosen randomly. The rendering process therefore supplies controlled views rather than relying on arbitrary user photographs.

3 Implementation Details

Implementation details include symmetry-aware orientation classes, common geometric refinement for baselines, domain-specific candidate positions, voxel sizes, and complete training hyperparameters.

3.1 Symmetry-aware rotation prediction.

Seven output classes jointly encode rotational symmetry and orientation: one class for quarter-turn-invariant bricks, two for half-turn symmetry, and four for asymmetric bricks. This prevents counting physically identical orientations as distinct errors.

Baseline models on LEGO datasets.

Baseline translations are rounded to half-unit coordinates, then a local neighborhood is searched for a connected, collision-free placement nearest the prediction. If no valid candidate is found, the rounded prediction is kept. Thus this postprocessing is not a guarantee that every baseline output satisfies the connection constraints.

Equation
{(x+δx,y+δy,z+δz)|δi∈{0,−0.5,0.5,−1,1}}\{(x+\delta_{x},y+\delta_{y},z+\delta_{z})|\quad\delta_{i}\in\{0,-0.5,0.5,-1,1\}\}

3D-Craft dataset.

For Minecraft, enumerate positions on the ground or next to existing blocks and select the one whose projection is closest to the predicted center. Baselines receive analogous local refinement. The rule replaces LEGO stud matching while retaining the same geometric candidate-selection idea.

Hyperparameters

Base assemblies use 130³ voxel grids and new components use 65³ grids. Keypoint and mask losses have weight one, and orientation loss has weight 0.1. LEGO MEPNet trains for fifteen epochs at batch size sixteen and learning rate 0.00025; the Minecraft version uses ten epochs and batch size eight. The original table preserves all baseline settings and optimizer differences.

Equation
ℒ=α⋅ℒkeypoint+β⋅ℒmask+γ⋅ℒrotation.\mathcal{L}=\alpha\cdot\mathcal{L}_{\text{keypoint}}+\beta\cdot\mathcal{L}_{\text{mask}}+\gamma\cdot\mathcal{L}_{\text{rotation}}.
Dataset Learning Rate Batch Size Epochs
LEGO MEPNet 0.000250.00025 1616 1515
Direct3D 0.000250.00025 1616 1515
PartAssembly 0.0010.001 88 1515
3D-Craft MEPNet 0.000250.00025 88 1010
Direct3D 0.000250.00025 88 1010
VoxelCNN 0.010.01 88 1010
Table 1: Hyperparameters of models.

4 Details of 3D Pose Inference.

The supplement derives why projected attachment offsets can be computed without unknown component depth and describes special cases for initial placements and ambiguous topmost keypoints.

Inferring 2D locations of anti-studs of the new component.

Scaled orthographic projection is affine. Subtracting the projections of two points cancels translation, so their image offset depends only on their known relative geometry, orientation, and camera scale/rotation. A detected keypoint therefore determines projected anti-stud locations for matching to the base. This argument relies on the stated projection model and does not automatically hold for arbitrary perspective images.

Equation
v′=A​Ki​n​t​Ke​x​t​v=(100010)​(s000s0001)​(RT)​vv^{\prime}=AK_{int}K_{ext}v=\begin{pmatrix}1&0&0\\ 0&1&0\\ \end{pmatrix}\begin{pmatrix}s&0&0\\ 0&s&0\\ 0&0&1\\ \end{pmatrix}\begin{pmatrix}R&T\\ \end{pmatrix}v

Empty base shape

When the base is empty, the component need not attach to an existing brick. The system searches allowed ground-plane translations and chooses the projected keypoint closest to the detection.

Handling submodules with multiple topmost primitive bricks.

If several submodule bricks share the maximum height, test each as the possible detected keypoint together with each of four rotations. With n topmost choices, this produces at most 4n candidate poses, and mask intersection over union selects the best. This handles a specific keypoint ambiguity without training a separate classifier for every possible submodule.

5 Camera Parameters

The camera procedure defines the rendering distribution and a separate estimator for orientation, scale, and image translation.

Camera parameters of the LEGO datasets.

A small set of original manuals and matching 3D shapes is used to fit camera parameters through differentiable mask overlap. Classics and Architecture use a common rendering camera; synthetic examples vary scale, translation, and angles around that range. Consequently, the camera distribution is designed to resemble the selected manuals rather than testing arbitrary viewpoints.

Camera parameter estimation.

The camera estimator takes a known 3D shape and target image, extending a prior orientation model with scale and translation heads. It is trained with synthetic parameter supervision. The authors report unchanged pose and Chamfer results when substituting its predictions for ground-truth parameters across the three LEGO datasets, within this controlled camera distribution.

6 Additional Results

Additional ablations compare the image backbone and isolate translation versus rotation errors.

Pose Acc ↑\uparrow (%) CD ↓\downarrow
MEPNet 88.69 72.79
ResNet+DeConv[5] 78.2878.28 96.4396.43
Table 2: Comparison with an ablation model where we replaced the Hourglass network with a encoder-decoder network using ResNet and deconvolutional layers [5]. Chamfer distance metrics are multiplied by a factor of 10510^{5}.
Pose Acc ↑\uparrow (%) CD ↓\downarrow
MEPNet (GT Trans.) 96.38 32.92
MEPNet (GT Rot.) 90.5890.58 67.2967.29
Table 3: An ablation study where we use ground-truth translation or rotation to replace MEPNet’s predictions. It’s harder for MEPNet to predict translation than rotation. Chamfer distance metrics are multiplied by a factor of 10510^{5}.

Ablation of the Hourglass architecture.

Replacing Hourglass with a ResNet-and-deconvolution backbone reduces Classics component accuracy from 88.69% to 78.28% and worsens scaled Chamfer distance from 72.79 to 96.43. This supports the chosen backbone under the reported training setup.

Pose accuracy with groundtruth translations or rotations.

Using ground-truth translations raises Classics accuracy to 96.38%, while ground-truth rotations raise it to 90.58%. Translation inference is therefore the larger remaining error source in this diagnostic, although the interventions are oracle substitutions rather than deployable improvements.

7 Discussions and Limitations

The method explicitly encodes domain-specific connection rules and currently operates in discrete placement spaces. Extending to peg-and-hole or mortise-and-tenon assemblies requires new geometry and may introduce continuous actions. Sequential predictions accumulate errors, motivating future inference that jointly uses multiple manual steps. These limits bound the generality of the machine-executable-plan claim.

8 Attribution and License for Assets

The source attributes brick geometry and some assembly sequences to LDraw and the LDraw Model Repository, imported with ImportLDraw. Manual renderings use LPub3D and Bricklink Studio. The authors state that the project assets use Creative Commons or other open licensing terms; the original attribution links are retained in the source material.

9 Step-by-Step Building Visualization

The complete supplementary assembly montage shows selected sets reconstructed with 100% accuracy, including submodule construction before the main assembly. These successful examples illustrate how predictions compose across steps; they are selected successes rather than a full-set success-rate measurement.

Refer to caption
Figure 1: Visualization of the full assembly process by MEPNet. Assembly of submodules are shown at the beginning of each set.
Concept demonstration

Open image ↗