At a glance
MEPNet translates a sequence of assembly diagrams into discrete poses for known parts and previously assembled submodules. Neural networks predict image keypoints, masks, and orientations; geometric attachment constraints then recover translations, and mask-based search refines submodule rotations. The method improves pose reconstruction on synthetic LEGO manuals, adapted manuals from real LEGO sets, and rendered Minecraft construction sequences. The output is an assembly representation, not a demonstrated robot motion, grasp, or insertion controller.
1 Introduction
A manual supplies intermediate assembly states that reduce ambiguity compared with reconstructing an object from its final image. Its components can include new combinations of known bricks, making direct pose regression difficult. MEPNet combines learned image localization with exact discrete connection geometry to handle those unseen submodules.
The input already includes the current 3D assembly and the identities, quantities, and geometry of new components. The task is to place them correctly relative to the base. It does not extract an arbitrary parts inventory from an unprocessed instruction page or solve the downstream physical manipulation problem.
2 Related Work
Prior work parses diagrams, reconstructs shape programs, or estimates part poses from images. The comparison to PartAssembly [23] adapts a single-image method to sequential assembly. MEPNet differs by conditioning on the partially assembled state and exploiting discrete attachments, while reusing ideas from keypoint detection [46] and associative embeddings [28].
3 Problem Formulation
Each step receives a voxelized base shape, voxel models of the components to add, and a target diagram. It predicts each component’s translation and rotation relative to the base. Subassemblies create a tree of dependencies because a component may itself require earlier assembly steps.
LEGO studs and anti-studs restrict valid placements to discrete connections. These constraints reduce the search space and rule out modeled interpenetration. The resulting geometric assembly does not itself certify grasp accessibility, insertion forces, or the success of a real robot executing the sequence.
4 Manual-to-Executable-Plan Network
The first stage estimates two-dimensional keypoints, instance masks, and discrete orientations. The second stage combines these predictions with attachment geometry and camera projection to infer precise component poses. The pipeline can be applied sequentially to the manual, feeding its own reconstructed shape into later steps.
4.1 Neural Pose Estimation
A 3D convolutional encoder projects the current voxelized assembly into the known camera frame and concatenates its features with the target image. Another encoder represents each distinct new component shape. A component-conditioned Hourglass network predicts keypoint heatmaps and offsets, masks with instance embeddings, and orientation classes.
The architecture supports at most five distinct component types per step, while multiple instances can share a type. Keypoint losses combine focal detection and offset regression; masks use semantic classification and contrastive instance grouping. Rotation uses discrete quarter-turn classes with a symmetry-aware implementation. Training has full synthetic keypoint, mask, and orientation supervision.
Equation
4.2 3D Pose Inference
Under scaled orthographic projection, the image displacement from a component’s keypoint to any anti-stud is known from its geometry and orientation, independently of depth. Match these inferred anti-stud positions to projected studs on the existing assembly, then vote over compatible placements to recover translation.
For submodules, test the four allowed rotations, infer connection-compatible translations for each, and compare rendered component masks with the predicted mask. Select the pose with maximum intersection over union. This uses geometric projection rather than needing the exact color-and-lighting renderer that generated the manual. An isolated later sentence says minimize IoU for the pure-search baseline; the matching criterion is otherwise described as maximizing agreement.
4.3 Implementation Details
A basic brick is represented by a 2×2×2 voxel grid to allow half-brick translations. Primitive keypoints lie at top-surface centers; a submodule uses a topmost brick, with ambiguity handled by searching possible topmost choices. Cameras use scaled orthographic projection. Camera parameters are supplied to all methods, and a separately trained estimator is also tested; the geometric inference depends on these camera estimates.
5 Experiments
Evaluation covers three LEGO datasets and a Minecraft-style construction task. Single-step metrics use correct previous states, while final-shape evaluation rolls predictions forward. These protocols measure different aspects of error accumulation.
5.1 Setup
PartAssembly is adapted with state-aware image features, the same voxel representation, coordinate features, and connection-based pose refinement. Direct3D uses the MEPNet backbone but regresses 3D poses directly, again with coordinate features and discrete refinement.
Component accuracy checks each discrete pose; step accuracy requires every added component in that step to be correct. Chamfer distance compares sampled surface points, with reported LEGO tables scaled by 100,000. Final-shape Chamfer distance uses autoregressive assembly, including predicted submodules. The normalized mistakes-to-complete score instead evaluates incorrect steps with ground-truth base shapes and submodules, so it is not the probability of an entire error-free rollout.
5.2 Results on LEGO
The synthetic LEGO set contains 8,000 training manuals, ten validation manuals, and twenty test manuals, covering about 200,000, 300, and 600 steps. It uses 72 primitive types and includes repeated bricks and composed submodules. The transfer sets contain eleven Classics and five Architecture models with roughly 200 steps each. Unsupported bricks are removed or replaced, and the manuals are re-rendered in the factorized format. This is transfer to adapted real-set designs, not unrestricted evaluation on original commercial manual pages.
MEPNet component accuracy is 96.96% on synthetic data, 88.69% on Classics, and 83.47% on Architecture; Direct3D achieves 88.51%, 34.84%, and 13.71%. The method also leads the reported stepwise and final-shape metrics. Errors remain, especially for heavily occluded or complex submodules.
Submodule-only accuracy rises from 22.27/20.00/23.08% without rotation search to 75.83/90.00/38.46% with it across the three sets. The Architecture submodule Chamfer metric nevertheless slightly worsens, 256.55 to 271.35, so the ablation does not improve every metric.
On Classics and Architecture, MEPNet averages 0.311 and 0.406 seconds per step, versus 10.94 and 70.77 seconds for pure analysis-by-synthesis. The latter exceeds a one-minute-per-component limit in 0.9% and 11.65% of cases. Direct3D is faster, 0.233 and 0.320 seconds, but less accurate. These are inference measurements, not physical assembly cycle times.
| Componentwise | Stepwise | Setwise | |||||
| Pose Acc (%) | CD | Pose Acc (%) | CD | CD | MTC (%) | ||
| Synthetic | PartAssembly [23] | ||||||
| Direct3D | |||||||
| MEPNet | |||||||
| Classics | PartAssembly [23] | ||||||
| Direct3D | |||||||
| MEPNet | |||||||
| Architecture | PartAssembly [23] | ||||||
| Direct3D | |||||||
| MEPNet | |||||||
| Synthetic | Classics | Architecture | ||||
| Pose Acc (%) | CD | Pose Acc (%) | CD | Pose Acc (%) | CD | |
| PartAssembly [23] | ||||||
| Direct3D | ||||||
| MEPNet (w.o. RS) | ||||||
| MEPNet | ||||||
| Classics | Architecture | |||||
| Pose Acc (%) | Time (s) | TLE (%) | Pose Acc (%) | Time (s) | TLE (%) | |
| Direct3D | ||||||
| MEPNet (a-by-s) | ||||||
| MEPNet | 88.69 | 0.311 | 0 | 83.47 | 0.406 | 0 |
5.3 Results on 3D-Craft
The 3D-Craft experiment renders human-built Minecraft houses as one-brick-per-step manuals, using eighty training houses and five each for validation and testing. Five common block types are retained, and orientation prediction is removed. MEPNet projects all ground or neighbor-supported candidate positions and matches them to the detected brick center. Translation accuracy is 86.3%, compared with 75.8% for Direct3D and 49.8% for VoxelCNN. This validates another discrete attachment rule, not general continuous furniture assembly.
6 Discussion
Combining image predictions with attachment geometry substantially improves pose reconstruction and transfer to new compositions in the tested domains. The method relies on supplied component models, discrete poses, camera information, and domain-specific connection rules. Errors can accumulate through sequential assembly. Broader connections, continuous placements, and joint reasoning across steps are open directions; physical robotic execution is outside the evaluation.
Supplementary Materials: Translating Visual LEGO Manuals to a Machine-Executable Plan
The supplement documents synthetic data construction, implementation and training, projection geometry, camera estimation, ablations, limitations, asset attribution, and full assembly visualizations.
1 Synthetic Dataset Generation
Synthetic manuals are generated by building valid random assemblies forward, then removing visible components backward to create feasible intermediate instruction images.
Forward Stage
The generator samples a bounding box, partitions it, and fills regions or grows boundaries with primitive bricks, stacked repetitions, and randomly composed submodules. Rejection sampling requires attachment to another component and prevents collisions. This produces varied valid structures but also imposes a specific procedural training distribution.
Backward Stage
Starting from a finished shape, the backward stage identifies removable components visible from top and side views. Components are grouped into steps with no more than ten instances and five distinct types. Removing groups and rendering the resulting states yields the manual sequence. This visibility rule helps ensure the added parts can be inferred from the diagrams.
2 3D-Craft Dataset Generation
Minecraft manuals preserve the crowdworker construction order and add one block per step. A predefined viewpoint is selected to show the new block when possible; otherwise one is chosen randomly. The rendering process therefore supplies controlled views rather than relying on arbitrary user photographs.
3 Implementation Details
Implementation details include symmetry-aware orientation classes, common geometric refinement for baselines, domain-specific candidate positions, voxel sizes, and complete training hyperparameters.
3.1 Symmetry-aware rotation prediction.
Seven output classes jointly encode rotational symmetry and orientation: one class for quarter-turn-invariant bricks, two for half-turn symmetry, and four for asymmetric bricks. This prevents counting physically identical orientations as distinct errors.
Baseline models on LEGO datasets.
Baseline translations are rounded to half-unit coordinates, then a local neighborhood is searched for a connected, collision-free placement nearest the prediction. If no valid candidate is found, the rounded prediction is kept. Thus this postprocessing is not a guarantee that every baseline output satisfies the connection constraints.
Equation
3D-Craft dataset.
For Minecraft, enumerate positions on the ground or next to existing blocks and select the one whose projection is closest to the predicted center. Baselines receive analogous local refinement. The rule replaces LEGO stud matching while retaining the same geometric candidate-selection idea.
Hyperparameters
Base assemblies use 130³ voxel grids and new components use 65³ grids. Keypoint and mask losses have weight one, and orientation loss has weight 0.1. LEGO MEPNet trains for fifteen epochs at batch size sixteen and learning rate 0.00025; the Minecraft version uses ten epochs and batch size eight. The original table preserves all baseline settings and optimizer differences.
Equation
4 Details of 3D Pose Inference.
The supplement derives why projected attachment offsets can be computed without unknown component depth and describes special cases for initial placements and ambiguous topmost keypoints.
Inferring 2D locations of anti-studs of the new component.
Scaled orthographic projection is affine. Subtracting the projections of two points cancels translation, so their image offset depends only on their known relative geometry, orientation, and camera scale/rotation. A detected keypoint therefore determines projected anti-stud locations for matching to the base. This argument relies on the stated projection model and does not automatically hold for arbitrary perspective images.
Equation
Empty base shape
When the base is empty, the component need not attach to an existing brick. The system searches allowed ground-plane translations and chooses the projected keypoint closest to the detection.
Handling submodules with multiple topmost primitive bricks.
If several submodule bricks share the maximum height, test each as the possible detected keypoint together with each of four rotations. With n topmost choices, this produces at most 4n candidate poses, and mask intersection over union selects the best. This handles a specific keypoint ambiguity without training a separate classifier for every possible submodule.
5 Camera Parameters
The camera procedure defines the rendering distribution and a separate estimator for orientation, scale, and image translation.
Camera parameters of the LEGO datasets.
A small set of original manuals and matching 3D shapes is used to fit camera parameters through differentiable mask overlap. Classics and Architecture use a common rendering camera; synthetic examples vary scale, translation, and angles around that range. Consequently, the camera distribution is designed to resemble the selected manuals rather than testing arbitrary viewpoints.
Camera parameter estimation.
The camera estimator takes a known 3D shape and target image, extending a prior orientation model with scale and translation heads. It is trained with synthetic parameter supervision. The authors report unchanged pose and Chamfer results when substituting its predictions for ground-truth parameters across the three LEGO datasets, within this controlled camera distribution.
6 Additional Results
Additional ablations compare the image backbone and isolate translation versus rotation errors.
Ablation of the Hourglass architecture.
Replacing Hourglass with a ResNet-and-deconvolution backbone reduces Classics component accuracy from 88.69% to 78.28% and worsens scaled Chamfer distance from 72.79 to 96.43. This supports the chosen backbone under the reported training setup.
Pose accuracy with groundtruth translations or rotations.
Using ground-truth translations raises Classics accuracy to 96.38%, while ground-truth rotations raise it to 90.58%. Translation inference is therefore the larger remaining error source in this diagnostic, although the interventions are oracle substitutions rather than deployable improvements.
7 Discussions and Limitations
The method explicitly encodes domain-specific connection rules and currently operates in discrete placement spaces. Extending to peg-and-hole or mortise-and-tenon assemblies requires new geometry and may introduce continuous actions. Sequential predictions accumulate errors, motivating future inference that jointly uses multiple manual steps. These limits bound the generality of the machine-executable-plan claim.
8 Attribution and License for Assets
The source attributes brick geometry and some assembly sequences to LDraw and the LDraw Model Repository, imported with ImportLDraw. Manual renderings use LPub3D and Bricklink Studio. The authors state that the project assets use Creative Commons or other open licensing terms; the original attribution links are retained in the source material.
9 Step-by-Step Building Visualization
The complete supplementary assembly montage shows selected sets reconstructed with 100% accuracy, including submodule construction before the main assembly. These successful examples illustrate how predictions compose across steps; they are selected successes rather than a full-set success-rate measurement.