Literature
Display
← All papers

iPack: Intuitive Bin Packing with Large Language Models

Yannik Blei · Michael Krawez · Adrian Göß · Devadas Vijayan Sheela · Tobias Jülg · Pierre Krack · Florian Walter · Wolfram Burgard

Preprint posted · arXiv (preprint) ↗

Robotics8 min text · CondensedOriginal paper ↗
Contents

At a glance

iPack combines visual object recognition, language-generated packing rules, mixed-integer spatial optimization, and robot execution. It knows the complete item set in advance and can choose among available container sizes. Language models supply preferences such as keeping a heavy item below a fragile one, which ordinary geometric packing does not express.

The method requires no task-specific model training, but relies on pretrained foundation models. In the reported planning benchmark, adding integrity constraints reduces mean human-preference violations from 2.4 to 0.2 per box while density falls from 47.8% to 44.5%. Physical trials report 92.6% manipulation success under constrained initial arrangements. Neither this score nor preference agreement measures actual product damage.

I Introduction

Dense, stable packing can still be inappropriate for groceries: weight, fragility, moisture, and item identity affect which products should touch or support one another. Manually listing every rule is difficult when product classes change.

iPack asks foundation models for object properties and pairwise integrity constraints, then solves the geometric arrangement explicitly. Its main contributions are this modular pipeline, a human-derived stacking metric, and simulation/robot evaluations. Open vocabulary means the system is not limited to a fixed trained grocery-class list; it does not establish reliable handling of arbitrary objects or novel domains.

The authors provide source code, data, and manipulation software.

Refer to caption
Fig. 1: iPack ensures product integrity during bin packing. It employs foundation models to identify items and packing constraints from a scene image. It then uses a mixed-integer linear programming approach to compute a spatially optimized packing scheme. A robot manipulator places each item at its target position.

II Related Work

The full architecture separates perception, constrained packing, and manipulation. Prior work supplies both the geometric packing algorithms and the vision/motion components used here.

Refer to caption
Fig. 2: Overview of iPack: We assume a random but separated arrangement of items on the table. We employ GroundingDINO [11] to detect all objects, followed by SAM 2 [12] to obtain segmentation masks. We employ a VLM to obtain estimates of the sizes and the weights of the objects. In the planning step, we utilize a large language model (LLM) to determine product integrity constraints. We apply a mixed-integer linear programming approach [13] to optimize the packing space. In the final execution step, we perform the manipulation tasks using a real robot.

II-A Bin Packing Methods

Online packers such as GOPT place items in arrival order without knowing future arrivals. Offline methods can optimize a complete known set. Existing work addresses density, stability, complex shapes, placement correction, and scalable layer-based optimization.

Grocery-specific approaches estimate fragility using tactile/visual sensing or learn human packing sequences from demonstrations. iPack instead obtains semantic rules from pretrained language models and applies them to an offline optimizer. Its comparison with online packing therefore includes an information-availability difference, not just a different planning algorithm.

II-B Grocery Store Automation

Earlier supermarket robots mainly navigate, identify requested products, pick from shelves, guide customers, or automate billing. iPack addresses the later packing stage. It is not evaluated as a complete store-navigation or autonomous-shopping system.

III The iPack approach

Perception estimates the visible items and their properties. Planning converts semantic restrictions and approximate box geometries into target placements. Execution then picks items in order of increasing target height so lower objects are placed first.

III-A Perception Step

GroundingDINO detects candidate objects, and SAM 2 segments them. Numbered visual prompts let a VLM reject distractors such as the container or gripper, name the groceries, and estimate size and weight in one call.

An optional agentic internet search, given the country of origin, supplies standardized package dimensions and weights; produce is estimated from appearance. These are estimates, not calibrated physical measurements. Planning adds padding to tolerate some error.

After manipulations, Hungarian matching combines mask intersection-over-union and centroid distance to maintain object identities. Mask overlap also checks whether a grasp removed the intended object. The pipeline still assumes initially separated objects.

Refer to caption
Fig. 3: Visualization of the perception step. We detect objects using GroundingDINO. A VLM then infers the object type and properties.

III-B Planning Step

The planner represents each item as an axis-aligned bounding box inside a rectangular container, with Cartesian positions and discrete 90° rotations. An LLM marks ordered pairs for which placing one item on the other may damage the lower item. If such a pair overlaps in horizontal projection, the protected item must be above the other. Separate contact-avoidance rules prohibit harmful side-by-side contact.

Geometric constraints keep all boxes inside the container and prevent box overlap. If multiple containers are available, the formulation can minimize selected container volume, relaxing placements in unselected containers with a big-M construction. The implementation uses the PySCIPOpt/SCIP mixed-integer optimization interface.

Optional execution restrictions add LLM-identified unstable stacks to the prohibited pairs, apply 5 mm collision padding, and restrict items to stable orientations, usually with their shortest dimension vertical. Minimizing summed vertical positions encourages low arrangements. It is not a complete mechanical stability or support proof.

Each solve has a 30-second limit and returns the best feasible incumbent if one exists. Thus “optimal” describes the optimization objective; a time-limited result is not necessarily a certified global optimum. Incorrect properties, omitted rules, or inconsistent constraints can still produce poor plans or infeasibility.

Example three-dimensional grocery packing, preserving all original box geometry, axis labels, ticks, and separation.
Fig. 4: Example packing scheme returned by the optimizer. The bread loaf (brown) and the frozen spinach (green) are separated by contact avoidance constraints. The set ℱ\mathcal{F} of critical item pairs ensures a protected position of the banana (yellow).

III-C Execution Step

Execution follows the target-height ordering. For each item, principal-component analysis of its segmentation mask sets the gripper’s horizontal orientation along the smaller principal component. The RGB-mask centroid sets horizontal position, mean masked depth sets height, and the approach is vertical.

A Contact-GraspNet-style point-cloud approach was considered but is sensitive to reflective/translucent surfaces in this setup. The authors also describe preliminary failures of Octo/OpenVLA on unseen items; these are not a controlled benchmark establishing general failure of those models.

An RRT planner executes a vertical lift, transit above the target, and downward placement. The system remains dependent on reliable depth, centroid grasps, clearance, and placement accuracy. The method is modular enough to substitute a better grasping or motion controller.

Refer to caption
Fig. 5: Visualization of the grasp pose construction. The grasping angle is aligned with the smaller principal component of the segmentation mask (left). The grasp position is the centroid of the RGB mask, with the zz-coordinate set to the average of the RGB-D depth values (top-right). The approach direction is set to be vertical. We display the point cloud samples in red and the gripper-object contact points in yellow.

IV Experimental Evaluation

Experiments separate model choice, geometric planning, and end-to-end robot execution. The retained figures and tables include both density and preference violations, since improving one can worsen the other.

IV-A Dataset

Simulation uses the HOPE dataset’s 28 scanned toy grocery items plus five newly scanned fragile objects. Physical trials use a separate set of store-bought household groceries. These are limited object collections, not evidence of unrestricted product coverage.

IV-B Hardware Setup

The physical system uses a Franka Research 3 arm, custom Fin Ray-inspired compliant fingers, and an embedded thin-film force sensor for closed-loop grip-force control. An Orbbec Femto Bolt supplies RGB-D data. Local model inference runs on two NVIDIA RTX A6000 GPUs; a separate real-time Linux machine handles robot control.

Refer to caption
Fig. 6: Visualization of our custom gripper. A thin-film force sensor is located in the left finger.

IV-C Packing Consistency Score ( C ), a data-driven metric for human-like packing

Eighty-four participants supplied 8,820 pairwise packing preferences across 14 classes. The estimated probability that class i belongs below class k defines the reference preference. Evaluation considers ordered pairs whose placed meshes overlap in horizontal projection and whose vertical order forms a stack.

A stacked pair counts as a violation when its human-preference probability is below 0.4. The Packing Consistency Score C sums these violations; lower is better. Mean aC is reported per box, alongside density and planning time. Despite the name, C is a violation count, not a normalized agreement rate.

The score depends on how many pairs are stacked and does not directly assess crushing, moisture transfer, physical stability, or side contact. The tables therefore also retain the number of overlaps. Density is summed object volume divided by container volume, not a manipulation-success score.

Equation
G={c1,…,cn}.G=\{c_{1},\dots,c_{n}\}.
Refer to caption
Fig. 7: Visualization of the web-based survey. Participants are asked to click on the objects in their preferred packing sequence. The example items (heavy water bottle and fragile noodles) are not part of the actual survey to avoid biasing the results.
Equations
𝕀viol​(i,k)={1,if ​P​(ci≤ck)<θ0,otherwise.\mathbb{I}_{\text{viol}}(i,k)=\begin{cases}1,&\text{if }P(c_{i}\leq c_{k})<\theta\\ 0,&\text{otherwise.}\end{cases}
C=∑(i,k)∈𝒪𝕀viol​(i,k).C=\sum_{(i,k)\in\mathcal{O}}\mathbb{I}_{\text{viol}}(i,k).

IV-D VLM Sensitivity Study

Each planning trial chooses four fragile and five non-fragile objects. Starting from a 50 cm cube, binary search shrinks the container to 2 mm precision while retaining space for each individual item. The paper reports ten repetitions. These are solver-based estimates of suitable size, without a guarantee that time-limited infeasibility proves no smaller packing exists.

GPT-5 yields the lowest reported mean violation count, 0.2, with density 0.445. GPT-5-mini gives 0.5/0.433; Claude Sonnet 4.5 gives 1.6/0.478; Gemini 3 Flash gives 0.75/0.466. Claude’s denser result comes with more violations. The source’s phrase “scores the lowest” means worst consistency, not the smallest numeric C.

The listed 2.5–3.7-second timings exclude variable API response delays. GPT-5 is selected for subsequent experiments. No confidence intervals or detailed repeated-run distribution are supplied.

Table I: Results of the VLM sensitivity study in terms of density, a​CaC and planning time. NoN_{o} denotes the average number of item overlaps. Optimization times do not include the response delay of the API.
Density aC(↓)aC(\downarrow) NoN_{o} Time [s]
GPT-5 0.445 0.2 6.5 2.990
GPT-5-mini 0.433 0.5 5.6 3.656
Claude Sonnet 4.5 0.478 1.6 7.1 2.516
Gemini 3 Flash 0.466 0.75 6.0 2.586

IV-E Planning Baseline Comparison

iPack achieves density 0.445 and aC 0.2, compared with 0.213/1.5 for Dube’s heuristic, 0.252/1.4 for online GOPT, and 0.167/0.4 for the layer-based Elhedhli method. The latter stacks fewer pairs, averaging 1.5 overlaps versus iPack’s 6.5, which also changes opportunities for violations.

Removing iPack’s integrity constraints increases density to 0.478 but raises aC to 2.4. This ablation most directly isolates the value of semantic restrictions within the same planner. The fastest baseline is Dube at 0.173 seconds; iPack is 2.990 seconds. Language-model/API delay is excluded from its optimization timing.

The reported 1,178.702-second Elhedhli result is a particular implementation/benchmark measurement, not a universal runtime bound. Solver cost grows with packing difficulty, and the 30-second limit means the returned plan can be an incumbent rather than an optimum.

The tested offline baseline implementations are Wadaboa/3d-bpp and jerry800416/3D-bin-packing.

Table II: Results of the packing evaluation in terms of density and a​CaC. iPack (vanilla) denotes our approach without product integrity constraints. NoN_{o} denotes the average number of item overlaps.
Density aC(↓)aC(\downarrow) NoN_{o} Time [s]
Dube et al. [17] 0.213 1.5 3.2 0.173
GOPT [14] 0.252 1.4 8.2 0.357
Elhedhli et al. [16] 0.167 0.4 1.5 1178.702
iPack 0.445 0.2 6.5 2.990
iPack (vanilla) 0.478 2.4 6.2 3.631

IV-F Full System Evaluation

Hardware trials require non-overlapping items lying flat, a centroid grasp that fits the gripper, and flat target orientations. Adding these assumptions and hardware restrictions reduces simulated average density to 26.4%, substantially below the unconstrained planning table.

The robot packs sets of 6, 8, 10, and 12 items, with five trials per size, choosing between two containers. It reports 92.6% manipulation success, defined as picking an item from the table and placing it in the box. Figure 9 shows zero mean preference violations at 6–10 items and about one at 12; it does not show a smooth increase at every size. The paper does not provide the complete per-trial counts behind the aggregate rate.

Failures arise from depth noise and slipping, gripper–wall collisions, and placement errors that create later collisions or preference violations. A successful pick-and-place does not establish correct pose, full-box success, or undamaged goods. Contact-rich pushing and closed-loop placement correction are proposed future improvements.

Refer to caption
Fig. 8: Results of the Full System Evaluation. We show packed boxes for 66 (left), 88 (middle) and 1010 (right) objects.
Original line chart of manipulation success and mean constraint violations for six, eight, ten and twelve objects.
Fig. 9: Results of the real-world experiments. We report the C metric and the manipulation success rate.

V Conclusions

iPack demonstrates that semantic constraints from pretrained models can improve agreement with human grocery-stacking preferences while preserving useful packing density. Its strongest evidence is the constraint-removal ablation plus limited physical trials.

Robust operation remains limited by estimated properties, object layout assumptions, approximate geometry, manipulation errors, and API/solver variability. Pushing and learned corrective actions are future work rather than capabilities demonstrated here.

Acknowledgment

The work acknowledges partial support from the German Federal Ministry of Research, Technology and Space through the Robotics Institute Germany.

Concept demonstration

Open image ↗