At a glance
iPack combines visual object recognition, language-generated packing rules, mixed-integer spatial optimization, and robot execution. It knows the complete item set in advance and can choose among available container sizes. Language models supply preferences such as keeping a heavy item below a fragile one, which ordinary geometric packing does not express.
The method requires no task-specific model training, but relies on pretrained foundation models. In the reported planning benchmark, adding integrity constraints reduces mean human-preference violations from 2.4 to 0.2 per box while density falls from 47.8% to 44.5%. Physical trials report 92.6% manipulation success under constrained initial arrangements. Neither this score nor preference agreement measures actual product damage.
I Introduction
Dense, stable packing can still be inappropriate for groceries: weight, fragility, moisture, and item identity affect which products should touch or support one another. Manually listing every rule is difficult when product classes change.
iPack asks foundation models for object properties and pairwise integrity constraints, then solves the geometric arrangement explicitly. Its main contributions are this modular pipeline, a human-derived stacking metric, and simulation/robot evaluations. Open vocabulary means the system is not limited to a fixed trained grocery-class list; it does not establish reliable handling of arbitrary objects or novel domains.
The authors provide source code, data, and manipulation software.
II Related Work
The full architecture separates perception, constrained packing, and manipulation. Prior work supplies both the geometric packing algorithms and the vision/motion components used here.
II-A Bin Packing Methods
Online packers such as GOPT place items in arrival order without knowing future arrivals. Offline methods can optimize a complete known set. Existing work addresses density, stability, complex shapes, placement correction, and scalable layer-based optimization.
Grocery-specific approaches estimate fragility using tactile/visual sensing or learn human packing sequences from demonstrations. iPack instead obtains semantic rules from pretrained language models and applies them to an offline optimizer. Its comparison with online packing therefore includes an information-availability difference, not just a different planning algorithm.
II-B Grocery Store Automation
Earlier supermarket robots mainly navigate, identify requested products, pick from shelves, guide customers, or automate billing. iPack addresses the later packing stage. It is not evaluated as a complete store-navigation or autonomous-shopping system.
III The iPack approach
Perception estimates the visible items and their properties. Planning converts semantic restrictions and approximate box geometries into target placements. Execution then picks items in order of increasing target height so lower objects are placed first.
III-A Perception Step
GroundingDINO detects candidate objects, and SAM 2 segments them. Numbered visual prompts let a VLM reject distractors such as the container or gripper, name the groceries, and estimate size and weight in one call.
An optional agentic internet search, given the country of origin, supplies standardized package dimensions and weights; produce is estimated from appearance. These are estimates, not calibrated physical measurements. Planning adds padding to tolerate some error.
After manipulations, Hungarian matching combines mask intersection-over-union and centroid distance to maintain object identities. Mask overlap also checks whether a grasp removed the intended object. The pipeline still assumes initially separated objects.
III-B Planning Step
The planner represents each item as an axis-aligned bounding box inside a rectangular container, with Cartesian positions and discrete 90° rotations. An LLM marks ordered pairs for which placing one item on the other may damage the lower item. If such a pair overlaps in horizontal projection, the protected item must be above the other. Separate contact-avoidance rules prohibit harmful side-by-side contact.
Geometric constraints keep all boxes inside the container and prevent box overlap. If multiple containers are available, the formulation can minimize selected container volume, relaxing placements in unselected containers with a big-M construction. The implementation uses the PySCIPOpt/SCIP mixed-integer optimization interface.
Optional execution restrictions add LLM-identified unstable stacks to the prohibited pairs, apply 5 mm collision padding, and restrict items to stable orientations, usually with their shortest dimension vertical. Minimizing summed vertical positions encourages low arrangements. It is not a complete mechanical stability or support proof.
Each solve has a 30-second limit and returns the best feasible incumbent if one exists. Thus “optimal” describes the optimization objective; a time-limited result is not necessarily a certified global optimum. Incorrect properties, omitted rules, or inconsistent constraints can still produce poor plans or infeasibility.
III-C Execution Step
Execution follows the target-height ordering. For each item, principal-component analysis of its segmentation mask sets the gripper’s horizontal orientation along the smaller principal component. The RGB-mask centroid sets horizontal position, mean masked depth sets height, and the approach is vertical.
A Contact-GraspNet-style point-cloud approach was considered but is sensitive to reflective/translucent surfaces in this setup. The authors also describe preliminary failures of Octo/OpenVLA on unseen items; these are not a controlled benchmark establishing general failure of those models.
An RRT planner executes a vertical lift, transit above the target, and downward placement. The system remains dependent on reliable depth, centroid grasps, clearance, and placement accuracy. The method is modular enough to substitute a better grasping or motion controller.
IV Experimental Evaluation
Experiments separate model choice, geometric planning, and end-to-end robot execution. The retained figures and tables include both density and preference violations, since improving one can worsen the other.
IV-A Dataset
Simulation uses the HOPE dataset’s 28 scanned toy grocery items plus five newly scanned fragile objects. Physical trials use a separate set of store-bought household groceries. These are limited object collections, not evidence of unrestricted product coverage.
IV-B Hardware Setup
The physical system uses a Franka Research 3 arm, custom Fin Ray-inspired compliant fingers, and an embedded thin-film force sensor for closed-loop grip-force control. An Orbbec Femto Bolt supplies RGB-D data. Local model inference runs on two NVIDIA RTX A6000 GPUs; a separate real-time Linux machine handles robot control.
IV-C Packing Consistency Score ( C ), a data-driven metric for human-like packing
Eighty-four participants supplied 8,820 pairwise packing preferences across 14 classes. The estimated probability that class i belongs below class k defines the reference preference. Evaluation considers ordered pairs whose placed meshes overlap in horizontal projection and whose vertical order forms a stack.
A stacked pair counts as a violation when its human-preference probability is below 0.4. The Packing Consistency Score C sums these violations; lower is better. Mean aC is reported per box, alongside density and planning time. Despite the name, C is a violation count, not a normalized agreement rate.
The score depends on how many pairs are stacked and does not directly assess crushing, moisture transfer, physical stability, or side contact. The tables therefore also retain the number of overlaps. Density is summed object volume divided by container volume, not a manipulation-success score.
Equation
Equations
IV-D VLM Sensitivity Study
Each planning trial chooses four fragile and five non-fragile objects. Starting from a 50 cm cube, binary search shrinks the container to 2 mm precision while retaining space for each individual item. The paper reports ten repetitions. These are solver-based estimates of suitable size, without a guarantee that time-limited infeasibility proves no smaller packing exists.
GPT-5 yields the lowest reported mean violation count, 0.2, with density 0.445. GPT-5-mini gives 0.5/0.433; Claude Sonnet 4.5 gives 1.6/0.478; Gemini 3 Flash gives 0.75/0.466. Claude’s denser result comes with more violations. The source’s phrase “scores the lowest” means worst consistency, not the smallest numeric C.
The listed 2.5–3.7-second timings exclude variable API response delays. GPT-5 is selected for subsequent experiments. No confidence intervals or detailed repeated-run distribution are supplied.
| Density | Time [s] | |||
|---|---|---|---|---|
| GPT-5 | 0.445 | 0.2 | 6.5 | 2.990 |
| GPT-5-mini | 0.433 | 0.5 | 5.6 | 3.656 |
| Claude Sonnet 4.5 | 0.478 | 1.6 | 7.1 | 2.516 |
| Gemini 3 Flash | 0.466 | 0.75 | 6.0 | 2.586 |
IV-E Planning Baseline Comparison
iPack achieves density 0.445 and aC 0.2, compared with 0.213/1.5 for Dube’s heuristic, 0.252/1.4 for online GOPT, and 0.167/0.4 for the layer-based Elhedhli method. The latter stacks fewer pairs, averaging 1.5 overlaps versus iPack’s 6.5, which also changes opportunities for violations.
Removing iPack’s integrity constraints increases density to 0.478 but raises aC to 2.4. This ablation most directly isolates the value of semantic restrictions within the same planner. The fastest baseline is Dube at 0.173 seconds; iPack is 2.990 seconds. Language-model/API delay is excluded from its optimization timing.
The reported 1,178.702-second Elhedhli result is a particular implementation/benchmark measurement, not a universal runtime bound. Solver cost grows with packing difficulty, and the 30-second limit means the returned plan can be an incumbent rather than an optimum.
The tested offline baseline implementations are Wadaboa/3d-bpp and jerry800416/3D-bin-packing.
| Density | Time [s] | |||
|---|---|---|---|---|
| Dube et al. [17] | 0.213 | 1.5 | 3.2 | 0.173 |
| GOPT [14] | 0.252 | 1.4 | 8.2 | 0.357 |
| Elhedhli et al. [16] | 0.167 | 0.4 | 1.5 | 1178.702 |
| iPack | 0.445 | 0.2 | 6.5 | 2.990 |
| iPack (vanilla) | 0.478 | 2.4 | 6.2 | 3.631 |
IV-F Full System Evaluation
Hardware trials require non-overlapping items lying flat, a centroid grasp that fits the gripper, and flat target orientations. Adding these assumptions and hardware restrictions reduces simulated average density to 26.4%, substantially below the unconstrained planning table.
The robot packs sets of 6, 8, 10, and 12 items, with five trials per size, choosing between two containers. It reports 92.6% manipulation success, defined as picking an item from the table and placing it in the box. Figure 9 shows zero mean preference violations at 6–10 items and about one at 12; it does not show a smooth increase at every size. The paper does not provide the complete per-trial counts behind the aggregate rate.
Failures arise from depth noise and slipping, gripper–wall collisions, and placement errors that create later collisions or preference violations. A successful pick-and-place does not establish correct pose, full-box success, or undamaged goods. Contact-rich pushing and closed-loop placement correction are proposed future improvements.
V Conclusions
iPack demonstrates that semantic constraints from pretrained models can improve agreement with human grocery-stacking preferences while preserving useful packing density. Its strongest evidence is the constraint-removal ablation plus limited physical trials.
Robust operation remains limited by estimated properties, object layout assumptions, approximate geometry, manipulation errors, and API/solver variability. Pushing and learned corrective actions are future work rather than capabilities demonstrated here.
Acknowledgment
The work acknowledges partial support from the German Federal Ministry of Research, Technology and Space through the Robotics Institute Germany.