Overview
This edition covers arXiv v2, dated 12 October 2026, including the complete supplementary material. MoMa-SG project. One-shot estimation uses one observed interaction per articulated object, together with pretrained perception models and RGB-D camera poses; it does not infer unseen joints without motion evidence.
At a glance
MoMa-SG turns interaction videos into a semantic–kinematic scene graph: segment interactions, track moving points, estimate a prismatic or revolute joint, associate its motion with a mapped object, and discover contained objects. A robot localizes in the resulting map, estimates handles and follows kinematically consistent opening/closing trajectories.
The paper introduces Arti4D-Semantic and reports improved articulation/part estimates plus HSR and Spot demonstrations. Mapping is offline, about 22 minutes for ten objects. “Open-world” concerns semantic retrieval and previously unmodeled objects; the modeled joints remain rigid, one-degree-of-freedom prismatic/revolute mechanisms.
1 Introduction
Conventional semantic scene graphs identify objects and relations but often omit the motion constraints needed to open a drawer or door. MoMa-SG learns those constraints from interaction, allowing the same graph to support object retrieval, articulation-state queries and robot manipulation.
Its contributions are a regularized twist estimator, geometry-based part mapping with globally constrained motion association, contained-object relations and an expanded benchmark. Reliable depth, poses, observed interactions and pretrained perception are substantial prerequisites; dynamic rearrangement and general multi-joint mechanisms remain outside the demonstrated scope.
2 Related Work
Prior articulation work uses geometric fitting, depth, tracking or learned models; ArtiPoint supplies the interaction-based benchmark and a principal baseline. Scene-graph systems such as ConceptGraphs provide open-vocabulary geometry but do not by themselves estimate how parts move.
Pandora and related articulated reconstruction methods use different state/depth assumptions. MoMa-SG combines motion estimation with part-level association and contained-object discovery, then uses an existing whole-body control framework rather than learning a new manipulation policy.
Equation
3 Approach
The input is posed RGB-D video. The output combines static semantic geometry, articulated parent parts, estimated motion axes/states and contained children. Interaction discovery, motion fitting and mapping remain separate components, so upstream tracking or segmentation errors can propagate into graph construction.
3.1 Interaction Discovery
An interaction prior from hand/end-effector segmentation is fused with depth-change evidence after camera-motion compensation. The resulting signal is normalized, filtered and segmented into intervals, rejecting durations outside configured bounds. This separates likely object motion from camera motion, but is not a general physical-contact detector.
Equation
3.2 Articulation Estimation
TapNext++ tracks a 64×64 grid forward and backward from interaction endpoints. Depth lifts tracks into world coordinates; smoothing, displacement filtering and rigidity clustering reject unreliable/static trajectories. A RANSAC loop repeatedly fits a constant twist and per-frame articulation states.
The twist objective penalizes rotation for prismatic motion and screw pitch for revolute motion. Pairwise displacement-direction similarity supplies a median score η; a shifted sigmoid around η*=0.995 selects the regularization balance. A single joint-type-aware objective replaces separately choosing two models, although RANSAC still repeats optimization.
GPT-5-mini classifies four keyframes as opening, closing or the two consecutive combinations; a trajectory heuristic checks the mode. This estimates semantic direction, not just axis geometry. The printed objective includes nonsmooth norms/absolute values and normalized terms whose zero-denominator handling is unspecified, despite the text’s fully differentiable characterization.
Equations 1
| (1) |
3.3 Articulated 3D Scene Graph Construction
Semantic-SAM masks on stationary keyframes become 3D parts with CLIP features. Geometric overlap merges detections; semantic similarity is deliberately excluded from merging so adjacent look-alike drawers remain distinct.
Each motion is scored against nearby parts by replaying candidate geometry and comparing projected masks with dynamic/static tracks. A binary integer program assigns exactly one object per motion and at most one motion per object, while penalizing overlapping selected parts. The printed constraints define pairwise selection indicators but do not explicitly connect each object-selection variable y to its assignment variables m; that implementation detail remains underspecified.
At maximum opening, new masks become candidate children. Overlap with the moving parent indicates ARTICULATED containment; overlap with its closed-state mask indicates STATIC containment. These are image/geometry heuristics, not verified physical containment. A child in a door can move with it; a jar behind a cabinet door can stay fixed.
Equations 2–3
| (2) |
| (3) |
4 Experimental Evaluation
Tables 1–2 use ground-truth interaction segments, with predicted-segment experiments in the appendix. Pandora receives privileged depth, segmentation and manually selected states; ArtGS is adapted to three hand-masked frames. Thus the methods do not share identical inputs.
On Arti4D, MoMa-SG reports prismatic/revolute axis errors of 10.374°/14.800°, revolute position error 0.087 m, joint-type accuracy 90.9% and mode accuracy 98.3%. ArtiPoint reports 23.272°/26.358°, 0.248 m and 77.6% type accuracy. A denser 96×96 grid improves some metrics but worsens revolute angle error, so density is not uniformly beneficial.
Free-object IoU is 0.533 versus ConceptGraphs’ 0.438; matched articulated-part IoU is 0.434 versus Pandora’s 0.065. The RH201 child results report IoU 11.14, recall 42.60% and relation accuracy 90.56%; the IoU scale is not marked as a percentage in the header. High relation accuracy is conditional on found/matched children and coexists with low recall.
HSR and Spot opening/closing tests use 50 trials per robot–task combination and exceed 80% success in the depicted settings. These are structured real-world trials, not a broad open-world reliability estimate. The full comparison tables, qualitative scenes and success plots remain below.
| Arti4D | DROID | |||||||||
| Method | Prismatic | Revolute | Joint Type [%] | Mode | Prismatic | Revolute | Type | |||
| [∘] | [∘] | [m] | Acc. | Recall Pris./Rev. | Acc. [%] | [∘] | [∘] | [m] | Acc.[%] | |
| Ours w/ CoTracker3 | 13.190 | 22.982 | 0.091 | 88.4 | 91.7 / 86.6 | 98.3 | ||||
| Ours w/ 32x32 queries | 14.982 | 18.424 | 0.110 | 86.1 | 82.5 / 92.9 | |||||
| Ours w/ 96x96 queries | 09.263 | 16.704 | 0.066 | 93.8 | 91.5 / 100.0 | |||||
| Ours w/o -regul. + w/o RANSAC | 34.089 | 23.175 | 0.103 | 90.9 | 87.7 / 97.0 | |||||
| Ours w/o -regular. + RANSAC | 21.427 | 17.119 | 0.103 | 90.2 | 87.4 / 95.2 | |||||
| Ours w/o RANSAC | 11.205 | 20.382 | 0.113 | 90.3 | 87.6 / 95.6 | |||||
| ArtGS [32] | 52.070 | 62.638 | 0.301 | 61.9 | 100.0 / 00.0 | - | - | - | - | - |
| Pandora [44] | 46.814 | 50.620 | 0.195 | 55.9 | 77.2 / 44.4 | - | - | - | - | - |
| ArtiPoint [15] w/ [29] | 17.754 | 28.467 | 0.313 | 87.2 | 85.4 / 90.5 | - | 28.47 | 27.33 | 0.333 | 62.2 |
| ArtiPoint w/o prior [15] | 22.953 | 27.057 | 0.269 | 77.6 | 71.3 / 89.9 | - | 34.68 | 26.10 | 0.284 | 61.8 |
| ArtiPoint [15] | 23.272 | 26.358 | 0.248 | 77.6 | 70.9 / 90.6 | - | 35.88 | 25.43 | 0.278 | 61.1 |
| MoMa-SG (ours) | 10.374 | 14.800 | 0.087 | 90.9 | 87.7 / 95.8 | 98.3 | 07.15 | 16.91 | 0.115 | 89.5 |
| 3D Part Mapping | Contained Object Discovery | |||||||
| Method | Object | IoU | Recall | IoU | Recall [%] | Relation Acc. [%] | ||
| Set | @0.25 | @0.50 | @0.75 | @IoU=0.25 | STATIC / ARTIC. | |||
| OpenMask3D [56] | 0.082 | 0.087 | 0.019 | 0.000 | ||||
| ConceptGraphs [4] | 0.438 | 0.332 | 0.127 | 0.018 | ||||
| Ours w/o bound.-resp. merging | 0.481 | 0.807 | 0.534 | 0.086 | ||||
| MoMa-SG (ours) | 0.533 | 0.824 | 0.646 | 0.163 | ||||
| Pandora [44] | 0.065 | 0.012 | 0.000 | 0.000 | 00.02 | 00.35 | 19.77 | |
| Ours w/ CoTracker3 | 0.221 | 0.370 | 0.218 | 0.027 | 09.14 | 16.67 | 59.23 | |
| Ours w/ Mobile-SAM | 0.417 | 0.672 | 0.453 | 0.112 | 09.31 | 35.66 | 67.08 | |
| Ours w/ greedy matching | 0.423 | 0.703 | 0.453 | 0.110 | 09.50 | 37.43 | 67.19 | |
| MoMa-SG (ours) | 0.434 | 0.720 | 0.471 | 0.111 | 11.14 | 42.60 | 90.56 | |
5 Conclusion
MoMa-SG integrates semantic geometry and observed kinematics sufficiently to support articulated-object manipulation with two mobile robot platforms. Its main limits are cascading perception/association errors, costly offline mapping, restricted joint families and semi-static scenes. Automatic updates after object rearrangement remain future work.
Acknowledgments
The authors acknowledge BrainLinks–BrainTools, an NVIDIA research grant and Toyota’s HSR program. Hermann Blum is also affiliated with the Lamarr Institute.
Appendix A.1 Approach Details on MoMa-SG
The supplementary method specifies temporal segmentation, regularized estimation, incremental part merging, association and the complete robot execution stack. All original formal expressions, the merging algorithm and visual examples are retained.
A.1-A Interaction Discovery
YOLOv11 supplies hand/end-effector masks. Depth from a previous frame is warped into the current view with known poses; normalized depth residual and interaction-prior signals are median-filtered.
The conditional interaction table is tuned on held-out data, and the fusion factorizes depth and hand evidence as independent probabilities. Calibration or independence is not established. Nonzero segments are length-filtered using minimum/maximum bounds whose numeric values are not supplied here.
Equations 1–3
| (1) | ||||
| (2) |
| (3) |
A.1-B Regularized Twist Estimation:
After lifting/smoothing tracks, reject trajectories shorter than 0.15 m and use DBSCAN rigidity/spatial filtering. RANSAC runs 100 iterations with a 30% point subset. Displacement-direction agreement distinguishes translation from rotation, and regularization suppresses implausible rotational/helical components.
Synthetic comparisons retain the privileged pure-prismatic/pure-revolute estimators alongside regularized and unconstrained twists. Mode recognition uses GPT-5-mini, whose reported accuracy is 98.3%, versus 69.3% for GPT-5-nano. The exact prompt is: “Is the following series of images a consecutive 'opening-closing', 'closing-opening', or just 'opening', or 'closing'. Please answer with one of these four options.”
A.1-C Open-Vocabulary 3D Part Mapping:
Select low-blur frames outside interactions using Laplacian sharpness and translation/rotation changes. Masks touching an image boundary are non-isolated; other masks are isolated. Incremental merging uses geometric similarity and containment, with different rules for isolated/partial observations, as preserved in Algorithm 1.
The optional dense mode reinitializes tracking at quarter, half and three-quarter interaction times. Masks with dynamic/static support ratio above 0.5 seed SAM2 propagation, giving denser collision geometry at additional computational cost.
The printed rotation-change formula, Eq. (4), uses arccos(tr(Rₖ₋₁Rₖ⁻¹)/2), omitting the usual subtraction of one. For identical rotations its argument is 1.5. The original expression is retained; the implementation’s intended formula is unresolved.
Equation 4
| (4) |
Copyable algorithm
A.1-D Object-to-Articulation Matching
For each estimated articulation, test the k nearest mapped part centroids. Replay each part through the estimated motion, project its visible geometry and compare inside/outside dynamic and static tracks, using convex projected masks when available.
Eq. (5) is printed as ratios of inside/outside counts, although the prose calls it a mean association cost. Its denominators can vanish, and its sign/normalization relative to the minimized objective is not explained. The exact equation is retained without silently repairing it. Global assignment then discourages mutually overlapping selected parts.
Equation 5
| (5) |
A.1-D.1 Contained Object Discovery
Use mode and fitted state to select a maximum-opening view. Segment visible objects, associate them with the opened parent and classify their motion relation from overlap with the current versus closed-state parent masks. A missed/opening-state error can therefore hide children or assign the wrong relation. Ten original discovery frames preserve both successful and difficult cases.
A.1-E Real-World Mobile Manipulation
Robot execution combines localization, local registration, online handle/grasp estimation and sampled kinematic trajectories. Learned perception supplies geometry; an adapted N2M2 whole-body controller executes end-effector motions. The pipeline does not establish formal contact-force or collision guarantees.
A.1-E.1 Global Localization in Ego-Centric Maps
Fit the floor with RANSAC and refine its normal by SVD, then transform the reconstruction to a gravity-aligned map. Points within 0.05 m of the floor mark free space; points at 0.15–0.30 m mark occupancy. Morphological operations clean the grid, and AMCL localizes the robot using 2D lidar.
Equation 6
| (6) |
A.1-E.2 Local Online Ego-centric Map Registration
Before each action, back-project depth within 1.5 m and register it against global-map geometry within a 0.8 m neighborhood. ICP estimates a refined world transform, optionally bounded in translation/rotation. Graph object poses are transformed through this corrected frame; this reduces accumulated pose error without rebuilding the scene graph.
Equations 7–9
| (7) |
| (8) |
| (9) |
A.1-E.3 Online Grasp Prediction
SAM3 prompted with “handle” supplies handle masks. Fit a box to each visible handle cloud; its longest axis defines gripper orientation and its midpoint defines the grasp position. Combine that direction with the articulated part’s surface normal at the current state to form the grasp transform.
A Hungarian assignment matches candidate handles to mapped objects using IoU, overlap and centroid distance. Partial views and already-open parts can cause incorrect associations; the grasp construction assumes suitable visible handle geometry.
Equations 10–12
| (10) |
| (11) |
| (12) |
A.1-E.4 End-Effector Trajectory Sampling
Propagate the initial grasp through sampled articulation states with the twist exponential, then transform it from graph coordinates through the global and refined world frames. Opening and closing reverse or negate the sampled state range. These end-effector paths respect the estimated joint, so model or registration error directly affects execution.
Equations 13
| (13) |
A.1-E.5 Language-enabled Mobile Manipulation
GPT-5 plans over open(o), inspect(o), close(o), retrieve(o,p) and done(), conditioned on semantic object states. It replans after actions; optional retries insert inspection and repeat failed opening/closing/retrieval, helping recover from grip loss.
For “get the milk from the fridge,” the robot opens the fridge, inspects its contents and retrieves the milk along a grasp-approach direction. The demonstrations show language-conditioned composition of predefined skills; they are not a systematic benchmark of unconstrained language planning.
Appendix A.2 Experimental Evaluation
The evaluation appendix defines metrics, separates ground-truth and predicted interactions, reports split-wise results and uncertainty, and analyzes real robot execution, failure propagation and runtime.
A.2-A Metrics
The original metric equations and full result tables remain. Their denominators, matching thresholds and conditional populations matter when comparing figures across tasks or dataset subsets.
A.2-A.1 Interaction Segmentation
Temporal segmentation uses global one-dimensional IoU and optimal bipartite matching of predicted/true intervals. Pairs below IoU 0.5 are discarded; precision/recall, matched-segment IoU and absolute onset/offset errors are then computed. Matched-only errors can look good even when many interactions are missed.
Equations 14–15
| (14) |
| (15) |
| Split | Paradigm | GT Poses | Prismatic joints | Revolute joints | Type Quality [%] | Mode | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| [deg] | [m] | [deg] | [m] | Accuracy | Prismatic Recall | Revolute Recall | Acc.[%] | |||
| RH078 | EGO | ✓ | 05.43 | – | 15.88 | 0.063 | 0.862 | 0.840 | 0.900 | 0.980 |
| RR080 | EGO | ✓ | 19.36 | – | 14.80 | 0.035 | 0.808 | 0.792 | 1.000 | 0.952 |
| DR080 | EGO | ✓ | 07.99 | – | 15.67 | 0.094 | 0.940 | 0.922 | 0.970 | 1.000 |
| RH201 | EGO | ✓ | 03.52 | – | 13.97 | 0.099 | 0.979 | 0.973 | 0.979 | 0.993 |
| MHZH | EGO | 42.01 | – | 21.34 | 0.271 | 0.917 | 0.727 | 1.000 | 0.972 | |
| MHZH-EXO | EXO | 30.24 | – | 36.56 | 0.172 | 0.778 | 0.500 | 0.917 | 1.000 | |
| HSR | ROBOT | 30.99 | – | 51.05 | 0.367 | 0.605 | 0.684 | 0.526 | 0.852 | |
| Overall | 19.94 | – | 24.18 | 0.157 | 0.841 | 0.777 | 0.899 | 0.964 | ||
A.2-A.2 Articulation Estimation
Axes are compared through angular error and shortest-axis position distance, using a special near-parallel branch at ε=10⁻⁴. Position error is reported only for revolute joints. Joint-type and opening/closing mode accuracy supplement geometric errors; an axis metric alone does not capture whether the robot moves in the correct direction.
Equations 16–17
| (16) |
| (17) |
A.2-A.3 Articulated 3D Part Segmentation
Hungarian matching uses IoU of axis-aligned 3D bounding boxes, not volumetric surface-mask overlap. The paper reports mean IoU and recall at thresholds 0.25/0.50/0.75. Semantic retrieval ranks CLIP text categories against optimally matched object features, reporting top-1/2/5/10 accuracy.
Across all seven splits, TapNext++ articulated-part IoU averages 0.417, versus CoTracker3’s 0.233. Free/matched-part category top-1 accuracy is 35.5%/28.7% and top-5 is 73.4%/63.3%; open vocabulary does not imply uniformly accurate semantic labels.
| Method | Split | Paradigm | GT Poses | IoU | Recall | ||
| @0.25 | @0.50 | @0.75 | |||||
| w/ CoTracker3 | RH078 | EGO | ✓ | 0.184 | 0.327 | 0.181 | 0.000 |
| RR080 | EGO | ✓ | 0.196 | 0.347 | 0.154 | 0.035 | |
| DR080 | EGO | ✓ | 0.243 | 0.387 | 0.241 | 0.019 | |
| RH201 | EGO | ✓ | 0.262 | 0.416 | 0.287 | 0.049 | |
| MHZH | EGO | 0.208 | 0.317 | 0.108 | 0.025 | ||
| MHZH-EXO | EXO | 0.238 | 0.396 | 0.229 | 0.000 | ||
| HSR | ROBOT | 0.300 | 0.537 | 0.237 | 0.025 | ||
| Overall | 0.233 | 0.391 | 0.212 | 0.025 | |||
| w/ TapNext++ | RH078 | EGO | ✓ | 0.436 | 0.740 | 0.402 | 0.144 |
| RR080 | EGO | ✓ | 0.459 | 0.744 | 0.552 | 0.131 | |
| DR080 | EGO | ✓ | 0.404 | 0.667 | 0.398 | 0.091 | |
| RH201 | EGO | ✓ | 0.434 | 0.717 | 0.505 | 0.087 | |
| MHZH | EGO | 0.393 | 0.565 | 0.269 | 0.090 | ||
| MHZH-EXO | EXO | 0.387 | 0.683 | 0.289 | 0.074 | ||
| HSR | ROBOT | 0.403 | 0.541 | 0.305 | 0.077 | ||
| Overall | 0.417 | 0.665 | 0.389 | 0.099 | |||
Equation 18
| (18) |
| Object Set | Split | ||||
|---|---|---|---|---|---|
| RH078 | 0.387 | 0.506 | 0.660 | 0.830 | |
| RR080 | 0.069 | 0.174 | 0.452 | 0.532 | |
| DR080 | 0.489 | 0.663 | 0.774 | 0.888 | |
| RH201 | 0.410 | 0.605 | 0.781 | 0.915 | |
| MHZH | 0.357 | 0.523 | 0.725 | 0.980 | |
| MHZH-EXO | 0.333 | 0.562 | 0.896 | 1.000 | |
| HSR | 0.444 | 0.669 | 0.850 | 0.975 | |
| Overall | 0.355 | 0.529 | 0.734 | 0.875 | |
| RH078 | 0.212 | 0.356 | 0.641 | 0.781 | |
| RR080 | 0.026 | 0.123 | 0.403 | 0.549 | |
| DR080 | 0.305 | 0.424 | 0.609 | 0.778 | |
| RH201 | 0.374 | 0.480 | 0.758 | 0.935 | |
| MHZH | 0.287 | 0.543 | 0.667 | 0.782 | |
| MHZH-EXO | 0.271 | 0.396 | 0.583 | 0.812 | |
| HSR | 0.531 | 0.662 | 0.769 | 0.875 | |
| Overall | 0.287 | 0.426 | 0.633 | 0.787 |
A.2-A.4 Contained Objects
Child objects are compared at observed articulation states, voxelizing at 1.5 cm. Each prediction receives its best IoU against children of the assigned parent; recall uses a 0.25 threshold. STATIC/ARTICULATED relation accuracy is evaluated for associated children. State selection and parent association affect all three metrics.
Equation 19
| (19) |
A.2-B Additional Experiments
Additional experiments test temporal discovery and downstream reconstruction using predicted segments, with split-wise tables and heavy-tail summaries supplementing the main aggregate results.
A.2-B.1 Interaction Segmentation
MoMa-SG’s temporal IoU is 0.649, precision 0.786 and recall 0.800, with onset/offset errors 0.816/0.742 s. GPT-5-mini yields 0.550 temporal IoU and lower recall 0.611, but higher matched-segment IoU 0.753 versus 0.718. Thus the narrative claim of the best segment IoU does not hold against every listed method.
The HMM baseline has higher precision, 0.871, but recall 0.532. Fusing depth and hand priors trades some precision for substantially better coverage; separate signals and matched-only metrics describe different failure modes.
| Method | 1D-IoU | P | R | Segment-IoU | [] | [] |
|---|---|---|---|---|---|---|
| GPT-5-mini | 0.55 | 0.862 | 0.6111 | 0.753 | 8.179 | 10.12 |
| Pandora [44] | 0.359 | 0.690 | 0.400 | 0.689 | 0.7442 | 0.725 |
| HMM | 0.471 | 0.871 | 0.532 | 0.683 | 0.770 | 0.762 |
| ArtiPoint [15] | 0.575 | 0.678 | 0.714 | 0.698 | 1.05 | 0.558 |
| MoMa-SG | 0.649 | 0.786 | 0.800 | 0.718 | 0.816 | 0.742 |
| Object | Trans. Error | Rot. Error | ||
|---|---|---|---|---|
| Type | [] | [] | [] | [] |
| Prismatic | 0.0174 | 0.006 | - | - |
| Revolute | - | - | 6.477 | 3.604 |
A.2-B.2 Articulation Estimation
With predicted segments, MoMa-SG’s prismatic/revolute angle errors are 20.66°/17.31°, revolute position error 0.071 m, fit/missing-segment failure rate 10.5%, joint-type accuracy 82.0% and mode accuracy 80.7%. Ground-truth interactions therefore materially simplify the main setup. Failed/missed interactions are counted using an IoU 0.25 association threshold.
New MHZH/MHZH-EXO recordings use estimated camera poses and include textureless objects and external-camera occlusions; HSR is robot-centric. Across all seven splits, ground-truth-segment angle errors rise to 19.94°/24.18° and position error to 0.157 m.
Errors are strongly heavy-tailed: mean prismatic angle 10.37° contrasts with median 2.56°, and mean revolute angle 14.80° with median 4.30°. Paired Wilcoxon tests favor MoMa-SG in the reported comparisons, including p=0.02 for one position comparison; these tests do not establish transfer to other environments.
| Method | Prismatic joints | Revolute joints | Failure | Joint Type [%] | Mode | |||
|---|---|---|---|---|---|---|---|---|
| [deg] | [m] | [deg] | [m] | Rate [%] | Accuracy | Pris./Rev. Recall | Acc.[%] | |
| Pandora [44] | 46.31 | – | 50.23 | 0.182 | 00.0 | 55.8 | 45.0 / 75.9 | - |
| ArtiPoint [15] | 24.88 | – | 30.24 | 0.293 | 23.2 | 77.4 | 88.2 / 72.2 | - |
| MoMa-SG (ours) | 20.66 | – | 17.31 | 0.071 | 10.5 | 82.0 | 73.8 / 98.4 | 80.7 |
| Split | Paradigm | GT Poses | Prismatic joints | Revolute joints | Type Quality [%] | Mode | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| [deg] | [m] | [deg] | [m] | Accuracy | Prismatic Recall | Revolute Recall | Acc.[%] | |||
| RH078 | EGO | ✓ | 17.58 | – | 06.42 | 0.035 | 0.794 | 0.667 | 1.000 | 0.825 |
| RR080 | EGO | ✓ | 29.84 | – | 32.76 | 0.118 | 0.657 | 0.626 | 1.000 | 0.879 |
| DR080 | EGO | ✓ | 16.32 | – | 21.57 | 0.062 | 0.889 | 1.000 | 0.816 | 0.815 |
| RH201 | EGO | ✓ | 12.94 | – | 17.34 | 0.085 | 0.919 | 0.877 | 0.966 | 0.734 |
| Overall | 20.66 | – | 17.31 | 0.071 | 0.820 | 0.738 | 0.984 | 0.807 | ||
| Method | Prismatic joints | Revolute joints | |
|---|---|---|---|
| [deg] | [deg] | [m] | |
| Pandora | 46.8137.49 | 50.6228.58 | 0.200.18 |
| {47.31, 85.58, 88.43} | {51.72, 79.69, 86.34} | {0.11, 0.31, 0.42} | |
| ArtiPoint | 23.2731.67 | 26.3625.53 | 0.250.26 |
| {04.85, 45.19, 81.80} | {12.86, 38.38, 62.22} | {0.05, 0.10, 0.31} | |
| MoMa-SG | 10.3724.10 | 14.8022.99 | 0.090.15 |
| {02.56, 03.90, 14.93} | {04.30, 11.21, 55.24} | {0.03, 0.07, 0.27} | |
The values below the errors denote 50,75,90% percentile error scores, respectively.
A.2-C 3D Part Mapping
Predicted-interaction part mapping reports IoU 0.448 across the four original splits. The all-seven-split value 0.417 uses a different population, so comparing those aggregates cannot show that predicted segmentation improves mapping. The reported standard deviations also indicate substantial scene/object variability.
Free objects are easier than associating articulated parts, and semantic accuracy drops after association. Repeated masks, tracking drift, inaccurate joint magnitudes and imperfect segmentation limit fine-grained children such as milk cartons or jars.
| Split | Paradigm | GT Poses | IoU | Recall | ||
|---|---|---|---|---|---|---|
| @0.25 | @0.50 | @0.75 | ||||
| RH078 | EGO | ✓ | 0.508 | 0.692 | 0.415 | 0.134 |
| RR080 | EGO | ✓ | 0.411 | 0.605 | 0.429 | 0.146 |
| DR080 | EGO | ✓ | 0.548 | 0.889 | 0.556 | 0.111 |
| RH201 | EGO | ✓ | 0.408 | 0.674 | 0.501 | 0.124 |
| Overall | 0.448 | 0.675 | 0.460 | 0.132 | ||
A.2-D Real-World Long-Horizon Mobile Manipulation
Across 54 real object states, mean prismatic displacement error is 0.0174 m and revolute angle error is 6.477°. One HSR sequence opens seven objects—three revolute and four prismatic—and a Spot sequence opens three. These qualitative long sequences supplement the separately counted robot trials.
Grip loss dominates observed execution failures; inaccurate handle association also occurs, especially when objects are already open. Closing is harder because current-state estimation and handle visibility become more demanding. The language milk-retrieval example uses the same localization and articulation stack.
A.2-D.1 Robustness Study
The RH201 analysis tracks failures through 3D tracking, trajectory extraction, articulation fitting, object association and child discovery. About 94.8% of parts are matched and mode accuracy is 99.3% under that study’s conditional definitions; these are not end-to-end child success rates.
Specular surfaces, failed filtering, drift and wrong motion magnitudes propagate downstream. The original Sankey chart preserves conditional stage percentages and losses, and the η sensitivity plots show the joint-type/axis tradeoff around the chosen threshold.
Chart mathematical labels
A.2-D.2 Runtime
A ten-object graph takes 21.89 minutes on an RTX A6000, 128 GB RAM and Threadripper Pro 3975WX. Mapping accounts for 42.1% of runtime; other stages include filtering, interaction segmentation, tracking, trajectory filtering, articulation/mode estimation and association. The authors recommend at least 64 GB RAM and 24 GB VRAM. This is offline graph construction, not real-time end-to-end scene understanding.
Appendix A.3 Arti4D-Semantic Dataset & Benchmark
Arti4D-Semantic contains 62 RGB-D sequences across five physical scenes and seven splits, adding exocentric and robot-centric observations to Arti4D. Original splits have ground-truth poses; new MHZH/MHZH-EXO/HSR poses are estimated with DROID-SLAM.
Annotators fuse closed-state parent masks from selected keyframes into 3D, assign categories and independently verify them. Maximum-opening views label children and distinguish parent-attached from static regions. The dataset has 29 parent and 54 child categories; annotation geometry and state choice remain potential error sources.
Table A.10 lists all 62 recordings. Summing its rows gives 502 object entries and 510 interactions, whereas the main text reports 600 interactions; the source does not reconcile this difference. Child labels are marked for the 45 original recordings and absent for the 17 new recordings. Both the complete table and the source’s broader label-coverage claims are preserved without conflating them.
| Split | GT | Paradigm | Sequence | Recording | # | # | # | # | Parent | Child |
| Poses | ID | Objects | Interactions | PRISM / REV | EASY / HARD | Labels | Labels | |||
| RH078 | ✓ | EGO | RH078-00 | scene_2025-04-04-19-14-38 | 7 | 7 | 3 / 4 | 7 / 0 | ✓ | ✓ |
| ✓ | EGO | RH078-01 | scene_2025-04-04-19-18-54 | 6 | 7 | 2 / 4 | 4 / 2 | ✓ | ✓ | |
| ✓ | EGO | RH078-02 | scene_2025-04-07-11-39-17 | 7 | 7 | 3 / 4 | 7 / 0 | ✓ | ✓ | |
| ✓ | EGO | RH078-03 | scene_2025-04-07-11-41-52 | 8 | 9 | 3 / 5 | 6 / 2 | ✓ | ✓ | |
| ✓ | EGO | RH078-04 | scene_2025-04-07-11-48-40 | 7 | 7 | 3 / 4 | 7 / 0 | ✓ | ✓ | |
| ✓ | EGO | RH078-05 | scene_2025-04-09-10-30-11 | 8 | 8 | 5 / 3 | 3 / 5 | ✓ | ✓ | |
| ✓ | EGO | RH078-06 | scene_2025-04-09-10-32-52 | 6 | 6 | 5 / 1 | 3 / 3 | ✓ | ✓ | |
| ✓ | EGO | RH078-07 | scene_2025-04-09-10-35-47 | 7 | 7 | 6 / 1 | 2 / 5 | ✓ | ✓ | |
| ✓ | EGO | RH078-08 | scene_2025-04-09-10-38-38 | 7 | 7 | 5 / 2 | 3 / 4 | ✓ | ✓ | |
| ✓ | EGO | RH078-09 | scene_2025-04-09-10-46-48 | 7 | 8 | 7 / 0 | 4 / 3 | ✓ | ✓ | |
| ✓ | EGO | RH078-10 | scene_2025-04-09-10-49-20 | 7 | 7 | 6 / 1 | 2 / 5 | ✓ | ✓ | |
| RR080 | ✓ | EGO | RR080-00 | scene_2025-04-10-13-11-16 | 17 | 17 | 14 / 3 | 9 / 8 | ✓ | ✓ |
| ✓ | EGO | RR080-01 | scene_2025-04-10-16-05-09 | 14 | 14 | 13 / 1 | 8 / 6 | ✓ | ✓ | |
| ✓ | EGO | RR080-02 | scene_2025-04-17-15-25-14 | 11 | 12 | 11 / 0 | 7 / 4 | ✓ | ✓ | |
| ✓ | EGO | RR080-03 | scene_2025-04-17-15-33-44 | 9 | 9 | 9 / 0 | 7 / 2 | ✓ | ✓ | |
| ✓ | EGO | RR080-04 | scene_2025-04-22-09-53-49 | 8 | 8 | 8 / 0 | 5 / 3 | ✓ | ✓ | |
| ✓ | EGO | RR080-05 | scene_2025-04-22-09-56-24 | 10 | 10 | 9 / 1 | 9 / 1 | ✓ | ✓ | |
| ✓ | EGO | RR080-06 | scene_2025-04-22-09-58-49 | 7 | 8 | 7 / 0 | 4 / 3 | ✓ | ✓ | |
| ✓ | EGO | RR080-07 | scene_2025-04-22-11-45-15 | 9 | 9 | 8 / 1 | 7 / 2 | ✓ | ✓ | |
| ✓ | EGO | RR080-08 | scene_2025-04-22-11-48-01 | 9 | 9 | 8 / 1 | 7 / 2 | ✓ | ✓ | |
| ✓ | EGO | RR080-09 | scene_2025-04-22-11-50-40 | 8 | 8 | 7 / 1 | 6 / 2 | ✓ | ✓ | |
| DR080 | ✓ | EGO | DR080-00 | scene_2025-04-11-11-44-32 | 11 | 11 | 7 / 4 | 5 / 6 | ✓ | ✓ |
| ✓ | EGO | DR080-01 | scene_2025-04-11-12-58-58 | 10 | 10 | 6 / 4 | 5 / 5 | ✓ | ✓ | |
| ✓ | EGO | DR080-02 | scene_2025-04-11-13-01-59 | 9 | 9 | 5 / 4 | 4 / 5 | ✓ | ✓ | |
| ✓ | EGO | DR080-03 | scene_2025-04-11-13-18-00 | 9 | 10 | 4 / 5 | 3 / 6 | ✓ | ✓ | |
| ✓ | EGO | DR080-04 | scene_2025-04-11-13-43-03 | 11 | 11 | 7 / 4 | 5 / 6 | ✓ | ✓ | |
| ✓ | EGO | DR080-05 | scene_2025-04-11-14-01-06 | 11 | 11 | 7 / 4 | 5 / 6 | ✓ | ✓ | |
| ✓ | EGO | DR080-06 | scene_2025-04-11-15-43-24 | 11 | 12 | 7 / 4 | 5 / 6 | ✓ | ✓ | |
| ✓ | EGO | DR080-07 | scene_2025-04-11-15-46-48 | 11 | 11 | 6 / 5 | 4 / 7 | ✓ | ✓ | |
| RH201 | ✓ | EGO | RH201-00 | scene_2025-04-24-17-52-21 | 11 | 11 | 5 / 6 | 6 / 5 | ✓ | ✓ |
| ✓ | EGO | RH201-01 | scene_2025-04-24-17-54-13 | 9 | 9 | 5 / 4 | 5 / 4 | ✓ | ✓ | |
| ✓ | EGO | RH201-02 | scene_2025-04-24-19-18-42 | 11 | 11 | 5 / 6 | 7 / 4 | ✓ | ✓ | |
| ✓ | EGO | RH201-03 | scene_2025-04-24-19-21-50 | 8 | 8 | 2 / 6 | 5 / 3 | ✓ | ✓ | |
| ✓ | EGO | RH201-04 | scene_2025-04-24-19-24-09 | 8 | 8 | 5 / 3 | 5 / 3 | ✓ | ✓ | |
| ✓ | EGO | RH201-05 | scene_2025-04-25-10-36-37 | 9 | 9 | 5 / 4 | 7 / 2 | ✓ | ✓ | |
| ✓ | EGO | RH201-06 | scene_2025-04-25-10-53-40 | 8 | 8 | 4 / 4 | 3 / 5 | ✓ | ✓ | |
| ✓ | EGO | RH201-07 | scene_2025-04-25-10-56-33 | 8 | 8 | 3 / 5 | 3 / 5 | ✓ | ✓ | |
| ✓ | EGO | RH201-08 | scene_2025-04-25-11-11-47 | 15 | 16 | 6 / 9 | 6 / 9 | ✓ | ✓ | |
| ✓ | EGO | RH201-09 | scene_2025-04-25-11-15-47 | 7 | 7 | 5 / 2 | 4 / 3 | ✓ | ✓ | |
| ✓ | EGO | RH201-10 | scene_2025-04-25-14-58-42 | 9 | 9 | 6 / 3 | 4 / 5 | ✓ | ✓ | |
| ✓ | EGO | RH201-11 | scene_2025-04-25-15-02-14 | 7 | 7 | 3 / 4 | 4 / 3 | ✓ | ✓ | |
| ✓ | EGO | RH201-12 | scene_2025-04-25-15-04-48 | 7 | 7 | 4 / 3 | 3 / 4 | ✓ | ✓ | |
| ✓ | EGO | RH201-13 | scene_2025-04-25-15-16-29 | 9 | 9 | 5 / 4 | 3 / 6 | ✓ | ✓ | |
| ✓ | EGO | RH201-14 | scene_2025-04-25-15-19-22 | 10 | 10 | 5 / 5 | 5 / 5 | ✓ | ✓ | |
| ✓ | EGO | RH201-15 | scene_2025-04-25-15-22-54 | 8 | 8 | 4 / 4 | 3 / 5 | ✓ | ✓ | |
| MHZH | EGO | MHZH-00 | scene_20251205_171539 | 6 | 6 | 2 / 4 | - / - | ✓ | - | |
| EGO | MHZH-01 | scene_20251206_152615 | 11 | 11 | 2 / 9 | - / - | ✓ | - | ||
| EGO | MHZH-02 | scene_20251206_153729 | 8 | 8 | 4 / 4 | - / - | ✓ | - | ||
| EGO | MHZH-03 | scene_20251206_154127 | 4 | 4 | 2 / 2 | - / - | ✓ | - | ||
| EGO | MHZH-04 | scene_20251206_154926 | 8 | 8 | 1 / 7 | - / - | ✓ | - | ||
| MHZH EXO | EXO | MHZH-EXO-00 | scene_20251210_123247 | 4 | 4 | 0 / 4 | - / - | ✓ | - | |
| EXO | MHZH-EXO-01 | scene_20251210_123710 | 4 | 4 | 1 / 3 | - / - | ✓ | - | ||
| EXO | MHZH-EXO-02 | scene_20251210_123920-0 | 6 | 6 | 3 / 3 | - / - | ✓ | - | ||
| EXO | MHZH-EXO-03 | scene_20251210_123920-1 | 6 | 6 | 2 / 4 | - / - | ✓ | - | ||
| HSR | ROBOT | HSR-00 | scene_2026-01-05-15-44-54 | 5 | 5 | 2 / 3 | - / - | ✓ | - | |
| ROBOT | HSR-01 | scene_2026-01-05-15-49-46 | 5 | 5 | 2 / 3 | - / - | ✓ | - | ||
| ROBOT | HSR-02 | scene_2026-01-05-15-53-13 | 5 | 5 | 2 / 3 | - / - | ✓ | - | ||
| ROBOT | HSR-03 | scene_2026-01-05-16-00-17 | 5 | 5 | 3 / 2 | - / - | ✓ | - | ||
| ROBOT | HSR-04 | scene_2026-01-05-16-02-59 | 5 | 5 | 3 / 2 | - / - | ✓ | - | ||
| ROBOT | HSR-05 | scene_2026-01-05-16-05-49 | 5 | 5 | 1 / 3 | - / - | ✓ | - | ||
| ROBOT | HSR-06 | scene_2026-01-05-16-09-34 | 4 | 4 | 2 / 2 | - / - | ✓ | - | ||
| ROBOT | HSR-07 | scene_2026-01-05-16-13-36 | 5 | 5 | 4 / 1 | - / - | ✓ | - |