Literature
Display
← All papers

Articulated 3D Scene Graphs for Open-World Mobile Manipulation

Martin Büchner · Adrian Röfer · Tim Engelbracht · Tim Welschehold · Zuria Bauer · Hermann Blum · Marc Pollefeys · Abhinav Valada

Preprint posted · CoRL 2026 (accepted) ↗

Robotics13 min text · CondensedOriginal paper ↗
Contents

Overview

This edition covers arXiv v2, dated 12 October 2026, including the complete supplementary material. MoMa-SG project. One-shot estimation uses one observed interaction per articulated object, together with pretrained perception models and RGB-D camera poses; it does not infer unseen joints without motion evidence.

Complete original S0.F1, including all panels and labels.
Figure 1: MoMa-SG enables the construction of accurate semantic-kinematic 3D scene graphs over articulated scenes and serves as a backbone for long-horizon and language-enabled interactive mobile manipulation.
Complete original A0.F1, including all panels and labels.
Figure A.1: We demonstrate the ability of MoMa-SG to map natural language commands (e.g., “get the milk from the fridge”) to an appropriate sequence of high-level actions, such as opening the fridge, inspecting its contents, and retrieving the target object. All of this is done from open-vocabulary language features and object articulation models estimated from ego-centric video in a one-shot fashion while not relying on any fiducial markers.

At a glance

MoMa-SG turns interaction videos into a semantic–kinematic scene graph: segment interactions, track moving points, estimate a prismatic or revolute joint, associate its motion with a mapped object, and discover contained objects. A robot localizes in the resulting map, estimates handles and follows kinematically consistent opening/closing trajectories.

The paper introduces Arti4D-Semantic and reports improved articulation/part estimates plus HSR and Spot demonstrations. Mapping is offline, about 22 minutes for ten objects. “Open-world” concerns semantic retrieval and previously unmodeled objects; the modeled joints remain rigid, one-degree-of-freedom prismatic/revolute mechanisms.

1 Introduction

Conventional semantic scene graphs identify objects and relations but often omit the motion constraints needed to open a drawer or door. MoMa-SG learns those constraints from interaction, allowing the same graph to support object retrieval, articulation-state queries and robot manipulation.

Its contributions are a regularized twist estimator, geometry-based part mapping with globally constrained motion association, contained-object relations and an expanded benchmark. Reliable depth, poses, observed interactions and pretrained perception are substantial prerequisites; dynamic rearrangement and general multi-joint mechanisms remain outside the demonstrated scope.

2 Related Work

Prior articulation work uses geometric fitting, depth, tracking or learned models; ArtiPoint supplies the interaction-based benchmark and a principal baseline. Scene-graph systems such as ConceptGraphs provide open-vocabulary geometry but do not by themselves estimate how parts move.

Pandora and related articulated reconstruction methods use different state/depth assumptions. MoMa-SG combines motion estimation with part-level association and contained-object discovery, then uses an existing whole-body control framework rather than learning a new manipulation policy.

Equation

𝔰​𝔢​(3)⊆ℝ6\mathfrak{se}(3)\subseteq\mathbb{R}^{6}

3 Approach

The input is posed RGB-D video. The output combines static semantic geometry, articulated parent parts, estimated motion axes/states and contained children. Interaction discovery, motion fitting and mapping remain separate components, so upstream tracking or segmentation errors can propagate into graph construction.

3.1 Interaction Discovery

An interaction prior from hand/end-effector segmentation is fused with depth-change evidence after camera-motion compensation. The resulting signal is normalized, filtered and segmented into intervals, rejecting durations outside configured bounds. This separates likely object motion from camera motion, but is not a general physical-contact detector.

Complete original S3.F2, including all panels and labels.
Figure 2: Overview of MoMa-SG: We first discover interaction segments (Sec. 3.1), then attain object articulation models 𝒜\mathcal{A} by estimating twists from point trajectories (Sec. 3.2). We match mapped objects 𝒪\mathcal{O} against articulations and discover objects contained in respective articulated parents (Sec. 3.3).
Equation

{𝐈t,𝐃t}t=1T\left\{\mathbf{I}_{t},\mathbf{D}_{t}\right\}_{t=1}^{T}

𝒮={(ts(n),te(n))}n=1N\mathcal{S}=\{(t_{s}^{(n)},t_{e}^{(n)})\}_{n=1}^{N}

3.2 Articulation Estimation

TapNext++ tracks a 64×64 grid forward and backward from interaction endpoints. Depth lifts tracks into world coordinates; smoothing, displacement filtering and rigidity clustering reject unreliable/static trajectories. A RANSAC loop repeatedly fits a constant twist and per-frame articulation states.

The twist objective penalizes rotation for prismatic motion and screw pitch for revolute motion. Pairwise displacement-direction similarity supplies a median score η; a shifted sigmoid around η*=0.995 selects the regularization balance. A single joint-type-aware objective replaces separately choosing two models, although RANSAC still repeats optimization.

GPT-5-mini classifies four keyframes as opening, closing or the two consecutive combinations; a trajectory heuristic checks the mode. This estimates semantic direction, not just axis geometry. The printed objective includes nonsmooth norms/absolute values and normalized terms whose zero-denominator handling is unspecified, despite the text’s fully differentiable characterization.

Equations 1
ξ^∗,θ1∗,…,θe∗\displaystyle\hat{\xi}^{*},\theta^{*}_{1},\ldots,\theta^{*}_{e} =minξ^,θs,…,θe∑m=se∑j=1|Pm|‖pt+ϵj−exp(ξ^θm)⋅ptj‖+α(λr​e​v|ω⊤​v‖ω‖​‖v‖|+λp​r​i​s∥ω∥),\displaystyle=\min_{\hat{\xi},\theta_{s},\ldots,\theta_{e}}\sum_{m=s}^{e}\sum_{j=1}^{|P_{m}|}\left\lVert p^{t+\epsilon}_{j}-\exp(\hat{\mathbf{\xi}}\theta_{m})\cdot p^{t}_{j}\right\rVert+\alpha\left(\lambda_{rev}\left|\frac{\omega^{\top}v}{\left\lVert\omega\right\rVert\left\lVert v\right\rVert}\right|+\lambda_{pris}\|\omega\|\right), (1)

𝒜i={ξ,θs,…,θe,M}\mathcal{A}_{i}=\{\xi,\theta_{s},\ldots,\theta_{e},M\}

𝒳n∈ℝTn×F×2\mathcal{X}_{n}\in\mathbb{R}^{T_{n}\times F\times 2}

𝒱n∈{0,1}Tn×F\mathcal{V}_{n}\in\{0,1\}^{T_{n}\times F}

ξ=⟨ω,v⟩∈𝔰​𝔢​(3)\xi=\left\langle\omega,v\right\rangle\in\mathfrak{se}(3)

exp:𝔰​𝔢​(3)→S​E​(3)\text{exp}:\mathfrak{se}(3)\rightarrow SE(3)

log:S​E​(3)→𝔰​𝔢​(3)\text{log}:SE(3)\rightarrow\mathfrak{se}(3)

𝐱i=𝐩it−𝐩i0\mathbf{x}_{i}=\mathbf{p}_{i}^{\text{t}}-\mathbf{p}_{i}^{\text{0}}

Ci​j=𝐱i⊤​𝐱j‖𝐱i‖​‖𝐱j‖C_{ij}=\frac{\mathbf{x}_{i}^{\top}\mathbf{x}_{j}}{\|\mathbf{x}_{i}\|\,\|\mathbf{x}_{j}\|}

λp​r​i​s=σ⁡(k⁡(η−η∗))\lambda_{pris}=\sigma(k(\eta-\eta^{*}))

λr​e​v=1−σ⁡(k⁡(η−η∗))\lambda_{rev}=1-\sigma(k(\eta-\eta^{*}))

3.3 Articulated 3D Scene Graph Construction

Semantic-SAM masks on stationary keyframes become 3D parts with CLIP features. Geometric overlap merges detections; semantic similarity is deliberately excluded from merging so adjacent look-alike drawers remain distinct.

Each motion is scored against nearby parts by replaying candidate geometry and comparing projected masks with dynamic/static tracks. A binary integer program assigns exactly one object per motion and at most one motion per object, while penalizing overlapping selected parts. The printed constraints define pairwise selection indicators but do not explicitly connect each object-selection variable y to its assignment variables m; that implementation detail remains underspecified.

At maximum opening, new masks become candidate children. Overlap with the moving parent indicates ARTICULATED containment; overlap with its closed-state mask indicates STATIC containment. These are image/geometry heuristics, not verified physical containment. A child in a door can move with it; a jar behind a cabinet door can stay fixed.

Equations 2–3
minm,y,z​∑i∈𝒜∑j∈𝒪pi​j​mi​j+λ​∑(j,k)∈𝒫qj​k​zj​k,\underset{m,y,z}{\operatorname{min}}\sum_{i\in\mathcal{A}}\sum_{j\in\mathcal{O}}{p_{ij}m_{ij}}+\lambda\sum_{(j,k)\in\mathcal{P}}{q_{jk}z_{jk}}, (2)
∑j∈𝒪mi​j=1∀i∈𝒜,∑i∈𝒜mi​j≤1∀j∈𝒪,zj​k≤yj,zj​k≤yk, and zj​k≥yj+yk−1,\displaystyle\sum_{j\in\mathcal{O}}m_{ij}=1\,\,\forall i\in\mathcal{A},\quad\sum_{i\in\mathcal{A}}m_{ij}\leq 1\,\,\forall j\in\mathcal{O},\quad z_{jk}\leq y_{j},z_{jk}\leq y_{k},\text{ and }z_{jk}\geq y_{j}+y_{k}-1, (3)

𝒢=(𝒱,ℰ)\mathcal{G}=(\mathcal{V},\mathcal{E})

𝒪𝒜⊆𝒪\mathcal{O}^{\mathcal{A}}\subseteq\mathcal{O}

mi​j∈{0,1}m_{ij}\in\{0,1\}

yj∈{0,1}y_{j}\in\{0,1\}

zj​k∈{0,1},j,k∈𝒪z_{jk}\in\{0,1\},j,k\in\mathcal{O}

zj​k=yj∧ykz_{jk}=y_{j}\,\wedge y_{k}

4 Experimental Evaluation

Tables 1–2 use ground-truth interaction segments, with predicted-segment experiments in the appendix. Pandora receives privileged depth, segmentation and manually selected states; ArtGS is adapted to three hand-masked frames. Thus the methods do not share identical inputs.

On Arti4D, MoMa-SG reports prismatic/revolute axis errors of 10.374°/14.800°, revolute position error 0.087 m, joint-type accuracy 90.9% and mode accuracy 98.3%. ArtiPoint reports 23.272°/26.358°, 0.248 m and 77.6% type accuracy. A denser 96×96 grid improves some metrics but worsens revolute angle error, so density is not uniformly beneficial.

Free-object IoU is 0.533 versus ConceptGraphs’ 0.438; matched articulated-part IoU is 0.434 versus Pandora’s 0.065. The RH201 child results report IoU 11.14, recall 42.60% and relation accuracy 90.56%; the IoU scale is not marked as a percentage in the header. High relation accuracy is conditional on found/matched children and coexists with low recall.

HSR and Spot opening/closing tests use 50 trials per robot–task combination and exceed 80% success in the depicted settings. These are structured real-world trials, not a broad open-world reliability estimate. The full comparison tables, qualitative scenes and success plots remain below.

Table 1: Articulated object estimation results on Arti4D. We report axis-angle errors θe​r​r\theta_{err}, pos. errors dL​2d_{L2} of rev. joints, type accuracy, and mode recognition quality. We conduct the ablation study on Arti4D, see top 6 rows.
Arti4D DROID
Method Prismatic Revolute Joint Type [%] Mode Prismatic Revolute Type
θe​r​r\theta_{err}[∘] θe​r​r\theta_{err}[∘] dL​2d_{L2}[m] Acc. Recall Pris./Rev. Acc. [%] θe​r​r\theta_{err}[∘] θe​r​r\theta_{err}[∘] dL​2d_{L2}[m] Acc.[%]
Ours w/ CoTracker3 13.190 22.982 0.091 88.4 91.7 / 86.6 98.3
Ours w/ 32x32 queries 14.982 18.424 0.110 86.1 82.5 / 92.9
Ours w/ 96x96 queries 09.263 16.704 0.066 93.8 91.5 / 100.0
Ours w/o ξ\xi-regul. + w/o RANSAC 34.089 23.175 0.103 90.9 87.7 / 97.0
Ours w/o ξ\xi-regular. + RANSAC 21.427 17.119 0.103 90.2 87.4 / 95.2
Ours w/o RANSAC 11.205 20.382 0.113 90.3 87.6 / 95.6
ArtGS [32] 52.070 62.638 0.301 61.9 100.0 / 00.0 - - - - -
Pandora [44] 46.814 50.620 0.195 55.9 77.2 / 44.4 - - - - -
ArtiPoint [15] w/ [29] 17.754 28.467 0.313 87.2 85.4 / 90.5 - 28.47 27.33 0.333 62.2
ArtiPoint w/o prior [15] 22.953 27.057 0.269 77.6 71.3 / 89.9 - 34.68 26.10 0.284 61.8
ArtiPoint [15] 23.272 26.358 0.248 77.6 70.9 / 90.6 - 35.88 25.43 0.278 61.1
MoMa-SG (ours) 10.374 14.800 0.087 90.9 87.7 / 95.8 98.3 07.15 16.91 0.115 89.5
Complete original S4.F3, including all panels and labels.
Figure 3: Qualitative results of MoMa-SG on Arti4D-Semantic: Estimated axis positions and corresponding object masks. As demonstrated, we observe minimal errors for a large number of prismatic objects and small errors on revolute objects. Revolute axes are depicted with a blue torus, prismatic axes without.
Complete original S4.F4, including all panels and labels.
Figure 4: Articulated 3D scene graphs: The top node represents the root node, the cube-shaped nodes represent articulated objects (parents), while the smaller, lower nodes represent discovered child nodes with respect to their articulated parents.
Table 2: Articulated 3D part segmentation (left) and contained object discovery (right) evaluation on Arti4D-Semantic: The upper four rows refer to free articul.-indep. objects 𝒪\mathcal{O}, the lower section contains objects 𝒪𝒜\mathcal{O}^{\mathcal{A}} matched against articulations. The IoU denotes mutual overlap under a linear assignment. The recall indicates the GT overlap at various IoUs. For object discovery (right), we only evaluate objects paired with GT objects on the, the IoU denotes the highest-achieved overlap between a predicted child node and all GT children. The relation accuracy measures correctly identified parent-child edges. Evaluation on RH201 split of Arti4D-Semantic.
3D Part Mapping Contained Object Discovery
Method Object IoU Recall IoU Recall [%] Relation Acc. [%]
Set @0.25 @0.50 @0.75 @IoU=0.25 STATIC / ARTIC.
OpenMask3D [56] 𝒪\mathcal{O} 0.082 0.087 0.019 0.000
ConceptGraphs [4] 0.438 0.332 0.127 0.018
Ours w/o bound.-resp. merging 0.481 0.807 0.534 0.086
MoMa-SG (ours) 0.533 0.824 0.646 0.163
Pandora [44] 𝒪𝒜\mathcal{O}^{\mathcal{A}} 0.065 0.012 0.000 0.000 00.02 00.35 19.77
Ours w/ CoTracker3 0.221 0.370 0.218 0.027 09.14 16.67 59.23
Ours w/ Mobile-SAM 0.417 0.672 0.453 0.112 09.31 35.66 67.08
Ours w/ greedy matching 0.423 0.703 0.453 0.110 09.50 37.43 67.19
MoMa-SG (ours) 0.434 0.720 0.471 0.111 11.14 42.60 90.56
Complete original S4.F5, including all panels and labels.
Figure 5: We portray our real-world mobile manipulation environments of the HSR as well as the Spot. We demonstrate that MoMa-SG allows for robust mobile manipulation even under various articulation states of objects. The sample size for each robot-task combination is 50.

5 Conclusion

MoMa-SG integrates semantic geometry and observed kinematics sufficiently to support articulated-object manipulation with two mobile robot platforms. Its main limits are cascading perception/association errors, costly offline mapping, restricted joint families and semi-static scenes. Automatic updates after object rearrangement remain future work.

Acknowledgments

The authors acknowledge BrainLinks–BrainTools, an NVIDIA research grant and Toyota’s HSR program. Hermann Blum is also affiliated with the Lamarr Institute.

Appendix A.1 Approach Details on MoMa-SG

The supplementary method specifies temporal segmentation, regularized estimation, incremental part merging, association and the complete robot execution stack. All original formal expressions, the merging algorithm and visual examples are retained.

A.1-A Interaction Discovery

YOLOv11 supplies hand/end-effector masks. Depth from a previous frame is warped into the current view with known poses; normalized depth residual and interaction-prior signals are median-filtered.

The conditional interaction table is tuned on held-out data, and the fusion factorizes depth and hand evidence as independent probabilities. Calibration or independence is not established. Nonzero segments are length-filtered using minimum/maximum bounds whose numeric values are not supplied here.

Equations 1–3
Δt\displaystyle\Delta_{t} =|𝐃t−𝐃~t←sH|\displaystyle=\left|\mathbf{D}_{t}-\mathbf{\tilde{D}}_{t\leftarrow s_{H}}\right| (1)
𝐃~t←sH\displaystyle\mathbf{\tilde{D}}_{t\leftarrow s_{H}} =proj​(𝐓t−1​𝐓t−sH​proj−1​(𝐃t−sH))\displaystyle=\textrm{proj}\left(\mathbf{T}_{t}^{-1}\mathbf{T}_{t-s_{H}}\textrm{proj}^{-1}(\mathbf{D}_{t-s_{H}})\right) (2)
p⁡(s)\displaystyle p\left(s\right) =∑d,h∈{⊤,⊥}p(s,d,h)=∑d,hp(s|d,h)p(d,h)=∑d,hp(s|d,h)p(d)p(h)\displaystyle=\sum_{d,h\in\{\top,\bot\}}p\left(s,d,h\right)=\sum_{d,h}p\left(s\;\middle|\;d,h\right)p\left(d,h\right)=\sum_{d,h}p\left(s\;\middle|\;d,h\right)p\left(d\right)p\left(h\right) (3)

𝐃=proj​(𝐩)\mathbf{D}=\textrm{proj}(\mathbf{p})

ht,δt∈[0,1]h_{t},\delta_{t}\in[0,1]

δt=p(d=⊤)\delta_{t}=p\left(d=\top\right)

ht=p(h=⊤)h_{t}=p\left(h=\top\right)

A.1-B Regularized Twist Estimation:

After lifting/smoothing tracks, reject trajectories shorter than 0.15 m and use DBSCAN rigidity/spatial filtering. RANSAC runs 100 iterations with a 30% point subset. Displacement-direction agreement distinguishes translation from rotation, and regularization suppresses implausible rotational/helical components.

Synthetic comparisons retain the privileged pure-prismatic/pure-revolute estimators alongside regularized and unconstrained twists. Mode recognition uses GPT-5-mini, whose reported accuracy is 98.3%, versus 69.3% for GPT-5-nano. The exact prompt is: “Is the following series of images a consecutive 'opening-closing', 'closing-opening', or just 'opening', or 'closing'. Please answer with one of these four options.”

Complete original A1.F2, including all panels and labels.
Figure A.2: Estimated twist components given an observed prismatic motion. We depict observed 3D point trajectories (left), sampled point pairs used for articulation estimation (middle), and the estimated articulation model (right).
Complete original A1.F3, including all panels and labels.
Figure A.3: Estimated twist components given an observed revolute motion of small angular range using our proposed regularized twist estimation scheme. We visualize point tracks (left), sampled point pairs (middle), and rotational and translation twist components (right).
Complete original A1.F4, including all panels and labels.
Figure A.4: Articulation mode prompt consisting of four concatenated keyframes along the interaction.
Complete original A1.F5, including all panels and labels.
Figure A.5: Axis and position prediction errors of a regularized twist estimator (ours), under a privileged purely prismatic/revolute model (prismatic/revolute twist), and a non-regularized twist at various articulation configurations sampled synthetically and subject to point-level Gaussian noise.

A.1-C Open-Vocabulary 3D Part Mapping:

Select low-blur frames outside interactions using Laplacian sharpness and translation/rotation changes. Masks touching an image boundary are non-isolated; other masks are isolated. Incremental merging uses geometric similarity and containment, with different rules for isolated/partial observations, as preserved in Algorithm 1.

The optional dense mode reinitializes tracking at quarter, half and three-quarter interaction times. Masks with dynamic/static support ratio above 0.5 seed SAM2 propagation, giving denser collision geometry at additional computational cost.

The printed rotation-change formula, Eq. (4), uses arccos(tr(Rₖ₋₁Rₖ⁻¹)/2), omitting the usual subtraction of one. For identical rotations its argument is 1.5. The original expression is retained; the implementation’s intended formula is unresolved.

Equation 4
γ=‖𝐭k−𝐭k-1‖,β=arccos⁡(tr⁡(𝐑k−1​𝐑k−1)2),\gamma=\left\|\mathbf{t}_{k}-\mathbf{t}_{\text{k-1}}\right\|\quad,\quad\beta=\operatorname{arccos}\left(\frac{\operatorname{tr}(\mathbf{R}_{k-1}\mathbf{R}_{k}^{-1})}{2}\right), (4)
Complete original algorithm1, including all panels and labels.
Algorithm 1 Incremental Merging
Copyable algorithm
Input : Detections 𝒟\mathcal{D}, Objects 𝒪\mathcal{O}, Similarity matrix 𝐒\mathbf{S}
Output : Updated free objects 𝒪\mathcal{O}
for i←0i\leftarrow 0 to |𝐒|r​o​w​s−1|\mathbf{S}|_{rows}-1 do
j←argmax​(Si)j\leftarrow\text{argmax}(S_{i})
si​j←𝐒⁡(i,j)s_{ij}\leftarrow\mathbf{S}({i,j})
d←𝒟id\leftarrow\mathcal{D}_{i}
o←𝒪jo\leftarrow\mathcal{O}_{j}
if o.isolatedo.\text{isolated} then
if si​j≥ϵi​s​o​l​a​t​e​ds_{ij}\geq\epsilon_{isolated} or overlap(d,od,o) >ϵo​v​e​r​l​a​p>\epsilon_{overlap} then
Oj←O_{j}\leftarrow Merge(o,do,d)
    end if
else
if d.isolatedd.\text{isolated} then
if overlap(o,do,d) >ϵo​v​e​r​l​a​p>\epsilon_{overlap} then
o←o\leftarrow Merge(o,do,d)
o.isolated←Trueo.\text{isolated}\leftarrow\text{True}
      end if
else
o←o\leftarrow Merge(o,do,d)
    end if
  end if
end for
return 𝒪\mathcal{O}
Complete original A1.F6, including all panels and labels.
Figure A.6: We depict dense 2D segmentation masks across two distinct keyframes and denote masks considered as isolated and non-isolated, respectively. The scene stems from the HSR split of Arti4D-Semantic.
Complete original A1.F7, including all panels and labels.
Figure A.7: Dense object segmentation for object reconstruction using SAM2 based on repeated TapNext++ initialization along each interaction.

A.1-D Object-to-Articulation Matching

For each estimated articulation, test the k nearest mapped part centroids. Replay each part through the estimated motion, project its visible geometry and compare inside/outside dynamic and static tracks, using convex projected masks when available.

Eq. (5) is printed as ratios of inside/outside counts, although the prose calls it a mean association cost. Its denominators can vanish, and its sign/normalization relative to the minimized objective is not explained. The exact equation is retained without silently repairing it. Global assignment then discourages mutually overlapping selected parts.

Equation 5
pi​j=∑t=tste∑k=1F𝟏int⁡(Ω)​(xk,td)∑k=1F𝟏ℝd∖Ω​(xk,td)+(1−∑k=1F𝟏ℝd∖Ω​(xk,ts)∑k=1F𝟏int⁡(Ω)​(xk,ts)),p_{ij}=\sum_{t=t_{s}}^{t_{e}}\frac{\sum_{k=1}^{F}\mathbf{1}_{\operatorname{int}(\Omega)}(x_{k,t}^{d})}{\sum_{k=1}^{F}\mathbf{1}_{\mathbb{R}^{d}\setminus\Omega}(x_{k,t}^{{d}})}+\left(1-\frac{\sum_{k=1}^{F}\mathbf{1}_{\mathbb{R}^{d}\setminus\Omega}(x_{k,t}^{{s}})}{\sum_{k=1}^{F}\mathbf{1}_{\operatorname{int}(\Omega)}(x_{k,t}^{s})}\right), (5)
Complete original A1.F8, including all panels and labels.
Figure A.8: Qualitative results of articulation-to-object association scheme introduced in Sec. 3.3. Across all depicted frames, the shaded mask represents the most appropriate identified free object when articulated under the estimated articulation model that is projected into the camera frame as a red axis. Solid green image boundaries denote successful pairings, dashed yellow intermediate cases represent either wrong segmentations or misidentified articulation magnitudes under accurate articulation models, and dotted red boundaries denote wrongly associated object masks under permissible estimated axes.

A.1-D.1 Contained Object Discovery

Use mode and fitted state to select a maximum-opening view. Segment visible objects, associate them with the opened parent and classify their motion relation from overlap with the current versus closed-state parent masks. A missed/opening-state error can therefore hide children or assign the wrong relation. Ten original discovery frames preserve both successful and difficult cases.

Complete original A1.F9, including all panels and labels.
Figure A.9: We depict the object discovery component of MoMa-SG. Each of the 10 frames displays identified maximum-opening object states. Instance masks represent identified child objects wrt. the articulated parent.

A.1-E Real-World Mobile Manipulation

Robot execution combines localization, local registration, online handle/grasp estimation and sampled kinematic trajectories. Learned perception supplies geometry; an adapted N2M2 whole-body controller executes end-effector motions. The pipeline does not establish formal contact-force or collision guarantees.

A.1-E.1 Global Localization in Ego-Centric Maps

Fit the floor with RANSAC and refine its normal by SVD, then transform the reconstruction to a gravity-aligned map. Points within 0.05 m of the floor mark free space; points at 0.15–0.30 m mark occupancy. Morphological operations clean the grid, and AMCL localizes the robot using 2D lidar.

Equation 6
|𝐧⊤​𝐩i+d|<ϵ,|\mathbf{n}^{\top}\mathbf{p}_{i}+d|<\epsilon, (6)

A.1-E.2 Local Online Ego-centric Map Registration

Before each action, back-project depth within 1.5 m and register it against global-map geometry within a 0.8 m neighborhood. ICP estimates a refined world transform, optionally bounded in translation/rotation. Graph object poses are transformed through this corrected frame; this reduces accumulated pose error without rebuilding the scene graph.

Equations 7–9
𝒫l​o​c​a​lC={π−1​(𝐮,𝐃t​(𝐮))∣𝐃t​(𝐮)<δm​a​x}{}^{C}\mathcal{P}_{local}=\{\pi^{-1}(\mathbf{u},\mathbf{D}_{t}(\mathbf{u}))\mid\mathbf{D}_{t}(\mathbf{u})<\delta_{max}\} (7)
𝒫s​u​bW={𝐪∈𝒫W∣‖𝐪−𝐜l​o​c​a​l‖2<rl​o​c​a​l}{}^{W}\mathcal{P}_{sub}=\{\mathbf{q}\in{}^{W}\mathcal{P}\mid\|\mathbf{q}-\mathbf{c}_{local}\|_{2}<r_{local}\} (8)
𝐓o​b​jW~=𝐓WW~⋅𝐓GW⋅𝐓o​b​jG{}^{\tilde{W}}\mathbf{T}_{obj}={}^{\tilde{W}}\mathbf{T}_{W}\cdot{}^{W}\mathbf{T}_{G}\cdot{}^{{G}}\mathbf{T}_{obj} (9)

𝒫l​o​c​a​lW=𝐓CW​𝒫l​o​c​a​lC{}^{W}\mathcal{P}_{local}={}^{W}\mathbf{T}_{C}{}^{C}\mathcal{P}_{local}

A.1-E.3 Online Grasp Prediction

SAM3 prompted with “handle” supplies handle masks. Fit a box to each visible handle cloud; its longest axis defines gripper orientation and its midpoint defines the grasp position. Combine that direction with the articulated part’s surface normal at the current state to form the grasp transform.

A Hungarian assignment matches candidate handles to mapped objects using IoU, overlap and centroid distance. Partial views and already-open parts can cause incorrect associations; the grasp construction assumes suitable visible handle geometry.

Complete original A1.F10, including all panels and labels.
Figure A.10: Selection of grasps constructed using our handle detection pipeline employing SAM3 [55] and predominant axis estimation. The blue axis of each grasp constitutes its approach axis.
Equations 10–12
𝐚p=𝐑:,argmaxiℓi,\mathbf{a}_{p}=\mathbf{R}_{:,\,\arg\max_{i}\ell_{i}}, (10)
𝐭g=12​(𝐩min+𝐩max),\mathbf{t}_{g}=\tfrac{1}{2}(\mathbf{p}_{\min}+\mathbf{p}_{\max}), (11)
𝐓g​r​a​s​p=[−𝐧θ−𝐚p×𝐧θ𝐚p𝐭𝐠.0001]\mathbf{T}_{grasp}=\begin{bmatrix}-\mathbf{n}_{\theta}&-\mathbf{a}_{p}\times\mathbf{n}_{\theta}&\mathbf{a}_{p}&\mathbf{t_{g}}.\\ 0&0&0&1\end{bmatrix} (12)
Complete original A1.F11, including all panels and labels.
Figure A.11: Qualitative full-scene results on articulated object estimation and object association on splits of Arti4D-Semantic.
Complete original A1.F12, including all panels and labels.
Figure A.12: Estimated articulated 3D objects: The colored arrow represents the articulation axis and the distinctly colored point cloud represents the object. A torus around an axis denotes a revolute axis.

A.1-E.4 End-Effector Trajectory Sampling

Propagate the initial grasp through sampled articulation states with the twist exponential, then transform it from graph coordinates through the global and refined world frames. Opening and closing reverse or negate the sampled state range. These end-effector paths respect the estimated joint, so model or registration error directly affects execution.

Equations 13
𝐓e​eW~​(θ)=𝐓WW~⋅𝐓GW⋅(exp⁡(ξ^​θ)⋅𝐓g​r​a​s​pG){}^{\tilde{W}}\mathbf{T}_{ee}(\theta)={}^{\tilde{W}}\mathbf{T}_{W}\cdot{}^{W}\mathbf{T}_{G}\cdot\left(\exp(\hat{\xi}\theta)\cdot{}^{G}\mathbf{T}_{grasp}\right) (13)

Θ={θ0,θ1,…,θn}\Theta=\{\theta_{0},\theta_{1},\dots,\theta_{n}\}

A.1-E.5 Language-enabled Mobile Manipulation

GPT-5 plans over open(o), inspect(o), close(o), retrieve(o,p) and done(), conditioned on semantic object states. It replans after actions; optional retries insert inspection and repeat failed opening/closing/retrieval, helping recover from grip loss.

For “get the milk from the fridge,” the robot opens the fridge, inspects its contents and retrieves the milk along a grasp-approach direction. The demonstrations show language-conditioned composition of predefined skills; they are not a systematic benchmark of unconstrained language planning.

Complete original A1.F13, including all panels and labels.
Figure A.13: We depict the ability of MoMa-SG to perform long-horizon mobile manipulation of articulated scenes using the Toyota HSR. Throughout this episode (left to right, top to bottom), the robot successfully opened 7 different articulated objects, 3 revolute objects, and 4 prismatic objects. Throughout our experiments, we found that grasping failures are much more dominant than errors due to mediocre articulation axes.
Complete original A1.F14, including all panels and labels.
Figure A.14: We depict three consecutive opening actions conducted using the Boston Dynamics Spot.

Appendix A.2 Experimental Evaluation

The evaluation appendix defines metrics, separates ground-truth and predicted interactions, reports split-wise results and uncertainty, and analyzes real robot execution, failure propagation and runtime.

A.2-A Metrics

The original metric equations and full result tables remain. Their denominators, matching thresholds and conditional populations matter when comparing figures across tasks or dataset subsets.

A.2-A.1 Interaction Segmentation

Temporal segmentation uses global one-dimensional IoU and optimal bipartite matching of predicted/true intervals. Pairs below IoU 0.5 are discarded; precision/recall, matched-segment IoU and absolute onset/offset errors are then computed. Matched-only errors can look good even when many interactions are missed.

Equations 14–15
IoU1​D={|P∩G||P∪G|if ​|P∪G|>00if ​|P∪G|=0\mathrm{IoU}_{\mathrm{1D}}=\begin{cases}\frac{|P\cap G|}{|P\cup G|}&\text{if }|P\cup G|>0\\ 0&\text{if }|P\cup G|=0\end{cases} (14)
IoU⁡(p,g)=max⁡(0,min⁡(ep,eg)−max⁡(sp,sg))max⁡(ep,eg)−min⁡(sp,sg).\mathrm{IoU}(p,g)=\frac{\max\!\left(0,\min(e_{p},e_{g})-\max(s_{p},s_{g})\right)}{\max(e_{p},e_{g})-\min(s_{p},s_{g})}. (15)
Table A.1: Split-wise articulated object estimation on Arti4D-Semantic using ground truth interaction segmentation: We report the axis-angle errors θe​r​r\theta_{err}, the positional errors dL​2d_{L2} of revolute joints, and the accuracy of predicted joint types, including their respective recalls as in Tab. 1.
Split Paradigm GT Poses Prismatic joints Revolute joints Type Quality [%] Mode
θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] Accuracy Prismatic Recall Revolute Recall Acc.[%]
RH078 EGO ✓ 05.43 – 15.88 0.063 0.862 0.840 0.900 0.980
RR080 EGO ✓ 19.36 – 14.80 0.035 0.808 0.792 1.000 0.952
DR080 EGO ✓ 07.99 – 15.67 0.094 0.940 0.922 0.970 1.000
RH201 EGO ✓ 03.52 – 13.97 0.099 0.979 0.973 0.979 0.993
MHZH EGO ×\times 42.01 – 21.34 0.271 0.917 0.727 1.000 0.972
MHZH-EXO EXO ×\times 30.24 – 36.56 0.172 0.778 0.500 0.917 1.000
HSR ROBOT ×\times 30.99 – 51.05 0.367 0.605 0.684 0.526 0.852
Overall 19.94 – 24.18 0.157 0.841 0.777 0.899 0.964

A.2-A.2 Articulation Estimation

Axes are compared through angular error and shortest-axis position distance, using a special near-parallel branch at ε=10⁻⁴. Position error is reported only for revolute joints. Joint-type and opening/closing mode accuracy supplement geometric errors; an axis metric alone does not capture whether the robot moves in the correct direction.

Equations 16–17
di={(𝐩^i−𝐩g​t)⊤​(𝐚^i×𝐚g​t)‖𝐚^i×𝐚g​t‖if ​‖𝐚^i×𝐚g​t‖>ϵ‖(𝐩^i−𝐩g​t)×𝐚g​t‖else ,d_{i}=\begin{cases}\frac{(\hat{\mathbf{p}}_{i}-\mathbf{p}_{gt})^{\top}(\hat{\mathbf{a}}_{i}\times\mathbf{a}_{gt})}{\left\lVert\hat{\mathbf{a}}_{i}\times\mathbf{a}_{gt}\right\rVert}&\text{if }\left\lVert\hat{\mathbf{a}}_{i}\times\mathbf{a}_{gt}\right\rVert>\epsilon\\ \left\lVert(\hat{\mathbf{p}}_{i}-\mathbf{p}_{gt})\times\mathbf{a}_{gt}\right\rVert&\text{else }\end{cases}, (16)
Θe​r​r,i=cos−1⁡(𝐚g​t⊤​𝐚^i‖𝐚g​t‖​‖𝐚^i‖).\Theta_{err,i}=\cos^{-1}\left(\frac{\mathbf{a}_{gt}^{\top}\hat{\mathbf{a}}_{i}}{\left\lVert\mathbf{a}_{gt}\right\rVert\left\lVert\hat{\mathbf{a}}_{i}\right\rVert}\right). (17)

A.2-A.3 Articulated 3D Part Segmentation

Hungarian matching uses IoU of axis-aligned 3D bounding boxes, not volumetric surface-mask overlap. The paper reports mean IoU and recall at thresholds 0.25/0.50/0.75. Semantic retrieval ranks CLIP text categories against optimally matched object features, reporting top-1/2/5/10 accuracy.

Across all seven splits, TapNext++ articulated-part IoU averages 0.417, versus CoTracker3’s 0.233. Free/matched-part category top-1 accuracy is 35.5%/28.7% and top-5 is 73.4%/63.3%; open vocabulary does not imply uniformly accurate semantic labels.

Table A.2: Split-wise articulated 3D part segmentation on Arti4D-Semantic: We report the same set of metrics as in Tab. 2. Below, we list the performance using either CoTracker3 [53] or TapNext++[46], which is employed in default MoMa-SG.
Method Split Paradigm GT Poses IoU Recall
@0.25 @0.50 @0.75
w/ CoTracker3 RH078 EGO ✓ 0.184 0.327 0.181 0.000
RR080 EGO ✓ 0.196 0.347 0.154 0.035
DR080 EGO ✓ 0.243 0.387 0.241 0.019
RH201 EGO ✓ 0.262 0.416 0.287 0.049
MHZH EGO ×\times 0.208 0.317 0.108 0.025
MHZH-EXO EXO ×\times 0.238 0.396 0.229 0.000
HSR ROBOT ×\times 0.300 0.537 0.237 0.025
Overall 0.233 0.391 0.212 0.025
w/ TapNext++ RH078 EGO ✓ 0.436 0.740 0.402 0.144
RR080 EGO ✓ 0.459 0.744 0.552 0.131
DR080 EGO ✓ 0.404 0.667 0.398 0.091
RH201 EGO ✓ 0.434 0.717 0.505 0.087
MHZH EGO ×\times 0.393 0.565 0.269 0.090
MHZH-EXO EXO ×\times 0.387 0.683 0.289 0.074
HSR ROBOT ×\times 0.403 0.541 0.305 0.077
Overall 0.417 0.665 0.389 0.099
Equation 18
S⁡(i,c)=𝐟i⋅𝐰c‖𝐟i‖​‖𝐰c‖S(i,c)=\frac{\mathbf{f}_{i}\cdot\mathbf{w}_{c}}{\|\mathbf{f}_{i}\|\|\mathbf{w}_{c}\|} (18)
Table A.3: Articulated 3D part semantics evaluation on Arti4D-Semantic: We report open-vocabulary top-k retrieval scores as outlined in Sec. A.2-A.
Object Set Split t​o​p1top_{1} t​o​p2top_{2} t​o​p5top_{5} t​o​p10top_{10}
𝒪\mathcal{O} RH078 0.387 0.506 0.660 0.830
RR080 0.069 0.174 0.452 0.532
DR080 0.489 0.663 0.774 0.888
RH201 0.410 0.605 0.781 0.915
MHZH 0.357 0.523 0.725 0.980
MHZH-EXO 0.333 0.562 0.896 1.000
HSR 0.444 0.669 0.850 0.975
Overall 0.355 0.529 0.734 0.875
𝒪𝒜\mathcal{O}^{\mathcal{A}} RH078 0.212 0.356 0.641 0.781
RR080 0.026 0.123 0.403 0.549
DR080 0.305 0.424 0.609 0.778
RH201 0.374 0.480 0.758 0.935
MHZH 0.287 0.543 0.667 0.782
MHZH-EXO 0.271 0.396 0.583 0.812
HSR 0.531 0.662 0.769 0.875
Overall 0.287 0.426 0.633 0.787

A.2-A.4 Contained Objects

Child objects are compared at observed articulation states, voxelizing at 1.5 cm. Each prediction receives its best IoU against children of the assigned parent; recall uses a 0.25 threshold. STATIC/ARTICULATED relation accuracy is evaluated for associated children. State selection and parent association affect all three metrics.

Equation 19
IoUm​a​x​(c)=maxg∈𝒞p​a​r​e​n​t⁡(Pc∩PgPc∪Pg)\text{IoU}_{max}(c)=\max_{g\in\mathcal{C}_{parent}}\left(\frac{P_{c}\cap P_{g}}{P_{c}\cup P_{g}}\right) (19)

A.2-B Additional Experiments

Additional experiments test temporal discovery and downstream reconstruction using predicted segments, with split-wise tables and heavy-tail summaries supplementing the main aggregate results.

A.2-B.1 Interaction Segmentation

MoMa-SG’s temporal IoU is 0.649, precision 0.786 and recall 0.800, with onset/offset errors 0.816/0.742 s. GPT-5-mini yields 0.550 temporal IoU and lower recall 0.611, but higher matched-segment IoU 0.753 versus 0.718. Thus the narrative claim of the best segment IoU does not hold against every listed method.

The HMM baseline has higher precision, 0.871, but recall 0.532. Fusing depth and hand priors trades some precision for substantially better coverage; separate signals and matched-only metrics describe different failure modes.

Table A.4: Temporal interaction segmentation evaluation on Arti4D-Semantic: The 1D-IoU metric denotes the intersection over union of the dense binary timeseries signals, P and R, as well as the Segment-IoU denote the precision and recall, as well as the segment-wise IoU, following a linear sum assignment between predicted and ground truth segments under an IoU >> 0.5. IonI_{\text{on}} and IoffI_{\text{off}} measure the absolute start and end errors of matched segments.
Method 1D-IoU P R Segment-IoU IonI_{\text{on}} [s\mathrm{s}] IoffI_{\text{off}} [s\mathrm{s}]
GPT-5-mini 0.55 0.862 0.6111 0.753 8.179 10.12
Pandora [44] 0.359 0.690 0.400 0.689 0.7442 0.725
HMM 0.471 0.871 0.532 0.683 0.770 0.762
ArtiPoint [15] 0.575 0.678 0.714 0.698 1.05 0.558
MoMa-SG 0.649 0.786 0.800 0.718 0.816 0.742
Table A.5: Real-world articulation state estimation: Experiments are conducted across 54 various object articulation states.
Object Trans. Error   Rot. Error  
Type Δ​d\Delta d [m\mathrm{m}] σ\sigma [m\mathrm{m}] Δ​θ\Delta\theta [°\mathrm{\SIUnitSymbolDegree}] σ\sigma [°\mathrm{\SIUnitSymbolDegree}]
Prismatic 0.0174 0.006 - -
Revolute - - 6.477 3.604

A.2-B.2 Articulation Estimation

With predicted segments, MoMa-SG’s prismatic/revolute angle errors are 20.66°/17.31°, revolute position error 0.071 m, fit/missing-segment failure rate 10.5%, joint-type accuracy 82.0% and mode accuracy 80.7%. Ground-truth interactions therefore materially simplify the main setup. Failed/missed interactions are counted using an IoU 0.25 association threshold.

New MHZH/MHZH-EXO recordings use estimated camera poses and include textureless objects and external-camera occlusions; HSR is robot-centric. Across all seven splits, ground-truth-segment angle errors rise to 19.94°/24.18° and position error to 0.157 m.

Errors are strongly heavy-tailed: mean prismatic angle 10.37° contrasts with median 2.56°, and mean revolute angle 14.80° with median 4.30°. Paired Wilcoxon tests favor MoMa-SG in the reported comparisons, including p=0.02 for one position comparison; these tests do not establish transfer to other environments.

Table A.6: Articulated object estimation on Arti4D across methods under predicted interaction segments: Our evaluation matched predicted articulations against ground truth articulations if a 1D-IoU of at least 0.25 is exceeded. We report the axis-angle errors θe​r​r\theta_{err}, the positional errors dL​2d_{L2} of revolute joints, and the accuracy of predicted joint types, including their respective recalls as in Tab. 1.
Method Prismatic joints Revolute joints Failure Joint Type [%] Mode
θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] Rate [%] Accuracy Pris./Rev. Recall Acc.[%]
Pandora [44] 46.31 – 50.23 0.182 00.0 55.8 45.0 / 75.9 -
ArtiPoint [15] 24.88 – 30.24 0.293 23.2 77.4 88.2 / 72.2 -
MoMa-SG (ours) 20.66 – 17.31 0.071 10.5 82.0 73.8 / 98.4 80.7
Complete original A2.F15, including all panels and labels.
Figure A.15: Contained object discovery given the articulated 3D parent objects.
Table A.7: Articulated object estimation on Arti4D across dataset splits: We report the axis-angle errors θe​r​r\theta_{err}, the positional errors dL​2d_{L2} of revolute joints, and the accuracy of predicted joint types, including their respective recalls as in Tab. 1.
Split Paradigm GT Poses Prismatic joints Revolute joints Type Quality [%] Mode
θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] Accuracy Prismatic Recall Revolute Recall Acc.[%]
RH078 EGO ✓ 17.58 – 06.42 0.035 0.794 0.667 1.000 0.825
RR080 EGO ✓ 29.84 – 32.76 0.118 0.657 0.626 1.000 0.879
DR080 EGO ✓ 16.32 – 21.57 0.062 0.889 1.000 0.816 0.815
RH201 EGO ✓ 12.94 – 17.34 0.085 0.919 0.877 0.966 0.734
Overall 20.66 – 17.31 0.071 0.820 0.738 0.984 0.807
Table A.8: Error variance and percentiles of articulated axes on Arti4D
Method Prismatic joints Revolute joints
θe​r​r\theta_{err}[deg]±σ\pm\sigma θe​r​r\theta_{err}[deg]±σ\pm\sigma dL​2d_{L2}[m]±σ\pm\sigma
Pandora 46.81±\pm37.49 50.62±\pm28.58 0.20±\pm0.18
{47.31, 85.58, 88.43} {51.72, 79.69, 86.34} {0.11, 0.31, 0.42}
ArtiPoint 23.27±\pm31.67 26.36±\pm25.53 0.25±\pm0.26
{04.85, 45.19, 81.80} {12.86, 38.38, 62.22} {0.05, 0.10, 0.31}
MoMa-SG 10.37±\pm24.10 14.80±\pm22.99 0.09±\pm0.15
{02.56, 03.90, 14.93} {04.30, 11.21, 55.24} {0.03, 0.07, 0.27}

The values below the errors denote 50,75,90% percentile error scores, respectively.

Complete original A2.F16, including all panels and labels.
Figure A.16: Tracked keypoints of CoTracker3 [53] throughout interactions of texture-less objects on the MHZH split (ego-centric) of Arti4D-Semantic.
Complete original A2.F17, including all panels and labels.
Figure A.17: Tracked keypoints of TapNext++ [46] on the MHZH-EXO split (exo-centric) of Arti4D-Semantic along an interaction (left to right, top to bottom).

A.2-C 3D Part Mapping

Predicted-interaction part mapping reports IoU 0.448 across the four original splits. The all-seven-split value 0.417 uses a different population, so comparing those aggregates cannot show that predicted segmentation improves mapping. The reported standard deviations also indicate substantial scene/object variability.

Free objects are easier than associating articulated parts, and semantic accuracy drops after association. Repeated masks, tracking drift, inaccurate joint magnitudes and imperfect segmentation limit fine-grained children such as milk cartons or jars.

Table A.9: Articulated 3D part segmentation performance of MoMa-SG on Arti4D-Semantic using predicted interaction segments. MoMa-SG employs TapNext++ point tracks. We report the same set of metrics as in Tab. 2.
Split Paradigm GT Poses IoU Recall
@0.25 @0.50 @0.75
RH078 EGO ✓ 0.508 0.692 0.415 0.134
RR080 EGO ✓ 0.411 0.605 0.429 0.146
DR080 EGO ✓ 0.548 0.889 0.556 0.111
RH201 EGO ✓ 0.408 0.674 0.501 0.124
Overall 0.448 0.675 0.460 0.132

A.2-D Real-World Long-Horizon Mobile Manipulation

Across 54 real object states, mean prismatic displacement error is 0.0174 m and revolute angle error is 6.477°. One HSR sequence opens seven objects—three revolute and four prismatic—and a Spot sequence opens three. These qualitative long sequences supplement the separately counted robot trials.

Grip loss dominates observed execution failures; inaccurate handle association also occurs, especially when objects are already open. Closing is harder because current-state estimation and handle visibility become more demanding. The language milk-retrieval example uses the same localization and articulation stack.

A.2-D.1 Robustness Study

The RH201 analysis tracks failures through 3D tracking, trajectory extraction, articulation fitting, object association and child discovery. About 94.8% of parts are matched and mode accuracy is 99.3% under that study’s conditional definitions; these are not end-to-end child success rates.

Specular surfaces, failed filtering, drift and wrong motion magnitudes propagate downstream. The original Sankey chart preserves conditional stage percentages and losses, and the η sensitivity plots show the joint-type/axis tradeoff around the chosen threshold.

Complete original A2.F18, including all panels and labels.
Figure A.18: Robustness study across the stages of MoMa-SG on the RH201 split of the Arti4D-Semantic dataset.
Complete original A2.F19, including all panels and labels.
Figure A.19: Sensitivity of the proposed regularized twist estimation to various η\eta on the RH201 split of Arti4D-Semantic.
Chart mathematical labels

0.9850.985

0.9870.987

0.9890.989

0.9910.991

0.9930.993

0.9950.995

0.9970.997

0.9990.999

4040

6060

8080

100100

η\eta

0.9850.985

0.9870.987

0.9890.989

0.9910.991

0.9930.993

0.9950.995

0.9970.997

0.9990.999

2020

4040

6060

8080

100100

η\eta

θpris\theta_{\mathrm{pris}}

θpris\theta_{\mathrm{pris}}

±σ\pm\,\sigma

0.9850.985

0.9870.987

0.9890.989

0.9910.991

0.9930.993

0.9950.995

0.9970.997

0.9990.999

2020

4040

6060

8080

100100

η\eta

θrev\theta_{\mathrm{rev}}

θrev\theta_{\mathrm{rev}}

±σ\pm\,\sigma

0.9850.985

0.9870.987

0.9890.989

0.9910.991

0.9930.993

0.9950.995

0.9970.997

0.9990.999

0.10.1

0.20.2

0.30.3

0.40.4

0.50.5

η\eta

dd

m\mathrm{m}

d±σd\pm\,\sigma

A.2-D.2 Runtime

A ten-object graph takes 21.89 minutes on an RTX A6000, 128 GB RAM and Threadripper Pro 3975WX. Mapping accounts for 42.1% of runtime; other stages include filtering, interaction segmentation, tracking, trajectory filtering, articulation/mode estimation and association. The authors recommend at least 64 GB RAM and 24 GB VRAM. This is offline graph construction, not real-time end-to-end scene understanding.

Appendix A.3 Arti4D-Semantic Dataset & Benchmark

Arti4D-Semantic contains 62 RGB-D sequences across five physical scenes and seven splits, adding exocentric and robot-centric observations to Arti4D. Original splits have ground-truth poses; new MHZH/MHZH-EXO/HSR poses are estimated with DROID-SLAM.

Annotators fuse closed-state parent masks from selected keyframes into 3D, assign categories and independently verify them. Maximum-opening views label children and distinguish parent-attached from static regions. The dataset has 29 parent and 54 child categories; annotation geometry and state choice remain potential error sources.

Table A.10 lists all 62 recordings. Summing its rows gives 502 object entries and 510 interactions, whereas the main text reports 600 interactions; the source does not reconcile this difference. Child labels are marked for the 45 original recordings and absent for the 17 new recordings. Both the complete table and the source’s broader label-coverage claims are preserved without conflating them.

Complete original A3.F20, including all panels and labels.
Figure A.20: We visualize the three new dataset splits of Arti4D-Semantic: The ego-centric MHZH / exo-centric MHZH-EXO splits (left) and the robot-centric HSR split (right).
Complete original A3.F21, including all panels and labels.
Figure A.21: We visualize articulated 3D parent instance labels for two scenes contained in RH078 (top) and DIN080 (bottom), respectively. Each distinctly colored mask represents a closed articulated object.
Table A.10: Overview of all demonstration sequences of Arti4D-Semantic. The RH078, RR080, DR080, RH201, and MHZH splits are ego-centric. The MHZH-EXO split is exo-centric and the HSR split is robot-centric. We list the distribution of articulation types (PRISM / REV), difficulties, and the number of distinct interactions vs. the number of articulated objects.
Split GT Paradigm Sequence Recording # # # # Parent Child
Poses ID Objects Interactions PRISM / REV EASY / HARD Labels Labels
RH078 ✓ EGO RH078-00 scene_2025-04-04-19-14-38 7 7 3 / 4 7 / 0 ✓ ✓
✓ EGO RH078-01 scene_2025-04-04-19-18-54 6 7 2 / 4 4 / 2 ✓ ✓
✓ EGO RH078-02 scene_2025-04-07-11-39-17 7 7 3 / 4 7 / 0 ✓ ✓
✓ EGO RH078-03 scene_2025-04-07-11-41-52 8 9 3 / 5 6 / 2 ✓ ✓
✓ EGO RH078-04 scene_2025-04-07-11-48-40 7 7 3 / 4 7 / 0 ✓ ✓
✓ EGO RH078-05 scene_2025-04-09-10-30-11 8 8 5 / 3 3 / 5 ✓ ✓
✓ EGO RH078-06 scene_2025-04-09-10-32-52 6 6 5 / 1 3 / 3 ✓ ✓
✓ EGO RH078-07 scene_2025-04-09-10-35-47 7 7 6 / 1 2 / 5 ✓ ✓
✓ EGO RH078-08 scene_2025-04-09-10-38-38 7 7 5 / 2 3 / 4 ✓ ✓
✓ EGO RH078-09 scene_2025-04-09-10-46-48 7 8 7 / 0 4 / 3 ✓ ✓
✓ EGO RH078-10 scene_2025-04-09-10-49-20 7 7 6 / 1 2 / 5 ✓ ✓
RR080 ✓ EGO RR080-00 scene_2025-04-10-13-11-16 17 17 14 / 3 9 / 8 ✓ ✓
✓ EGO RR080-01 scene_2025-04-10-16-05-09 14 14 13 / 1 8 / 6 ✓ ✓
✓ EGO RR080-02 scene_2025-04-17-15-25-14 11 12 11 / 0 7 / 4 ✓ ✓
✓ EGO RR080-03 scene_2025-04-17-15-33-44 9 9 9 / 0 7 / 2 ✓ ✓
✓ EGO RR080-04 scene_2025-04-22-09-53-49 8 8 8 / 0 5 / 3 ✓ ✓
✓ EGO RR080-05 scene_2025-04-22-09-56-24 10 10 9 / 1 9 / 1 ✓ ✓
✓ EGO RR080-06 scene_2025-04-22-09-58-49 7 8 7 / 0 4 / 3 ✓ ✓
✓ EGO RR080-07 scene_2025-04-22-11-45-15 9 9 8 / 1 7 / 2 ✓ ✓
✓ EGO RR080-08 scene_2025-04-22-11-48-01 9 9 8 / 1 7 / 2 ✓ ✓
✓ EGO RR080-09 scene_2025-04-22-11-50-40 8 8 7 / 1 6 / 2 ✓ ✓
DR080 ✓ EGO DR080-00 scene_2025-04-11-11-44-32 11 11 7 / 4 5 / 6 ✓ ✓
✓ EGO DR080-01 scene_2025-04-11-12-58-58 10 10 6 / 4 5 / 5 ✓ ✓
✓ EGO DR080-02 scene_2025-04-11-13-01-59 9 9 5 / 4 4 / 5 ✓ ✓
✓ EGO DR080-03 scene_2025-04-11-13-18-00 9 10 4 / 5 3 / 6 ✓ ✓
✓ EGO DR080-04 scene_2025-04-11-13-43-03 11 11 7 / 4 5 / 6 ✓ ✓
✓ EGO DR080-05 scene_2025-04-11-14-01-06 11 11 7 / 4 5 / 6 ✓ ✓
✓ EGO DR080-06 scene_2025-04-11-15-43-24 11 12 7 / 4 5 / 6 ✓ ✓
✓ EGO DR080-07 scene_2025-04-11-15-46-48 11 11 6 / 5 4 / 7 ✓ ✓
RH201 ✓ EGO RH201-00 scene_2025-04-24-17-52-21 11 11 5 / 6 6 / 5 ✓ ✓
✓ EGO RH201-01 scene_2025-04-24-17-54-13 9 9 5 / 4 5 / 4 ✓ ✓
✓ EGO RH201-02 scene_2025-04-24-19-18-42 11 11 5 / 6 7 / 4 ✓ ✓
✓ EGO RH201-03 scene_2025-04-24-19-21-50 8 8 2 / 6 5 / 3 ✓ ✓
✓ EGO RH201-04 scene_2025-04-24-19-24-09 8 8 5 / 3 5 / 3 ✓ ✓
✓ EGO RH201-05 scene_2025-04-25-10-36-37 9 9 5 / 4 7 / 2 ✓ ✓
✓ EGO RH201-06 scene_2025-04-25-10-53-40 8 8 4 / 4 3 / 5 ✓ ✓
✓ EGO RH201-07 scene_2025-04-25-10-56-33 8 8 3 / 5 3 / 5 ✓ ✓
✓ EGO RH201-08 scene_2025-04-25-11-11-47 15 16 6 / 9 6 / 9 ✓ ✓
✓ EGO RH201-09 scene_2025-04-25-11-15-47 7 7 5 / 2 4 / 3 ✓ ✓
✓ EGO RH201-10 scene_2025-04-25-14-58-42 9 9 6 / 3 4 / 5 ✓ ✓
✓ EGO RH201-11 scene_2025-04-25-15-02-14 7 7 3 / 4 4 / 3 ✓ ✓
✓ EGO RH201-12 scene_2025-04-25-15-04-48 7 7 4 / 3 3 / 4 ✓ ✓
✓ EGO RH201-13 scene_2025-04-25-15-16-29 9 9 5 / 4 3 / 6 ✓ ✓
✓ EGO RH201-14 scene_2025-04-25-15-19-22 10 10 5 / 5 5 / 5 ✓ ✓
✓ EGO RH201-15 scene_2025-04-25-15-22-54 8 8 4 / 4 3 / 5 ✓ ✓
MHZH ×\times EGO MHZH-00 scene_20251205_171539 6 6 2 / 4 - / - ✓ -
×\times EGO MHZH-01 scene_20251206_152615 11 11 2 / 9 - / - ✓ -
×\times EGO MHZH-02 scene_20251206_153729 8 8 4 / 4 - / - ✓ -
×\times EGO MHZH-03 scene_20251206_154127 4 4 2 / 2 - / - ✓ -
×\times EGO MHZH-04 scene_20251206_154926 8 8 1 / 7 - / - ✓ -
MHZH EXO ×\times EXO MHZH-EXO-00 scene_20251210_123247 4 4 0 / 4 - / - ✓ -
×\times EXO MHZH-EXO-01 scene_20251210_123710 4 4 1 / 3 - / - ✓ -
×\times EXO MHZH-EXO-02 scene_20251210_123920-0 6 6 3 / 3 - / - ✓ -
×\times EXO MHZH-EXO-03 scene_20251210_123920-1 6 6 2 / 4 - / - ✓ -
HSR ×\times ROBOT HSR-00 scene_2026-01-05-15-44-54 5 5 2 / 3 - / - ✓ -
×\times ROBOT HSR-01 scene_2026-01-05-15-49-46 5 5 2 / 3 - / - ✓ -
×\times ROBOT HSR-02 scene_2026-01-05-15-53-13 5 5 2 / 3 - / - ✓ -
×\times ROBOT HSR-03 scene_2026-01-05-16-00-17 5 5 3 / 2 - / - ✓ -
×\times ROBOT HSR-04 scene_2026-01-05-16-02-59 5 5 3 / 2 - / - ✓ -
×\times ROBOT HSR-05 scene_2026-01-05-16-05-49 5 5 1 / 3 - / - ✓ -
×\times ROBOT HSR-06 scene_2026-01-05-16-09-34 4 4 2 / 2 - / - ✓ -
×\times ROBOT HSR-07 scene_2026-01-05-16-13-36 5 5 4 / 1 - / - ✓ -
Concept demonstration

Open image ↗