Literature
Display
← All papers

Articulated Object Estimation in the Wild

Abdelrhman Werby · Martin Büchner · Adrian Röfer · Chenguang Huang · Wolfram Burgard · Abhinav Valada

Preprint posted · CoRL 2025 (accepted) ↗

Robotics7 min text · CondensedOriginal paper ↗
Contents

Overview

The supplementary material details matching and error metrics, estimated-camera and interaction-segment experiments, tracking ablations, scene-level results, and the Arti4D collection protocol. Its final sections and media are restored here from the full PDF because the available HTML stops partway through the supplement.

Refer to caption
Figure 1: We present ArtiPoint, our novel approach to articulated object estimation in the wild. ArtiPoint makes use of deep point tracking and factor graph optimization. It does not rely on costly scene-wise optimization or deep models that are prone to overfit. We evaluate our approach on the Arti4D dataset, a novel scene-level articulated object dataset that is recorded with a moving, ego-centric camera, compared to previous benchmarks that are limited to isolated objects and static camera poses.

At a glance

ArtiPoint estimates articulated-object motion from egocentric RGB-D videos of human interaction. Hand-guided point tracking supplies object trajectories, depth and camera poses lift them into a common frame, and articulation-regularized optimization estimates joint axes and types. The accompanying Arti4D benchmark contains 45 sequences and 414 interactions across four scenes. Main results use ground-truth camera poses; an additional experiment substitutes estimated odometry. The work evaluates perception of simple two-body joints, not closed-loop robot manipulation.

1 Introduction

Changing viewpoints, occlusion, clutter, and missing depth make unconstrained articulation estimation harder than observing an isolated object from a fixed camera. ArtiPoint uses the interacting hand to select when and where to track, then separates camera motion from object motion. Its contributions are this estimation pipeline and a scene-level dataset with axis, temporal, difficulty, and camera-pose annotations.

2 Related Work

Earlier probabilistic methods fit joint models to part trajectories, while neural and reconstruction methods infer articulation from depth, point clouds, or multiview object states. ArtiPoint combines CoTracker3 [39] with the factor-graph estimator of Buchanan et al. [9]. The distinction is a tracking front end and regularized motion estimation for moving-camera human demonstrations, whose visibility conditions differ from the isolated-object setups of several baselines.

3 Approach

The four stages are interaction detection, hand-adjacent point tracking, depth-based trajectory filtering, and articulation estimation. Camera poses and intrinsics are required inputs alongside RGB-D measurements.

Refer to caption
Figure 2: Overview of our method: We take an ego-centric RGB-D video as input and employ hand tracking as a trigger signal to identify interaction segments (top left). We uniformly sample points around the hand masks and prompt a class-agnostic instance segmentation model (MobileSAM [40]), which yields object masks in the immediate vicinity that may be undergoing articulation. Given those masks, we detect stable keypoints (bottom left) that are fed into an any-point tracking model (CoTracker3 [39]) in order to obtain point trajectories throughout each entire articulation segment (top right). Finally, we estimate the underlying articulation model of the object through a factor graph formulation that operates on the obtained point trajectories (bottom right).

3.1 Extraction of Interaction Intervals

A pretrained hand segmenter produces a binary visibility signal. A moving-average threshold opens and closes interaction windows, and duration limits discard unsuitable segments. This provides a practical trigger but can merge different actions if the hand remains visible between them, or split an action when the hand is occluded.

3.2 Deep Point Tracking

Points around the hand prompt MobileSAM to identify candidate object masks. Shi–Tomasi features within those masks become queries for CoTracker3, which tracks them through the interaction and estimates visibility. The hand selects relevant regions; it does not itself provide an object’s motion model.

3.3 3D Track Estimation and Filtering

Depth lifts image tracks into 3D, and camera odometry transforms them to a world frame. Static, invalid-depth, and heavily occluded tracks are filtered. A smoothing objective balances measurement fidelity against velocity and jerk penalties. The main text describes static filtering in 3D; the supplemental ablation treats 3D filtering as a variant of a 2D default, an unresolved implementation-description mismatch.

Equation 1
E⁡(𝐩)=∑t=1Tvt​‖𝐩t−𝐩^t‖2+λv​e​l​‖𝐩t−𝐩t−1‖2+λj​e​r​k​‖𝐩t−3​𝐩t−1+3​𝐩t−2−𝐩t−3‖2,E(\mathbf{p})=\sum_{t=1}^{T}v_{t}||\mathbf{p}_{t}-\mathbf{\hat{p}}_{t}||^{2}+\lambda_{vel}||\mathbf{p}_{t}-\mathbf{p}_{t-1}||^{2}+\lambda_{jerk}||\mathbf{p}_{t}-3\mathbf{p}_{t-1}+3\mathbf{p}_{t-2}-\mathbf{p}_{t-3}||^{2}, (1)

3.4 Exploiting the Articulation Prior

A one-degree-of-freedom joint is represented by a shared twist and changing scalar configuration. Rather than fitting unrelated rigid transforms to each frame pair, the proposed estimator constrains the complete trajectory to share a common articulation. GTSAM solves this regularized pose-fitting problem, after which relative part poses feed the articulation-model estimator. The equations retain the full correspondence and transformation objectives; experiments focus on prismatic and revolute joints.

Equations 2–4
𝒵={Pm={⟨pft,pft+ϵ⟩|𝒱n,ft∧𝒱n,ft+ϵ}}m=1M,\mathcal{Z}=\left\{P_{m}=\left\{\left\langle p^{t}_{f},p^{t+\epsilon}_{f}\right\rangle\;\middle|\;\mathcal{V}^{t}_{n,f}\wedge\mathcal{V}^{t+\epsilon}_{n,f}\right\}\right\}_{m=1}^{M}, (2)
𝐓∗t=t+ϵ,mmint𝐓t+ϵ,m∑j=1|Pm|‖pjt+ϵ−𝐓t⋅t+ϵ,mpjt‖,{{}^{t}}{\mathbf{T^{*}}}{{}_{t+\epsilon,m}}=\min_{{{}^{t}}{\mathbf{T}}{{}_{t+\epsilon,m}}}\sum_{j=1}^{|P_{m}|}\left\lVert p^{t+\epsilon}_{j}-{{}^{t}}{\mathbf{T}}{{}_{t+\epsilon,m}}\cdot p^{t}_{j}\right\rVert, (3)
ξ^∗,θ1∗,…,θM∗=minξ^,θ1,…,θM∑m=1M∑j=1|Pm|‖pjt+ϵ−exp(ξ^θm)⋅pjt‖,\hat{\xi}^{*},\theta^{*}_{1},\ldots,\theta^{*}_{M}=\min_{\hat{\xi},\theta_{1},\ldots,\theta_{M}}\sum_{m=1}^{M}\sum_{j=1}^{|P_{m}|}\left\lVert p^{t+\epsilon}_{j}-\exp(\hat{\xi}\theta_{m})\cdot p^{t}_{j}\right\rVert, (4)

4 Arti4D Dataset

Arti4D contains 45 handheld RGB-D sequences and 414 interactions across four indoor scenes. Objects remain embedded in cluttered environments, with moving viewpoints and partial visibility. Labels include axis parameters, interaction windows, difficulty, reconstructed scenes, and externally tracked camera poses with stated centimeter-level accuracy. Difficulty depends on hand visibility, depth quality, separability of actions, and occlusion, rather than only joint type.

5 Experimental Results

Baselines include adapted Ditto and ArtGS pipelines and classical factor-graph estimators supplied with bounding-box or point-based trajectories. Predictions are associated with ground truth when temporal intersection over union exceeds 0.5; errors describe matched interactions, not a complete interaction-detection recall metric. Evaluation measures sign-invariant axis angle, revolute-axis position, and joint-type accuracy. The implementation reports 20 GB system memory and 16 GB video memory, without establishing real-time throughput.

5.1 Quantitative Results on Arti4D

With ground-truth camera poses, ArtiPoint reports mean angular errors of 14.54° for prismatic and 17.14° for revolute joints, and 0.07 m revolute-axis distance. Independent frame-pair transforms give 15.60°, 18.61°, and 0.15 m, supporting the shared-articulation constraint. Joint-type entries are 0.68 and 0.98, apparently fractions despite the table’s percent header. Difficulty splits show larger angular errors on hard examples.

Some source tables have inconsistent units or values: Table 3 labels distance in centimeters where Table 1 labels the same 0.07 result in meters, and Table 2 prints 0.87 m for hard revolute cases. The original tables remain intact; these entries are not silently rescaled or reconciled.

Table 1: Overall comparison of classical and deep articulation estimation methods on the Arti4D dataset. We report the axis-angle errors θe​r​r\theta_{err}, the positional errors dL​2d_{L2} of revolute joints, and the accuracy of joint type detection. We note that ArtiPoint achieves the lowest error in estimating articulation parameters.
Method Prismatic joints Revolute joints Type accuracy [%]
θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] Prismatic Revolute
ArtGS 52.29 – 56.82 0.25 1.00 0.00
Ditto 55.03 – 60.89 0.29 0.00 1.00
Sturm et al. [18] w/ bbox 63.15 – 57.98 1.34 0.00 1.00
Buchanan  [9] w/ bbox 49.67 – 58.82 0.22 0.00 1.00
ArtiPoint w/ Sturm et al. [18] 26.85 – 18.32 1.40 0.70 0.96
ArtiPoint w/ indep. transforms 15.60 – 18.61 0.15 0.65 0.96
ArtiPoint (ours) 14.54 – 17.14 0.07 0.68 0.98
Table 2: Difficulty-level comparison of estimation methods on Arti4D.
Prismatic joints Revolute joints Type accuracy [%]
Diff. Method θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] Prismatic Revolute
EASY ArtGS 53.65 - 54.06 0.30 1.00 0.00
Ditto 54.46 - 51.83 0.37 0.00 1.00
ArtiPoint w/ Sturm et al. [18] 20.24 - 13.11 0.96 0.82 0.95
ArtiPoint 12.63 - 11.66 0.05 0.75 0.95
HARD ArtGS 50.12 - 59.48 0.20 1.00 0.00
Ditto 56.09 - 67.11 0.23 0.00 1.00
ArtiPoint w/ Sturm et al. [18] 38.94 - 22.10 1.80 0.48 0.96
ArtiPoint 18.03 - 21.10 0.87 0.54 1.00
Table 3: Ablations of key components of our approach on the Arti4D dataset. As before, we report the same set of metrics as in Tab. 1.
Prismatic joints Revolute joints Type Accuracy    [%\mathrm{\%}]
θe​r​r\theta_{err} [deg\deg] dL​2d_{L2} [cm\mathrm{cm}] θe​r​r\theta_{err} [deg\deg] dL​2d_{L2} [cm\mathrm{cm}] Prismatic Revolute
w/ ORB keypoints 19.22 - 25.61 0.10 0.55 0.92
w/ keyframe stride 4 14.67 - 20.48 0.08 0.67 0.94
w/o trajectory smoothing 16.24 - 19.38 0.11 0.58 0.92
w/ unreliable tracks 14.61 - 18.86 0.09 0.67 0.93
ArtiPoint (ours) 14.54 – 17.14 0.07 0.68 0.98

5.2 Qualitative Results on Arti4D

The qualitative examples show two prismatic drawers and a revolute storage case. They illustrate useful estimates with few keypoints or missing depth, while the aggregate evaluation and limitations define the broader reliability claim.

Refer to caption Refer to caption Refer to caption
Figure 3: Qualitative results on Arti4D: Estimated joint axis and pose trajectory of a drawer in a kitchen scene (RH201), a revolute joint of a storage case (RH078), and another drawer in RH201.

5.3 Ablation Study

Replacing Shi–Tomasi points with ORB, increasing keyframe stride to four, omitting smoothing, or admitting unreliable tracks worsens the reported aggregate axis estimates. Smoothing removal raises revolute angular error from 17.14° to 19.38°. These ablations support the pipeline components within the tested dataset, with the original distance-unit discrepancy retained.

6 Conclusion

Hand-guided tracking and globally constrained motion fitting extend articulation estimation to moving-camera human videos. Arti4D enables evaluation of this setting and related odometry challenges. The results establish a perception pipeline and benchmark; downstream robot execution is outside the reported experiments.

7 Limitations

The method needs trackable visual features, sufficiently reliable depth, accurate camera motion, and a usable temporal trigger. Reflective, dark, or transparent surfaces remain difficult, and the current model handles simple two-body articulation rather than complex joint chains. Hand detection can fail when interactions overlap or the hand is not withdrawn. Depth completion and more robust action recognition are proposed future work.

Acknowledgments

The authors acknowledge the University of Freiburg’s BrainLinks–BrainTools center and an NVIDIA academic grant.

S.1 Experimental Evaluation

The supplement expands the association protocol, camera-pose experiment, segment supervision, tracking ablations, and per-scene evaluation.

S.1.1 Evaluation Protocol & Metrics

Temporal windows associate repeated interactions with the appropriate labeled object axis when their intersection over union exceeds 0.5. Axis angle uses the absolute normalized dot product, so reversing axis direction does not change the error. Revolute position uses the shortest distance between axis lines, with a parallel-line special case. Prismatic axis position is not scored because translating that axis does not change its motion.

Equations 1–2
di={(𝐩^i−𝐩g​t)⊤​(𝐚^i×𝐚g​t)‖𝐚^i×𝐚g​t‖if ​‖𝐚^i×𝐚g​t‖>ϵ‖(𝐩^i−𝐩g​t)×𝐚g​t‖else ,d_{i}=\begin{cases}\frac{(\hat{\mathbf{p}}_{i}-\mathbf{p}_{gt})^{\top}(\hat{\mathbf{a}}_{i}\times\mathbf{a}_{gt})}{\left\lVert\hat{\mathbf{a}}_{i}\times\mathbf{a}_{gt}\right\rVert}&\text{if }\left\lVert\hat{\mathbf{a}}_{i}\times\mathbf{a}_{gt}\right\rVert>\epsilon\\ \left\lVert(\hat{\mathbf{p}}_{i}-\mathbf{p}_{gt})\times\mathbf{a}_{gt}\right\rVert&\text{else }\end{cases}, (1)
Θe​r​r,i=cos−1⁡(𝐚g​t⊤​𝐚^i‖𝐚g​t‖​‖𝐚^i‖).\Theta_{err,i}=\cos^{-1}\left(\frac{\mathbf{a}_{gt}^{\top}\hat{\mathbf{a}}_{i}}{\left\lVert\mathbf{a}_{gt}\right\rVert\left\lVert\hat{\mathbf{a}}_{i}\right\rVert}\right). (2)

S.1.2 Performance Under Estimated Camera Poses

Main-paper results use external ground-truth odometry. For estimated poses, DROID-SLAM supplies frame-level camera motion; its reconstructed map is registered to the ground-truth point cloud with KISS-Matcher for evaluation. Angular errors become 14.67° and 18.12°, and revolute-axis distance 0.10 m, versus 14.54°, 17.14°, and 0.07 m with ground truth. This supports tolerance to the tested estimated poses while retaining an evaluation alignment step and challenging camera-tracking requirements.

Table S.1: Comparison of estimated object axis under estimated and ground truth camera poses
Method Prismatic joints Revolute joints Type accuracy [%]
θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] Prismatic Revolute
w/ DROID-SLAM poses 14.67 – 18.12 0.10 0.71 0.94
w/ Arti4D odometry 14.54 – 17.14 0.07 0.68 0.98

S.1.3 Evaluation Using Ground-Truth Interaction Segments

Ground-truth interaction windows lower prismatic angular error from 14.54° to 11.99° but increase revolute error from 17.14° to 20.24° and axis distance from 0.07 to 0.09 m. The mixed change does not establish that temporal labeling is irrelevant; window contents and retained trajectories also change.

Table S.2: Evaluation under ground-truth interaction segments
Method Prismatic joints Revolute joints Type accuracy [%]
θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] Prismatic Revolute
w/ GT segments 11.99 – 20.24 0.09 0.74 0.97
w/ pred. segments 14.54 – 17.14 0.07 0.68 0.98

S.1.4 Additional Ablation Study

Further ablations vary the hand-trigger timing, trajectory filters, tracker version, backward tracking, and query-point density.

S.1.4.1 Ablation on Hand Detection

Longer hand-detection smoothing can include idle phases and worsen estimates; shorter minimum windows can admit false positives. Removing outlier filtering or backward tracking raises revolute angular and position errors, and CoTracker2 performs worse than CoTracker3. The table also lists 3D static filtering as an ablation, in tension with the main-text description. Figure S.1 retains the full hand-mask and timing visualization.

Table S.3: Ablations of key parameters for extracting the interaction intervals component Sec. 3.1 on the Arti4D dataset. As before, we report the same set of metrics as in Tab. 1. Default values: Tm​a​x=90,Tm​i​n=30,wh=6T_{max}=90,T_{min}={30},w_{h}=6.
Method Prismatic joints Revolute joints Type accuracy [%]
θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] θe​r​r\theta_{err}[deg] dL​2d_{L2}[m] Prismatic Revolute
w/o filtering outlier tracks 13.17 – 22.80 0.14 0.64 0.84
w/o backward tracking 13.92 – 20.67 0.13 0.71 0.94
CoTracker2 15.78 – 21.04 0.10 0.66 0.94
static traj. filter in 3D 15.18 – 18.23 0.10 0.65 0.92
wh=12w_{h}=12 15.58 – 19.88 0.08 0.66 0.97
Tm​a​x=120T_{max}=120 14.57 – 18.13 0.07 0.68 0.96
Tm​i​n=15T_{min}=15 14.57 – 17.88 0.07 0.68 0.96
ArtiPoint 14.54 – 17.14 0.07 0.68 0.98
Refer to caption
Refer to caption
Figure S.1: Hand detection and interaction extraction: We visualize a live frame of an interaction, including a hand mask marked violet (top). In addition, we visualize the frequency of raw hand detections over time up to the live frame as well as its moving average (bottom). The horizontal red line indicates the threshold at which an interaction segment is created, given the moving average signal. The vertical dashed red line indicates the current frame.

S.1.4.2 Ablation on Point Tracking

Keyframe stride controls how many point queries are tracked. The curves favor a stride of two for angular error and show a tradeoff between feature coverage and computation. Supplemental prose is cautious about positional error, while Figure S.2’s caption calls stride two optimal; the plotted curve and original caption are preserved.

Original Figure S.2, restored from the PDF with every panel and label.
Figure S.2: Impact of keyframe stride on tracking accuracy and error. (a) The revolute joint angular error θerr[deg] exhibits a minimum at stride 2. (b) The revolute joint positional error dL2[m] is lowest at a stride of 2, with higher errors observed for both smaller and larger strides. (c) Similarly, the prismatic joint angular error θerr exhibits its minimum at stride of 2. We conclude that a stride of 2 is optimal in terms of point density and computational efficiency.

S.1.5 Scene-Respective Results

Scene results reveal different failure patterns. RH078 has the highest prismatic angular error, 35.11°, but not the worst revolute angular error; RH201 reaches 20.88° there. RH078 often contains hand-visible transitions between interactions and textureless metallic surfaces. The table therefore supports scene-specific weaknesses rather than uniformly ranking one scene hardest on every metric.

Original Table S.4, restored from the PDF with every panel and label.
Table S.4: Scene-respective results: We report the scene-wise results using the established set of metrics. We find that RH078 constitutes the most difficult split of the Arti4D dataset.

S.1.6 Qualitative Results

Figures S.3 and S.4 retain filtered point trajectories and a complete scene-level prediction. They show successful local tracking despite missing depth and the placement of multiple estimated axes and part poses. These examples complement the matched-interaction metrics without supplying an independent quantitative detection-recall estimate.

Original Figure S.3, restored from the PDF with every panel and label.
Figure S.3: Smoothed point trajectories: We visualize a cabinet (revolute) and its tracked keypoints (left) as well as a linear slider shelf (prismatic) on the right. Each keypoint trajectory is represented with a unique color. Both sets of point trajectories visualized constitute the output of our track filtering introduced in Sec. 3.3.
Original Figure S.4, restored from the PDF with every panel and label.
Figure S.4: Scene-level prediction: We depict a full scene-level output of the ArtiPoint framework on sequence scene 2025-04-11-11-44-32 of the DR080 scene. Yellow arrows denote axes of motion of predicted object interactions while coordinate frames represent the estimate part poses throughout articulation based on the proposed estimation framework (see Sec. 3.4).

S.2 Arti4D Dataset

The released recordings contain aligned RGB, depth, and camera poses at 15 Hz, with higher-frequency transforms in the rosbags. An Azure Kinect is handheld while the operator interacts; HTC Vive trackers provide external pose ground truth. Sequences with substantial tracking glitches caused by occlusion or reflections are removed. Ground-truth poses drive volumetric scene reconstruction, whose axes are labeled in Blender and checked by a second reviewer.

Object counts differ from interaction counts because an object can be manipulated repeatedly. Some consecutive actions are difficult to separate because the hand stays visible. The complete 45-sequence inventory preserves these counts and difficulty splits. Figures S.5–S.7 show all four reconstructed environments, depth failures on a microwave, and the full multi-scene interaction montage.

Original Figure S.5, restored from the PDF with every panel and label.
Figure S.5: We visualize the reconstructed scenes of the four Arti4D environments: DR080, RH078, RH201, and RR080.
Original Figure S.6, restored from the PDF with every panel and label.
Figure S.6: We visualize an exemplary interaction with the depth projected onto the RGB observation in red wherever a depth reading was unavailable for the particular image coordinate. As depicted, metallic, semi- transparent, or dark-colored objects do not produce reliable depth estimates, ultimately complicating the lifting of 3D point trajectories used to estimate the underlying articulation.
Original Table S.5, restored from the PDF with every panel and label.
Table S.5: Overview of all Arti4D demonstration sequences: We assign sequence identifiers to each recording and list the number of objects interacted with per sequence as well as the distribution of prismatic vs. revolute joints (PRISM / REV) and objects classified as either EASY or HARD. The number of objects corresponds to the number of annotated axes per sequence, whereas the number of interactions denotes the number of performed articulations, thus including objects that are articulated multiple times per sequence.
Original Figure S.7, restored from the PDF with every panel and label.
Figure S.7: We visualize several object interactions across the four different environments (DR080, RH078, RH201, RR080) captured as part of the Arti4D dataset.
Concept demonstration

Open image ↗