Overview
The supplementary material details matching and error metrics, estimated-camera and interaction-segment experiments, tracking ablations, scene-level results, and the Arti4D collection protocol. Its final sections and media are restored here from the full PDF because the available HTML stops partway through the supplement.
At a glance
ArtiPoint estimates articulated-object motion from egocentric RGB-D videos of human interaction. Hand-guided point tracking supplies object trajectories, depth and camera poses lift them into a common frame, and articulation-regularized optimization estimates joint axes and types. The accompanying Arti4D benchmark contains 45 sequences and 414 interactions across four scenes. Main results use ground-truth camera poses; an additional experiment substitutes estimated odometry. The work evaluates perception of simple two-body joints, not closed-loop robot manipulation.
1 Introduction
Changing viewpoints, occlusion, clutter, and missing depth make unconstrained articulation estimation harder than observing an isolated object from a fixed camera. ArtiPoint uses the interacting hand to select when and where to track, then separates camera motion from object motion. Its contributions are this estimation pipeline and a scene-level dataset with axis, temporal, difficulty, and camera-pose annotations.
2 Related Work
Earlier probabilistic methods fit joint models to part trajectories, while neural and reconstruction methods infer articulation from depth, point clouds, or multiview object states. ArtiPoint combines CoTracker3 [39] with the factor-graph estimator of Buchanan et al. [9]. The distinction is a tracking front end and regularized motion estimation for moving-camera human demonstrations, whose visibility conditions differ from the isolated-object setups of several baselines.
3 Approach
The four stages are interaction detection, hand-adjacent point tracking, depth-based trajectory filtering, and articulation estimation. Camera poses and intrinsics are required inputs alongside RGB-D measurements.
3.1 Extraction of Interaction Intervals
A pretrained hand segmenter produces a binary visibility signal. A moving-average threshold opens and closes interaction windows, and duration limits discard unsuitable segments. This provides a practical trigger but can merge different actions if the hand remains visible between them, or split an action when the hand is occluded.
3.2 Deep Point Tracking
Points around the hand prompt MobileSAM to identify candidate object masks. Shi–Tomasi features within those masks become queries for CoTracker3, which tracks them through the interaction and estimates visibility. The hand selects relevant regions; it does not itself provide an object’s motion model.
3.3 3D Track Estimation and Filtering
Depth lifts image tracks into 3D, and camera odometry transforms them to a world frame. Static, invalid-depth, and heavily occluded tracks are filtered. A smoothing objective balances measurement fidelity against velocity and jerk penalties. The main text describes static filtering in 3D; the supplemental ablation treats 3D filtering as a variant of a 2D default, an unresolved implementation-description mismatch.
Equation 1
| (1) |
3.4 Exploiting the Articulation Prior
A one-degree-of-freedom joint is represented by a shared twist and changing scalar configuration. Rather than fitting unrelated rigid transforms to each frame pair, the proposed estimator constrains the complete trajectory to share a common articulation. GTSAM solves this regularized pose-fitting problem, after which relative part poses feed the articulation-model estimator. The equations retain the full correspondence and transformation objectives; experiments focus on prismatic and revolute joints.
Equations 2–4
| (2) |
| (3) |
| (4) |
4 Arti4D Dataset
Arti4D contains 45 handheld RGB-D sequences and 414 interactions across four indoor scenes. Objects remain embedded in cluttered environments, with moving viewpoints and partial visibility. Labels include axis parameters, interaction windows, difficulty, reconstructed scenes, and externally tracked camera poses with stated centimeter-level accuracy. Difficulty depends on hand visibility, depth quality, separability of actions, and occlusion, rather than only joint type.
5 Experimental Results
Baselines include adapted Ditto and ArtGS pipelines and classical factor-graph estimators supplied with bounding-box or point-based trajectories. Predictions are associated with ground truth when temporal intersection over union exceeds 0.5; errors describe matched interactions, not a complete interaction-detection recall metric. Evaluation measures sign-invariant axis angle, revolute-axis position, and joint-type accuracy. The implementation reports 20 GB system memory and 16 GB video memory, without establishing real-time throughput.
5.1 Quantitative Results on Arti4D
With ground-truth camera poses, ArtiPoint reports mean angular errors of 14.54° for prismatic and 17.14° for revolute joints, and 0.07 m revolute-axis distance. Independent frame-pair transforms give 15.60°, 18.61°, and 0.15 m, supporting the shared-articulation constraint. Joint-type entries are 0.68 and 0.98, apparently fractions despite the table’s percent header. Difficulty splits show larger angular errors on hard examples.
Some source tables have inconsistent units or values: Table 3 labels distance in centimeters where Table 1 labels the same 0.07 result in meters, and Table 2 prints 0.87 m for hard revolute cases. The original tables remain intact; these entries are not silently rescaled or reconciled.
| Method | Prismatic joints | Revolute joints | Type accuracy [%] | |||
|---|---|---|---|---|---|---|
| [deg] | [m] | [deg] | [m] | Prismatic | Revolute | |
| ArtGS | 52.29 | – | 56.82 | 0.25 | 1.00 | 0.00 |
| Ditto | 55.03 | – | 60.89 | 0.29 | 0.00 | 1.00 |
| Sturm et al. [18] w/ bbox | 63.15 | – | 57.98 | 1.34 | 0.00 | 1.00 |
| Buchanan [9] w/ bbox | 49.67 | – | 58.82 | 0.22 | 0.00 | 1.00 |
| ArtiPoint w/ Sturm et al. [18] | 26.85 | – | 18.32 | 1.40 | 0.70 | 0.96 |
| ArtiPoint w/ indep. transforms | 15.60 | – | 18.61 | 0.15 | 0.65 | 0.96 |
| ArtiPoint (ours) | 14.54 | – | 17.14 | 0.07 | 0.68 | 0.98 |
| Prismatic joints | Revolute joints | Type accuracy [%] | |||||
|---|---|---|---|---|---|---|---|
| Diff. | Method | [deg] | [m] | [deg] | [m] | Prismatic | Revolute |
| EASY | ArtGS | 53.65 | - | 54.06 | 0.30 | 1.00 | 0.00 |
| Ditto | 54.46 | - | 51.83 | 0.37 | 0.00 | 1.00 | |
| ArtiPoint w/ Sturm et al. [18] | 20.24 | - | 13.11 | 0.96 | 0.82 | 0.95 | |
| ArtiPoint | 12.63 | - | 11.66 | 0.05 | 0.75 | 0.95 | |
| HARD | ArtGS | 50.12 | - | 59.48 | 0.20 | 1.00 | 0.00 |
| Ditto | 56.09 | - | 67.11 | 0.23 | 0.00 | 1.00 | |
| ArtiPoint w/ Sturm et al. [18] | 38.94 | - | 22.10 | 1.80 | 0.48 | 0.96 | |
| ArtiPoint | 18.03 | - | 21.10 | 0.87 | 0.54 | 1.00 | |
| Prismatic joints | Revolute joints | Type Accuracy [] | ||||
|---|---|---|---|---|---|---|
| [] | [] | [] | [] | Prismatic | Revolute | |
| w/ ORB keypoints | 19.22 | - | 25.61 | 0.10 | 0.55 | 0.92 |
| w/ keyframe stride 4 | 14.67 | - | 20.48 | 0.08 | 0.67 | 0.94 |
| w/o trajectory smoothing | 16.24 | - | 19.38 | 0.11 | 0.58 | 0.92 |
| w/ unreliable tracks | 14.61 | - | 18.86 | 0.09 | 0.67 | 0.93 |
| ArtiPoint (ours) | 14.54 | – | 17.14 | 0.07 | 0.68 | 0.98 |
5.2 Qualitative Results on Arti4D
The qualitative examples show two prismatic drawers and a revolute storage case. They illustrate useful estimates with few keypoints or missing depth, while the aggregate evaluation and limitations define the broader reliability claim.
![]() |
![]() |
![]() |
5.3 Ablation Study
Replacing Shi–Tomasi points with ORB, increasing keyframe stride to four, omitting smoothing, or admitting unreliable tracks worsens the reported aggregate axis estimates. Smoothing removal raises revolute angular error from 17.14° to 19.38°. These ablations support the pipeline components within the tested dataset, with the original distance-unit discrepancy retained.
6 Conclusion
Hand-guided tracking and globally constrained motion fitting extend articulation estimation to moving-camera human videos. Arti4D enables evaluation of this setting and related odometry challenges. The results establish a perception pipeline and benchmark; downstream robot execution is outside the reported experiments.
7 Limitations
The method needs trackable visual features, sufficiently reliable depth, accurate camera motion, and a usable temporal trigger. Reflective, dark, or transparent surfaces remain difficult, and the current model handles simple two-body articulation rather than complex joint chains. Hand detection can fail when interactions overlap or the hand is not withdrawn. Depth completion and more robust action recognition are proposed future work.
Acknowledgments
The authors acknowledge the University of Freiburg’s BrainLinks–BrainTools center and an NVIDIA academic grant.
S.1 Experimental Evaluation
The supplement expands the association protocol, camera-pose experiment, segment supervision, tracking ablations, and per-scene evaluation.
S.1.1 Evaluation Protocol & Metrics
Temporal windows associate repeated interactions with the appropriate labeled object axis when their intersection over union exceeds 0.5. Axis angle uses the absolute normalized dot product, so reversing axis direction does not change the error. Revolute position uses the shortest distance between axis lines, with a parallel-line special case. Prismatic axis position is not scored because translating that axis does not change its motion.
Equations 1–2
| (1) |
| (2) |
S.1.2 Performance Under Estimated Camera Poses
Main-paper results use external ground-truth odometry. For estimated poses, DROID-SLAM supplies frame-level camera motion; its reconstructed map is registered to the ground-truth point cloud with KISS-Matcher for evaluation. Angular errors become 14.67° and 18.12°, and revolute-axis distance 0.10 m, versus 14.54°, 17.14°, and 0.07 m with ground truth. This supports tolerance to the tested estimated poses while retaining an evaluation alignment step and challenging camera-tracking requirements.
| Method | Prismatic joints | Revolute joints | Type accuracy [%] | |||
|---|---|---|---|---|---|---|
| [deg] | [m] | [deg] | [m] | Prismatic | Revolute | |
| w/ DROID-SLAM poses | 14.67 | – | 18.12 | 0.10 | 0.71 | 0.94 |
| w/ Arti4D odometry | 14.54 | – | 17.14 | 0.07 | 0.68 | 0.98 |
S.1.3 Evaluation Using Ground-Truth Interaction Segments
Ground-truth interaction windows lower prismatic angular error from 14.54° to 11.99° but increase revolute error from 17.14° to 20.24° and axis distance from 0.07 to 0.09 m. The mixed change does not establish that temporal labeling is irrelevant; window contents and retained trajectories also change.
| Method | Prismatic joints | Revolute joints | Type accuracy [%] | |||
|---|---|---|---|---|---|---|
| [deg] | [m] | [deg] | [m] | Prismatic | Revolute | |
| w/ GT segments | 11.99 | – | 20.24 | 0.09 | 0.74 | 0.97 |
| w/ pred. segments | 14.54 | – | 17.14 | 0.07 | 0.68 | 0.98 |
S.1.4 Additional Ablation Study
Further ablations vary the hand-trigger timing, trajectory filters, tracker version, backward tracking, and query-point density.
S.1.4.1 Ablation on Hand Detection
Longer hand-detection smoothing can include idle phases and worsen estimates; shorter minimum windows can admit false positives. Removing outlier filtering or backward tracking raises revolute angular and position errors, and CoTracker2 performs worse than CoTracker3. The table also lists 3D static filtering as an ablation, in tension with the main-text description. Figure S.1 retains the full hand-mask and timing visualization.
| Method | Prismatic joints | Revolute joints | Type accuracy [%] | |||
|---|---|---|---|---|---|---|
| [deg] | [m] | [deg] | [m] | Prismatic | Revolute | |
| w/o filtering outlier tracks | 13.17 | – | 22.80 | 0.14 | 0.64 | 0.84 |
| w/o backward tracking | 13.92 | – | 20.67 | 0.13 | 0.71 | 0.94 |
| CoTracker2 | 15.78 | – | 21.04 | 0.10 | 0.66 | 0.94 |
| static traj. filter in 3D | 15.18 | – | 18.23 | 0.10 | 0.65 | 0.92 |
| 15.58 | – | 19.88 | 0.08 | 0.66 | 0.97 | |
| 14.57 | – | 18.13 | 0.07 | 0.68 | 0.96 | |
| 14.57 | – | 17.88 | 0.07 | 0.68 | 0.96 | |
| ArtiPoint | 14.54 | – | 17.14 | 0.07 | 0.68 | 0.98 |


S.1.4.2 Ablation on Point Tracking
Keyframe stride controls how many point queries are tracked. The curves favor a stride of two for angular error and show a tradeoff between feature coverage and computation. Supplemental prose is cautious about positional error, while Figure S.2’s caption calls stride two optimal; the plotted curve and original caption are preserved.
S.1.5 Scene-Respective Results
Scene results reveal different failure patterns. RH078 has the highest prismatic angular error, 35.11°, but not the worst revolute angular error; RH201 reaches 20.88° there. RH078 often contains hand-visible transitions between interactions and textureless metallic surfaces. The table therefore supports scene-specific weaknesses rather than uniformly ranking one scene hardest on every metric.
S.1.6 Qualitative Results
Figures S.3 and S.4 retain filtered point trajectories and a complete scene-level prediction. They show successful local tracking despite missing depth and the placement of multiple estimated axes and part poses. These examples complement the matched-interaction metrics without supplying an independent quantitative detection-recall estimate.
S.2 Arti4D Dataset
The released recordings contain aligned RGB, depth, and camera poses at 15 Hz, with higher-frequency transforms in the rosbags. An Azure Kinect is handheld while the operator interacts; HTC Vive trackers provide external pose ground truth. Sequences with substantial tracking glitches caused by occlusion or reflections are removed. Ground-truth poses drive volumetric scene reconstruction, whose axes are labeled in Blender and checked by a second reviewer.
Object counts differ from interaction counts because an object can be manipulated repeatedly. Some consecutive actions are difficult to separate because the hand stays visible. The complete 45-sequence inventory preserves these counts and difficulty splits. Figures S.5–S.7 show all four reconstructed environments, depth failures on a microwave, and the full multi-scene interaction montage.


