Literature
Display
← All papers

Hierarchical Motion Understanding via Motion Programs

Sumith Kulal · Jiayuan Mao · Alex Aiken · Jiajun Wu

· CVPR 2021 ↗

Computer Vision8 min text · CondensedOriginal paper ↗
Contents

At a glance

Motion Programs represent human movement at three levels: tracked two-dimensional keypoints, sequences of geometric motion primitives, and probabilistic loops over repeated primitives. Stationary, linear, and circular segments provide a compact description, while repeated start-middle-end trajectories capture variation across cycles.

The representation improves some sparse video-interpolation and repetitive-motion prediction results when combined with existing image generators. Evaluation uses golf, pose-warping clips, and three long exercise videos. These are explicit geometric descriptions of tracked movement, not general semantic action understanding; their success depends on pose quality and on the motion fitting the chosen primitives or repetition pattern.

1 Introduction

Keypoint sequences capture local motion but do not directly express a backswing, a repeated squat, or a loop of jumping jacks. The proposed hierarchy groups trajectories into longer primitives and then groups recurring primitives into loops. Figure 1 illustrates induction, editing, and execution.

A concrete domain-specific language (DSL) specifies geometry, timing, and duration. An abstract language represents distributions over primitives and for-loops. Synthesizing these programs allows denoising, interpolation within a primitive, repetition detection, and continued execution of a recurring pattern.

Refer to caption
Figure 1: Overview of our hierarchical motion understanding framework. We start with low-level keypoints extracted from raw videos and build higher level motion understanding. Higher levels of abstraction consist of a sequence of motion primitives and their repetitive patterns. Similar motion in repetitive patterns are captured using a probabilistic for-loop body (refer Figure 3).

2 Related Work

The work connects procedural representations of visual data with video synthesis and motion modeling.

Program-like representations for visual data.

Procedural graphics and program-like image representations describe complex visuals with reusable structure. Earlier motion systems also use parameterized curves and recombinable segments. Here, programs are inferred from keypoint tracks extracted from video, with probabilistic loops adding a level for repeated temporal structure.

Video synthesis.

Neural interpolation and prediction methods synthesize frames using learned features, optical flow, blending kernels, or pose dynamics. Motion Programs supply higher-level pose structure to those generators rather than replacing image synthesis. The interpolation experiments explicitly combine the representation with SepConv.

3 Hierarchical Motion Understanding

The three levels trade direct observation for compact structure. Concrete programs are fitted to trajectories; abstract programs are inferred from similarities between consecutive concrete primitives.

Level 1—Keypoints.

Off-the-shelf tools such as AlphaPose or OpenPose recover one two-dimensional trajectory per body joint. These tracks are the input to program synthesis, so occlusions and detector errors propagate into the representation.

Level 2—Concrete Motion Programs.

A concrete primitive is stationary, linear, or circular motion with explicit spatial and temporal parameters. A program executes a sequence of these primitives for their specified durations. Table 1 retains the full grammar, and Figure 2 shows golf and squat decompositions. The vocabulary is intentionally small; richer motion could require additional primitives such as splines.

Refer to caption
Figure 2: Examples of synthesized motion primitives. a) This golf swing has three primitives: back-swing, pause and down-swing. b) The squats sequence has a repeating subsequence of three similar primitives: squating down, standing up and a brief rest in the standing pose.

Level 3—Abstract Motion Programs.

An abstract primitive records start, middle, and end points plus duration, with probability distributions for repeated variants. This common representation can group instances whose best concrete types differ. Execution samples a trajectory description, chooses the concrete primitive that fits it best, and executes it.

Table 2 retains the abstract grammar. A for-loop repeats a body of such distributions, capturing approximate recurrence rather than requiring identical cycles. Three points distinguish curved from straight motion in this representation; they do not describe arbitrary articulated dynamics.

Table 1: DSL of Concrete Programs (Level 2) for describing motion as a sequence of motion primitives.
Program ⟶\longrightarrow Prim; …; Prim
Prim ⟶\longrightarrow circle (center: Point, radius: R,
angle=(vel: R, start: R), time: Int)
Prim ⟶\longrightarrow line (x=(vel: R, start: R),
y=(slope: R, intercept: R), time: Int)
Prim ⟶\longrightarrow stationary(point: Point, time: Int)
Point ⟶\longrightarrow (x: R, y: R)
Int ⟶\longrightarrow a positive integer
R ⟶\longrightarrow a real number
Table 2: DSL of Abstract Programs (Level 3) for describing motion as a sequence of probabilistic motion primitives with loops.
Program ⟶\longrightarrow Stmt; …; Stmt
Stmt ⟶\longrightarrow DetPrim || For-Loop (iter: Int) {
ProbPrim; …; ProbPrim }
DetPrim ⟶\longrightarrow {<start, middle, end>:
<Point, Point, Point>, time: Int}
ProbPrim ⟶\longrightarrow {<start, middle, end>:
Gaussian(<Point, Point, Point>),
), time: FreqDist(Int)}
Point ⟶\longrightarrow (x: R, y: R)
Int ⟶\longrightarrow a positive integer
R ⟶\longrightarrow a real number

3.1 Concrete Motion Programs

Concrete program induction first fits individual candidate primitives, then chooses temporal boundaries across the observed sequence.

Synthesizing single primitives.

For a circular candidate, fit a circle to the tracked positions, project the points onto it, and estimate temporal parameters. Linear and stationary candidates are fitted similarly. Select the type with the smallest reconstruction error. The authors report that separating geometric and temporal fitting was faster than joint optimization with comparable results.

Segmentation and synthesis.

Dynamic programming finds a segmentation minimizing summed fit errors plus a penalty per primitive, as specified in Equation 1. A small penalty allows many short segments; a large one favors a single coarse segment. The implementation adapts the penalty to the local covariance of keypoints, making granularity depend on motion scale.

Equation 1
Errorn=mink<n[Errork+fit(keypoints[k:n])+λ].\mathrm{Error}_{\text{n}}=\min\limits_{k<n}\left[\mathrm{Error}_{\text{k}}+\mathrm{fit}(\text{keypoints}[k:n])+\lambda\right]. (1)
Refer to caption
Figure 3: Illustration of rolling up 6 repetitive (alternating) statements into a for-loop of body size 2. We first translate concrete primitives to deterministic abstract primitives and then synthesize for-loops with probabilistic primitives in the body. Concrete primitives are sampled from the probabilistic abstract primitives during execution.

Multiple keypoints.

For several joints, sum fitting errors across trajectories and use one shared segmentation so all joints change primitives simultaneously. Other than this boundary selection, the programs are constructed per keypoint. The method therefore does not impose a full-body kinematic or physical model.

3.2 Abstract Motion Programs

Abstract synthesis converts concrete segments to a common representation, searches for repeated groups, and replaces accepted groups with probabilistic loops.

Concrete to abstract translation.

Replace each fitted concrete primitive with its start, middle, and end positions and duration. This deterministic intermediate form allows comparison across different concrete primitive types.

Loop detection.

Slide a window over the primitive sequence and try increasing loop-body lengths. For a candidate length, group primitives by their position modulo that length, as in Equation 2, then fit a Gaussian to each group’s point representation.

The average norm of the fitted covariance matrices measures loop consistency. A threshold accepts sufficiently similar cycles; the window expands while the criterion remains satisfied. A maximum body length bounds the search. This is geometric similarity detection, not a learned action classifier.

Equation 2
Si={wj|j%​l=i}for ​0≤i<l.S_{i}=\{w_{j}|j\%l=i\}\ \ \mbox{for\ }0\leq i<l. (2)

Loop synthesis.

Replace the accepted interval by a loop whose iteration count matches the number of grouped primitives. Figure 3 preserves the concrete sequence, intermediate point representation, probabilistic loop body, and sampling example.

4 Experiments

Experiments separately assess trajectory reconstruction, video interpolation, repetition detection, and prediction of repetitive motion. Figure 4 shows how program-supplied poses enter the interpolation pipeline.

Refer to caption
Figure 4: Evaluation of three video interpolation variants: Top row (SepConv [23]), middle row (SepConv w/ heatmaps) and bottom row (SepConv w/ programs) which is our model where poses for intermediate frames are supplied by motion programs. Middle row (SepConv w/ heatmaps) is evaluated as an ablation study.

4.1 Motion Program Induction

Data.

GolfDB provides 242 face-on, real-time golf-swing videos cropped from swing start to end. The experiment uses an equal train/test split and frames scaled to 512 × 512.

Analysis.

On the training set used for the representation analysis, an average 58-frame clip is described by 3.33 primitives and 253 parameters, compared with 1,623 parameters for the pose sequence. The fitted keypoints differ by 6.25 pixels on average from the reference/input poses.

The maximum adjacent-frame joint displacement decreases from 100.10 to 76.15 pixels, consistent with smoothing some tracking noise. Qualitative segments often resemble backswing, downswing, and follow-through. These measurements demonstrate compact approximation of detected trajectories, not independently verified three-dimensional motion or semantic segmentation accuracy.

4.2 Video Interpolation

The interpolation experiment adds motion-derived target poses to a learned frame-interpolation network.

Data.

Evaluation uses GolfDB and 105 continuous videos assembled from PoseWarp, resized to 512 × 512. PoseWarp retains its original training split.

Setup.

Three trained variants compare ordinary SepConv, SepConv with endpoint-pose heatmaps, and SepConv with endpoint and target poses. At test time, Motion Programs provide the target poses by executing fitted primitives at finer temporal resolution.

The described evaluation generates programs on test clips, then subsamples frames within their primitives at factors of two, four, or eight. The proposed system consequently uses clip-level pose structure beyond the two endpoint images; this is relevant when interpreting comparisons with ordinary two-frame interpolation. The paper does not fully spell out the information boundary between program fitting and frame subsampling.

Original figure
Figure 5: Human evaluation of video interpolation across different rates on both datasets. Incorporating motion programs significantly improves upon the baseline for 8x interpolation.
Refer to caption
Figure 6: Qualitative results for 8x video interpolation. We observe that using program information results in uniformly higher image quality.
Table 3: Comparison of metrics for video interpolation of 2x / 4x / 8x subsampling on the GolfDB and PoseWarp test sets. Metrics include keypoint difference in left/right elbow and wrist (lower is better), PSNR, SSIM (higher is better), and LPIPS (lower is better).
Method Dataset Keypoint Difference↓\downarrow Perceptual Metrics
l-elbow l-wrist r-elbow r-wrist average PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow
SepConv [23] PoseWarp 2x 2.556 3.606 2.330 3.124 1.791 32.285 0.967 0.023
SepConv w/ heatmaps 2.357 3.490 2.267 2.910 1.788 32.246 0.968 0.025
SepConv w/ programs 2.238 2.982 1.965 2.552 1.650 32.699 0.964 0.021
SepConv [23] GolfDB 2x 3.263 3.348 3.077 3.048 2.308 34.860 0.981 0.017
SepConv w/ heatmaps 3.395 3.289 3.211 2.935 2.355 35.046 0.982 0.017
SepConv w/ programs 3.210 3.263 3.325 2.900 2.432 35.597 0.982 0.015
SepConv [23] PoseWarp 4x 3.319 4.625 2.926 3.933 2.151 30.834 0.955 0.034
SepConv w/ heatmaps 3.011 4.305 2.708 3.699 2.088 30.82 0.955 0.036
SepConv w/ programs 2.788 3.441 2.495 3.068 1.938 30.344 0.951 0.032
SepConv [23] GolfDB 4x 4.987 5.791 4.620 4.797 3.598 30.488 0.958 0.035
SepConv w/ heatmaps 4.927 5.351 4.670 4.783 3.532 30.606 0.959 0.034
SepConv w/ programs 4.466 4.704 4.085 4.093 3.091 30.990 0.959 0.031
SepConv [23] PoseWarp 8x 5.520 9.349 4.132 7.527 3.275 27.659 0.923 0.059
SepConv w/ heatmaps 5.367 9.214 4.672 8.425 3.411 27.666 0.923 0.06
SepConv w/ programs 4.371 6.611 3.955 6.475 2.840 27.587 0.92 0.054
SepConv [23] GolfDB 8x 10.762 15.852 8.340 14.075 6.900 25.672 0.908 0.071
SepConv w/ heatmaps 10.914 16.135 8.839 15.484 7.317 25.918 0.913 0.068
SepConv w/ programs 8.496 11.134 7.134 9.929 5.669 26.563 0.914 0.062

Metrics.

Image quality is assessed with PSNR, SSIM, and LPIPS. Keypoint difference measures distances between poses detected in generated and reference frames, so it also depends on the pose detector. A human study asks which video looks more realistic: thirty participants each judge thirty random test videos for each task.

Results.

Table 3 preserves every rate, dataset, joint error, and image metric. Benefits are clearest at eightfold interpolation: average keypoint difference falls from 6.900 to 5.669 on GolfDB and from 3.275 to 2.840 on PoseWarp. Human preference also favors the program-assisted model more strongly at this rate.

The improvement is not uniform across metrics: PoseWarp’s PSNR and SSIM are slightly worse at eightfold interpolation, and GolfDB’s average keypoint error is worse at twofold interpolation. Figures 5 and 6 retain the preference and qualitative comparisons, including hand details and difference maps.

4.3 Extracting Repetitive Segments

Loop detection is evaluated against manually annotated repetitive intervals in long exercise videos.

Data.

The dataset contains three YouTube exercise videos: two cardio routines and one Tai Chi routine, each longer than fifteen minutes. This is a small, manually selected sample rather than a broad action benchmark.

Results.

A detected interval counts as a match when intersection-over-minimum exceeds 0.5. Table 4 reports precision/recall of 88.57/58.06, 69.35/90.47, and 100/67.64% for the three videos. The paper also mentions qualitative examples from the AIST++ dance database. The linked project page containing additional examples was unavailable during this review.

Error analysis.

Small motions can be overwhelmed by keypoint noise, reducing recall for jogging-like actions. Occlusion produces poor tracks, and movements largely along the camera’s depth axis are not well represented by two-dimensional coordinates. These failures constrain both primitive fitting and repetition extraction.

Table 4: Comparing the number of detected segments (# det) with ground-truth segments (# gt) on three long exercise videos.
Video ID Duration # det # gt Precision Recall
cardio00 37:10 36 62 88.57 58.06
cardio01 17:51 19 21 69.35 90.47
taichi02 23:22 23 34 100 67.64

4.4 Video Prediction

Executing a detected loop for more or fewer iterations supports extending or shortening a repetitive segment. Indefinite program execution is possible, but continued repetition is an assumed future pattern rather than a guarantee of what a person will do.

Data.

Detected loops from each long video form a dataset with an 80/20 train/test split. This evaluates future motion within selected repetitive segments, not prediction of arbitrary action changes.

Setup.

HierchVid and HistryRep are trained on the loop clips. At test time, the first half of a clip supplies context and the second half supplies reference future motion. The program method infers a loop from the first half and continues it. All methods use Everybody Dance Now to turn predicted poses into frames, isolating pose dynamics under a common generator.

Original figure
Original figure
Original figure
Original figure
Figure 7: Comparison of HierchVid [29] (blue), HistryRep [32] (orange) and ours (green) on video prediction. Top row: KD↓\downarrow (keypoint difference), bottom row: LPIPS↓\downarrow on loops extracted from cardio00 and cardio01.
Original figure
Figure 8: Human evaluation of video prediction across different future frame slices on two datasets. We observe that extrapolating via motion programs leads to more consistent future prediction of repetitive segments

Metrics.

The authors report keypoint and image metrics averaged over ten runs. Human evaluation asks which generated video resembles the reference more closely: ten participants per dataset judge all test videos in three successive 300-frame intervals. This preference criterion differs from the realism question used for interpolation.

Results.

Figures 7 and 8 show results for the two cardio datasets. As prediction extends farther into repetitive segments, program continuation generally maintains the pattern better than the sequence-model baselines, whose outputs drift. HierchVid remains competitive early on one dataset. These results support the repetition prior in the selected setting, not general long-horizon prediction outside detected loops.

5 Discussion

Geometric primitives and probabilistic loops offer a compact, editable representation between raw tracking and full semantic action descriptions. The benefits are clearest when motion is well tracked, approximately piecewise geometric, and repetitive. Limitations include two-dimensional pose errors, limited coupling between joints, small evaluation sets, and the assumption that detected patterns continue. The authors also call for responsible use of generated video.

Acknowledgements.

The work received support from the Brown Institute’s Magic Grant, Samsung’s Global Research Outreach program, Autodesk, Amazon Web Services, and Stanford HAI cloud credits.

Concept demonstration

Open image ↗