At a glance
Motion Programs represent human movement at three levels: tracked two-dimensional keypoints, sequences of geometric motion primitives, and probabilistic loops over repeated primitives. Stationary, linear, and circular segments provide a compact description, while repeated start-middle-end trajectories capture variation across cycles.
The representation improves some sparse video-interpolation and repetitive-motion prediction results when combined with existing image generators. Evaluation uses golf, pose-warping clips, and three long exercise videos. These are explicit geometric descriptions of tracked movement, not general semantic action understanding; their success depends on pose quality and on the motion fitting the chosen primitives or repetition pattern.
1 Introduction
Keypoint sequences capture local motion but do not directly express a backswing, a repeated squat, or a loop of jumping jacks. The proposed hierarchy groups trajectories into longer primitives and then groups recurring primitives into loops. Figure 1 illustrates induction, editing, and execution.
A concrete domain-specific language (DSL) specifies geometry, timing, and duration. An abstract language represents distributions over primitives and for-loops. Synthesizing these programs allows denoising, interpolation within a primitive, repetition detection, and continued execution of a recurring pattern.
2 Related Work
The work connects procedural representations of visual data with video synthesis and motion modeling.
Program-like representations for visual data.
Procedural graphics and program-like image representations describe complex visuals with reusable structure. Earlier motion systems also use parameterized curves and recombinable segments. Here, programs are inferred from keypoint tracks extracted from video, with probabilistic loops adding a level for repeated temporal structure.
Video synthesis.
Neural interpolation and prediction methods synthesize frames using learned features, optical flow, blending kernels, or pose dynamics. Motion Programs supply higher-level pose structure to those generators rather than replacing image synthesis. The interpolation experiments explicitly combine the representation with SepConv.
3 Hierarchical Motion Understanding
The three levels trade direct observation for compact structure. Concrete programs are fitted to trajectories; abstract programs are inferred from similarities between consecutive concrete primitives.
Level 1—Keypoints.
Off-the-shelf tools such as AlphaPose or OpenPose recover one two-dimensional trajectory per body joint. These tracks are the input to program synthesis, so occlusions and detector errors propagate into the representation.
Level 2—Concrete Motion Programs.
A concrete primitive is stationary, linear, or circular motion with explicit spatial and temporal parameters. A program executes a sequence of these primitives for their specified durations. Table 1 retains the full grammar, and Figure 2 shows golf and squat decompositions. The vocabulary is intentionally small; richer motion could require additional primitives such as splines.
Level 3—Abstract Motion Programs.
An abstract primitive records start, middle, and end points plus duration, with probability distributions for repeated variants. This common representation can group instances whose best concrete types differ. Execution samples a trajectory description, chooses the concrete primitive that fits it best, and executes it.
Table 2 retains the abstract grammar. A for-loop repeats a body of such distributions, capturing approximate recurrence rather than requiring identical cycles. Three points distinguish curved from straight motion in this representation; they do not describe arbitrary articulated dynamics.
| Program | Prim; …; Prim | |
| Prim | circle (center: Point, radius: R, | |
| angle=(vel: R, start: R), time: Int) | ||
| Prim | line (x=(vel: R, start: R), | |
| y=(slope: R, intercept: R), time: Int) | ||
| Prim | stationary(point: Point, time: Int) | |
| Point | (x: R, y: R) | |
| Int | a positive integer | |
| R | a real number |
| Program | Stmt; …; Stmt | |
| Stmt | DetPrim For-Loop (iter: Int) { | |
| ProbPrim; …; ProbPrim } | ||
| DetPrim | {<start, middle, end>: | |
| <Point, Point, Point>, time: Int} | ||
| ProbPrim | {<start, middle, end>: | |
| Gaussian(<Point, Point, Point>), | ||
| ), time: FreqDist(Int)} | ||
| Point | (x: R, y: R) | |
| Int | a positive integer | |
| R | a real number |
3.1 Concrete Motion Programs
Concrete program induction first fits individual candidate primitives, then chooses temporal boundaries across the observed sequence.
Synthesizing single primitives.
For a circular candidate, fit a circle to the tracked positions, project the points onto it, and estimate temporal parameters. Linear and stationary candidates are fitted similarly. Select the type with the smallest reconstruction error. The authors report that separating geometric and temporal fitting was faster than joint optimization with comparable results.
Segmentation and synthesis.
Dynamic programming finds a segmentation minimizing summed fit errors plus a penalty per primitive, as specified in Equation 1. A small penalty allows many short segments; a large one favors a single coarse segment. The implementation adapts the penalty to the local covariance of keypoints, making granularity depend on motion scale.
Equation 1
| (1) |
Multiple keypoints.
For several joints, sum fitting errors across trajectories and use one shared segmentation so all joints change primitives simultaneously. Other than this boundary selection, the programs are constructed per keypoint. The method therefore does not impose a full-body kinematic or physical model.
3.2 Abstract Motion Programs
Abstract synthesis converts concrete segments to a common representation, searches for repeated groups, and replaces accepted groups with probabilistic loops.
Concrete to abstract translation.
Replace each fitted concrete primitive with its start, middle, and end positions and duration. This deterministic intermediate form allows comparison across different concrete primitive types.
Loop detection.
Slide a window over the primitive sequence and try increasing loop-body lengths. For a candidate length, group primitives by their position modulo that length, as in Equation 2, then fit a Gaussian to each group’s point representation.
The average norm of the fitted covariance matrices measures loop consistency. A threshold accepts sufficiently similar cycles; the window expands while the criterion remains satisfied. A maximum body length bounds the search. This is geometric similarity detection, not a learned action classifier.
Equation 2
| (2) |
Loop synthesis.
Replace the accepted interval by a loop whose iteration count matches the number of grouped primitives. Figure 3 preserves the concrete sequence, intermediate point representation, probabilistic loop body, and sampling example.
4 Experiments
Experiments separately assess trajectory reconstruction, video interpolation, repetition detection, and prediction of repetitive motion. Figure 4 shows how program-supplied poses enter the interpolation pipeline.
4.1 Motion Program Induction
Data.
GolfDB provides 242 face-on, real-time golf-swing videos cropped from swing start to end. The experiment uses an equal train/test split and frames scaled to 512 × 512.
Analysis.
On the training set used for the representation analysis, an average 58-frame clip is described by 3.33 primitives and 253 parameters, compared with 1,623 parameters for the pose sequence. The fitted keypoints differ by 6.25 pixels on average from the reference/input poses.
The maximum adjacent-frame joint displacement decreases from 100.10 to 76.15 pixels, consistent with smoothing some tracking noise. Qualitative segments often resemble backswing, downswing, and follow-through. These measurements demonstrate compact approximation of detected trajectories, not independently verified three-dimensional motion or semantic segmentation accuracy.
4.2 Video Interpolation
The interpolation experiment adds motion-derived target poses to a learned frame-interpolation network.
Data.
Evaluation uses GolfDB and 105 continuous videos assembled from PoseWarp, resized to 512 × 512. PoseWarp retains its original training split.
Setup.
Three trained variants compare ordinary SepConv, SepConv with endpoint-pose heatmaps, and SepConv with endpoint and target poses. At test time, Motion Programs provide the target poses by executing fitted primitives at finer temporal resolution.
The described evaluation generates programs on test clips, then subsamples frames within their primitives at factors of two, four, or eight. The proposed system consequently uses clip-level pose structure beyond the two endpoint images; this is relevant when interpreting comparisons with ordinary two-frame interpolation. The paper does not fully spell out the information boundary between program fitting and frame subsampling.
| Method | Dataset | Keypoint Difference | Perceptual Metrics | ||||||
| l-elbow | l-wrist | r-elbow | r-wrist | average | PSNR | SSIM | LPIPS | ||
| SepConv [23] | PoseWarp 2x | 2.556 | 3.606 | 2.330 | 3.124 | 1.791 | 32.285 | 0.967 | 0.023 |
| SepConv w/ heatmaps | 2.357 | 3.490 | 2.267 | 2.910 | 1.788 | 32.246 | 0.968 | 0.025 | |
| SepConv w/ programs | 2.238 | 2.982 | 1.965 | 2.552 | 1.650 | 32.699 | 0.964 | 0.021 | |
| SepConv [23] | GolfDB 2x | 3.263 | 3.348 | 3.077 | 3.048 | 2.308 | 34.860 | 0.981 | 0.017 |
| SepConv w/ heatmaps | 3.395 | 3.289 | 3.211 | 2.935 | 2.355 | 35.046 | 0.982 | 0.017 | |
| SepConv w/ programs | 3.210 | 3.263 | 3.325 | 2.900 | 2.432 | 35.597 | 0.982 | 0.015 | |
| SepConv [23] | PoseWarp 4x | 3.319 | 4.625 | 2.926 | 3.933 | 2.151 | 30.834 | 0.955 | 0.034 |
| SepConv w/ heatmaps | 3.011 | 4.305 | 2.708 | 3.699 | 2.088 | 30.82 | 0.955 | 0.036 | |
| SepConv w/ programs | 2.788 | 3.441 | 2.495 | 3.068 | 1.938 | 30.344 | 0.951 | 0.032 | |
| SepConv [23] | GolfDB 4x | 4.987 | 5.791 | 4.620 | 4.797 | 3.598 | 30.488 | 0.958 | 0.035 |
| SepConv w/ heatmaps | 4.927 | 5.351 | 4.670 | 4.783 | 3.532 | 30.606 | 0.959 | 0.034 | |
| SepConv w/ programs | 4.466 | 4.704 | 4.085 | 4.093 | 3.091 | 30.990 | 0.959 | 0.031 | |
| SepConv [23] | PoseWarp 8x | 5.520 | 9.349 | 4.132 | 7.527 | 3.275 | 27.659 | 0.923 | 0.059 |
| SepConv w/ heatmaps | 5.367 | 9.214 | 4.672 | 8.425 | 3.411 | 27.666 | 0.923 | 0.06 | |
| SepConv w/ programs | 4.371 | 6.611 | 3.955 | 6.475 | 2.840 | 27.587 | 0.92 | 0.054 | |
| SepConv [23] | GolfDB 8x | 10.762 | 15.852 | 8.340 | 14.075 | 6.900 | 25.672 | 0.908 | 0.071 |
| SepConv w/ heatmaps | 10.914 | 16.135 | 8.839 | 15.484 | 7.317 | 25.918 | 0.913 | 0.068 | |
| SepConv w/ programs | 8.496 | 11.134 | 7.134 | 9.929 | 5.669 | 26.563 | 0.914 | 0.062 | |
Metrics.
Image quality is assessed with PSNR, SSIM, and LPIPS. Keypoint difference measures distances between poses detected in generated and reference frames, so it also depends on the pose detector. A human study asks which video looks more realistic: thirty participants each judge thirty random test videos for each task.
Results.
Table 3 preserves every rate, dataset, joint error, and image metric. Benefits are clearest at eightfold interpolation: average keypoint difference falls from 6.900 to 5.669 on GolfDB and from 3.275 to 2.840 on PoseWarp. Human preference also favors the program-assisted model more strongly at this rate.
The improvement is not uniform across metrics: PoseWarp’s PSNR and SSIM are slightly worse at eightfold interpolation, and GolfDB’s average keypoint error is worse at twofold interpolation. Figures 5 and 6 retain the preference and qualitative comparisons, including hand details and difference maps.
4.3 Extracting Repetitive Segments
Loop detection is evaluated against manually annotated repetitive intervals in long exercise videos.
Data.
The dataset contains three YouTube exercise videos: two cardio routines and one Tai Chi routine, each longer than fifteen minutes. This is a small, manually selected sample rather than a broad action benchmark.
Results.
A detected interval counts as a match when intersection-over-minimum exceeds 0.5. Table 4 reports precision/recall of 88.57/58.06, 69.35/90.47, and 100/67.64% for the three videos. The paper also mentions qualitative examples from the AIST++ dance database. The linked project page containing additional examples was unavailable during this review.
Error analysis.
Small motions can be overwhelmed by keypoint noise, reducing recall for jogging-like actions. Occlusion produces poor tracks, and movements largely along the camera’s depth axis are not well represented by two-dimensional coordinates. These failures constrain both primitive fitting and repetition extraction.
| Video ID | Duration | # det | # gt | Precision | Recall |
| cardio00 | 37:10 | 36 | 62 | 88.57 | 58.06 |
| cardio01 | 17:51 | 19 | 21 | 69.35 | 90.47 |
| taichi02 | 23:22 | 23 | 34 | 100 | 67.64 |
4.4 Video Prediction
Executing a detected loop for more or fewer iterations supports extending or shortening a repetitive segment. Indefinite program execution is possible, but continued repetition is an assumed future pattern rather than a guarantee of what a person will do.
Data.
Detected loops from each long video form a dataset with an 80/20 train/test split. This evaluates future motion within selected repetitive segments, not prediction of arbitrary action changes.
Setup.
HierchVid and HistryRep are trained on the loop clips. At test time, the first half of a clip supplies context and the second half supplies reference future motion. The program method infers a loop from the first half and continues it. All methods use Everybody Dance Now to turn predicted poses into frames, isolating pose dynamics under a common generator.
Metrics.
The authors report keypoint and image metrics averaged over ten runs. Human evaluation asks which generated video resembles the reference more closely: ten participants per dataset judge all test videos in three successive 300-frame intervals. This preference criterion differs from the realism question used for interpolation.
Results.
Figures 7 and 8 show results for the two cardio datasets. As prediction extends farther into repetitive segments, program continuation generally maintains the pattern better than the sequence-model baselines, whose outputs drift. HierchVid remains competitive early on one dataset. These results support the repetition prior in the selected setting, not general long-horizon prediction outside detected loops.
5 Discussion
Geometric primitives and probabilistic loops offer a compact, editable representation between raw tracking and full semantic action descriptions. The benefits are clearest when motion is well tracked, approximately piecewise geometric, and repetitive. Limitations include two-dimensional pose errors, limited coupling between joints, small evaluation sets, and the assumption that detected patterns continue. The authors also call for responsible use of generated video.
Acknowledgements.
The work received support from the Brown Institute’s Magic Grant, Samsung’s Global Research Outreach program, Autodesk, Amazon Web Services, and Stanford HAI cloud credits.