Overview
At a glance
Programmatic Motion Concepts represents human actions as named groups of spline-based joint trajectories. A recognition model segments repeated exercise motions, and a generative model learns distributions over the resulting spline parameters. The same representation supports description, editing, and synthesis with limited interval annotation. Experiments favor it over the selected baselines on six exercise classes, while relying on reliable two-dimensional pose estimates and repetitive motion.
1 Introduction
Recognition and generation can share an interpretable motion representation. Here each action consists of a small sequence of parametric joint trajectories, grouped under a concept such as a jumping jack. Sparse annotations mark a repeated-action interval and three individual repetitions; learned alignment expands those examples into training instances. Users can edit primitive parameters or the concept sequence, then generate poses and render video through a separate neural model.
2 Related Work
Action recognition and localization.
Related action-recognition and localization systems often learn directly from many frames; weakly supervised alternatives infer alignment from sequence labels. PMC adapts a CTC-style objective to compact motion primitives. For synthesis, it fits and samples action-specific curve distributions rather than learning a single sequence generator over poses. Its program-like representation also follows earlier work on interpretable visual primitives and concept learning. The comparisons adapt existing motion generators to the paper’s two-dimensional setting.
3 Motion Concept Learning Framework
The hierarchy has three levels: parametric joint-trajectory primitives, named motion concepts composed of primitives, and programs that order the concepts. This structure supports edits at different scales, from changing a curve to inserting or removing an action.
3.1 Primitive-based Motion Representation
Earlier motion programs approximate a joint trajectory using stationary, linear, or circular segments. Dynamic programming chooses segment boundaries by balancing fit error against a per-primitive penalty. PMC retains that segmentation strategy but expands the primitive family to cubic splines.
3.2 Refined Spline-based Primitives
A spline primitive has cubic x(t) and y(t) coordinates plus a duration. This compact form can represent joint paths that do not fit a line or circle well. The retained examples compare right-wrist motion in jumping jacks and show individual high-knee primitives.
Equation 1
| (1) |
3.3 Human Motion Description
A pretrained pose estimator extracts two-dimensional joint tracks, which are segmented into K primitives. One shared MLP encodes each primitive and another encodes each adjacent pair, producing 2K−1 alternating features. A local recurrent window of 13 steps supplies context. Label predictions include a blank token; merging repeated labels and removing blanks produces the concept sequence. Transition features are crucial because consecutive repetitions of the same exercise require intervening blanks. The paper uses a CTC-style alignment objective and dynamic-programming decoding.
3.4 Low-Resource Training of Motion Concepts
Annotation specifies the outer repeated-action interval and three example repetition intervals, requiring eight boundary clicks. Joint-trajectory plots help identify boundaries. The mean annotated duration estimates the total repetition count and creates a pseudo-label sequence. Training combines CTC loss with primitive-class and transition-boundary losses. The two auxiliary losses initially have unit weight, then are disabled because their duration-based pseudo-labels are approximate.
3.5 Human Motion Synthesis
The description model aligns each training sequence to action instances. For each concept, the synthesis stage retains the most common primitive count and filters examples against annotated repetitions. It then fits Gaussian distributions over corresponding spline parameters and samples them to generate new instances. Longer descriptions are composed by sampling concepts in order; the start pose of each segment is manually set to the previous endpoint. This modular construction supports new action sequences, but does not learn arbitrary inter-action transitions.
4 Experiments
Evaluation covers spline reconstruction, concept recognition/localization, single-action generation, and generation from ordered descriptions. Interactive editing is illustrated qualitatively.
4.1 MotiCon Dataset
MotiCon contains six cardio-exercise classes from YouTube videos selected for a stable single camera, 1920×1080 resolution, and 30 frames per second. The complete class counts, durations, and repetition counts are retained. Sparse annotation and repetitive structure reduce labeling effort; the narrow acquisition conditions constrain conclusions about broader human motion.
| Class | # vids | # mins | # reps |
| Arm Cross Swings | 12 | 6.68 | 127 |
| Buttkickers | 9 | 5.94 | 425 |
| High Knee March | 8 | 3.70 | 81 |
| Jumping Jack | 23 | 13.03 | 853 |
| Toe Touch Sweeps | 11 | 5.00 | 35 |
| Torso Twists | 8 | 3.90 | 79 |
4.2 Evaluation of Spline Primitives
On GolfDB and MotiCon, cubic splines reduce average per-joint reconstruction error by about 57% relative to the earlier motion-program primitives and fit faster than circles. The tradeoff is representation size: spline models sometimes require 1.5–2 times as many parameters. The table retains both error and runtime results.
| Method | GolfDB[33] | MotiCon | ||
| KD | Time | KD | Time | |
| Motion Programs[23] | 1.171% | 48.82 | 0.575% | 57.65 |
| PMC (Ours) | 0.529% | 3.22 | 0.229% | 15.92 |
| Ground-Truth | 0% | ✗ | 0% | ✗ |
4.3 Human Motion Description
Description evaluation reports normalized sequence edit distance, its derived sequence-accuracy score, and repetition-localization mAP across IoU thresholds 0.50–0.95. CTC models outperform the tested sequence-to-sequence models on recognition. Primitive inputs and both auxiliary losses improve localization substantially. The LSTM primitive model reaches 45.43% localization mAP and the GRU version 38.74%, so high sequence accuracy should not be mistaken for near-perfect temporal boundaries. Results average three runs.
| Method | NormED | SeqAcc | mAP |
| seq2seq-LSTM-Pose | 0.64840 | 35.16% | ✗ |
| seq2seq-GRU-Pose | 0.31353 | 68.65% | ✗ |
| seq2seq-LSTM-Prim | 0.60810 | 39.19% | ✗ |
| seq2seq-GRU-Prim | 0.59770 | 40.23% | ✗ |
| LSTM-Pose | 0.22990 | 77.01% | 0.00% |
| GRU-Pose | 0.18057 | 81.94% | 0.00% |
| GRU-Prim w/o & | 0.11072 | 88.93% | 0.10% |
| GRU-Prim w/o | 0.10303 | 89.70% | 20.58% |
| GRU-Prim w/o | 0.14763 | 85.24% | 17.87% |
| LSTM-Prim (Ours) | 0.08831 | 91.17% | 45.43% |
| GRU-Prim (Ours) | 0.08469 | 91.53% | 38.74% |
4.4 Action-Conditioned Motion Synthesis
Action-conditioned synthesis compares against two-dimensional Action2Motion variants using the same inferred repetition segments. A learned action classifier supplies recognition and feature-space distribution metrics. PMC obtains lower FID and higher action recognition while producing less within-class variation than the baseline under its conservative covariance setting. Increasing variance trades visual quality and recognizability for diversity.
| Method | FID | Acc | Div | MM |
| Real motions | 0.220 | 98.72% | 8.363 | 1.891 |
| Act2Mot-2D [3] | 23.465 | 84.39% | 7.748 | 2.001 |
| Act2Mot-2D-Prim | 16.971 | 83.49% | 7.872 | 2.794 |
| PMC (Ours) | 2.406 | 97.48% | 8.182 | 1.141 |
4.5 Controlled Motion and Video Synthesis
For controlled synthesis, concept sequences generate poses, then two separately trained skeleton-to-human GANs render video. Comparisons include description-to-pose baselines and stronger versions predicting spline primitives. PMC improves the reported position, variance, and video reconstruction metrics in this setting. These comparisons use the dataset’s action inventory and rendering models; they do not establish open-ended text-to-video generation.
| Method | APE | AVE |
| Lin et al. [4] | 0.2470 | 0.0457 |
| JL2P [1] | 0.2504 | 0.0440 |
| Lin et al.-Prim | 0.2075 | 0.0359 |
| JL2P-Prim | 0.2052 | 0.0358 |
| PMC (Ours) | 0.1531 | 0.0174 |
| Method | PSNR | SSIM | LPIPS | MSE |
| Lin et al. [4] | 17.791 | 0.925 | 0.112 | 1230.732 |
| JL2P [1] | 17.898 | 0.926 | 0.109 | 1192.657 |
| Lin et al.-Prim | 18.574 | 0.930 | 0.094 | 1067.636 |
| JL2P-Prim | 18.637 | 0.931 | 0.093 | 1067.194 |
| PMC (Ours) | 19.355 | 0.934 | 0.085 | 916.898 |
5 Conclusion
The shared primitive/concept hierarchy supports recognition, localization, synthesis, and editing with sparse interval annotations. Its main evidence comes from controlled exercise videos, and the supplement explains dependence on pose quality, repetitive structure, and the chosen variance/accuracy tradeoff.
Acknowledgements.
The authors acknowledge the listed university, industry, and aging-research support, including the SAGE Center and NIH/NIA grant.
Programmatic Concept Learning for Human Motion Description and Synthesis Supplementary Materials
1 Details of Data Filtering and Model Fitting
Predicted alignments provide many candidate repetitions but may be noisy. Two filtering steps build a cleaner, fixed-length sample set for each concept before fitting its generator.
Length Filtering.
Only instances with the modal number of spline primitives are retained. This simplifies fitting to corresponding primitive positions, but discards alternative action decompositions rather than modeling several lengths.
Similarity Filtering.
Each candidate is compared with the annotated repetitions using average L2 distance between four equally spaced samples along each corresponding spline. Candidates whose closest reference exceeds threshold 8 are removed.
Model Fitting.
A separate Gaussian is fitted at each primitive position from the filtered parameter samples. The covariance is multiplied by a user-controlled factor, set to 0.01 in the experiments. Sampling the distributions in order generates a new instance; the complete parameter and covariance expressions are retained.
Equation
2 Inference with Dynamic Programming
CTC-style decoding groups alignments that collapse to the same label sequence and accumulates their probabilities through prefix states. The supplement describes dynamic programming as the practical alternative to enumerating every alignment. Transition-label predictions distinguish repeated occurrences of the same action.
3 Ablation of Covariance Factor
The covariance sweep retains all eight plot panels. Higher covariance increases diversity and multimodality, but produces visibly poorer motion even when classifier accuracy decreases only slightly. The chosen low covariance favors quality over within-action variability.
4 Ablation of Window Size
Temporal-window ablations show gains that largely plateau around nine steps. Jumping jacks are recognized well over a broad range, whereas torso twists benefit more from extra context. The default model uses a 13-step window.
5 Details of the Evaluation Metrics
All synthesis metrics use established comparisons, but each captures only a specific aspect of generated motion or reconstruction.
5.1 Action-Conditioned Motion Synthesis
An RNN action classifier trained on annotated single repetitions supplies labels and feature vectors. Synthesis metrics are averaged over 20 independent runs.
FID score:
FID compares feature distributions from 1,000 generated and 1,000 real test motions, sampling the real set with replacement. It evaluates similarity in this learned motion-feature space rather than directly measuring physical plausibility.
Accuracy (Acc):
Recognition accuracy asks whether the classifier assigns 1,000 generated examples to their requested action classes. It depends on the trained classifier’s decision boundaries.
Diversity (Div):
Diversity is the mean feature-space distance between paired samples from two generated sets of 200 motions, mixing classes. The original formula is retained; matching real-motion diversity is the intended comparison.
Equation
Multimodality (MM):
Multimodality applies the analogous paired-distance computation within each concept, using two sets of 20 samples per class. It distinguishes within-action variability from variation caused merely by switching action labels.
Equation
5.2 Controlled Motion Synthesis
Controlled-motion evaluation compares generated and reference joint trajectories using position and variance measures.
Average Positional Error (APE):
Average Positional Error averages L2 keypoint distance over joints and time. Dynamic Time Warping aligns generated and reference sequences of different lengths before comparison, so the metric permits temporal realignment.
Equation
Average Variance Error (AVE):
Average Variance Error compares each joint’s temporal positional variance with the reference variance, then aggregates over joints. The retained equations define the temporal mean, variance, and final error.
Equations
6 Limitations and Societal Impact
Occlusions and noisy pose detections can break primitive extraction. Transfer to other moving entities or less repetitive activities is unestablished. The authors discuss potential uses in training and rehabilitation, along with misuse risks from visual-content generation, and propose usage restrictions in their release.