Literature
Display
← All papers

Programmatic Concept Learning for Human Motion Description and Synthesis

Sumith Kulal · Jiayuan Mao · Alex Aiken · Jiajun Wu

· CVPR 2022 ↗

Computer Vision7 min text · CondensedOriginal paper ↗
Contents

Overview

At a glance

Programmatic Motion Concepts represents human actions as named groups of spline-based joint trajectories. A recognition model segments repeated exercise motions, and a generative model learns distributions over the resulting spline parameters. The same representation supports description, editing, and synthesis with limited interval annotation. Experiments favor it over the selected baselines on six exercise classes, while relying on reliable two-dimensional pose estimates and repetitive motion.

1 Introduction

Recognition and generation can share an interpretable motion representation. Here each action consists of a small sequence of parametric joint trajectories, grouped under a concept such as a jumping jack. Sparse annotations mark a repeated-action interval and three individual repetitions; learned alignment expands those examples into training instances. Users can edit primitive parameters or the concept sequence, then generate poses and render video through a separate neural model.

Refer to caption
Figure 1: (I) We present a hierarchical human motion description framework. Each video is represented as a sequence of motion concepts and each motion concept is further grounded as a sequence of motion primitives. (II) Motion concepts can be learned from very small amount of human annotation: human annotators label a repetition range of a motion concept and just three instance ranges with in this group of repetitions. (III) Motion concept supports interactive editing and video synthesis. Human editors can flexibly edit a human motion video at both the concept level or the primitive level. We use neural generative models to render the output video.

2 Related Work

Action recognition and localization.

Related action-recognition and localization systems often learn directly from many frames; weakly supervised alternatives infer alignment from sequence labels. PMC adapts a CTC-style objective to compact motion primitives. For synthesis, it fits and samples action-specific curve distributions rather than learning a single sequence generator over poses. Its program-like representation also follows earlier work on interpretable visual primitives and concept learning. The comparisons adapt existing motion generators to the paper’s two-dimensional setting.

3 Motion Concept Learning Framework

The hierarchy has three levels: parametric joint-trajectory primitives, named motion concepts composed of primitives, and programs that order the concepts. This structure supports edits at different scales, from changing a curve to inserting or removing an action.

3.1 Primitive-based Motion Representation

Earlier motion programs approximate a joint trajectory using stationary, linear, or circular segments. Dynamic programming chooses segment boundaries by balancing fit error against a per-primitive penalty. PMC retains that segmentation strategy but expands the primitive family to cubic splines.

Refer to caption
Figure 2: Four detected motion primitives in a High Knees clip. The third order coefficient is small in all four primitives (absolute value is smaller than 0.010.01) so we omitted them in the visualization.
Original figure
Figure 3: Refined spline primitives can better capture the right-wrist motion in the Jumping Jacks class compared to circular primitives.

3.2 Refined Spline-based Primitives

A spline primitive has cubic x(t) and y(t) coordinates plus a duration. This compact form can represent joint paths that do not fit a line or circle well. The retained examples compare right-wrist motion in jumping jacks and show individual high-knee primitives.

Equation 1
spline=[X=ax​t3+bx​t2+cx​t+dY=ay​t3+by​t2+cy​t+dT=t].\displaystyle\mathrm{spline}=\begin{bmatrix}X=a_{x}t^{3}+b_{x}t^{2}+c_{x}t+d\\ Y=a_{y}t^{3}+b_{y}t^{2}+c_{y}t+d\\ T=t\end{bmatrix}. (1)
Refer to caption
Figure 4: The model for generating motion descriptions. Given the input video, we first generate motion primitives and feed in the primitives to a simple neural architecture containing feed-forward and recurrent layers to compute concept label predictions for each primitive. We use a connectionist temporal classification-style (CTC) objective to extract the most probable motion description.

3.3 Human Motion Description

A pretrained pose estimator extracts two-dimensional joint tracks, which are segmented into K primitives. One shared MLP encodes each primitive and another encodes each adjacent pair, producing 2K−1 alternating features. A local recurrent window of 13 steps supplies context. Label predictions include a blank token; merging repeated labels and removing blanks produces the concept sequence. Transition features are crucial because consecutive repetitions of the same exercise require intervening blanks. The paper uses a CTC-style alignment objective and dynamic-programming decoding.

3.4 Low-Resource Training of Motion Concepts

Annotation specifies the outer repeated-action interval and three example repetition intervals, requiring eight boundary clicks. Joint-trajectory plots help identify boundaries. The mean annotated duration estimates the total repetition count and creates a pseudo-label sequence. Training combines CTC loss with primitive-class and transition-boundary losses. The two auxiliary losses initially have unit weight, then are disabled because their duration-based pseudo-labels are approximate.

Refer to caption
Figure 5: Learning a generative model of motion concepts. We leverage the learnt alignments to extract all single repetition examples for each action class. By taking few ground-truth repetitions as reference, we filter them to resolve ambiguities and then learn a primitive distribution as our concept representation.

3.5 Human Motion Synthesis

The description model aligns each training sequence to action instances. For each concept, the synthesis stage retains the most common primitive count and filters examples against annotated repetitions. It then fits Gaussian distributions over corresponding spline parameters and samples them to generate new instances. Longer descriptions are composed by sampling concepts in order; the start pose of each segment is manually set to the previous endpoint. This modular construction supports new action sequences, but does not learn arbitrary inter-action transitions.

4 Experiments

Evaluation covers spline reconstruction, concept recognition/localization, single-action generation, and generation from ordered descriptions. Interactive editing is illustrated qualitatively.

4.1 MotiCon Dataset

MotiCon contains six cardio-exercise classes from YouTube videos selected for a stable single camera, 1920×1080 resolution, and 30 frames per second. The complete class counts, durations, and repetition counts are retained. Sparse annotation and repetitive structure reduce labeling effort; the narrow acquisition conditions constrain conclusions about broader human motion.

Table 1: MotiCon dataset statistics. For each class, we list # vids (total number of videos), # mins (total number of minutes) and # reps (total number of repetitions).
Class # vids # mins # reps
Arm Cross Swings 12 6.68 127
Buttkickers 9 5.94 425
High Knee March 8 3.70 81
Jumping Jack 23 13.03 853
Toe Touch Sweeps 11 5.00 35
Torso Twists 8 3.90 79

4.2 Evaluation of Spline Primitives

On GolfDB and MotiCon, cubic splines reduce average per-joint reconstruction error by about 57% relative to the earlier motion-program primitives and fit faster than circles. The tradeoff is representation size: spline models sometimes require 1.5–2 times as many parameters. The table retains both error and runtime results.

Table 2: Evaluating expressive power of motion primitives. On two datasets, we compute KD (keypoint difference w/ ground-truth), and Time (synthesis time in seconds).
Method GolfDB[33] MotiCon
KD↓\downarrow Time↓\downarrow KD↓\downarrow Time↓\downarrow
Motion Programs[23] 1.171% 48.82 0.575% 57.65
PMC (Ours) 0.529% 3.22 0.229% 15.92
Ground-Truth 0% ✗ 0% ✗

4.3 Human Motion Description

Description evaluation reports normalized sequence edit distance, its derived sequence-accuracy score, and repetition-localization mAP across IoU thresholds 0.50–0.95. CTC models outperform the tested sequence-to-sequence models on recognition. Primitive inputs and both auxiliary losses improve localization substantially. The LSTM primitive model reaches 45.43% localization mAP and the GRU version 38.74%, so high sequence accuracy should not be mistaken for near-perfect temporal boundaries. Results average three runs.

Table 3: Evaluating human motion description. We compute the NormED (normalized edit distance) and SeqAcc (sequence accuracy) for recognition and mAP (repetition mAP) for localization.
Method NormED↓\downarrow SeqAcc↑\uparrow mAP↑\uparrow
seq2seq-LSTM-Pose 0.64840 35.16% ✗
seq2seq-GRU-Pose 0.31353 68.65% ✗
seq2seq-LSTM-Prim 0.60810 39.19% ✗
seq2seq-GRU-Prim 0.59770 40.23% ✗
LSTM-Pose 0.22990 77.01% 0.00%
GRU-Pose 0.18057 81.94% 0.00%
GRU-Prim w/o ℒP\mathcal{L}_{P} & ℒT\mathcal{L}_{T} 0.11072 88.93% 0.10%
GRU-Prim w/o ℒP\mathcal{L}_{P} 0.10303 89.70% 20.58%
GRU-Prim w/o ℒT\mathcal{L}_{T} 0.14763 85.24% 17.87%
LSTM-Prim (Ours) 0.08831 91.17% 45.43%
GRU-Prim (Ours) 0.08469 91.53% 38.74%
Refer to caption
Figure 6: Visualising the localized “torso twists” concept in a video. Different colors represent different segments of the localized by models. Even if they all predict the correct number of repetitions of “torso twists”, models using our primitive-based representations and ℒP+ℒT\mathcal{L}_{\text{P}}+\mathcal{L}_{\text{T}} finds the segments that maximally align with the ground-truth.

4.4 Action-Conditioned Motion Synthesis

Action-conditioned synthesis compares against two-dimensional Action2Motion variants using the same inferred repetition segments. A learned action classifier supplies recognition and feature-space distribution metrics. PMC obtains lower FID and higher action recognition while producing less within-class variation than the baseline under its conservative covariance setting. Increasing variance trades visual quality and recognizability for diversity.

Table 4: Evaluating action-conditioned motion synthesis. We compare the FID score, Acc (recognition accuracy), Div (variance across action classes) and MM (variance within action classes).
Method FID↓\downarrow Acc↑\uparrow Div→\rightarrow MM→\rightarrow
Real motions 0.220 98.72% 8.363 1.891
Act2Mot-2D [3] 23.465 84.39% 7.748 2.001
Act2Mot-2D-Prim 16.971 83.49% 7.872 2.794
PMC (Ours) 2.406 97.48% 8.182 1.141

4.5 Controlled Motion and Video Synthesis

For controlled synthesis, concept sequences generate poses, then two separately trained skeleton-to-human GANs render video. Comparisons include description-to-pose baselines and stronger versions predicting spline primitives. PMC improves the reported position, variance, and video reconstruction metrics in this setting. These comparisons use the dataset’s action inventory and rendering models; they do not establish open-ended text-to-video generation.

Table 5: Evaluating controlled motion synthesis. We compare the APE (average position error) and AVE (average variance error).
  Method   APE↓\downarrow   AVE↓\downarrow
  Lin et al. [4]   0.2470   0.0457
  JL2P [1]   0.2504   0.0440
  Lin et al.-Prim   0.2075   0.0359
  JL2P-Prim   0.2052   0.0358
  PMC (Ours)   0.1531   0.0174
Table 6: Evaluating controlled video synthesis. We compare the PSNR, SSIM, LPIPS and MSE metrics for the generated videos.
Method PSNR↑\uparrow SSIM↑\uparrow LPIPS↓\downarrow MSE↓\downarrow
Lin et al. [4] 17.791 0.925 0.112 1230.732
JL2P [1] 17.898 0.926 0.109 1192.657
Lin et al.-Prim 18.574 0.930 0.094 1067.636
JL2P-Prim 18.637 0.931 0.093 1067.194
PMC (Ours) 19.355 0.934 0.085 916.898

5 Conclusion

The shared primitive/concept hierarchy supports recognition, localization, synthesis, and editing with sparse interval annotations. Its main evidence comes from controlled exercise videos, and the supplement explains dependence on pose quality, repetitive structure, and the chosen variance/accuracy tradeoff.

Acknowledgements.

The authors acknowledge the listed university, industry, and aging-research support, including the SAGE Center and NIH/NIA grant.

Programmatic Concept Learning for Human Motion Description and Synthesis Supplementary Materials

1 Details of Data Filtering and Model Fitting

Predicted alignments provide many candidate repetitions but may be noisy. Two filtering steps build a cleaner, fixed-length sample set for each concept before fitting its generator.

Length Filtering.

Only instances with the modal number of spline primitives are retained. This simplifies fitting to corresponding primitive positions, but discards alternative action decompositions rather than modeling several lengths.

Similarity Filtering.

Each candidate is compared with the annotated repetitions using average L2 distance between four equally spaced samples along each corresponding spline. Candidates whose closest reference exceeds threshold 8 are removed.

Model Fitting.

A separate Gaussian is fitted at each primitive position from the filtered parameter samples. The covariance is multiplied by a user-controlled factor, set to 0.01 in the experiments. Sampling the distributions in order generates a new instance; the complete parameter and covariance expressions are retained.

Equation
Gcℓ∼𝒩⁡(μ⁡(tl),cov_f⋅σ​(tl)2)G^{\ell}_{c}\sim\mathcal{N}(\mu(t^{l}),\,\textit{cov\_f}\cdot\sigma(t^{l})^{2})

2 Inference with Dynamic Programming

CTC-style decoding groups alignments that collapse to the same label sequence and accumulates their probabilities through prefix states. The supplement describes dynamic programming as the practical alternative to enumerating every alignment. Transition-label predictions distinguish repeated occurrences of the same action.

3 Ablation of Covariance Factor

The covariance sweep retains all eight plot panels. Higher covariance increases diversity and multimodality, but produces visibly poorer motion even when classifier accuracy decreases only slightly. The chosen low covariance favors quality over within-action variability.

Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Original figure
Figure 1: Ablating the covariance factor for evaluating action-conditioned motion synthesis. We compare the FID score, Acc (recognition accuracy), Div (variance across action classes) and MM (variance within action classes).

4 Ablation of Window Size

Temporal-window ablations show gains that largely plateau around nine steps. Jumping jacks are recognized well over a broad range, whereas torso twists benefit more from extra context. The default model uses a 13-step window.

Original figure
Original figure
Figure 2: Ablating the window size for evaluating human motion description (SeqAcc). Left: ablation across all classes. Right: performance breakdown for two classes – Jumping Jacks (JJ, easy) and Torso Twists (TT, hard).

5 Details of the Evaluation Metrics

All synthesis metrics use established comparisons, but each captures only a specific aspect of generated motion or reconstruction.

5.1 Action-Conditioned Motion Synthesis

An RNN action classifier trained on annotated single repetitions supplies labels and feature vectors. Synthesis metrics are averaged over 20 independent runs.

FID score:

FID compares feature distributions from 1,000 generated and 1,000 real test motions, sampling the real set with replacement. It evaluates similarity in this learned motion-feature space rather than directly measuring physical plausibility.

Accuracy (Acc):

Recognition accuracy asks whether the classifier assigns 1,000 generated examples to their requested action classes. It depends on the trained classifier’s decision boundaries.

Diversity (Div):

Diversity is the mean feature-space distance between paired samples from two generated sets of 200 motions, mixing classes. The original formula is retained; matching real-motion diversity is the intended comparison.

Equation
Div=1200​∑n=1200‖un−vn‖2.\textnormal{Div}=\frac{1}{200}\sum_{n=1}^{200}||u_{n}-v_{n}||_{2}.

Multimodality (MM):

Multimodality applies the analogous paired-distance computation within each concept, using two sets of 20 samples per class. It distinguishes within-action variability from variation caused merely by switching action labels.

Equation
MM=1C×20​∑c=1C∑n=120‖uc,n−vc,n‖2\textnormal{MM}=\frac{1}{C\times 20}\sum_{c=1}^{C}\sum_{n=1}^{20}||u_{c,n}-v_{c,n}||_{2}

5.2 Controlled Motion Synthesis

Controlled-motion evaluation compares generated and reference joint trajectories using position and variance measures.

Average Positional Error (APE):

Average Positional Error averages L2 keypoint distance over joints and time. Dynamic Time Warping aligns generated and reference sequences of different lengths before comparison, so the metric permits temporal realignment.

Equation
APE=1J×T​∑t=1T‖pt−p¯t‖2.\textnormal{APE}=\frac{1}{J\times T}\sum_{t=1}^{T}\|p_{t}-\bar{p}_{t}\|_{2}.

Average Variance Error (AVE):

Average Variance Error compares each joint’s temporal positional variance with the reference variance, then aggregates over joints. The retained equations define the temporal mean, variance, and final error.

Equations
σ⁡(j)=1T−1​∑t=1T‖ptj−μpj‖2,\sigma(j)=\frac{1}{T-1}\sum_{t=1}^{T}\|p^{j}_{t}-\mu_{p}^{j}\|_{2},
AVE=1J​∑j=1J‖σ⁡(j)−σ¯​(j)‖2.\textnormal{AVE}=\frac{1}{J}\sum_{j=1}^{J}\|\sigma(j)-\bar{\sigma}(j)\|_{2}.

6 Limitations and Societal Impact

Occlusions and noisy pose detections can break primitive extraction. Transfer to other moving entities or less repetitive activities is unestablished. The authors discuss potential uses in training and rehabilitation, along with misuse risks from visual-content generation, and propose usage restrictions in their release.

Concept demonstration

Open image ↗