Literature
Display
← All papers

The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion

Changan Chen · Juze Zhang · Shrinidhi K. Lakshmikanth · Yusu Fang · Ruizhi Shao · Gordon Wetzstein · Li Fei-Fei · Ehsan Adeli

· CVPR 2025 ↗

Computer vision7 min text · CondensedOriginal paper ↗
Contents

Overview

The original opening montage illustrates the model’s audio, text, and body-part motion interfaces and the four demonstrated task families.

Original Figure 1 framework montage, restored from PDF page 1.
Figure 1: We introduce a language-model-based motion understanding and generation framework that takes in any of the audio/motion/text modalities and outputs the desired target modality. Coupled with our generative pre-training strategy, our model demonstrates competitive performance on an array of tasks, showing promising signs toward unified verbal and non-verbal language of human motions.

At a glance

The Language of Motion extends a pretrained text model with discrete speech and compositional motion tokens. Spatial and temporal motion prediction plus audio–text alignment precede instruction tuning on paired downstream data. On the tested BEATv2 speaker protocol, it improves gesture-distribution distance and diversity over the compared models, while one baseline retains better speech–motion beat correlation. Editable motion and emotion-label output are also demonstrated, with important limits on joint conditioning and evaluation.

1 Introduction

Most motion generators specialize in one input modality and dataset. A shared token interface could transfer motion structure and speech semantics between tasks. This paper combines separate face, hand, upper-body, and lower-body vocabularies with audio and text, then trains a common encoder–decoder language model. The approach still needs modality-specific tokenizers and paired downstream data; its demonstrations do not establish unrestricted any-to-any generation.

2 Related Work

Related work covers speech-conditioned gestures, text-conditioned movement, and multimodal language models.

2.1 Speech-Driven Motion Generation

Speech-driven methods generate listener behavior, gestures, or facial expressions from audio, often for specific speakers. The proposed shared model aims to use additional motion and text resources rather than restrict learning to one speech–motion pairing.

2.2 Text to Motion Generation

Text-to-motion methods use diffusion or token-based generation to produce body movement from descriptions. This work emphasizes expressive rotations, gestures, and facial motion alongside text and audio inputs. Qualitative text-following evidence is reported later, without a comparable quantitative text-to-motion benchmark score.

2.3 Multimodal Language Models

Multimodal language models combine modality-specific representations with a shared token-processing backbone. This model applies that idea to text, speech, and human motion, with principal evaluated outputs being motion sequences and text.

3 Multimodal Language Model for Motion Generation and Understanding

The method comprises body-model representation, modality tokenization, alignment pretraining, instruction-based downstream tuning, and autoregressive token prediction.

Refer to caption
Figure 2: Method overview. We employ modality-specific tokenizers to process various input modalities. Specifically, we train a compositional body motion VQ-VAE to tokenize face, hands, upper body, and lower body motions into discrete tokens, combining these modality-specific vocabularies(audio and text) into a unified multimodal vocabulary. During training, mixed tokens from different modalities are used as input, and the output is generated through an encoder-decoder language model. The mixed tokens are fed into the transformer encoder, while the decoder predicts the probability distribution of the next token in an autoregressive manner at each step.

3.1 Preliminaries

The neutral SMPL-X model with FLAME facial representation describes body shape, joint pose, facial expression, and global translation over time. Separating these parameters supports expressive body-part control but also requires consistent reconstruction into a complete body sequence.

3.2 Tokenization

Four vector-quantized autoencoders encode face, hands, upper body, and lower body separately. Training combines pose, velocity, acceleration, mesh, and codebook-commitment losses. Six-dimensional rotations retain twisting information that the authors argue is underrepresented in the commonly used HumanML3D format.

HuBERT [20] converts 16 kHz audio into fifty discrete tokens per second. Text uses the pretrained T5 vocabulary; motion/audio code vocabularies and modality-boundary tokens extend it. Motion is normally represented at thirty frames per second. The resulting tokens share a language-model interface, while their encoders and decoders remain modality-specific.

Equations 1–2
𝐪t=𝒬⁡(𝐳t):=arg⁡min𝐪k∈Q⁡‖𝐳t−𝐪k‖2\displaystyle\begin{split}\mathbf{q}^{t}=\mathcal{Q}(\mathbf{z}^{t}):=\arg\min_{\mathbf{q}^{k}\in Q}\|\mathbf{z}^{t}-\mathbf{q}^{k}\|^{2}\end{split} (1)
ℒt​o​t​a​l=ℒrec​(𝐠,𝐠^)+ℒvel​(𝐠′,𝐠^′)+ℒacc​(𝐠′′,𝐠^′′)+ℒmrec​(𝐠,𝐠^)+ℒmvel​(𝐠′,𝐠^′)+ℒmacc​(𝐠′′,𝐠^′′)+ℒcomm​(𝐠,𝐪),\displaystyle\begin{split}\mathcal{L}_{total}=&\mathcal{L}_{\text{rec}}(\mathbf{g},\mathbf{\hat{g}})+\mathcal{L}_{\text{vel}}(\mathbf{g^{\prime}},\mathbf{\hat{g}^{\prime}})+\\ &\mathcal{L}_{\text{acc}}(\mathbf{g^{\prime\prime}},\mathbf{\hat{g}^{\prime\prime}})+\mathcal{L}_{\text{mrec}}(\mathbf{g},\mathbf{\hat{g}})+\\ &\mathcal{L}_{\text{mvel}}(\mathbf{g^{\prime}},\mathbf{\hat{g}^{\prime}})+\mathcal{L}_{\text{macc}}(\mathbf{g^{\prime\prime}},\mathbf{\hat{g}^{\prime\prime}})+\\ &\mathcal{L}_{\text{comm}}(\mathbf{g},\mathbf{{q}}),\end{split} (2)

3.3 Pre-training for Modality Alignment

Pretraining predicts one subset of body parts from another, reconstructs temporally masked motion, and translates between audio and text. The gesture experiment uses about sixty hours of motion and one thousand hours of audio–text data. No audio-to-motion pairs are used in this alignment stage, but audio–text alignment is itself paired supervision; calling the entire stage unpaired would be misleading. Correlations among body parts are learned from these datasets, not shown to be universal across cultures or people.

Original figure
Figure 3: Illustration of pre-training. We pre-train our language model by translating one modality to another using paired data.

3.4 Post-training with Instruction Following

Instruction tuning uses paired downstream examples for tasks such as audio-to-motion, text-to-motion, and emotion-conditioned motion. Many prompt templates vary how a task is requested. The reported thousand-plus instruction variants are not a thousand independently benchmarked capabilities. The original task prompts, conditions, and output formats remain alongside the summaries.

3.5 Language Model Training Details

The backbone is a 220-million-parameter Flan-T5-base encoder–decoder derived from T5 [55]. Inputs are limited to 512 tokens, and the decoder predicts output tokens autoregressively. Both alignment and downstream stages update the full model rather than low-rank adapters. Modality tokens identify codebook entries and sequence boundaries.

Equation 3
ℒL​M=−∑k=0Lt−1logpθ(stk|st<k,si),\mathcal{L}_{LM}=-\sum_{k=0}^{L_{t}-1}\log p_{\theta}(s_{t}^{k}|s_{t}^{<k},s_{i}), (3)

4 Experiments

Experiments measure speech-conditioned gestures, alignment/data ablations, editable body-part generation, and textual emotion-label prediction.

4.1 Co-Speech Gesture Generation

The primary protocol trains and tests on BEATv2 speaker 2 and adopts the motion tokenizer/evaluation procedure of EMAGE [43]. This is speaker-specific evaluation with paired fine-tuning, not zero-shot transfer to unseen people.

In the table’s reported units, the proposed method has gesture distance 5.301 versus EMAGE’s 5.512, beat correlation 7.780 versus 7.724, and diversity 15.167 versus 13.06. SynTalker’s beat correlation is higher at 7.971, so the source’s claim to win every metric is too broad. Higher diversity is a movement-variation statistic, not proof of semantic correctness or naturalness.

Removing language pretraining substantially worsens distribution distance to 7.470. Removing multimodal pretraining gives 5.408 in Table 1. These ablations support the usefulness of pretrained structure within the reported setting; different baselines also receive different conditioning signals.

Refer to caption
Figure 4: Qualitative example on co-speech gesture generation. Given a speech, we visualize the ground truth 3D motion accompanying the audio, the motion generated by the baseline EMAGE [43], SynTalker [8] and our method. Our model generates more diverse and expressive motion compared to the baseline, especially when the speaker emphasizes on certain words such as “tired” and “because”.
FGD ↓\downarrow BC↑\uparrow Diversity↑\uparrow Condition Signal
DisCo [40] 9.417 6.439 9.912 audio
CaMN [41] 6.644 6.769 10.86 audio, text, facial
DiffStyleGesture [69] 8.811 7.241 11.49 audio, style
Habibie et al. [19] 9.040 7.716 8.213 audio, text
TalkSHOW [71] 6.209 6.947 13.47 audio
SynTalker [8] 6.413 7.971 12.721 audio, text
EMAGE [43] 5.512 7.724 13.06 audio, text
Ours w/o language pre-training 7.470 6.148 14.162 audio
Ours w/o multimodal pre-training 5.408 7.742 14.418 audio
Ours 5.301 7.780 15.167 audio
Table 1: Co-speech gesture generation results on BEATv2 benchmark. We report FGD ×10−1\times 10^{-1}, BC ×10−1\times 10^{-1}, Diversity, which measures realism, speech-motion synchronization, and diversity respectively. Our model outperforms state-of-the-art methods on this benchmark.

4.2 Effect of Generative Pre-training

Removing audio-to-text, spatial, or temporal objectives worsens the reported full-model result. Table 2 gives 5.501 for its no-pretraining variant, differing from the 5.408 row in Table 1; the source does not reconcile those values. The data-scarcity plot varies available paired post-training examples, down to one thirty-second, and favors the pretrained model over its ablation and EMAGE. The surrounding prose inconsistently calls this pretraining data, while the figure labels it post-training data. These results reduce required paired adaptation data in the tested speaker setting; they do not remove the large pretraining datasets or demonstrate broad unseen-speaker generalization.

FGD ↓\downarrow BC↑\uparrow Diversity↑\uparrow
W/o pre-training 5.501 7.721 14.281
W/o A2T 5.443 7.721 14.499
W/o spatial 6.336 7.381 14.173
W/o temporal 6.800 7.341 13.810
W/o motion 7.776 7.344 14.640
Ours 5.301 7.780 15.165
Table 2: Ablations of pre-training.
Original figure
Figure 5: Generation performance vs. the amount of post-training data. Our model learns a stronger motion prior from pre-training and thus shows much better under data scarcity.

4.3 Unifying Audio-to-Motion and Text-to-Motion for Editable Generation

The editable-generation setup trains motion tokenizers on BEATv2 and AMASS and uses HumanML3D text annotations for instruction tuning. In the actual demonstration, the model is called twice: audio conditions upper-body motion and text conditions lower-body motion, then the outputs are combined. The supplement explicitly leaves simultaneous multi-source prompting to future larger-scale training. Thus the evidence is compositional body-part editing, not a single jointly conditioned full-body generation pass.

Refer to caption
Figure 6: Editable gesture generation. We prompt the language with text and audio information and it outputs motions that are both expressional gesture motion as well as general movement motion.

4.4 Predicting Emotion from Motion

Instruction tuning adds the eight BEATv2 emotion labels. Predicted text is compared with reference labels using language-generation metrics because outputs do not reliably follow a fixed single-label format. The proposed model exceeds random output and MotionGPT [23], but MotionGPT was trained for general captions rather than this supervised emotion task, limiting that comparison.

The table reports 14.71 for BLEU-1, 26.67 under the combined “Rouge Cider” heading, and 16.94 for BERTScore. These are not emotion-classification accuracy percentages. The inconsistent combined metric label is preserved rather than interpreted as two separately reported measurements. Dataset-label prediction from motion does not validate reading a person’s internal feelings or a clinical use.

Bleu@1↑\uparrow Rouge Cider↑\uparrow BertScore↑\uparrow
GT 100 100 99.9
Random 2.45 4.44 0.19
MotionGPT 1.68 10.67 2.31
Ours 14.71 26.67 16.94
Table 3: Motion to emotion. We prompt our model to predict emotion given a motion sequence.
Refer to caption
Figure 7: Qualitative example of emotion prediction.

5 Discussion

Discrete motion tokenization sometimes produces incoherent movement, motivating the authors’ suggestion to investigate continuous representations. Additional limits include speaker-specific quantitative evaluation, qualitative-only edited/text-conditioned generation, and the use of text-similarity metrics for emotion labels. The shared model demonstrates useful transfer and flexibility without establishing uniformly natural or semantically correct motion.

Acknowledgments:

The project acknowledges NIH and UST funding and feedback from colleagues.

7 Supplementary

The supplement adds task templates, the precise editing procedure, emotion-metric details, qualitative text-to-motion comparisons, implementation settings, and another gesture montage.

7.1 Supplementary Video

The paper points to a supplementary video covering the framework, four task families, comparisons, and failures. The retained static figures document the PDF examples; this edition does not substitute still images for a quantitative temporal-quality evaluation or claim a separate video review.

7.2 Additional Details on Post-training

The complete original template table includes full-body and body-part audio/text requests, emotion-to-motion, motion-to-emotion, and captioning formats. These templates are constructed from existing paired datasets to teach instruction following. Repeated or mislabeled task headings in the original table are preserved.

Task Input Output
Audio-to-Full Motion Based on [audio], generate a synchronized movement sequence involving both face, hands, upper and lower body. Listen to [audio] and produce movements that involve both the upper and lower body in harmony. [face][hands] [upper][lower]
Audio-to-Full Motion Based on [audio], generate a synchronized movement sequence involving both face, hands, upper and lower body. Listen to [audio] and produce movements that involve both the upper and lower body in harmony. [face][hands] [upper][lower]
Audio&Transcript-to-Full Motion Generate a set of movements for face, hand, upper, and lower body that correspond to the timestamped alignment in [audio&transcript] Using the precise timestamp match in [audio&transcript], generate corresponding face, hand, upper, and lower body movements. [face][hands] [upper][lower]
Audio-to-Upper Body Motion Using [audio], produce upper body movements that capture the tone and energy. From [audio], create a series of gestures that use the upper body to reflect its flow. [upper]
Audio-to-Lower Body Motion Interpret [audio] with lower body gestures that reflect its tempo. Create leg and foot movements that align with the intensity shifts in [audio]. [lower]
Audio-to-Hands Body Motion Develop a set of hand movements that respond dynamically to [audio]. Generate expressive hand gestures that reflect the cues in [audio]. [hand]
Audio-to-Face Body Motion Create expressions that correspond to the varying sentiments in [audio]. Listen to [audio] and generate a sequence of facial expressions that match its energy. [face]
Emotion-to-Motion Generate a movement sequence that fully embodies the emotion of [emotion] using the face, hands, upper body, and lower body. Express the emotion [emotion] through a series of actions involving the face, hands, upper, and lower body.” [face][hands] [upper][lower]
Motion-to-Emotion What emotion is conveyed by the movements in the face, hands, upper body, and lower body within [face][hands][upper][lower]? Examine the face, hand, upper, and lower body movements in [face][hands] [upper][lower] to interpret the emotional tone. [emotion]
Text-to-Full Motion Give me gestures involving the face, hands, upper body, and lower body that correspond to [caption] Show me gestures involving the face, hands, upper body, and lower body that capture the essence of Input: [caption]. [face][hands] [upper][lower]
Text-to-Upper Body Motion Create an upper body gesture that aligns with the sentiment of [caption]. Develop an upper body action sequence that mirrors the tone in [caption]. [upper]
Text-to-Lower Body Motion Illustrate the message in [caption] with lower body motions. Translate [caption] into a lower body movement sequence. [lower]
Text-to-Lower Body Motion Describe the motion represented by [upper][lower] using plain English. What does the [upper][lower] communicate? Please describe it in words. [caption]
Table 4: Examples of instruction prompt templates during post-training. For each task, we show two examples of the input prompts and the output format.

7.3 Additional Details on Editable Gesture Generation

Editing uses body-part keywords and two separate calls: generate upper-body motion from audio and lower-body motion from text. Combining those outputs produces the showcased walking/talking or similar examples. A model that jointly follows multiple simultaneous prompt sources is identified as future work.

7.4 Additional Details on Emotion Understanding

Since instruction-tuned outputs can contain free-form emotion descriptions, evaluation uses BLEU, ROUGE, CIDEr, and BERTScore-style text comparisons rather than exact label classification. This measures resemblance to reference wording and does not by itself establish calibrated or accurate psychological inference.

7.5 Results for Text-to-motion Generation

The retained montage compares qualitative text-conditioned motion with existing models. The authors decline standard quantitative metrics because their feature evaluators depend on the HumanML3D representation, whereas this method emphasizes expressive rotation and body-part structure. Consequently, statements of parity or superiority in text-to-motion remain qualitative and cannot be inferred from the co-speech results.

Refer to caption
Figure 8: Qualitative examples for text-to-motion generation. Given a text caption, we compare the 3D motion generated by our method with those generated by state-of-the-art methods, including MDM [59], T2M-GPT [77], and MotionGPT [23]. Our model produces smooth, natural, and sometimes better motion in comparison with existing methods, which do not model the audio modality.

7.6 Additional Implementation Details

Alignment training uses eight H100 accelerators with AdamW at learning rate 0.0002 until convergence. Post-training uses eight 3090 accelerators at 0.0001 for 350 epochs in ablations. A separate four-layer temporal-convolutional variational autoencoder predicts global translations from lower-body motion, supplementing the local body-part reconstruction. The full pipeline therefore includes more than the language-model backbone.

7.7 Additional Qualitative Example of Co-speech Gesture Generation

The final montage retains ground truth, EMAGE, SynTalker, and proposed-model frames for a second speech example. It illustrates gesture variation and alignment around emphasized words, while providing no additional numerical success rate or perceptual-study result.

Refer to caption
Figure 9: Additional qualitative example on co-speech gesture generation. Given an input speech, we visualize the ground truth 3D motion accompanying the audio, the motion generated by two baselines: EMAGE [43], SynTalker [8], and our method. Our model generates more diverse and expressive motion compared to existing methods, especially when the speaker emphasizes on words such as “angered” and “upset”.
Concept demonstration

Open image ↗