Overview
The original opening montage illustrates the model’s audio, text, and body-part motion interfaces and the four demonstrated task families.
At a glance
The Language of Motion extends a pretrained text model with discrete speech and compositional motion tokens. Spatial and temporal motion prediction plus audio–text alignment precede instruction tuning on paired downstream data. On the tested BEATv2 speaker protocol, it improves gesture-distribution distance and diversity over the compared models, while one baseline retains better speech–motion beat correlation. Editable motion and emotion-label output are also demonstrated, with important limits on joint conditioning and evaluation.
1 Introduction
Most motion generators specialize in one input modality and dataset. A shared token interface could transfer motion structure and speech semantics between tasks. This paper combines separate face, hand, upper-body, and lower-body vocabularies with audio and text, then trains a common encoder–decoder language model. The approach still needs modality-specific tokenizers and paired downstream data; its demonstrations do not establish unrestricted any-to-any generation.
2 Related Work
Related work covers speech-conditioned gestures, text-conditioned movement, and multimodal language models.
2.1 Speech-Driven Motion Generation
Speech-driven methods generate listener behavior, gestures, or facial expressions from audio, often for specific speakers. The proposed shared model aims to use additional motion and text resources rather than restrict learning to one speech–motion pairing.
2.2 Text to Motion Generation
Text-to-motion methods use diffusion or token-based generation to produce body movement from descriptions. This work emphasizes expressive rotations, gestures, and facial motion alongside text and audio inputs. Qualitative text-following evidence is reported later, without a comparable quantitative text-to-motion benchmark score.
2.3 Multimodal Language Models
Multimodal language models combine modality-specific representations with a shared token-processing backbone. This model applies that idea to text, speech, and human motion, with principal evaluated outputs being motion sequences and text.
3 Multimodal Language Model for Motion Generation and Understanding
The method comprises body-model representation, modality tokenization, alignment pretraining, instruction-based downstream tuning, and autoregressive token prediction.
3.1 Preliminaries
The neutral SMPL-X model with FLAME facial representation describes body shape, joint pose, facial expression, and global translation over time. Separating these parameters supports expressive body-part control but also requires consistent reconstruction into a complete body sequence.
3.2 Tokenization
Four vector-quantized autoencoders encode face, hands, upper body, and lower body separately. Training combines pose, velocity, acceleration, mesh, and codebook-commitment losses. Six-dimensional rotations retain twisting information that the authors argue is underrepresented in the commonly used HumanML3D format.
HuBERT [20] converts 16 kHz audio into fifty discrete tokens per second. Text uses the pretrained T5 vocabulary; motion/audio code vocabularies and modality-boundary tokens extend it. Motion is normally represented at thirty frames per second. The resulting tokens share a language-model interface, while their encoders and decoders remain modality-specific.
Equations 1–2
| (1) | ||||
| (2) | ||||
3.3 Pre-training for Modality Alignment
Pretraining predicts one subset of body parts from another, reconstructs temporally masked motion, and translates between audio and text. The gesture experiment uses about sixty hours of motion and one thousand hours of audio–text data. No audio-to-motion pairs are used in this alignment stage, but audio–text alignment is itself paired supervision; calling the entire stage unpaired would be misleading. Correlations among body parts are learned from these datasets, not shown to be universal across cultures or people.
3.4 Post-training with Instruction Following
Instruction tuning uses paired downstream examples for tasks such as audio-to-motion, text-to-motion, and emotion-conditioned motion. Many prompt templates vary how a task is requested. The reported thousand-plus instruction variants are not a thousand independently benchmarked capabilities. The original task prompts, conditions, and output formats remain alongside the summaries.
3.5 Language Model Training Details
The backbone is a 220-million-parameter Flan-T5-base encoder–decoder derived from T5 [55]. Inputs are limited to 512 tokens, and the decoder predicts output tokens autoregressively. Both alignment and downstream stages update the full model rather than low-rank adapters. Modality tokens identify codebook entries and sequence boundaries.
Equation 3
| (3) |
4 Experiments
Experiments measure speech-conditioned gestures, alignment/data ablations, editable body-part generation, and textual emotion-label prediction.
4.1 Co-Speech Gesture Generation
The primary protocol trains and tests on BEATv2 speaker 2 and adopts the motion tokenizer/evaluation procedure of EMAGE [43]. This is speaker-specific evaluation with paired fine-tuning, not zero-shot transfer to unseen people.
In the table’s reported units, the proposed method has gesture distance 5.301 versus EMAGE’s 5.512, beat correlation 7.780 versus 7.724, and diversity 15.167 versus 13.06. SynTalker’s beat correlation is higher at 7.971, so the source’s claim to win every metric is too broad. Higher diversity is a movement-variation statistic, not proof of semantic correctness or naturalness.
Removing language pretraining substantially worsens distribution distance to 7.470. Removing multimodal pretraining gives 5.408 in Table 1. These ablations support the usefulness of pretrained structure within the reported setting; different baselines also receive different conditioning signals.
| FGD | BC | Diversity | Condition Signal | |
| DisCo [40] | 9.417 | 6.439 | 9.912 | audio |
| CaMN [41] | 6.644 | 6.769 | 10.86 | audio, text, facial |
| DiffStyleGesture [69] | 8.811 | 7.241 | 11.49 | audio, style |
| Habibie et al. [19] | 9.040 | 7.716 | 8.213 | audio, text |
| TalkSHOW [71] | 6.209 | 6.947 | 13.47 | audio |
| SynTalker [8] | 6.413 | 7.971 | 12.721 | audio, text |
| EMAGE [43] | 5.512 | 7.724 | 13.06 | audio, text |
| Ours w/o language pre-training | 7.470 | 6.148 | 14.162 | audio |
| Ours w/o multimodal pre-training | 5.408 | 7.742 | 14.418 | audio |
| Ours | 5.301 | 7.780 | 15.167 | audio |
4.2 Effect of Generative Pre-training
Removing audio-to-text, spatial, or temporal objectives worsens the reported full-model result. Table 2 gives 5.501 for its no-pretraining variant, differing from the 5.408 row in Table 1; the source does not reconcile those values. The data-scarcity plot varies available paired post-training examples, down to one thirty-second, and favors the pretrained model over its ablation and EMAGE. The surrounding prose inconsistently calls this pretraining data, while the figure labels it post-training data. These results reduce required paired adaptation data in the tested speaker setting; they do not remove the large pretraining datasets or demonstrate broad unseen-speaker generalization.
| FGD | BC | Diversity | |
| W/o pre-training | 5.501 | 7.721 | 14.281 |
| W/o A2T | 5.443 | 7.721 | 14.499 |
| W/o spatial | 6.336 | 7.381 | 14.173 |
| W/o temporal | 6.800 | 7.341 | 13.810 |
| W/o motion | 7.776 | 7.344 | 14.640 |
| Ours | 5.301 | 7.780 | 15.165 |
4.3 Unifying Audio-to-Motion and Text-to-Motion for Editable Generation
The editable-generation setup trains motion tokenizers on BEATv2 and AMASS and uses HumanML3D text annotations for instruction tuning. In the actual demonstration, the model is called twice: audio conditions upper-body motion and text conditions lower-body motion, then the outputs are combined. The supplement explicitly leaves simultaneous multi-source prompting to future larger-scale training. Thus the evidence is compositional body-part editing, not a single jointly conditioned full-body generation pass.
4.4 Predicting Emotion from Motion
Instruction tuning adds the eight BEATv2 emotion labels. Predicted text is compared with reference labels using language-generation metrics because outputs do not reliably follow a fixed single-label format. The proposed model exceeds random output and MotionGPT [23], but MotionGPT was trained for general captions rather than this supervised emotion task, limiting that comparison.
The table reports 14.71 for BLEU-1, 26.67 under the combined “Rouge Cider” heading, and 16.94 for BERTScore. These are not emotion-classification accuracy percentages. The inconsistent combined metric label is preserved rather than interpreted as two separately reported measurements. Dataset-label prediction from motion does not validate reading a person’s internal feelings or a clinical use.
| Bleu@1 | Rouge Cider | BertScore | |
| GT | 100 | 100 | 99.9 |
| Random | 2.45 | 4.44 | 0.19 |
| MotionGPT | 1.68 | 10.67 | 2.31 |
| Ours | 14.71 | 26.67 | 16.94 |
5 Discussion
Discrete motion tokenization sometimes produces incoherent movement, motivating the authors’ suggestion to investigate continuous representations. Additional limits include speaker-specific quantitative evaluation, qualitative-only edited/text-conditioned generation, and the use of text-similarity metrics for emotion labels. The shared model demonstrates useful transfer and flexibility without establishing uniformly natural or semantically correct motion.
Acknowledgments:
The project acknowledges NIH and UST funding and feedback from colleagues.
7 Supplementary
The supplement adds task templates, the precise editing procedure, emotion-metric details, qualitative text-to-motion comparisons, implementation settings, and another gesture montage.
7.1 Supplementary Video
The paper points to a supplementary video covering the framework, four task families, comparisons, and failures. The retained static figures document the PDF examples; this edition does not substitute still images for a quantitative temporal-quality evaluation or claim a separate video review.
7.2 Additional Details on Post-training
The complete original template table includes full-body and body-part audio/text requests, emotion-to-motion, motion-to-emotion, and captioning formats. These templates are constructed from existing paired datasets to teach instruction following. Repeated or mislabeled task headings in the original table are preserved.
| Task | Input | Output |
| Audio-to-Full Motion | Based on [audio], generate a synchronized movement sequence involving both face, hands, upper and lower body. Listen to [audio] and produce movements that involve both the upper and lower body in harmony. | [face][hands] [upper][lower] |
| Audio-to-Full Motion | Based on [audio], generate a synchronized movement sequence involving both face, hands, upper and lower body. Listen to [audio] and produce movements that involve both the upper and lower body in harmony. | [face][hands] [upper][lower] |
| Audio&Transcript-to-Full Motion | Generate a set of movements for face, hand, upper, and lower body that correspond to the timestamped alignment in [audio&transcript] Using the precise timestamp match in [audio&transcript], generate corresponding face, hand, upper, and lower body movements. | [face][hands] [upper][lower] |
| Audio-to-Upper Body Motion | Using [audio], produce upper body movements that capture the tone and energy. From [audio], create a series of gestures that use the upper body to reflect its flow. | [upper] |
| Audio-to-Lower Body Motion | Interpret [audio] with lower body gestures that reflect its tempo. Create leg and foot movements that align with the intensity shifts in [audio]. | [lower] |
| Audio-to-Hands Body Motion | Develop a set of hand movements that respond dynamically to [audio]. Generate expressive hand gestures that reflect the cues in [audio]. | [hand] |
| Audio-to-Face Body Motion | Create expressions that correspond to the varying sentiments in [audio]. Listen to [audio] and generate a sequence of facial expressions that match its energy. | [face] |
| Emotion-to-Motion | Generate a movement sequence that fully embodies the emotion of [emotion] using the face, hands, upper body, and lower body. Express the emotion [emotion] through a series of actions involving the face, hands, upper, and lower body.” | [face][hands] [upper][lower] |
| Motion-to-Emotion | What emotion is conveyed by the movements in the face, hands, upper body, and lower body within [face][hands][upper][lower]? Examine the face, hand, upper, and lower body movements in [face][hands] [upper][lower] to interpret the emotional tone. | [emotion] |
| Text-to-Full Motion | Give me gestures involving the face, hands, upper body, and lower body that correspond to [caption] Show me gestures involving the face, hands, upper body, and lower body that capture the essence of Input: [caption]. | [face][hands] [upper][lower] |
| Text-to-Upper Body Motion | Create an upper body gesture that aligns with the sentiment of [caption]. Develop an upper body action sequence that mirrors the tone in [caption]. | [upper] |
| Text-to-Lower Body Motion | Illustrate the message in [caption] with lower body motions. Translate [caption] into a lower body movement sequence. | [lower] |
| Text-to-Lower Body Motion | Describe the motion represented by [upper][lower] using plain English. What does the [upper][lower] communicate? Please describe it in words. | [caption] |
7.3 Additional Details on Editable Gesture Generation
Editing uses body-part keywords and two separate calls: generate upper-body motion from audio and lower-body motion from text. Combining those outputs produces the showcased walking/talking or similar examples. A model that jointly follows multiple simultaneous prompt sources is identified as future work.
7.4 Additional Details on Emotion Understanding
Since instruction-tuned outputs can contain free-form emotion descriptions, evaluation uses BLEU, ROUGE, CIDEr, and BERTScore-style text comparisons rather than exact label classification. This measures resemblance to reference wording and does not by itself establish calibrated or accurate psychological inference.
7.5 Results for Text-to-motion Generation
The retained montage compares qualitative text-conditioned motion with existing models. The authors decline standard quantitative metrics because their feature evaluators depend on the HumanML3D representation, whereas this method emphasizes expressive rotation and body-part structure. Consequently, statements of parity or superiority in text-to-motion remain qualitative and cannot be inferred from the co-speech results.
7.6 Additional Implementation Details
Alignment training uses eight H100 accelerators with AdamW at learning rate 0.0002 until convergence. Post-training uses eight 3090 accelerators at 0.0001 for 350 epochs in ablations. A separate four-layer temporal-convolutional variational autoencoder predicts global translations from lower-body motion, supplementing the local body-part reconstruction. The full pipeline therefore includes more than the language-model backbone.
7.7 Additional Qualitative Example of Co-speech Gesture Generation
The final montage retains ground truth, EMAGE, SynTalker, and proposed-model frames for a second speech example. It illustrates gesture variation and alignment around emphasized words, while providing no additional numerical success rate or perceptual-study result.