Literature
Display
← All papers

Morpheus: A Neural-driven Animatronic Face with Hybrid Actuation and Diverse Emotion Control

Zongzheng Zhang · Jiawen Yang · Ziqiao Peng · Meng Yang · Jianzhu Ma · Lin Cheng · Huazhe Xu · Hang Zhao · Hao Zhao

· Robotics: Science and Systems 2025 ↗

Robotics9 min text · CondensedOriginal paper ↗
Contents

Overview

Morpheus combines a physical animatronic face with speech-conditioned animation and learned motor control. The retained opening figure shows the hardware and matching virtual and physical expression frames. The supplementary sections document training, actuator ranges, hardware variants, user perception, materials, noise, and timing.

[Uncaptioned image]
Fig. 1: Morpheus: an animatronic face with diverse expressions. (a) Front view. Blue markers indicate the attachment points between the underlying mechanical structure and the soft skin, while yellow arrows denote the directions of movement. (b) Side view highlighting motion capabilities. Blue arrows indicate the three-axis neck movement: nodding, shaking, and rotation. The green arrow illustrates the jaw’s ability for horizontal movement in addition to typical opening and closing motions, enabling more diverse expressions. (c) Four sequential frames of the ”happy” expression as the animatronic face says ”welcome!” The first row illustrates the virtual expressions generated by our algorithm rendered in Blender, while the second row displays the corresponding real-world expressions reproduced by the animatronic face.

At a glance

Morpheus uses 33 actuators to animate a silicone-covered robotic face. Rigid mechanisms drive the eyebrows, eyes, mouth, and neck; four tendon drives shape the cheeks and nose. A speech model generates emotional blendshape animation, and a learned forward model of the face supplies gradients for matching virtual landmarks with motor commands. Experiments assess animation error, classifier recognition of four emotions, actuation variants, and user preferences. The evidence concerns expressive face generation, not the robot understanding or experiencing emotion.

I Introduction

Speech-synchronized animatronic faces need both useful mechanical motion and a mapping from desired expression to actuator settings. Rigid mechanisms offer well-defined trajectories but occupy space; tendon drives fit small facial regions while complicating control. Morpheus combines them and learns the soft skin’s motor-to-landmark relationship from randomized commands.

A separate model converts emotional speech to virtual facial motion. Connecting that animation to the learned face model allows continuous expressive playback. The paper motivates social interaction applications, but does not evaluate learning outcomes, clinical benefit, trust, or long-term human–robot interaction.

II Related Works

The related work covers speech-driven digital animation, mechanical facial platforms, and methods for recovering actuator commands from desired expressions.

II-A Audio-Driven Facial Animation

Audio-driven animation has progressed from phoneme–viseme matching to learned temporal models and separate content and emotion representations. Morpheus builds on emotional animation work such as EmoTalk [45], adding time alignment of repeated speech with different emotions and transferring the resulting motion to physical hardware.

II-B Mechanical Face

Existing robotic faces use rigid, pneumatic, tendon, or other compliant actuation, with varying facial and body complexity. The closest comparisons include Emo’s facial coexpression [19] and speech-driven tendon actuation [32]. Morpheus combines 29 rigid-driven actuators with four tendon-driven actuators; the experiments isolate selected modules of its own platform rather than conducting matched mechanical tests of every prior robot.

II-C Inverse Expression Learning

Prior inverse expression methods include programmed control, kinematics, search, and learned visual models. Morpheus uses a differentiable neural approximation of its own physical face to optimize commands against image landmarks, avoiding an explicit detailed model of silicone deformation and attachment geometry.

III Method

The system has three coupled components: speech-to-animation, the hybrid mechanical face, and a learned mapping used to recover physical actuator commands. The retained equations specify temporal alignment and the reconstruction objectives.

III-A Architecture Overview

Emotional speech first produces a sequence of 33 blendshape coefficients. Virtual faces rendered from those coefficients provide landmark targets for the inverse-control stage, which generates motor commands for synchronized physical playback. Blendshape dimensionality and the number of expression-controlled motors differ.

Refer to caption
Fig. 2: Overview of the Mechanism for Generating Realistic Speech-driven Facial Expressions. We propose a method that disentangles content and emotion from expressive speech and generates blendshape coefficients using a transformer-based decoder with emotion-guided attention. These coefficients are then mapped to actuator commands through a face inverse learning module, ultimately producing realistic facial expressions that match the emotional tone of the speech.

III-B Emotional Facial Animation

Separate pretrained audio encoders represent speech content and emotional tone. Pseudo-paired examples mix the two factors, while dynamic time warping aligns recordings of the same content spoken at different rates. Alignment uses features from wav2vec 2.0 [3] and a monotone path minimizing feature distance.

A transformer decoder fuses the aligned representations with positional information and emotion-guided attention, then predicts 33 blendshape coefficients per frame. The design aims to vary expressive style and intensity while preserving speech articulation. The reported experiments assess resulting animation and facial outputs rather than proving complete disentanglement of emotion from content.

Equation 1
min∑(i,j)∈Pd(𝑺a[i],𝑺b[j]),\displaystyle\min\sum_{(i,j)\in P}d(\bm{S}_{a}[i],\bm{S}_{b}[j]), (1)

III-C Hardware Design

The 3D-printed structure contains four eyebrow, six eye, sixteen mouth, and three neck actuators using rigid mechanisms, plus four tendon drives for the nose and cheeks. Silicone skin couples the mechanical motion into a continuous facial surface. This physical platform is demonstrated through photographs and generated expression sequences.

Refer to caption
Fig. 3: Hardware Design of Our Mechanical Face Platform. (a) Front view of the mechanical skeletal structure. (b) Transparent side view of the mechanical structure, highlighting tendon-driven movements in the nose and cheek areas (indicated by blue arrows). (c) Four rigid-driven modules (indicated by green arrows), including the eyebrow module (4 actuators), eye module (6 actuators), mouth module (16 actuators), and neck module (3 actuators), work together to form the complete mechanical face platform.

III-C 1 Module Design and Actuation

Separate mechanisms move eyebrow centers and peaks, upper and lower eyelids, and the linked eyeballs. Miniature eye cameras support face detection. Eight lip control points include two mouth corners with five-bar mechanisms and six upper/lower lip points. The jaw can translate laterally as well as open and close, adding articulation beyond a simple hinged mouth.

III-C 2 Hybrid Actuation Control

Tendons deform the small nose and cheek regions while rigid linkages drive the larger eyebrow, eye, and mouth movements. The intended tradeoff is space-efficient local deformation combined with more constrained motion elsewhere. Expressiveness is evaluated on this combined implementation, with the tendon contribution measured through an internal hardware ablation.

III-D Face Inverse learning

Two gaze motors, three neck motors, and two tongue motors are excluded from learned expression control, leaving 26 motors. A multilayer perceptron predicts 468 two-dimensional landmarks from their commands using measurements from MediaPipe [39]. Upper and lower face models are separated to reduce model size and optimization effort.

After fitting this forward model, freeze its weights and backpropagate landmark error with respect to motor inputs. This optimizes commands matching the virtual face. The loss measures a frontal image-space proxy; matching detected landmarks does not establish identical three-dimensional skin deformation or perceptual realism.

III-E Loss Function

Cross-reconstruction combines content from one utterance with emotion from another and compares the decoded blendshapes with the matching target combination. Self-reconstruction preserves the original animation when content and emotion come from the same input; both terms have weight one. The motor-control stage minimizes an L1 discrepancy between physical and virtual landmarks through the learned forward model. These objectives supervise reconstruction rather than directly optimizing human preference.

Equations 2–5
ℒ=λ1​ℒcross+λ2​ℒself,\mathcal{L}=\lambda_{1}\mathcal{L}_{\text{cross}}+\lambda_{2}\mathcal{L}_{\text{self}}, (2)
ℒcross=\displaystyle\mathcal{L}_{\text{cross}}= ‖𝑫⁡(𝑬c​(𝑨c1,e2),𝑬e​(𝑨c2,e1))−𝑩c1,e1‖22\displaystyle\left\|\boldsymbol{D}\left(\boldsymbol{E}_{c}\left(\boldsymbol{A}_{c_{1},e_{2}}\right),\boldsymbol{E}_{e}\left(\boldsymbol{A}_{c_{2},e_{1}}\right)\right)-\boldsymbol{B}_{c_{1},e_{1}}\right\|_{2}^{2} (3)
+‖𝑫⁡(𝑬c​(𝑨c2,e1),𝑬e​(𝑨c1,e2))−𝑩c2,e2‖22,\displaystyle+\left\|\boldsymbol{D}\left(\boldsymbol{E}_{c}\left(\boldsymbol{A}_{c_{2},e_{1}}\right),\boldsymbol{E}_{e}\left(\boldsymbol{A}_{c_{1},e_{2}}\right)\right)-\boldsymbol{B}_{c_{2},e_{2}}\right\|_{2}^{2},
ℒself=‖𝑫⁡(𝑬c​(𝑨c1,e2),𝑬e​(𝑨c1,e2))−𝑩c1,e2‖22.\mathcal{L}_{\text{self}}=\left\|\boldsymbol{D}\left(\boldsymbol{E}_{c}\left(\boldsymbol{A}_{c_{1},e_{2}}\right),\boldsymbol{E}_{e}\left(\boldsymbol{A}_{c_{1},e_{2}}\right)\right)-\boldsymbol{B}_{c_{1},e_{2}}\right\|_{2}^{2}. (4)
ℒL1​(Cr,Cv)=1N​∑i=1N‖cri−cvi‖1,\displaystyle\mathcal{L}_{\mathit{L_{1}}}(C_{r},C_{v})=\frac{1}{N}\sum_{i=1}^{N}\|c_{r_{i}}-c_{v_{i}}\|_{1}, (5)

IV Experiments

The evaluation combines digital-animation benchmarks, recordings of the physical face, internal actuation ablations, and supplementary user judgments. The original section titles label the image demonstrations quantitative and the numerical metrics qualitative; the results below describe their actual measurement types.

IV-A Implementation Details

The speech module uses five hours of 3D-ETF animation derived from the RAVDESS emotional and HDTF audiovisual datasets. For the physical forward model, 5,000 randomized safe-range motor configurations are observed by a frontal RGB camera and converted to landmark targets. Command ranges are constrained to avoid structural interference.

The neck uses larger TD-8035MG servos, other regions use small GUOHUA servos, and a Jetson Xavier handles processing and control. The main text describes zero-degree liquid silicone, while the supplementary fabrication section specifies A00-30 silicone; the edition retains both source descriptions without resolving that material inconsistency.

IV-B Diverse Expressions

Figure 4 displays 25 named expressions spanning basic emotions and subtler labels. The gallery demonstrates the available range of facial configurations. Naming a displayed expression does not by itself show that viewers consistently recognize all 25 categories; the classifier experiment below tests four.

Refer to caption
Fig. 4: Expressions generated by the mechanical face platform. The platform demonstrates 25 emotional expressions, from ”happy” to ”funny,” showcasing its ability to replicate a variety of human facial movements, providing a basis for realistic human-robot interaction.

IV-C Quantitative Results

Virtual and physical sequences show happy, angry, and fearful delivery of different complete sentences. Their eyebrow and mouth trajectories differ while retaining visible speech articulation. These are qualitative correspondence examples; still frames cannot alone establish temporal synchronization accuracy across the whole utterance.

Refer to caption
Fig. 5: Complete sentences with diverse facial expressions. The first row illustrates the virtual expressions generated by our algorithm rendered in Blender, while the second row displays the corresponding real-world expressions reproduced by the animatronic face. (a) Happy expression while saying ”Kids are sitting by the door.” (b) Angry expression while saying ”I lost my keys.” (c) Fear expression while saying ”The weather is changing.”

IV-D Qualitative Results

A video expression classifier [36] evaluates 400 approximately two-second videos: 100 randomly selected sentences paired with each of four expressions. Recognition is 90% happy, 91% angry, 66% disgust, and 73% fear, averaging 80%. Fear and disgust are often confused with one another. These rates describe the classifier, not human recognition or calibrated emotional understanding.

On the digital-animation benchmarks, Morpheus reports lip/emotional-region vertex errors of 2.426/2.389 mm on RAVDESS and 2.491/2.252 mm on HDTF, compared with EmoTalk’s 2.762/2.493 and 2.892/2.364 mm. The listed baselines are higher on these metrics. Vertex error quantifies reconstruction, not the entire subjective quality of expression, and these digital metrics are distinct from the physical face’s landmark error.

Equations 6–7
LVE=maxi⁡‖𝑳gt,i−𝑳pred,i‖2.\text{LVE}=\max_{i}\|\boldsymbol{L}_{\text{gt},i}-\boldsymbol{L}_{\text{pred},i}\|_{2}. (6)
EVE=maxj⁡‖𝑽gt,j−𝑽pred,j‖2,\text{EVE}=\max_{j}\|\boldsymbol{V}_{\text{gt},j}-\boldsymbol{V}_{\text{pred},j}\|_{2}, (7)
TABLE I: Confusion Matrix of recognizing four basic expressions
True \Predicted Happy (%) Angry (%) Disgust (%) Fear (%)
Happy (%) 90 2 3 5
Angry (%) 3 91 4 2
Disgust (%) 6 3 66 25
Fear (%) 5 2 20 73
TABLE II: Quantitative evaluation results on RAVDESS [38] and HDTF datasets (3D-ETF dataset) [56]
RAVDESS [38] (emotion) HDTF [56] (no emotion)
Method
LVE (mm)↓\downarrow
EVE (mm)↓\downarrow
LVE (mm)↓\downarrow
EVE (mm)↓\downarrow
VOCA [11] 5.091 4.188 4.447 3.286
MeshTalk [47] 3.459 3.386 3.886 3.124
FaceFormer [12] 3.247 3.757 3.374 3.142
EmoTalk [45] 2.762 2.493 2.892 2.364
Ours 2.426 2.389 2.491 2.252

IV-E Analysis of Hybrid Actuation Hardware Design

Each hardware configuration generates 400 videos under the same four-expression setup. Mean classifier accuracy is 43% with mouth-only control, 44.75% with eyebrows only, 64.5% with the rigid system, and 80% with hybrid actuation. The paper calls the last improvement mAP in prose, but Table IV reports the arithmetic mean of classification accuracies.

The hybrid design reduces the reported cheek landmark discrepancy from 2.10 to 0.95 mm and nose discrepancy from 1.18 to 0.26 mm relative to rigid-only actuation. Pixel distances are converted using camera-to-face calibration and known facial dimensions. These measurements support the contribution of tendon-driven regions on this platform; they do not establish superiority over all tendon-only hardware.

TABLE III: Comparison of Facial Landmark Displacement
Subtle Area Rigid-Only System (mm) ↓\downarrow Hybrid System (mm) ↓\downarrow
Cheek 2.10 0.95
Nose 1.18 0.26
TABLE IV: Classification Accuracy for Videos Generated by Different Hardware Configurations
Hardware Configuration Happy (%) Angry (%) Disgust (%) Fear (%) Mean (%)
Mouth-Only System 65 41 31 35 43
Eyebrow-Only System 47 50 39 43 44.75
Rigid-Driven System w/o tendon driven 72 75 51 60 64.5
Hybrid-Driven System (Ours) 90 91 66 73 80
Refer to caption
Fig. 6: Comparison of happy expression using different actuation systems. (a) Mouth-Only System. (b) Eyebrow-Only System. (c) Rigid-Driven System w/o tendon driven. (d) Hybrid-Driven System (Ours).

V Limitations

Adhesive skin attachments make adjustment and debugging difficult; replaceable attachment points are proposed. Skin material and thickness affect expression quality and need further testing. The main limitations section also reports high inference times and identifies near-real-time dialogue as future work. The supplementary component rates do not provide an end-to-end conversational latency measurement resolving that limitation.

VI Conclusion

Hybrid actuation, learned self-modeling, and emotional speech animation jointly produce varied physical facial motions. Classifier results, vertex errors, and a small user study support the tested expressive capability. Broader claims about natural interaction remain limited by selected stimuli, camera-based proxies, incomplete timing detail, and the absence of long-term interaction experiments.

VI-A Emotional Facial Animation Module Training

The supplementary speech-training recipe uses 30 fps video, 16 kHz audio, and 33 blendshape coefficients per frame. Adam training uses learning rate 0.0001, batch size sixteen, and eighty epochs, taking about four hours on one A100 GPU.

VI-B Description of the Mechanical Face Hardware Platform

Figure 7 preserves the complete actuator-location diagram and labels. Motors 31–33 provide neck motion and are omitted from that facial diagram. The text states that 26 motors control learned expressions, consistent with excluding gaze, tongue, and neck motors; its accompanying numbered range includes an eye-gaze index, so that range should not be read as an independently consistent list of the 26 learned controls.

Refer to caption
Fig. 7: Motor Distribution and Control Areas of the Mechanical Face Hardware Platform. 1 - Left Cheek (tendon drive). 2 - Left Nose (tendon drive). 3 - Left Eyebrow Center. 4 - Left Eyebrow Peak. 5 - Left Upper Eyelid. 6 - Left Lower Eyelid. 7 - Eyes Up/Down Gaze. 8 - Upper Lip Center. 9 - Left Upper Corner Mouth Movement. 10 - Upper Lip Right. 11 - Tongue Extension/Retraction. 12 - Tongue Up/Down. 13 - Chin (passive). 14 - Left Lower Corner Mouth Movement. 15 - Lower Lip Center. 16 - Lower Lip Left. 17 - Lower Lip Right. 18 - Chin (active). 19 - Right Lower Corner Mouth Movement. 20 - Chin Left/Right. 21 - Upper Lip Left. 22 - Right Upper Corner Mouth Movement. 23 - Mouth Opening/Closing. 24 - Eyes Left/Right Gaze. 25 - Right Lower Eyelid. 26 - Right Upper Eyelid. 27 - Right Eyebrow Peak. 28 - Right Eyebrow Center. 29 - Right Nose (tendon drive). 30 - Right Cheek (tendon drive).

VI-C Face Inverse Learning Module Training

Table V retains all 33 motor control ranges and starting values. Commands are normalized to [0,1] before fitting the forward model with L1 loss, using learning rate 0.00001 and ten to fifteen minutes on an RTX 3090. Several table entries labeled lower and upper bounds appear in decreasing numerical order, reflecting the original presentation; the values are retained as reported rather than silently reordered.

TABLE V: Motor Control Parameter Range
Motor ID Control Position Lower Bound Start Value Upper Bound
1 Left Cheek 1800 1800 1400
2 Left Nose 800 800 0
3 Left Eyebrow Center 980 1080 1480
4 Left Eyebrow Peak 1100 800 500
5 Left Upper Eyelid 1700 1300 600
6 Left Lower Eyelid 1000 1200 1800
7 Eyes Up/Down Gaze 300 1000 1500
8 Upper Lip Center 1000 1000 600
9 Left Upper Corner Mouth Movement 500 1000 1500
10 Upper Lip Right 1000 1000 1200
11 Tongue Extension/Retraction 800 1000 1200
12 Tongue Up/Down 800 500 300
13 Chin (passive) 1200 1600 1600
14 Left Lower Corner Mouth Movement 1000 1000 1500
15 Lower Lip Center 1300 1300 1900
16 Lower Lip Left 1500 1500 800
17 Lower Lip Right 800 800 1200
18 Chin (active) 1200 1600 1600
19 Right Lower Corner Mouth Movement 1000 1000 500
20 Jaw Left/Right 400 1000 1600
21 Upper Lip Left 1000 1000 800
22 Right Upper Corner Mouth Movement 1500 1000 500
23 Mouth Opening/Closing 1150 1050 1050
24 Eyes Left/Right Gaze 800 1150 1500
25 Right Lower Eyelid 1200 1000 400
26 Right Upper Eyelid 500 1000 1600
27 Right Eyebrow Peak 900 1200 1500
28 Right Eyebrow Center 1300 1200 800
29 Right Nose 1200 1200 2000
30 Right Cheek 0 0 600
31 Neck Rotation 1500 1750 2000
32 Neck Nodding/Shaking (left) 900 1200 1300
33 Neck Nodding/Shaking (right) 1200 900 800

VI-D Visualization of Virtual and Real Mechanical Faces with Emotions Speaking Specific Sentences

The supplementary sequences show the same sentence with happy, fearful, angry, and disgusted delivery. Each pairs virtual and physical face frames, exposing both expression changes and residual transfer differences. The full figure is preserved, including every temporal frame and emotion row.

Refer to caption
Fig. 8: Visualization of Morpheus interpreting the sentence ”look at the sky” with happy, fear, angry and disgust expressions, respectively.

VI-E Generalizability of the Self-Modeling Network.

Four hardware revisions have 17, 16, 23, and 26 expression degrees of freedom. The network is adapted to each actuator configuration and evaluated by landmark error: 1.652, 1.313, 1.206, and 0.748 mm, respectively. This demonstrates applying the self-modeling approach to related hardware variants; it is not evidence that one unchanged trained model transfers zero-shot to new mechanisms.

TABLE VI: Landmark Distances for Different Versions of the Self-Modeling Network
Version V1 V2 V3 V4
Landmark Distance (mm) 1.652 1.313 1.206 0.748
Refer to caption
Fig. 9: Morpheus design evolution across four iterations. (a) Internal skeletal structures. (b) External facial appearances with silicone skin applied.

VI-F User Perception Study

Fifty participants, aged 15–50 with balanced gender as reported, compare static expressions, speech-driven clips, and realism. Morpheus is preferred in 84% of static comparison cases with mean rank 1.18; 96% prefer its speech expressiveness over the selected prior system, and its realism score averages 7.6/10. Competitor material comes from published papers and public demonstrations, so recording and stimulus conditions are not matched. The paper does not provide uncertainty intervals or demonstrate generalization to sustained live interaction.

VI-G Hardware Noise Discussion

The supplement reports 45–50 dB for the selected small servos and a 20 dB reduction from a two-millimeter butyl soundproof layer around larger servos. Measurement geometry and a full acoustic protocol are not supplied. These figures should be treated as reported component observations, not a general environmental-noise or health-compliance finding. Lubrication, higher-frequency control, and brushless servos are proposed improvements.

VI-H Material Selection and Expressiveness

The supplementary fabrication recipe specifies A00-30 silicone mixed 1:1, vacuum degassing, casting in a printed mold, curing at 25°C, and an additional resting period. Four tested thicknesses are 2, 2.5, 4, and 5 mm. Images suggest four millimeters provides smoother tendon-driven deformation than the localized thin-skin response or the rigid five-millimeter response; eyelid areas are thinned to two millimeters. This is a qualitative material comparison rather than a systematic durability or force characterization.

Refer to caption
Fig. 10: Impact of silicone skin thickness on tendon-driven facial deformation. (a-b) For thinner skins (2mm and 2.5mm), the deformation is unnaturally localized around the pasted region (indicated by the base of the arrow). (c) At 4mm thickness (our selected configuration), the skin exhibits smooth and natural movement, enabling expressive actuation. (d) A 5mm thickness is overly rigid, significantly limiting visible deformation.

VI-I Latency in Real-Time Human-Robot Interaction

The supplement reports about 150 fps inference on Jetson AGX Xavier, virtual animation at 30 fps, and servo updates at 50 Hz. These are component throughput or update rates, with no stated audio window, buffering delay, or full speech-to-motion latency distribution. They therefore do not establish the response delay of a live conversational system.

Concept demonstration

Open image ↗