Overview
Morpheus combines a physical animatronic face with speech-conditioned animation and learned motor control. The retained opening figure shows the hardware and matching virtual and physical expression frames. The supplementary sections document training, actuator ranges, hardware variants, user perception, materials, noise, and timing.
![[Uncaptioned image]](https://storage.literature.yihao.one/media/sha256/69/698f8b291e83e8d745c13aaad1988b7765a9361466721c2865c54b58da72f0be.webp)
At a glance
Morpheus uses 33 actuators to animate a silicone-covered robotic face. Rigid mechanisms drive the eyebrows, eyes, mouth, and neck; four tendon drives shape the cheeks and nose. A speech model generates emotional blendshape animation, and a learned forward model of the face supplies gradients for matching virtual landmarks with motor commands. Experiments assess animation error, classifier recognition of four emotions, actuation variants, and user preferences. The evidence concerns expressive face generation, not the robot understanding or experiencing emotion.
I Introduction
Speech-synchronized animatronic faces need both useful mechanical motion and a mapping from desired expression to actuator settings. Rigid mechanisms offer well-defined trajectories but occupy space; tendon drives fit small facial regions while complicating control. Morpheus combines them and learns the soft skin’s motor-to-landmark relationship from randomized commands.
A separate model converts emotional speech to virtual facial motion. Connecting that animation to the learned face model allows continuous expressive playback. The paper motivates social interaction applications, but does not evaluate learning outcomes, clinical benefit, trust, or long-term human–robot interaction.
II Related Works
The related work covers speech-driven digital animation, mechanical facial platforms, and methods for recovering actuator commands from desired expressions.
II-A Audio-Driven Facial Animation
Audio-driven animation has progressed from phoneme–viseme matching to learned temporal models and separate content and emotion representations. Morpheus builds on emotional animation work such as EmoTalk [45], adding time alignment of repeated speech with different emotions and transferring the resulting motion to physical hardware.
II-B Mechanical Face
Existing robotic faces use rigid, pneumatic, tendon, or other compliant actuation, with varying facial and body complexity. The closest comparisons include Emo’s facial coexpression [19] and speech-driven tendon actuation [32]. Morpheus combines 29 rigid-driven actuators with four tendon-driven actuators; the experiments isolate selected modules of its own platform rather than conducting matched mechanical tests of every prior robot.
II-C Inverse Expression Learning
Prior inverse expression methods include programmed control, kinematics, search, and learned visual models. Morpheus uses a differentiable neural approximation of its own physical face to optimize commands against image landmarks, avoiding an explicit detailed model of silicone deformation and attachment geometry.
III Method
The system has three coupled components: speech-to-animation, the hybrid mechanical face, and a learned mapping used to recover physical actuator commands. The retained equations specify temporal alignment and the reconstruction objectives.
III-A Architecture Overview
Emotional speech first produces a sequence of 33 blendshape coefficients. Virtual faces rendered from those coefficients provide landmark targets for the inverse-control stage, which generates motor commands for synchronized physical playback. Blendshape dimensionality and the number of expression-controlled motors differ.
III-B Emotional Facial Animation
Separate pretrained audio encoders represent speech content and emotional tone. Pseudo-paired examples mix the two factors, while dynamic time warping aligns recordings of the same content spoken at different rates. Alignment uses features from wav2vec 2.0 [3] and a monotone path minimizing feature distance.
A transformer decoder fuses the aligned representations with positional information and emotion-guided attention, then predicts 33 blendshape coefficients per frame. The design aims to vary expressive style and intensity while preserving speech articulation. The reported experiments assess resulting animation and facial outputs rather than proving complete disentanglement of emotion from content.
Equation 1
| (1) |
III-C Hardware Design
The 3D-printed structure contains four eyebrow, six eye, sixteen mouth, and three neck actuators using rigid mechanisms, plus four tendon drives for the nose and cheeks. Silicone skin couples the mechanical motion into a continuous facial surface. This physical platform is demonstrated through photographs and generated expression sequences.
III-C 1 Module Design and Actuation
Separate mechanisms move eyebrow centers and peaks, upper and lower eyelids, and the linked eyeballs. Miniature eye cameras support face detection. Eight lip control points include two mouth corners with five-bar mechanisms and six upper/lower lip points. The jaw can translate laterally as well as open and close, adding articulation beyond a simple hinged mouth.
III-C 2 Hybrid Actuation Control
Tendons deform the small nose and cheek regions while rigid linkages drive the larger eyebrow, eye, and mouth movements. The intended tradeoff is space-efficient local deformation combined with more constrained motion elsewhere. Expressiveness is evaluated on this combined implementation, with the tendon contribution measured through an internal hardware ablation.
III-D Face Inverse learning
Two gaze motors, three neck motors, and two tongue motors are excluded from learned expression control, leaving 26 motors. A multilayer perceptron predicts 468 two-dimensional landmarks from their commands using measurements from MediaPipe [39]. Upper and lower face models are separated to reduce model size and optimization effort.
After fitting this forward model, freeze its weights and backpropagate landmark error with respect to motor inputs. This optimizes commands matching the virtual face. The loss measures a frontal image-space proxy; matching detected landmarks does not establish identical three-dimensional skin deformation or perceptual realism.
III-E Loss Function
Cross-reconstruction combines content from one utterance with emotion from another and compares the decoded blendshapes with the matching target combination. Self-reconstruction preserves the original animation when content and emotion come from the same input; both terms have weight one. The motor-control stage minimizes an L1 discrepancy between physical and virtual landmarks through the learned forward model. These objectives supervise reconstruction rather than directly optimizing human preference.
Equations 2–5
| (2) |
| (3) | ||||
| (4) |
| (5) |
IV Experiments
The evaluation combines digital-animation benchmarks, recordings of the physical face, internal actuation ablations, and supplementary user judgments. The original section titles label the image demonstrations quantitative and the numerical metrics qualitative; the results below describe their actual measurement types.
IV-A Implementation Details
The speech module uses five hours of 3D-ETF animation derived from the RAVDESS emotional and HDTF audiovisual datasets. For the physical forward model, 5,000 randomized safe-range motor configurations are observed by a frontal RGB camera and converted to landmark targets. Command ranges are constrained to avoid structural interference.
The neck uses larger TD-8035MG servos, other regions use small GUOHUA servos, and a Jetson Xavier handles processing and control. The main text describes zero-degree liquid silicone, while the supplementary fabrication section specifies A00-30 silicone; the edition retains both source descriptions without resolving that material inconsistency.
IV-B Diverse Expressions
Figure 4 displays 25 named expressions spanning basic emotions and subtler labels. The gallery demonstrates the available range of facial configurations. Naming a displayed expression does not by itself show that viewers consistently recognize all 25 categories; the classifier experiment below tests four.
IV-C Quantitative Results
Virtual and physical sequences show happy, angry, and fearful delivery of different complete sentences. Their eyebrow and mouth trajectories differ while retaining visible speech articulation. These are qualitative correspondence examples; still frames cannot alone establish temporal synchronization accuracy across the whole utterance.
IV-D Qualitative Results
A video expression classifier [36] evaluates 400 approximately two-second videos: 100 randomly selected sentences paired with each of four expressions. Recognition is 90% happy, 91% angry, 66% disgust, and 73% fear, averaging 80%. Fear and disgust are often confused with one another. These rates describe the classifier, not human recognition or calibrated emotional understanding.
On the digital-animation benchmarks, Morpheus reports lip/emotional-region vertex errors of 2.426/2.389 mm on RAVDESS and 2.491/2.252 mm on HDTF, compared with EmoTalk’s 2.762/2.493 and 2.892/2.364 mm. The listed baselines are higher on these metrics. Vertex error quantifies reconstruction, not the entire subjective quality of expression, and these digital metrics are distinct from the physical face’s landmark error.
Equations 6–7
| (6) |
| (7) |
| True \Predicted | Happy (%) | Angry (%) | Disgust (%) | Fear (%) |
|---|---|---|---|---|
| Happy (%) | 90 | 2 | 3 | 5 |
| Angry (%) | 3 | 91 | 4 | 2 |
| Disgust (%) | 6 | 3 | 66 | 25 |
| Fear (%) | 5 | 2 | 20 | 73 |
IV-E Analysis of Hybrid Actuation Hardware Design
Each hardware configuration generates 400 videos under the same four-expression setup. Mean classifier accuracy is 43% with mouth-only control, 44.75% with eyebrows only, 64.5% with the rigid system, and 80% with hybrid actuation. The paper calls the last improvement mAP in prose, but Table IV reports the arithmetic mean of classification accuracies.
The hybrid design reduces the reported cheek landmark discrepancy from 2.10 to 0.95 mm and nose discrepancy from 1.18 to 0.26 mm relative to rigid-only actuation. Pixel distances are converted using camera-to-face calibration and known facial dimensions. These measurements support the contribution of tendon-driven regions on this platform; they do not establish superiority over all tendon-only hardware.
| Subtle Area | Rigid-Only System (mm) | Hybrid System (mm) |
|---|---|---|
| Cheek | 2.10 | 0.95 |
| Nose | 1.18 | 0.26 |
| Hardware Configuration | Happy (%) | Angry (%) | Disgust (%) | Fear (%) | Mean (%) |
|---|---|---|---|---|---|
| Mouth-Only System | 65 | 41 | 31 | 35 | 43 |
| Eyebrow-Only System | 47 | 50 | 39 | 43 | 44.75 |
| Rigid-Driven System w/o tendon driven | 72 | 75 | 51 | 60 | 64.5 |
| Hybrid-Driven System (Ours) | 90 | 91 | 66 | 73 | 80 |
V Limitations
Adhesive skin attachments make adjustment and debugging difficult; replaceable attachment points are proposed. Skin material and thickness affect expression quality and need further testing. The main limitations section also reports high inference times and identifies near-real-time dialogue as future work. The supplementary component rates do not provide an end-to-end conversational latency measurement resolving that limitation.
VI Conclusion
Hybrid actuation, learned self-modeling, and emotional speech animation jointly produce varied physical facial motions. Classifier results, vertex errors, and a small user study support the tested expressive capability. Broader claims about natural interaction remain limited by selected stimuli, camera-based proxies, incomplete timing detail, and the absence of long-term interaction experiments.
VI-A Emotional Facial Animation Module Training
The supplementary speech-training recipe uses 30 fps video, 16 kHz audio, and 33 blendshape coefficients per frame. Adam training uses learning rate 0.0001, batch size sixteen, and eighty epochs, taking about four hours on one A100 GPU.
VI-B Description of the Mechanical Face Hardware Platform
Figure 7 preserves the complete actuator-location diagram and labels. Motors 31–33 provide neck motion and are omitted from that facial diagram. The text states that 26 motors control learned expressions, consistent with excluding gaze, tongue, and neck motors; its accompanying numbered range includes an eye-gaze index, so that range should not be read as an independently consistent list of the 26 learned controls.
VI-C Face Inverse Learning Module Training
Table V retains all 33 motor control ranges and starting values. Commands are normalized to [0,1] before fitting the forward model with L1 loss, using learning rate 0.00001 and ten to fifteen minutes on an RTX 3090. Several table entries labeled lower and upper bounds appear in decreasing numerical order, reflecting the original presentation; the values are retained as reported rather than silently reordered.
| Motor ID | Control Position | Lower Bound | Start Value | Upper Bound |
|---|---|---|---|---|
| 1 | Left Cheek | 1800 | 1800 | 1400 |
| 2 | Left Nose | 800 | 800 | 0 |
| 3 | Left Eyebrow Center | 980 | 1080 | 1480 |
| 4 | Left Eyebrow Peak | 1100 | 800 | 500 |
| 5 | Left Upper Eyelid | 1700 | 1300 | 600 |
| 6 | Left Lower Eyelid | 1000 | 1200 | 1800 |
| 7 | Eyes Up/Down Gaze | 300 | 1000 | 1500 |
| 8 | Upper Lip Center | 1000 | 1000 | 600 |
| 9 | Left Upper Corner Mouth Movement | 500 | 1000 | 1500 |
| 10 | Upper Lip Right | 1000 | 1000 | 1200 |
| 11 | Tongue Extension/Retraction | 800 | 1000 | 1200 |
| 12 | Tongue Up/Down | 800 | 500 | 300 |
| 13 | Chin (passive) | 1200 | 1600 | 1600 |
| 14 | Left Lower Corner Mouth Movement | 1000 | 1000 | 1500 |
| 15 | Lower Lip Center | 1300 | 1300 | 1900 |
| 16 | Lower Lip Left | 1500 | 1500 | 800 |
| 17 | Lower Lip Right | 800 | 800 | 1200 |
| 18 | Chin (active) | 1200 | 1600 | 1600 |
| 19 | Right Lower Corner Mouth Movement | 1000 | 1000 | 500 |
| 20 | Jaw Left/Right | 400 | 1000 | 1600 |
| 21 | Upper Lip Left | 1000 | 1000 | 800 |
| 22 | Right Upper Corner Mouth Movement | 1500 | 1000 | 500 |
| 23 | Mouth Opening/Closing | 1150 | 1050 | 1050 |
| 24 | Eyes Left/Right Gaze | 800 | 1150 | 1500 |
| 25 | Right Lower Eyelid | 1200 | 1000 | 400 |
| 26 | Right Upper Eyelid | 500 | 1000 | 1600 |
| 27 | Right Eyebrow Peak | 900 | 1200 | 1500 |
| 28 | Right Eyebrow Center | 1300 | 1200 | 800 |
| 29 | Right Nose | 1200 | 1200 | 2000 |
| 30 | Right Cheek | 0 | 0 | 600 |
| 31 | Neck Rotation | 1500 | 1750 | 2000 |
| 32 | Neck Nodding/Shaking (left) | 900 | 1200 | 1300 |
| 33 | Neck Nodding/Shaking (right) | 1200 | 900 | 800 |
VI-D Visualization of Virtual and Real Mechanical Faces with Emotions Speaking Specific Sentences
The supplementary sequences show the same sentence with happy, fearful, angry, and disgusted delivery. Each pairs virtual and physical face frames, exposing both expression changes and residual transfer differences. The full figure is preserved, including every temporal frame and emotion row.
VI-E Generalizability of the Self-Modeling Network.
Four hardware revisions have 17, 16, 23, and 26 expression degrees of freedom. The network is adapted to each actuator configuration and evaluated by landmark error: 1.652, 1.313, 1.206, and 0.748 mm, respectively. This demonstrates applying the self-modeling approach to related hardware variants; it is not evidence that one unchanged trained model transfers zero-shot to new mechanisms.
| Version | V1 | V2 | V3 | V4 |
| Landmark Distance (mm) | 1.652 | 1.313 | 1.206 | 0.748 |
VI-F User Perception Study
Fifty participants, aged 15–50 with balanced gender as reported, compare static expressions, speech-driven clips, and realism. Morpheus is preferred in 84% of static comparison cases with mean rank 1.18; 96% prefer its speech expressiveness over the selected prior system, and its realism score averages 7.6/10. Competitor material comes from published papers and public demonstrations, so recording and stimulus conditions are not matched. The paper does not provide uncertainty intervals or demonstrate generalization to sustained live interaction.
VI-G Hardware Noise Discussion
The supplement reports 45–50 dB for the selected small servos and a 20 dB reduction from a two-millimeter butyl soundproof layer around larger servos. Measurement geometry and a full acoustic protocol are not supplied. These figures should be treated as reported component observations, not a general environmental-noise or health-compliance finding. Lubrication, higher-frequency control, and brushless servos are proposed improvements.
VI-H Material Selection and Expressiveness
The supplementary fabrication recipe specifies A00-30 silicone mixed 1:1, vacuum degassing, casting in a printed mold, curing at 25°C, and an additional resting period. Four tested thicknesses are 2, 2.5, 4, and 5 mm. Images suggest four millimeters provides smoother tendon-driven deformation than the localized thin-skin response or the rigid five-millimeter response; eyelid areas are thinned to two millimeters. This is a qualitative material comparison rather than a systematic durability or force characterization.
VI-I Latency in Real-Time Human-Robot Interaction
The supplement reports about 150 fps inference on Jetson AGX Xavier, virtual animation at 30 fps, and servo updates at 50 Hz. These are component throughput or update rates, with no stated audio window, buffering delay, or full speech-to-motion latency distribution. They therefore do not establish the response delay of a live conversational system.