Literature
Display
← All papers

Automated Synthesis of Facial Mechanisms for Conversational Animatronic Robots

Zongzheng Zhang · Zi Lin · Jiawen Yang · Ziqiao Peng · Junyan Lao · Lin Cheng · Huazhe Xu · Hang Zhao · Hao Zhao

· Robotics: Science and Systems 2026 ↗

Robotics18 min text · CondensedOriginal paper ↗
Contents

Overview

Complete original PDF block S0.F1
Fig. 1: We demonstrate our end-to-end physical conversational face system across diverse multi-round interactions: (a) reenactments of Star Wars (Yoda–Luke Skywalker) and (b) Titanic (Rose–Jack); (c) a dialogue between a real elf and a virtual elf; (d) a three-character encounter from the Chinese folktale Legend of the White Snake; and (e) human–robot dialogue.

At a glance

This edition follows the complete thirty-one-page arXiv v1, including thirteen numbered figures, seven tables, the design algorithm and all seventy-five numbered equations. The catalog’s RSS publication link is retained; the reviewed text and illustrations are explicitly the arXiv version.

A portrait-to-hardware pipeline adapts a modular linkage template to different face geometries, then generates speaking/listening expressions from paired audio and maps them to calibrated robot actuators. The mechanical optimizer achieves collision-free sampled assemblies for ten of fifteen tested heads. A hundred-participant video study favors the complete interaction system over the tested ablations.

The contribution is automated parameter adaptation within a fixed mechanism topology, coupled to conversational animation and physical calibration. The experiments do not establish arbitrary-face manufacturability, continuous-motion collision guarantees, identity-disjoint conversational generalization or measured end-to-end response latency.

I Introduction

Existing expressive robot faces generally require custom mechanical redesign for each geometry. This work makes a reusable template the starting point: reconstruct and scale a face, synthesize module geometry, resolve assembly interference, then generate and retarget interactive motion.

Conversational embodiment requires listener behavior as well as speech articulation. The proposed audio model conditions on both interlocutors and switches between speaking and listening; region-specific regressors convert animation parameters into physical commands. Large-scale personalized deployment is a motivation, not an evaluated fleet-scale result.

II Related Work

II-A Mechanical Face Platform

Pneumatic, tendon, linkage and soft actuation have enabled progressively richer robot faces. Platforms such as EMO and Morpheus demonstrate expressive control, but adapting their geometry remains a mechanical-design problem. The proposed template trades unrestricted architectural choice for repeatable parameter optimization.

II-B Automatic Robot Design

Computational mechanism synthesis, including mechanical characters and linkage-based characters, provides the design precedent. Robot morphology/control optimization additionally considers dynamics and tasks. Facial packaging demands many small mechanisms fit together inside a constrained shell, motivating explicit assembly collision refinement.

II-C Talking Head Generation and Expression Mapping

Audio-driven 3D animation supplies expression coordinates more directly usable by robots than rendered 2D video. DualTalk models reciprocal conversational behavior; learned animatronic mapping such as Smile Like You Mean It addresses the avatar–hardware gap. This system combines paired-audio generation with semantic regional mapping.

III Automated Scalable Mechanism Design

The hardware pipeline initializes a fixed modular architecture from reconstructed facial landmarks, optimizes selected mechanisms independently, and then revisits them after assembly-level collision checks. It optimizes continuous parameters and motion amplitudes rather than searching new linkage topologies.

III-A Parametric Face Template Design

Two-DoF spatial six-bar brow mechanisms control inner-brow and ridge motion. The eyes have eight DoFs: four eyelids plus two rotations per eyeball. Four-bar upper/lower lips, two-input five-bar mouth corners and a four-bar jaw provide mouth motion. The neck combines two parallel pitch/roll inputs with separate yaw.

The hardware section states twenty-one facial actuators, while the explicitly enumerated two brows, eight eye DoFs and seven mouth/jaw outputs sum to nineteen. The source does not reconcile this count; the separate three neck actuators should not silently resolve it.

Complete original PDF block S3.F2
Fig. 2: Mechanical face template. (a) Full assembly of the linkage-driven robotic face with a soft skin and a 3-DoF neck module. (b) Exploded view of the four modular facial mechanisms—eyebrow, eyes, mouth, and jaw.

III-B Hierarchical Automatic Design

The outer loop alternates kinematic synthesis and geometric repair. Small base translations are accepted; excessive corrections instead reduce allowable expression amplitude. Termination at the iteration limit can retain a failed assembly, as the reported failures demonstrate.

Complete original PDF block S3.F3
Fig. 3: Overview of the hierarchical automatic design pipeline. (a) Initialization: From a 2D portrait, we reconstruct a 3D head mesh, semantic landmarks, and initialize module base poses. (b) Inner loop: We perform module-wise kinematic synthesis under anatomy-guided feasible volumes and AU-derived trajectories to maximize expressiveness (illustrated with the mouth-corner five-bar mechanism). (c) Outer loop: We assemble the global CAD model, detect interferences, and apply MTV-guided base-pose updates (or, if necessary, reduce the amplitude limit) until a collision-free assembly is obtained.

III-B 1 Coarse initialization

Metrical face reconstruction produces a mesh and semantic landmarks from the portrait. A chosen physical scale places the template in CAD. Eye and jaw geometry are fixed after initialization; brow and mouth bases use scaled template offsets and their link lengths undergo subsequent optimization.

III-B 2 Module-wise kinematic design optimization

Eight canonical trajectories derived from facial Action Units specify vertical brow/lip motion and raising, lateral and lowering corner motion. Each trajectory has an amplitude coefficient and ten sampled configurations. Regional anatomical envelopes constrain the attachment points.

Equation 1 minimizes inverse amplitude and trajectory error while subtracting a manipulability reward. It jointly fits link lengths, sampled joint configurations and amplitudes. This is a nonconvex, sampled kinematic objective; the term encourages usable local motion without proving a global optimum or modeling silicone elasticity and actuator torque.

Equation 1
min𝚽,𝚯,𝜶\displaystyle\min_{\mathbf{\Phi},\,\mathbf{\Theta},\,\boldsymbol{\alpha}} ℒ=ωamp​ℒamp+ωfit​ℒfit−ωman​ℒman\displaystyle\mathcal{L}=\omega_{\mathrm{amp}}\mathcal{L}_{\mathrm{amp}}+\omega_{\mathrm{fit}}\mathcal{L}_{\mathrm{fit}}-\omega_{\mathrm{man}}\mathcal{L}_{\mathrm{man}} (1)
s.t.\displaystyle\text{s.t.} 𝐏(𝜽k,m)⊂Ωfeas,∀m∈{1,…,M},\displaystyle\mathbf{P}(\boldsymbol{\theta}_{k,m})\subset\Omega_{\mathrm{feas}},\quad\forall m\in\{1,\dots,M\},
0<αk≤αlimitk,∀k∈{1,…,K},\displaystyle 0<\alpha_{k}\leq\alpha^{\mathrm{limit}}_{k},\quad\forall k\in\{1,\dots,K\},
Complete original PDF block alg1
Algorithm 1 Hierarchical Automatic Design
Original selectable algorithm
1: 2D portrait image II; template face design
2: Final assembly 𝒜\mathcal{A}
3: (ℳ,𝐏)←Reconstruct3D​(I)(\mathcal{M},\mathbf{P})\leftarrow\textsc{Reconstruct3D}(I)
4: {𝐩base}←CoarseInitialization​(ℳ,𝐏)\{\mathbf{p}^{\mathrm{base}}\}\leftarrow\textsc{CoarseInitialization}(\mathcal{M},\mathbf{P}) ⊳\triangleright Initialization
5: for iteration s=1s=1 to SS do
6:   for mechanism k∈{brow,mouth}k\in\{\mathrm{brow},\mathrm{mouth}\} do ⊳\triangleright Inner loop
7:    {𝐓k}←GlobalToLocal​({𝐓k},𝐩kbase)\{\mathbf{T}_{k}\}\leftarrow\textsc{GlobalToLocal}(\{\mathbf{T}_{k}\},\mathbf{p}_{k}^{\mathrm{base}})
8:    (𝚽k,𝚯k,𝜶k)←SolveEq. 1​(Ωfeas,{𝐓k},{αklimit})(\mathbf{\Phi}_{k},\mathbf{\Theta}_{k},\boldsymbol{\alpha}_{k})\leftarrow\textsc{SolveEq.~\ref{eq:innerloop_overall_obj_constrained}}(\Omega_{\mathrm{feas}},\{\mathbf{T}_{k}\},\{\alpha_{k}^{\mathrm{limit}}\})
9:   end for
10:   αklimit←min⁡(αklimit,αk∗),∀k\alpha_{k}^{\mathrm{limit}}\leftarrow\min(\alpha_{k}^{\mathrm{limit}},\alpha_{k}^{*}),\ \forall k ⊳\triangleright Outer loop
11:   𝒜←AssembleCAD​({𝐩kbase},{𝚽k})\mathcal{A}\leftarrow\textsc{AssembleCAD}(\{\mathbf{p}_{k}^{\mathrm{base}}\},\{\mathbf{\Phi}_{k}\})
12:   𝒞←CollisionDetection​(𝒜,{𝚯k})\mathcal{C}\leftarrow\textsc{CollisionDetection}(\mathcal{A},\{\mathbf{\Theta}_{k}\})
13:   if 𝒞=∅\mathcal{C}=\emptyset then
14:    break
15:   end if
16:   for each mechanism kk with constraints 𝒞k\mathcal{C}_{k} do
17:    ({𝐧j,δj}j∈𝒞k)←MTV​(𝒞k)(\{\mathbf{n}_{j},\delta_{j}\}_{j\in\mathcal{C}_{k}})\leftarrow\textsc{MTV}(\mathcal{C}_{k})
18:    Δ​𝐩k←SolveEq. 2​({𝐧j,δj}j∈𝒞k,ϵ)\Delta\mathbf{p}_{k}\leftarrow\textsc{SolveEq.~\ref{eq:outloop_optimization}}(\{\mathbf{n}_{j},\delta_{j}\}_{j\in\mathcal{C}_{k}},\epsilon)
19:    if ‖Δ​𝐩k‖2>Dmax\|\Delta\mathbf{p}_{k}\|_{2}>D_{\max} then ⊳\triangleright Amplitude scheduling
20:       αklimit←γ​αklimit\alpha_{k}^{\mathrm{limit}}\leftarrow\gamma\,\alpha_{k}^{\mathrm{limit}}
21:       continue ⊳\triangleright Skip update, re-optimization next iter
22:    end if
23:    𝐩kbase←𝐩kbase+Δ​𝐩k\mathbf{p}^{\mathrm{base}}_{k}\leftarrow\mathbf{p}^{\mathrm{base}}_{k}+\Delta\mathbf{p}_{k} ⊳\triangleright Pose adjustment
24:   end for
25: end for
26: return 𝒜\mathcal{A}

III-B 3 Global CAD assembly and collision-driven refinement

FCL detects assembly interference along the sampled motion. Each contact yields a separation normal and penetration depth; a quadratic program finds the minimum-norm base translation satisfying all linearized clearance constraints. A shift above three millimetres instead multiplies the amplitude limit by 0.9 and triggers reoptimization.

At most ten outer iterations use a 0.5 mm clearance. New interference can follow a local repair, and opposing constraints can make the translation problem infeasible. Checking ten trajectory waypoints does not certify continuous paths or arbitrary combinations of actuator commands.

Equation 2
minΔ​𝐩k\displaystyle\min_{\Delta\mathbf{p}_{k}} ‖Δ​𝐩k‖22s.t.\displaystyle\|\Delta\mathbf{p}_{k}\|^{2}_{2}\quad\text{s.t.}\quad 𝐧j⊤Δ𝐩k≥δj+ϵ,∀j∈𝒞k,\displaystyle\mathbf{n}_{j}^{\top}\Delta\mathbf{p}_{k}\geq\delta_{j}+\epsilon,\quad\forall j\in\mathcal{C}_{k}, (2)

IV Interaction Synthesis and Control

The control pipeline separates conversational motion generation from embodiment-specific inverse mapping. The first predicts an animated face; the second uses calibration data for each manufactured head, while analytical neck inverse kinematics handles head pose.

Complete original PDF block S4.F4
Fig. 4: Overview of interaction synthesis and control. (a) Given dual-speaker audio, the talking-head model predicts multi-round facial motion for both speaking and listening. (b) The predicted coefficients are mapped to robot motor commands via region-wise regressors and neck IK, enabling physically feasible multi-round expressions on the mechanical face.

IV-A Interactive Talking Head Synthesis

A shared wav2vec 2.0 audio encoder and linear projection produce two aligned feature streams. Temporal/style embeddings precede intensity-derived gating: the target’s audio dominates while it speaks and the partner’s audio dominates while it listens. A transformer predicts fifty-two blendshapes and three head angles.

Training uses paired conversational video with MediaPipe motion labels. The mostly nonoverlapping-turn assumption simplifies gating; overlapped speech may be imperfectly represented by a single weighted mixture. Unlike the cited DualTalk setup, partner future motion is not an input. The supplied architecture nevertheless does not specify a complete causal masking/window/lookahead protocol or measured end-to-end latency.

Equation 3
𝐙A=𝐖a​(Eaud​(𝐀A)),𝐙B=𝐖a​(Eaud​(𝐀B)),\mathbf{Z}_{A}=\mathbf{W}_{a}\big(E_{\mathrm{aud}}(\mathbf{A}_{A})\big),\qquad\mathbf{Z}_{B}=\mathbf{W}_{a}\big(E_{\mathrm{aud}}(\mathbf{A}_{B})\big), (3)

IV-B Human-to-Robot Facial Expression Mapping

Facial coefficients are partitioned into eyebrow, eye, mouth and jaw regions. Separate MLPs learn motor commands from corresponding coefficients, while neck angles use inverse kinematics. For each head, three thousand Latin-hypercube motor configurations are photographed and labeled with MediaPipe after actuation bounds are calibrated.

This is supervised per-head inverse mapping, not zero-shot transfer to an uncalibrated mechanism. Individual coefficient bounds or actuator limits do not by themselves guarantee feasibility of every combined facial expression.

V Experiment

V-A Experimental Setup

The fabricated heads use PLA components, silicone skin, 9 g facial servos, three NEMA 17 neck motors and a Jetson AGX Orin with 32 GB memory. Commands update at 30 Hz through the stated I²C/PWM interface. The twenty-one facial-actuator claim is retained with the module-count qualification above.

V-B Evaluation of the Automatic Design Flow

Across fifteen heads, success is 20% for local-only synthesis, 0% for joint optimization, 33.3% for heuristic repair and 66.7% for the proposed hierarchy. Successful-run convergence times are 25.7, unavailable, 1098.2 and 591.3 seconds respectively; expressiveness is 8.69, unavailable, 8.57 and 8.53.

Convergence time and expressiveness include only successful runs, hence different successful head subsets. Intersection volume averages use zero for successes and final residual overlap for failures. The reported 1.9× runtime advantage over heuristic repair is conditional on success, not an all-head time-to-valid-design comparison.

Two senior engineers required an average 22.8 hours to design one compact head manually, versus 11.7 minutes of optimization. This selected design-time comparison excludes printing, skin fabrication and assembly. Qualitative corner-motion improvement does not establish the same gain for every geometry.

Complete original PDF block S5.T1
TABLE I: Comparison of optimization strategies for automatic mechanism design. We report success rate (SR), convergence time (CT), expressiveness score (Exp =∑kαk=\sum_{k}\alpha_{k}), and intersection volume (IV). CT and Exp are reported only for successful runs; IV is 00 for successful runs and otherwise computed from the final design at the maximum iteration.
Original selectable table
Method SR (%)(\%) ↑\uparrow CT (s) ↓\downarrow Exp ↑\uparrow IV (m​m3)(mm^{3}) ↓\downarrow
Ours w/o outer loop 20.0 25.7 8.69 2050.7
Global Joint-Opt 0.0 – – 9690.5
Local + Heuristic 33.3 1098.2 8.57 1690.6
Ours 66.7 591.3 8.53 460.4
Complete original PDF block S5.F5
Fig. 5: (a) Algorithm design versatility: CAD Assemblies for Jack and Yoda. (b) Algorithm vs. Manual: Comparison of mouth-corner mechanisms (red circles) on a compact head.

V-C Evaluation of Conversational Motion Synthesis

The authors train FaceFormer, EmoTalk, SelfTalk, DualTalk and the proposed model on their dataset. The proposed speaker MSE/PDD are 0.38/5.63 and listener FDD/PDD 21.12/5.81, improving those reported comparisons. Its LVE 3.04 is worse than EmoTalk’s 2.98, and SID 3.29 is below DualTalk’s 3.36.

The table labels real-time support as a capability distinction; it does not supply complete pipeline latency. Motion-distribution statistics and coefficient error are complementary metrics, not direct measurements of conversational understanding or physical interaction safety.

Complete original PDF block S5.T2
TABLE II: Comparison under speaker and listener modes on our dataset. “RT” means whether real-time use is supported.
Original selectable table
Method Speaker Listener RT
LVE ↓\downarrow MSE ↓\downarrow PDD ↓\downarrow SID ↑\uparrow FDD ↓\downarrow PDD ↓\downarrow
FaceFormer 4.25 0.69 7.32 0.71 – – ✓
EmoTalk 2.98 0.68 6.88 3.03 – – ✓
SelfTalk 5.34 0.73 7.03 2.98 – – ✓
DualTalk 3.79 0.57 8.74 3.36 29.77 9.45 ✗
Ours 3.04 0.38 5.63 3.29 21.12 5.81 ✓

V-D Evaluation of Expression Mapping Results

Region-wise mapping gives LSE-D 10.61 and LSE-C 3.19, compared with 12.68/2.58 for a global blendshape regressor and weaker landmark/FLAME baselines. Its 2396.31 FPS is a mapping-network measurement on Orin, not the full audio-to-motion rate or the 30 Hz physical command rate. Global blendshape and FLAME regressors are slightly faster.

Landmark methods include rendering and re-detection, unlike coefficient-based inputs, so these timings compare complete stated mapping routes rather than matched neural-network compute. LSE searches temporal offsets, and therefore cannot independently establish low response latency.

Complete original PDF block S5.T3
TABLE III: Comparison for mapping grouped by facial representations. FPS is measured on Jetson AGX Orin.
Original selectable table
Metric Landmarks FLAME Blendshapes
NNR Norm. [10] + MLP Norm. [41] + MLP MLP MLP Ours
LSE-D ↓\downarrow 15.59 14.52 13.74 13.06 12.68 10.61
LSE-C ↑\uparrow 0.34 1.94 2.39 2.42 2.58 3.19
FPS ↑\uparrow 6.36 6.31 6.19 2408.55 2478.26 2396.31

V-E End-to-End Conversation Demonstration

Demonstrations combine Doubao Voice dialogue with the facial pipeline and stage dyadic fictional-character scenes. Figures retain speaking and listening expressions across human–robot, robot–robot and avatar–robot settings. These are selected interaction examples, not a quantitative language-by-language robustness test.

The time breakdown totals 25.84 hours by overlapping 21.47 hours of printing with 20.50 hours of skin production. Other stages are 0.16 hours optimization, 3.83 assembly, 0.25 calibration collection and 0.13 mapping training. Sequentially adding every stage gives 46.34 hours; the reported total explicitly depends on parallel fabrication.

Complete original PDF block S5.F6
Fig. 6: End-to-end system visualization across interaction scenarios. (a) Human-Robot Interaction: Real-time conversational gestures during a user-robot dialogue. The top row is the expression synthesis. (b) Dyadic Role-Play (Titanic): Reenacting the scene dynamics with expressions corresponding to the dialogue between Rose (speaker) and Jack (listener).
Complete original PDF block S5.T4
TABLE IV: End-to-end time breakdown. 1: automatic design optimization; 2: 3D printing; 3: silicone face skin; 4: assembly; 5: mapping dataset collection; 6: mapping network training. The total time assumes parallel execution of Stages 2 and 3.
Original selectable table
Stage 1 2 3 4 5 6 Total
Time (h) 0.16 21.47 20.50 3.83 0.25 0.13 25.84

V-F User study

A hundred participants view reference clips and randomized, method-blinded robot variants. The complete system receives the highest mean ratings across four interaction excerpts and three mapping sentences. The comparisons support the tested contribution of listener cues, coordinated neck motion and full-face mapping; no variability intervals or inferential tests accompany these reported means.

Complete original PDF block S5.F7
Fig. 7: User study results. (a) Overall interaction performance on four excerpts. (b) Mapping quality on three sentences.

VI Conclusion

The work connects automated adaptation of a reusable facial mechanism with paired-audio motion and per-head retargeting. Its demonstrated scope is a fifteen-head design experiment, selected physical dialogues and short video-based perceptual evaluations. More general geometry, material-aware synthesis and broader interaction quality remain open.

-A Overview

The appendix supplies kinematic derivations, optimization and network settings, fabrication details, extended demonstrations, demographic counts and limitations. These details clarify the sampled nature of feasibility checks, calibration requirements and the distinction between network throughput and physical interaction timing.

-B Extended Related Work

-B 1 Mechanical Face Platform

The historical survey spans FACS-oriented actuation, compliant materials, full humanoids and high-fidelity android faces. Prior platforms illustrate several viable architectures, but each has geometry-specific mechanisms and calibration. The proposed optimizer addresses reuse within its own template; the survey is not an empirical comparison against every listed platform.

-B 2 Automatic Robot Design

Linkage synthesis and motion retargeting motivate parameterized mechanical design. Dynamics-aware co-optimization elsewhere solves different task and packaging problems. Facial methods that tune actuator locations or skin properties motivate combining trajectory fitting with assembly collision checks here.

-B 3 Talking Head Generation and Expression Mapping

Three-dimensional speech animation has progressed from lip motion to emotional, cross-identity and dyadic behavior. Physical retargeting uses landmarks, action semantics and learned inverse models. The present modular pipeline retains intermediate coefficients, making errors and calibration needs at both stages relevant.

-C Mechanism Kinematic Analysis

The derivations give forward models for linkage synthesis and inverse models for the neck. Mobility counts describe generic configurations; closed-loop branch selection, singularities and geometric feasibility still matter. Every original displayed relation is retained.

Complete original PDF block A0.F1
Fig. A.1: Detailed mechanical of the facial robot template.
Complete original PDF block A0.F2
Fig. A.2: Kinematic schematics of the facial actuation modules. The CAD renderings are overlaid with simplified linkage diagrams, where labeled nodes denote joints and lil_{i} denote link lengths. (a) Eyebrow module: a symmetric six-bar linkage (left/right share the same design). (b) Eyes module: two four-bar linkages, with the left linkage actuating eyelid blinking and the right linkage actuating eyeball rotation. (c) Mouth module: a four-bar linkage for lip motion and a five-bar linkage for mouth-corner actuation. (d) Jaw module: a four-bar linkage that controls jaw opening/closing.

-C 1 Eyebrow module

The brow loop has three revolute and three spherical joints. The stated spatial mobility count is six; removing two passive axial twists and imposing two attachment-plane constraints yields two effective DoFs, assuming independent constraints.

Given two motor angles, six unknown coordinates for joints C and D satisfy three link-distance constraints, a revolute-axis constraint and two planar skin-attachment constraints. Newton–Raphson solves this closure system using the previous configuration as initialization. Its 6×6 closure Jacobian is monitored through determinant or minimum singular value. Local solvability and branch continuation require nonsingularity; a mobility count alone does not prove a globally unique assembly.

Equations A.1–A.16
𝒫:={𝐱∈ℝ3∣𝐧⊤​𝐱+d=0},\mathcal{P}:=\{\mathbf{x}\in\mathbb{R}^{3}\mid\mathbf{n}^{\top}\mathbf{x}+d=0\}, (A.1)
Mraw=6​(n−1−j)+∑i=1jfi,M_{\mathrm{raw}}=6(n-1-j)+\sum_{i=1}^{j}f_{i}, (A.2)
Mraw=6​(6−1−6)+(3⋅1+3⋅3)=6.M_{\mathrm{raw}}=6(6-1-6)+(3\cdot 1+3\cdot 3)=6. (A.3)
Meff=4−2=2,M_{\mathrm{eff}}=4-2=2, (A.4)
𝐛⁡(φ1)=𝐚+𝐑A​(φ1)​𝐫A​B,\mathbf{b}(\varphi_{1})=\mathbf{a}+\mathbf{R}_{A}(\varphi_{1})\,\mathbf{r}_{AB}, (A.5)
𝐟⁡(φ2)=𝐞+𝐑E​(φ2)​𝐫E​F,\mathbf{f}(\varphi_{2})=\mathbf{e}+\mathbf{R}_{E}(\varphi_{2})\,\mathbf{r}_{EF}, (A.6)
Φ1​(𝐜)=‖𝐜−𝐛⁡(φ1)‖22−lb​22=0,\Phi_{1}(\mathbf{c})=\|\mathbf{c}-\mathbf{b}(\varphi_{1})\|_{2}^{2}-l_{b2}^{2}=0, (A.7)
Φ2​(𝐜,𝐝)=‖𝐝−𝐜‖22−lb​32=0,\Phi_{2}(\mathbf{c},\mathbf{d})=\|\mathbf{d}-\mathbf{c}\|_{2}^{2}-l_{b3}^{2}=0, (A.8)
Φ3​(𝐝)=‖𝐝−𝐟⁡(φ2)‖22−lb​42=0.\Phi_{3}(\mathbf{d})=\|\mathbf{d}-\mathbf{f}(\varphi_{2})\|_{2}^{2}-l_{b4}^{2}=0. (A.9)
Φ4​(𝐝)=𝐬^F​(φ2)⊤​(𝐝−𝐟⁡(φ2))−hF=0.\Phi_{4}(\mathbf{d})=\hat{\mathbf{s}}_{F}(\varphi_{2})^{\top}\big(\mathbf{d}-\mathbf{f}(\varphi_{2})\big)-h_{F}=0. (A.10)
𝐩⁡(𝐜,𝐝)=𝐜+ηP​(𝐝−𝐜),𝐪⁡(𝐜,𝐝)=𝐜+ηQ​(𝐝−𝐜).\mathbf{p}(\mathbf{c},\mathbf{d})=\mathbf{c}+\eta_{P}(\mathbf{d}-\mathbf{c}),\qquad\mathbf{q}(\mathbf{c},\mathbf{d})=\mathbf{c}+\eta_{Q}(\mathbf{d}-\mathbf{c}). (A.11)
Ψ1​(𝐜,𝐝)=𝐧⊤​𝐩​(𝐜,𝐝)+d=0,\Psi_{1}(\mathbf{c},\mathbf{d})=\mathbf{n}^{\top}\mathbf{p}(\mathbf{c},\mathbf{d})+d=0, (A.12)
Ψ2​(𝐜,𝐝)=𝐧⊤​𝐪​(𝐜,𝐝)+d=0.\Psi_{2}(\mathbf{c},\mathbf{d})=\mathbf{n}^{\top}\mathbf{q}(\mathbf{c},\mathbf{d})+d=0. (A.13)
𝐅⁡(𝐱,𝐪)=𝟎,𝐱:=[𝐜⊤𝐝⊤]⊤,𝐪:=[φ1φ2]⊤,\mathbf{F}(\mathbf{x};\mathbf{q})=\mathbf{0},\qquad\mathbf{x}:=\begin{bmatrix}\mathbf{c}^{\top}&\mathbf{d}^{\top}\end{bmatrix}^{\top},\;\;\mathbf{q}:=\begin{bmatrix}\varphi_{1}&\varphi_{2}\end{bmatrix}^{\top}, (A.14)
𝐱(t+1)=𝐱(t)−𝐉​(𝐱(t),𝐪)−1​𝐅​(𝐱(t),𝐪),\mathbf{x}^{(t+1)}=\mathbf{x}^{(t)}-\mathbf{J}(\mathbf{x}^{(t)};\mathbf{q})^{-1}\,\mathbf{F}(\mathbf{x}^{(t)};\mathbf{q}), (A.15)
𝐉​(𝐱,𝐪)=∂𝐅∂𝐱​(𝐱,𝐪).\mathbf{J}(\mathbf{x};\mathbf{q})=\frac{\partial\mathbf{F}}{\partial\mathbf{x}}(\mathbf{x};\mathbf{q}). (A.16)

-C 2 Eyes module

Each eye uses four four-bar linkages: two eyelids and two eyeball rotation inputs. The bilateral assembly therefore has eight DoFs, allowing independent coordinated gaze and eyelid control. The paper does not supply an equally detailed equation-by-equation eye closure derivation.

-C 3 Mouth module

Upper/lower lips use planar four-bar circle-intersection solutions followed by a fixed follower-link attachment offset. The two intersection branches correspond to alternate assemblies, selected by continuity and joint limits. A positive center separation and feasible link lengths are required; clamping a negative square-root argument to zero is a numerical convention, not proof that an infeasible loop closes.

Each mouth corner is a two-input planar five-bar mechanism. Two driven crank endpoints define a second circle intersection, with the explicit triangle inequality limiting reachability. A rigid extension maps the floating joint to the skin attachment. Varying link ratios, input angles and base orientation changes the feasible motion region used in synthesis.

Equations A.17–A.32
xBm=xAm+lm​1​cos⁡γ,yBm=yAm+lm​1​sin⁡γ.x_{B}^{m}=x_{A}^{m}+l_{m1}\cos\gamma,\qquad y_{B}^{m}=y_{A}^{m}+l_{m1}\sin\gamma. (A.17)
(xCm−xBm)2+(yCm−yBm)2=lm​22,(x_{C}^{m}-x_{B}^{m})^{2}+(y_{C}^{m}-y_{B}^{m})^{2}=l_{m2}^{2}, (A.18)
(xCm−xDm)2+(yCm−yDm)2=lm​32.(x_{C}^{m}-x_{D}^{m})^{2}+(y_{C}^{m}-y_{D}^{m})^{2}=l_{m3}^{2}. (A.19)
a=lm​22−lm​32+s22​s,h=max⁡(lm​22−a2, 0),a=\frac{l_{m2}^{2}-l_{m3}^{2}+s^{2}}{2s},\qquad h=\sqrt{\max(l_{m2}^{2}-a^{2},\,0)}, (A.20)
𝐜=𝐛+as​𝐫±hs​𝐫⟂,𝐫⟂=[−ry,rx]⊤,\mathbf{c}=\mathbf{b}+\frac{a}{s}\mathbf{r}\ \pm\ \frac{h}{s}\mathbf{r}_{\perp},\qquad\mathbf{r}_{\perp}=[-r_{y},\ r_{x}]^{\top}, (A.21)
θC​D=atan2⁡(yCm−yDm,xCm−xDm).\theta_{CD}=\mathrm{atan2}(y_{C}^{m}-y_{D}^{m},\ x_{C}^{m}-x_{D}^{m}). (A.22)
xEm=xDm+lm​5​cos⁡(θC​D+δ),yEm=yDm+lm​5​sin⁡(θC​D+δ).x_{E}^{m}=x_{D}^{m}+l_{m5}\cos(\theta_{CD}+\delta),\qquad y_{E}^{m}=y_{D}^{m}+l_{m5}\sin(\theta_{CD}+\delta). (A.23)
lm​10=(xFm−xJm)2+(yFm−yJm)2,l_{m10}=\sqrt{(x_{F}^{m}-x_{J}^{m})^{2}+(y_{F}^{m}-y_{J}^{m})^{2}}, (A.24)
Γ=atan2⁡(yJm−yFm,xJm−xFm).\Gamma=\mathrm{atan2}\!\left(y_{J}^{m}-y_{F}^{m},\;x_{J}^{m}-x_{F}^{m}\right). (A.25)
𝐫G=[xGmyGm]=[xFmyFm]+lm​6​[cos⁡(Γ+γ2)sin⁡(Γ+γ2)],\mathbf{r}_{G}=\begin{bmatrix}x_{G}^{m}\\ y_{G}^{m}\end{bmatrix}=\begin{bmatrix}x_{F}^{m}\\ y_{F}^{m}\end{bmatrix}+l_{m6}\begin{bmatrix}\cos(\Gamma+\gamma_{2})\\ \sin(\Gamma+\gamma_{2})\end{bmatrix}, (A.26)
𝐫I=[xImyIm]=[xJmyJm]+lm​9​[cos⁡(Γ+γ3)sin⁡(Γ+γ3)].\mathbf{r}_{I}=\begin{bmatrix}x_{I}^{m}\\ y_{I}^{m}\end{bmatrix}=\begin{bmatrix}x_{J}^{m}\\ y_{J}^{m}\end{bmatrix}+l_{m9}\begin{bmatrix}\cos(\Gamma+\gamma_{3})\\ \sin(\Gamma+\gamma_{3})\end{bmatrix}. (A.27)
‖𝐫H−𝐫G‖2=lm​7,‖𝐫H−𝐫I‖2=lm​8.\|\mathbf{r}_{H}-\mathbf{r}_{G}\|_{2}=l_{m7},\qquad\|\mathbf{r}_{H}-\mathbf{r}_{I}\|_{2}=l_{m8}. (A.28)
|lm​7−lm​8|≤‖𝐫I−𝐫G‖2≤lm​7+lm​8.|l_{m7}-l_{m8}|\leq\|\mathbf{r}_{I}-\mathbf{r}_{G}\|_{2}\leq l_{m7}+l_{m8}. (A.29)
a=lm​72−lm​82+s22​s,h=max⁡(lm​72−a2, 0),a=\frac{l_{m7}^{2}-l_{m8}^{2}+s^{2}}{2s},\qquad h=\sqrt{\max(l_{m7}^{2}-a^{2},\,0)}, (A.30)
𝐫H=𝐫G+as​𝐫±hs​𝐫⟂,𝐫⟂=[−ryrx],\mathbf{r}_{H}=\mathbf{r}_{G}+\frac{a}{s}\mathbf{r}\ \pm\ \frac{h}{s}\mathbf{r}_{\perp},\qquad\mathbf{r}_{\perp}=\begin{bmatrix}-r_{y}\\ r_{x}\end{bmatrix}, (A.31)
𝐫K=𝐫H+lm​8+lm​11lm​8​(𝐫I−𝐫H).\mathbf{r}_{K}=\mathbf{r}_{H}+\frac{l_{m8}+l_{m11}}{l_{m8}}\left(\mathbf{r}_{I}-\mathbf{r}_{H}\right). (A.32)

-C 4 Jaw module

Jaw opening uses a compact four-bar mechanism with two ground pivots. Its placement and geometry are fixed after coarse initialization rather than optimized together with the brow/lip/corner mechanisms.

-C 5 Neck module

The neck assumes a fixed platform origin with rotational motion. In a yaw-compensated frame, each crank-and-rod closure reduces to a scalar sinusoidal equation for one motor angle. A real solution requires the normalized right-hand side to lie within ±1 and a nonzero sinusoid amplitude.

Each side admits two inverse-sine branches; the controller chooses a branch inside joint limits nearest the previous command. At zero amplitude the closure is either infeasible or underdetermined, and the authors exclude that singular case from nominal operation. The original Euler-order and coordinate equations remain visible; the notes do not substitute a different convention.

Complete original PDF block A0.F3
Fig. A.3: Neck module mechanism.
Equations A.33–A.47
𝐑⁡(θ,φ,ψ)=𝐑x​(φ)​𝐑y​(θ)​𝐑z​(ψ),\mathbf{R}(\theta,\varphi,\psi)=\mathbf{R}_{x}(\varphi)\,\mathbf{R}_{y}(\theta)\,\mathbf{R}_{z}(\psi), (A.33)
𝐂=𝐌+𝐑​𝐜M,𝐃=𝐌+𝐑​𝐝M,\mathbf{C}=\mathbf{M}+\mathbf{R}\,{}^{M}\mathbf{c},\qquad\mathbf{D}=\mathbf{M}+\mathbf{R}\,{}^{M}\mathbf{d}, (A.34)
𝐀⁡(α)=𝐎1+[0r​cos⁡αr​sin⁡α].\mathbf{A}(\alpha)=\mathbf{O}_{1}+\begin{bmatrix}0\\ r\cos\alpha\\ r\sin\alpha\end{bmatrix}. (A.35)
‖𝐂−𝐎1−𝐫1​(α)‖2=l12,𝐫1​(α):=[0r​cos⁡αr​sin⁡α].\|\mathbf{C}-\mathbf{O}_{1}-\mathbf{r}_{1}(\alpha)\|^{2}=l_{1}^{2},\qquad\mathbf{r}_{1}(\alpha):=\begin{bmatrix}0\\ r\cos\alpha\\ r\sin\alpha\end{bmatrix}. (A.36)
Δ​y​cos⁡α+Δ​z​sin⁡α=Z1,\Delta y\cos\alpha+\Delta z\sin\alpha=Z_{1}, (A.37)
Z1:=Δ​x2+Δ​y2+Δ​z2+r2−l122​r.Z_{1}:=\frac{\Delta x^{2}+\Delta y^{2}+\Delta z^{2}+r^{2}-l_{1}^{2}}{2r}. (A.38)
K1:=Δ​y2+Δ​z2,ϕ1:=atan2⁡(Δ​y,Δ​z),K_{1}:=\sqrt{\Delta y^{2}+\Delta z^{2}},\qquad\phi_{1}:=\operatorname{atan2}(\Delta y,\Delta z), (A.39)
K1​sin⁡(α+ϕ1)=Z1.K_{1}\sin(\alpha+\phi_{1})=Z_{1}. (A.40)
|Z1K1|≤1andK1>0.\left|\frac{Z_{1}}{K_{1}}\right|\leq 1\quad\text{and}\quad K_{1}>0. (A.41)
α1=arcsin⁡(Z1K1)−ϕ1,α2=π−arcsin⁡(Z1K1)−ϕ1.\alpha_{1}=\arcsin\!\left(\frac{Z_{1}}{K_{1}}\right)-\phi_{1},\qquad\alpha_{2}=\pi-\arcsin\!\left(\frac{Z_{1}}{K_{1}}\right)-\phi_{1}. (A.42)
Δ​𝐃:=𝐃−𝐎2=[Δ¯​x,Δ¯​y,Δ¯​z]⊤,\Delta\mathbf{D}:=\mathbf{D}-\mathbf{O}_{2}=[\bar{\Delta}x,\bar{\Delta}y,\bar{\Delta}z]^{\top}, (A.43)
Δ¯​y​cos⁡β+Δ¯​z​sin⁡β=Z2,\bar{\Delta}y\cos\beta+\bar{\Delta}z\sin\beta=Z_{2}, (A.44)
Z2:=Δ¯​x2+Δ¯​y2+Δ¯​z2+r2−l222​r.Z_{2}:=\frac{\bar{\Delta}x^{2}+\bar{\Delta}y^{2}+\bar{\Delta}z^{2}+r^{2}-l_{2}^{2}}{2r}. (A.45)
K2:=Δ¯​y2+Δ¯​z2,ϕ2:=atan2⁡(Δ¯​y,Δ¯​z),K_{2}:=\sqrt{\bar{\Delta}y^{2}+\bar{\Delta}z^{2}},\qquad\phi_{2}:=\operatorname{atan2}(\bar{\Delta}y,\bar{\Delta}z), (A.46)
β1=arcsin⁡(Z2K2)−ϕ2,β2=π−arcsin⁡(Z2K2)−ϕ2.\beta_{1}=\arcsin\!\left(\frac{Z_{2}}{K_{2}}\right)-\phi_{2},\qquad\beta_{2}=\pi-\arcsin\!\left(\frac{Z_{2}}{K_{2}}\right)-\phi_{2}. (A.47)

-C 6 Architecture matters

Applying the same synthesis procedure to Morpheus yields 2/15 successes. Its tendon components are excluded because the current framework supports linkages only. This is evidence about compatibility of the retained linkage subset with this optimizer, not a comparison of complete Morpheus capabilities against the new hardware.

-D Optimization Algorithm Details

-D 1 Initialization of geometric parameters and layout

Brow bases follow scaled eye-to-brow template offsets. Eye centers and radii are fitted from landmark spheres, then adjusted for skin, eyelid and assembly clearance. Lip/corner attachments align with corresponding landmarks while initially retaining template link lengths. The jaw follows a scaled eye-to-jaw offset and remains fixed.

These initialization rules encode geometric priors; reconstruction, chosen physical scaling and the assumption that template relative placement transfers to a new face remain inputs to success.

Complete original PDF block A0.F4
Fig. A.4: (a) AU-guided target motions. Canonical AU-derived motion directions for the optimized facial attachment points (eyebrow, lips, and mouth corners), defining the target trajectory primitives used in the inner-loop synthesis. (b) Anatomy-guided feasible volumes. Region-specific anatomically plausible feasible volumes that bound the optimizer’s search space, including the eyebrow volume, lip volume, and mouth-corner volume.

-D 2 Determine anatomy-guided feasible motion volumes

Brow attachment motion lies between planes 10 mm below and 50 mm above the lowest eyebrow landmark and the local face boundary. A cheek plane extruded 50 mm inward bounds corner motion. Lips remain within ±10 mm of the midsagittal plane and no more than 30 mm above neutral.

These envelopes constrain geometric motion; they are not a measured silicone deformation, stress or torque model. They retain sufficient freedom for search without proving that every enclosed configuration is physically comfortable or dynamically realizable.

-D 3 Determine AU-derived trajectories

Inner-brow and brow-peak attachments follow vertical targets, as do upper/lower lip midpoints. Corner targets span upward raising, lateral translation and downward pulling. Optimization scales these prescribed directions, rather than learning arbitrary expressions or changing mechanism topology.

-D 4 Optimization parameters

Table A.1 specifies eight trajectories, ten waypoints, amplitude floor 0.1 and initial ceiling 2.0. Amplitude, fit and manipulability weights are 1, 10 and 0.1. Inner L-BFGS uses at most one hundred iterations with tolerance 10⁻⁶; outer repair uses at most ten rounds, 0.5 mm clearance, a 3 mm shift threshold and 0.9 amplitude decay.

Complete original PDF block A0.T1
TABLE A.1: Hyperparameters for hierarchical optimization.
Original selectable table
Hyperparameter Meaning Value
β\beta Global mesh scaling factor used to bring the reconstructed head to metric scale. per-head
KK Number of canonical AU-derived target trajectories. 8
MM Number of sampled waypoints per trajectory (also used for collision checking along motion). 10
αmin\alpha_{\min} Lower bound on amplitude coefficient αk\alpha_{k} to avoid numerical instability. 0.1
αinitlimit\alpha^{\mathrm{limit}}_{\mathrm{init}} Initial amplitude limit αklimit\alpha_{k}^{\mathrm{limit}} before outer-loop scheduling. 2.0
ωamp\omega_{\mathrm{amp}} Weight for amplitude term ℒamp\mathcal{L}_{\mathrm{amp}} (encourages larger expression range). 1.0
ωfit\omega_{\mathrm{fit}} Weight for trajectory fitting term ℒfit\mathcal{L}_{\mathrm{fit}} (tracking accuracy). 10
ωman\omega_{\mathrm{man}} Weight for manipulability term ℒman\mathcal{L}_{\mathrm{man}} (avoids kinematic singularities / dead zones). 0.1
SS Maximum number of outer-loop refinement iterations. 10
ϵ\epsilon Collision-resolution clearance in QP constraints. 0.5 mm
DmaxD_{\max} Maximum accepted base-pose update magnitude ‖Δ​𝐩k‖2\|\Delta\mathbf{p}_{k}\|_{2} before triggering amplitude scheduling. 3.0 mm
γ\gamma Amplitude decay factor for scheduling when ‖Δ​𝐩k‖2>Dmax\|\Delta\mathbf{p}_{k}\|_{2}>D_{\max}. 0.9
L-BFGS iters Maximum iterations for each inner-loop solve. 100
L-BFGS tol Termination tolerance (gradient norm / relative improvement). 10−610^{-6}

-D 5 Baselines

Local-only optimization omits inter-module coupling. The global baseline merges mechanism objectives with base-position and squared collision-volume penalties in one problem. The heuristic baseline sums centroid-based repulsion directions and applies a fixed step rather than solving the minimum-translation program.

The printed global objective explicitly contains position and overlap penalties; the prose broadly describes relaxation of geometric constraints. These implementations evaluate particular solver/penalty choices, not the impossibility of every joint-optimization strategy.

Equations A.48–A.51
min𝐗ℒjoint=ℒmech+ωpos​ℒpos+ωc​o​l​l​ℒcoll.\min_{\mathbf{X}}\quad\mathcal{L}_{\mathrm{joint}}=\mathcal{L}_{\mathrm{mech}}+\omega_{\mathrm{pos}}\mathcal{L}_{\mathrm{pos}}+\omega_{coll}\mathcal{L}_{\mathrm{coll}}. (A.48)
ℒmech=∑k=1K(ωamp​ℒamp(k)+ωfit​ℒfit(k)−ωman​ℒman(k)),\mathcal{L}_{\mathrm{mech}}=\sum_{k=1}^{K}\left(\omega_{\mathrm{amp}}\mathcal{L}_{\mathrm{amp}}^{(k)}+\omega_{\mathrm{fit}}\mathcal{L}_{\mathrm{fit}}^{(k)}-\omega_{\mathrm{man}}\mathcal{L}_{\mathrm{man}}^{(k)}\right), (A.49)
ℒpos=∑k=1K‖𝐩base,k−𝐩base,kinit‖22.\mathcal{L}_{\mathrm{pos}}=\sum_{k=1}^{K}\|\mathbf{p}_{\mathrm{base},k}-\mathbf{p}_{\mathrm{base,}k}^{\mathrm{init}}\|^{2}_{2}. (A.50)
ℒc​o​l​l=∑(i,j)∈𝒞active(Vint​(Ωi,Ωj))2.\mathcal{L}_{coll}=\sum_{(i,j)\in\mathcal{C}_{\mathrm{active}}}\left(V_{\mathrm{int}}(\Omega_{i},\Omega_{j})\right)^{2}. (A.51)
Complete original PDF block A0.F5
Fig. A.5: Automated mechanism synthesis across a spectrum of facial morphologies. We visualize the optimized internal CAD assemblies for eight distinct identities, rendered at relative physical scale to highlight volumetric diversity. It spans cinematic characters (Yoda, Luke, Rose, Jack), a stylized Elf, and characters from Chinese folk (Xu Xian, Bai Suzhen, Xiaoqing).
Equations A.52–A.53
𝐯rep(j)=𝐜k−𝐜l‖𝐜k−𝐜l‖+ϵ.\mathbf{v}_{\mathrm{rep}}^{(j)}=\frac{\mathbf{c}_{k}-\mathbf{c}_{l}}{\|\mathbf{c}_{k}-\mathbf{c}_{l}\|+\epsilon}. (A.52)
𝐩k(s+1)←𝐩k(s)+γ​∑j∈𝒞k𝐯rep(j).\mathbf{p}_{k}^{(s+1)}\leftarrow\mathbf{p}_{k}^{(s)}+\gamma\sum_{j\in\mathcal{C}_{k}}\mathbf{v}_{\mathrm{rep}}^{(j)}. (A.53)

-D 6 Diverse facial mechanism synthesis visualization

Eight selected successful assemblies illustrate a range of human, stylized and nonhuman faces, including Yoda, Luke, Rose, Jack, an elf and three White Snake characters. The relative-scale montage preserves all heads; it is a successful subset rather than the complete fifteen-case outcome distribution.

-D 7 Failure cases

A pointed chin can leave insufficient room for the fixed mouth mechanisms. Multi-module interlocking can also require topology changes rather than small translations, with mutually inconsistent separation constraints. The algorithm may stop with residual collisions or strongly reduced amplitudes; parameter adaptation alone cannot repair an empty feasible set.

-E Manufacture Details

-E 1 3D printed materials

PLA parts are produced on a Bambu Lab P1S with 0.1–0.2 mm joint clearances. Printing time sums component batches on one printer. These fabrication tolerances are supplied design choices, not a reported durability or repeatability characterization.

-E 2 Silicone face skin

Skin casting uses a printed outer mold and inner core with release agent. The two-part RTV silicone is mixed 1:1, vacuum-degassed for ten minutes at the stated −0.1 MPa and cured for four hours at 25 °C. The skin attaches magnetically to permit access and replacement.

-E 3 Actuator choice and noise testing

The chosen 180-degree microservos and three neck steppers serve different packaging/torque demands. The physical command loop is synchronized to 30 FPS animation. The paper reports an idle noise floor of 30 dB at 0.5 m.

Its claim that 60–65 dB conversational speech masks actuation noise is not accompanied by a measured active-motion noise level or controlled masking study. Idle noise alone does not establish that the robot is perceptually silent during movement.

-F Talking Head Synthesis Details

-F 1 Network architecture

Projected paired-audio features receive learnable time and identity-style embeddings. The activity gate interpolates the target and partner streams. Three-layer encoder and decoder stacks use width 256, four heads and feed-forward width 512. A turn-change-derived bias modifies attention logits, while motion queries carry sinusoidal positions.

An MLP expands to 512 then returns to 256 before the 55-coordinate output. A sigmoid bounds the fifty-two blendshapes; three Euler angles form the pose branch. The exact construction of the attention-bias matrix, causal protocol and complete latency are not specified by the supplied equation alone.

Equations A.54–A.56
𝐙~A=𝐙A+𝐏+𝐒A,𝐙~B=𝐙B+𝐏+𝐒B.\tilde{\mathbf{Z}}_{A}=\mathbf{Z}_{A}+\mathbf{P}+\mathbf{S}_{A},\qquad\tilde{\mathbf{Z}}_{B}=\mathbf{Z}_{B}+\mathbf{P}+\mathbf{S}_{B}. (A.54)
𝐌A​(t)=gA​(t)⋅𝐙~A​(t)+(1−gA​(t))⋅𝐙~B​(t).\mathbf{M}_{A}(t)=g_{A}(t)\cdot\tilde{\mathbf{Z}}_{A}(t)+\big(1-g_{A}(t)\big)\cdot\tilde{\mathbf{Z}}_{B}(t). (A.55)
Attention​(𝐐,𝐊,𝐕)=softmax​(𝐐𝐊⊤d−λ⋅𝐁𝐭𝐮𝐫𝐧)​𝐕.\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}}-\lambda\cdot\mathbf{B}_{\mathbf{turn}}\right)\mathbf{V}. (A.56)

-F 2 Loss function

Training combines framewise squared error over the 55-dimensional target with squared error between predicted and target first differences. This encourages matching observed motion and velocities. It is not simply a penalty forcing every velocity toward zero; the loss weights are named without numerical values here.

Equations A.57–A.60
ℒrec=1T​∑t=1T‖𝐲^t−𝐲t‖22.\mathcal{L}_{\mathrm{rec}}=\frac{1}{T}\sum_{t=1}^{T}\left\|\hat{\mathbf{y}}_{t}-\mathbf{y}_{t}\right\|_{2}^{2}. (A.57)
𝐯^t=𝐲^t−𝐲^t−1,𝐯t=𝐲t−𝐲t−1,t=2,…,T,\hat{\mathbf{v}}_{t}=\hat{\mathbf{y}}_{t}-\hat{\mathbf{y}}_{t-1},\qquad\mathbf{v}_{t}=\mathbf{y}_{t}-\mathbf{y}_{t-1},\qquad t=2,\dots,T, (A.58)
ℒvel=1T−1​∑t=2T‖𝐯^t−𝐯t‖22.\mathcal{L}_{\mathrm{vel}}=\frac{1}{T-1}\sum_{t=2}^{T}\left\|\hat{\mathbf{v}}_{t}-\mathbf{v}_{t}\right\|_{2}^{2}. (A.59)
ℒ=λrec​ℒrec+λvel​ℒvel,\mathcal{L}=\lambda_{\mathrm{rec}}\mathcal{L}_{\mathrm{rec}}+\lambda_{\mathrm{vel}}\mathcal{L}_{\mathrm{vel}}, (A.60)

-F 3 Training details

The conversational model uses Adam at learning rate 10⁻⁴ for two hundred epochs, batch size thirty-two, on four NVIDIA L20 GPUs. Reported training time is about twenty hours. The per-head inverse mapper has a separate, much shorter training stage.

-F 4 Evaluation metrics

LVE averages each frame’s maximum lip-vertex distance; coefficient MSE measures reconstruction. PDD compares the standard deviations of head-angle velocities, and FDD does the same for upper-face coefficients. Matching these statistics does not require matching the complete motion sequence.

SID is cluster-assignment entropy, favoring diverse generated motion. The appendix describes listener diversity although Table II places SID under the speaker columns; both source presentations are retained. Cluster count and uncertainty intervals are not supplied in this definition.

Equations A.61–A.63
PDD=‖stdt​(𝐫˙tgen)−stdt​(𝐫˙tgt)‖1,\text{PDD}=\left\|\mathrm{std}_{t}(\dot{\mathbf{r}}^{\text{gen}}_{t})-\mathrm{std}_{t}(\dot{\mathbf{r}}^{\text{gt}}_{t})\right\|_{1}, (A.61)
Original PDF equation A.62: entropy over motion clusters
Original selectable MathML
SID=−∑k=1Kpklogpk.\text{SID}=-\sum_{k=1}^{K}p_{k}\log p_{k}. (A.62)
FDD=‖stdt​(𝐛˙gen,tupper)−stdt​(𝐛˙gt,tupper)‖1.\text{FDD}=\left\|\mathrm{std}_{t}(\dot{\mathbf{b}}^{\text{upper}}_{\text{gen},t})-\mathrm{std}_{t}(\dot{\mathbf{b}}^{\text{upper}}_{\text{gt},t})\right\|_{1}. (A.63)

-F 5 Dataset details

The data contain fifty hours, 1,052 identities and 5,858 clips averaging 2.5 turns, split into 5,282 training and 576 test clips. Interview-style YouTube material and RealTalk are filtered for visible faces, usable audio and natural turn-taking.

TransNetV2 segments scenes, tracked ordering keeps identities aligned, and SAM Audio separates synchronized 16 kHz streams. MediaPipe supplies fifty-two expression coefficients and three head angles per frame at 30 FPS. The stated split is by clips; an identity-disjoint test is not explicitly documented. Pseudolabel and audio-separation errors remain possible.

-G Mapping Network Details

-G 1 Baseline details

Landmark baselines use 468 two-dimensional points: nearest-neighbor command retrieval, min–max alignment plus a global MLP, or rest-relative scaling/clamping plus a global MLP. FLAME fitting yields shape, expression and pose descriptors, dropping fixed identity and global head rotation for the inverse regressor. The full fifty-two blendshapes provide another global-MLP baseline.

These alternatives differ in representation, preprocessing and interference between regions. Regional mapping uses semantic structure while retaining the same supervised calibration setting; no baseline provides a zero-calibration comparison.

Equations A.64–A.68
i⋆=arg⁡mini∈{1,…,N}⁡‖𝐥tH−𝐥iR‖22,𝐮^t=𝐮i⋆.i^{\star}=\arg\min_{i\in\{1,\dots,N\}}\left\|\mathbf{l}^{H}_{t}-\mathbf{l}^{R}_{i}\right\|_{2}^{2},\qquad\hat{\mathbf{u}}_{t}=\mathbf{u}_{i^{\star}}. (A.64)
𝐥~H=(𝐥H−𝐡min)​𝐫max−𝐫min𝐡max−𝐡min+𝐫min,\tilde{\mathbf{l}}^{H}=\left(\mathbf{l}^{H}-\mathbf{h}_{\min}\right)\frac{\mathbf{r}_{\max}-\mathbf{r}_{\min}}{\mathbf{h}_{\max}-\mathbf{h}_{\min}}+\mathbf{r}_{\min}, (A.65)
ℒMLP=1N​∑i=1N‖π⁡(𝐥iR)−𝐮i‖22.\mathcal{L}_{\text{MLP}}=\frac{1}{N}\sum_{i=1}^{N}\left\|\pi(\mathbf{l}^{R}_{i})-\mathbf{u}_{i}\right\|_{2}^{2}. (A.66)
𝐥~H=𝐫s+K×clip⁡(Δ,𝐫min−𝐫s,𝐫max−𝐫s),\tilde{\mathbf{l}}^{H}=\mathbf{r}_{s}+K\times\mathrm{clip}\!\left(\Delta,\ \mathbf{r}_{\min}-\mathbf{r}_{s},\ \mathbf{r}_{\max}-\mathbf{r}_{s}\right), (A.67)
ℳ⁡(θ,β,ψ)→𝐕∈ℝN×3\mathcal{M}(\theta,\beta,\psi)\rightarrow\mathbf{V}\in\mathbb{R}^{N\times 3} (A.68)

-G 2 Evaluation metrics

Audio and mouth-video windows are embedded by a pretrained synchronization model. LSE-D averages the minimum audio–visual distance over candidate temporal offsets. LSE-C averages the margin between the median offset distance and the best distance.

The search compensates for temporal misalignment, so a favorable score can coexist with constant delay inside the offset window. These metrics evaluate alignment quality under the stated matching rule rather than direct turn-response latency.

Equations A.69–A.72
d⁡(t,Δ)=‖𝐯t−𝐚t+Δ‖2.d(t,\Delta)=\|\mathbf{v}_{t}-\mathbf{a}_{t+\Delta}\|_{2}. (A.69)
dmin​(t)=minΔ⁡d⁡(t,Δ).d_{\text{min}}(t)=\min_{\Delta}d(t,\Delta). (A.70)
LSE-D=1T​∑t=1Tdmin​(t),\text{LSE-D}=\frac{1}{T}\sum_{t=1}^{T}d_{\text{min}}(t), (A.71)
LSE-C=1T​∑t=1T(medianΔ​d​(t,Δ)−dmin​(t)).\text{LSE-C}=\frac{1}{T}\sum_{t=1}^{T}(\text{median}_{\Delta}\ d(t,\Delta)-d_{\text{min}}(t)). (A.72)

-G 3 Semantic region-wise mapping network

The mouth/jaw mapper takes twenty-three blendshapes through 256/128 hidden layers with layer normalization, ReLU and 0.1 dropout. Seven outputs drive upper lip, lower lip, two motors for each corner and jaw. Other regional mappers reuse the architecture with different input/output sizes.

Calibration samples are randomly split 80/10/10. Adam uses learning rate 10⁻³, weight decay 10⁻⁵, batch 128, five hundred warmup steps and five hundred epochs. The reported approximately 0.02 MSE is a training loss, not a physical tracking distance or a documented held-out-test error.

Complete original PDF block A0.T2
TABLE A.2: Parameter configuration for the mouth and jaw region mapping network. The output dimension is specifically designed to control these regions with high fidelity.
Original selectable table
Category Parameter Value / Description
Architecture Input Dimension 23 (Blendshape coefficients)
Hidden Layers [256,128][256,128]
Output Dimension 7 (Motor commands)
Activation ReLU
Normalization Layer Normalization (LN)
Dropout Rate 0.1
Training Dataset Split Train: 80%, Val: 10%, Test: 10%
Optimizer Adam
Learning Rate 1×10−31\times 10^{-3}
Batch Size 128
Max Epochs 500
Weight Decay 1×10−51\times 10^{-5}
Warmup Steps 500 (Linear strategy)

-H Extended Visualization

The extended examples cover live human dialogue, paired physical characters and mixed digital/physical embodiment, with complete visual sequences retained. Narrative transcripts illustrate selected exchanges; they do not independently measure emotion understanding, social benefit or the robot’s subjective experiences.

Complete original PDF block A0.F6
Fig. A.6: Visualization. (a) Yoda (speaker) and Luke (listener); (b) a real elf interacting with a virtual one; (c) a Mandarin dialogue in a Chinese myth.

-H 1 Real-time human-robot interaction

A semi-structured human–robot conversation explores the fictional theme of learning to be human. It demonstrates a selected live dialogue using the voice system, but the accompanying transcript provides no timed response distribution or controlled cross-language accuracy measurement.

-H 2 Dyadic robot-robot dialogue

The Titanic reunion and Yoda–Luke scenes stage affectionate and reflective dialogue with different speaking styles. Alternating facial activity and listener responses illustrate synchronized embodiment. The creative scenes are demonstrations, not a quantitative test of unrestricted autonomous conversation.

-H 3 Hybrid avatar-robot interaction

A screen avatar converses with a physical elf about remaining in the real world. This mixed embodiment illustrates reuse of the audiovisual/retargeting pipeline. Statements about feeling, smell or perception belong to the scripted character dialogue and are not measurements of robot experience or sensory capabilities.

Complete original PDF block A0.T3
TABLE A.3: Participant demographics for the user study (N=100N=100). Background is self-reported and indicates participants’ primary domain exposure.
Original selectable table
Gender Age Group Background
Female Male 18–30 31–45 45–60 Robotics CV/ML Film/Media HCI/UX Other
Count 68 32 33 38 29 30 23 14 28 5

-I User Study Details

The hundred participants comprise sixty-eight women and thirty-two men; age groups 18–30, 31–45 and 45–60 contain thirty-three, thirty-eight and twenty-nine people. Self-reported backgrounds are robotics thirty, CV/ML twenty-three, film/media fourteen, HCI/UX twenty-eight and other five. This is the reported study sample, not a representative population claim.

-I 1 Study 1: Interaction performance on movie dialogue excerpts

Four reference-conditioned excerpts compare the complete system with speaker-only, no-neck, random-neck and mouth-only variants in randomized, single-blind order. The ten-point scale anchors poor through excellent overall naturalness, expression and coherence.

Complete-system means are 7.9–8.3, versus 6.4–7.2 speaker-only, 6.1–6.6 no-neck, 6.2–6.8 random-neck and 5.5–6.1 mouth-only. The source presents mean bars without dispersion or significance tests. These short video judgments support the tested combination of cues, not longitudinal engagement or a general causal explanation of every individual rating.

-I 2 Study 2: Perceived mapping quality.

Three short utterances compare six mappings after a reference avatar animation. Method order is randomized and identities concealed. Regional mapping receives the highest mean scores, followed by the stronger coefficient-based alternatives; nearest-neighbor landmarks are weakest.

The results align with the supplied synchronization metrics, while varying representation and regional structure together. They do not separately prove that a particular semantic decomposition is optimal across all faces or dialogue styles.

-J Limitations and Future Work

-J 1 Mechanical reconfigurability and modular extensibility

Rigid CAD-level placement burdens optimization when global facial proportions vary. Mechanically adjustable interpupillary spacing or module separation could ease adaptation, but this reconfigurable hardware is proposed rather than implemented in the reported template.

-J 2 Unified end-to-end learning framework

The audio, animation and retargeting stages can accumulate errors and add computation. Direct audio/text-to-actuator learning is proposed to reduce these costs; no such end-to-end physical learner or corresponding latency improvement is evaluated here.

-J 3 Differentiable simulation and sim-to-real adaptation

A material-aware differentiable simulator could model rigid–silicone coupling and supply synthetic calibration data. Accurate contact, skin properties and sim-to-real transfer remain future work, rather than existing guarantees of the geometric design optimizer.

-J 4 Towards full-body embodiment and interaction

The current platform actuates face and neck only. Coordinating torso posture and co-speech gestures is a proposed extension for whole-body interaction, requiring additional hardware, training and evaluation.

Concept demonstration

Open image ↗