Audio-visual Neural Face Generation with Emotional Stimuli

Abstract

In communication, it is important to generate realistic 3D facial avatars that can express various expressions. However, existing models have limitations in that they cannot show facial expressions in sufficient detail. To this end, we propose an audio-visual volumetric avatar generation method that reconstructs detailed facial expressions using two emotional stimuli. Emotional features extracted from the input and the rendered image induce the model to generate an avatar with the same emotion. At the same time, a speech clip is used as input along with the images, helping the model recognize the emotion of the target that is difficult to understand with the images. We showed qualitatively and qualitatively that our method achieved significantly more realistic and higher-quality avatar generation than current state-of-the-art methods. © 2023 IEEE.