Singing Lip Sync Animation System Using Audio Spectrum

  • Namjung KimEmail author
  • Kyoungju Park
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 536)


When humans sing, the changes in pitch and volume are usually significantly greater than when they speak. The shape and size of the mouth changes dramatically when singing compared with when talking, depending on the strength of the voice. In this study, we propose a new model that effectively generates the change in mouth shape in the vocal environment based on the voice strength estimated by analyzing the audio spectrum. We estimate the voice strength by analyzing the spectrum of the voice using a Fast Fourier Transform-based numerical technique for each frame. We apply the intensity of the estimated voice to the morph targets associated with the mouth shape as the weight of the blendshape. Experimental results show a visually convincing lip-synching animation that changes the mouth shape significantly depending on the pitch and volume of the voice.


Singing lip sync animation Audio spectrum Blendshape activation Fast Fourier Transform 



This research is supported by Ministry of Culture, Sports and Tourism (MCST) and Korea Creative Content Agency (KOCCA) in the Culture Technology (CT) Research & Development Program 2018, and is supported by the Mid-Career Researcher Program through an NRF grant funded by the MEST (No. NRF-2016R1A2B4016239).


  1. 1.
    Chao, Y., Faloutsos, P., Kohler, E., PIGHIN, F.: Real-time speech motion synthesis from recorded motions. In: Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 347–355 (2004)Google Scholar
  2. 2.
    Xie, L., Liu, Z.-Q.: Realistic mouth-synching for speech-driven talking face using articulatory modeling. IEEE Trans. Multimedia 9(3), 500–510 (2007)CrossRefGoogle Scholar
  3. 3.
    Pif, E., Chris, L., Eugene, F., Karan, S.: JALI: an animatorcentric viseme model for expressive lip synchronization. ACM Trans. Graph. (Proc. SIGGRAPH 2016) 35(4), 127:1–127:11 (2016)Google Scholar
  4. 4.
    Taylor, S., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A.G., Hodgins, J., Matthews, I.: A deep learning approach for generalized speech animation. ACM Trans. Graph. (Proc. SIGGRAPH 2017) 36(4), 93 (2017)Google Scholar
  5. 5.
    Suwajanakorn, S., Seitz, S.M., Kemelmacher-Schlizerman, I.: Synthesizing Obama: learning lip sync from audio. ACM Trans. Graph. (Proc. SIGGRAPH 2017) 36(4), 95 (2017)Google Scholar
  6. 6.
    Karras, T., Aila, T., Laine, S., Herva, A., Lehtinen, J.: Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Trans. Graph. (Proc. SIGGRAPH 2017) 36(4), 94:1–94:12 (2017)Google Scholar
  7. 7.
    Zhang, F.S., Geng, Z.X., Yuan, W.: The algorithm of interpolating windowed FFT for harmonic analysis of power system. IEEE Trans. Power Deliv. 16(2), 160–164 (2001)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  1. 1.Chung-Ang University Industry Academic Cooperation FoundationSeoulKorea
  2. 2.Department of SoftwareChung-Ang UniversitySeoulKorea

Personalised recommendations