Singing Lip Sync Animation System Using Audio Spectrum
When humans sing, the changes in pitch and volume are usually significantly greater than when they speak. The shape and size of the mouth changes dramatically when singing compared with when talking, depending on the strength of the voice. In this study, we propose a new model that effectively generates the change in mouth shape in the vocal environment based on the voice strength estimated by analyzing the audio spectrum. We estimate the voice strength by analyzing the spectrum of the voice using a Fast Fourier Transform-based numerical technique for each frame. We apply the intensity of the estimated voice to the morph targets associated with the mouth shape as the weight of the blendshape. Experimental results show a visually convincing lip-synching animation that changes the mouth shape significantly depending on the pitch and volume of the voice.
KeywordsSinging lip sync animation Audio spectrum Blendshape activation Fast Fourier Transform
This research is supported by Ministry of Culture, Sports and Tourism (MCST) and Korea Creative Content Agency (KOCCA) in the Culture Technology (CT) Research & Development Program 2018, and is supported by the Mid-Career Researcher Program through an NRF grant funded by the MEST (No. NRF-2016R1A2B4016239).
- 1.Chao, Y., Faloutsos, P., Kohler, E., PIGHIN, F.: Real-time speech motion synthesis from recorded motions. In: Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 347–355 (2004)Google Scholar
- 3.Pif, E., Chris, L., Eugene, F., Karan, S.: JALI: an animatorcentric viseme model for expressive lip synchronization. ACM Trans. Graph. (Proc. SIGGRAPH 2016) 35(4), 127:1–127:11 (2016)Google Scholar
- 4.Taylor, S., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A.G., Hodgins, J., Matthews, I.: A deep learning approach for generalized speech animation. ACM Trans. Graph. (Proc. SIGGRAPH 2017) 36(4), 93 (2017)Google Scholar
- 5.Suwajanakorn, S., Seitz, S.M., Kemelmacher-Schlizerman, I.: Synthesizing Obama: learning lip sync from audio. ACM Trans. Graph. (Proc. SIGGRAPH 2017) 36(4), 95 (2017)Google Scholar
- 6.Karras, T., Aila, T., Laine, S., Herva, A., Lehtinen, J.: Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Trans. Graph. (Proc. SIGGRAPH 2017) 36(4), 94:1–94:12 (2017)Google Scholar