Dance to the beat: Synchronizing motion to audio
- 35 Downloads
In this paper we introduce a video post-processing method that enhances the rhythm of a dancing performance, in the sense that the dancing movements are more in time to the beat of the music. The dancing performance as observed in a video is analyzed and segmented into motion intervals delimited by motion beats. We present an image-space method to extract the motion beats of a video by detecting frames at which there is a significant change in direction or motion stops. The motion beats are then synchronized with the music beats such that as many beats as possible are matched with as little as possible time-warping distortion to the video. We show two applications for this cross-media synchronization: one where a given dance performance is enhanced to be better synchronized with its original music, and one where a given dance video is automatically adapted to be synchronized with different music.
Keywordsvideo processing synchronization motion segmentation video analysis
- Zhou, S.; Fu, H.; Liu, L.; Cohen-Or, D.; Han, X. Parametric reshaping of human bodies in images. ACM Transactions on Graphics Vol. 29, No. 4, Article No. 126, 2010.Google Scholar
- Zhou, F.; Torre, F. Canonical time warping for alignment of human behavior. In: Proceedings of the Advances in Neural Information Processing Systems, 2286–2294, 2009.Google Scholar
- Zhou, F.; De la Torre, F. Generalized time warping for multi-modal alignment of human motion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1282–1289, 2012.Google Scholar
- Yoon, J.-C.; Lee, I.-K.; Lee, H.-C. Feature-based synchronization of video and background music. In: Advances in Machine Vision, Image Processing, and Pattern Analysis. Lecture Notes in Computer Science, Vol. 4153. Zheng, N.; Jiang, X.; Lan, X. Eds. Springer Berlin Heidelberg, 205–214, 2006.CrossRefGoogle Scholar
- Jehan, T.; Lew, M.; Vaucelle, C. Cati dance: Self-edited, self-synchronized music video. In: Proceedings of the ACM SIGGRAPH 2003 Sketches & Applications, 1–1, 2003.Google Scholar
- Suwajanakorn, S.; Seitz, S. M.; Kemelmacher-Shlizerman, I. Synthesizing Obama: Learning lip sync from audio. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 95, 2017.Google Scholar
- Slaney, M.; Covell, M. FaceSync: A linear operator for measuring synchronization of video facial images and audio tracks. In: Proceedings of the 13th International Conference on Neural Information Processing Systems, 784–790, 2000.Google Scholar
- Wang, O.; Schroers, C.; Zimmer, H.; Gross, M.; Sorkine-Hornung, A. VideoSnapping: Interactive synchronization of multiple videos. ACM Transactions on Graphics Vol. 33, No. 4, Article No. 77, 2014.Google Scholar
- Shiratori, T.; Nakazawa, A.; Ikeuchi, K. Detecting dance motion structure through music analysis. In: Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition, 857–862, 2004.Google Scholar
- Denman, H.; Doyle, E.; Kokaram, A.; Lennon, D.; Dahyot, R.; Fuller, R. Exploiting temporal discontinuities for event detection and manipulation in video streams. In: Proceedings of the 7th ACM SIGMM International Workshop on Multimedia Information Retrieval, 183–192, 2005.Google Scholar
- Patel, A. D.; Iversen, J. R.; Bregman, M. R.; Schulz, I.; Schulz, C. Investigating the human-specificity of synchronization to music. In: Proceedings of the 10th International Conference on Music Perception and Cognition, 100–104, 2008.Google Scholar
Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.