Estimation of Guitar Fingering and Plucking Controls Based on Multimodal Analysis of Motion, Audio and Musical Score
This work presents a method for the extraction of instrumental controls during guitar performances. The method is based on the analysis of multimodal data consisting of a combination of motion capture, audio analysis and musical score. High speed video cameras based on marker identification are used to track the position of finger bones and articulations and audio is recorded with a transducer measuring vibration on the guitar body. The extracted parameters are divided into left hand controls, i.e. fingering (which string and fret is pressed with a left hand finger) and right hand controls, i.e. the plucked string, the plucking finger and the characteristics of the pluck (position, velocity and angles with respect to the string). Controls are estimated based on probability functions of low level features, namely, the plucking instants (i.e. note onsets), the pitch and the distances of the fingers (both hands) to strings and frets. Note onsets are detected via audio analysis, the pitch is extracted from the score and distances are computed from 3D Euclidean Geometry. Results show that by combination of multimodal information, it is possible to estimate such a comprehensive set of control features, with special high performance for the fingering and plucked string estimation. Regarding the plucking finger and the pluck characteristics, their accuracy gets lower but improvements are foreseen including a hand model and the use of high-speed cameras for calibration and evaluation.
KeywordsGuitar Instrumental control Motion capture Audio analysis
A. Perez-Carrillo was supported by a Beatriu de Pinos grant 2010 BP-A 00209 by the Catalan Research Agency (AGAUR) and J. Ll. Arcos was supported by ICT -2011-8-318770 and 2009-SGR-1434 projects.
- 1.Abesser, G.S.J., Lukashevich, H.: Feature-based extraction of plucking and expression styles of the electric bass. In: Proceedings of the ICCASP Conference, Kyoto, Japan (2012)Google Scholar
- 2.Bevilacqua, F., Schnell, N., Rasamimanana, N., Zamborlin, B., Guédy, F.: Online gesture analysis and control of audio processing. In: Solis, J., Ng, K. (eds.) Musical Robots and Interactive Multimodal Systems. Springer Tracts in Advanced Robotics, vol. 74, pp. 127–142. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-22291-78CrossRefGoogle Scholar
- 3.Burns, A.M., Wanderley, M.M.: Visual methods for the retrieval of guitarist fingering. In: Proceedings of NIME, pp. 196–199. IRCAM, Paris (2006). http://dl.acm.org/citation.cfm?id=1142215.1142263
- 6.Collins, N., Kiefer, C., Patoli, Z., White, M.: Musical exoskeletons: experiments with a motion capture suit. In: New Interfaces for Musical Expression, Sydney, Australia (2010)Google Scholar
- 7.Duxbury, C., Bello, J.P., Davies, M., Sandler, M.: Complex domain onset detection for musical signals. In: Proceedings of the DAFx. Queen Mary University, London (2003)Google Scholar
- 8.Eberly, D.H.: 3D Game Engine Design: A Practical Approach to Real-time Computer Graphics. Morgan Kaufmann Publishers Inc., San Francisco (2000)Google Scholar
- 9.Erkut, C., Välimäki, V., Karjalainen, M., Laurson, M.: Extraction of physical and expressive parameters for model-based sound synthesis of the classical guitar. In: Audio Engineering Society Convention 108, February 2000. http://www.aes.org/e-lib/browse.cfm?elib=9224
- 10.Guaus, E., Arcos, J.L.: Analyzing left hand fingering in guitar playing. In: Proceedings of the SMC. Universitat Pompeu Fabra, Barcelona, July 2010Google Scholar
- 12.Linden, J.V.D., Schoonderwaldt, E., Bird, J.: Towards a real-time system for teaching novices correct violin bowing technique. In: IEEE International Workshop on Haptic Audio visual Environments and Games, pp. 81–86, November 2009Google Scholar
- 13.Maestre, E., Blaauw, M., Bonada, J., Guaus, E., Pérez, A.: Statistical modeling of bowing control applied to sound synthesis. IEEE Trans. Audio Speech Lang. Process. Special Issue on Virtual Analog Audio Effects and Musical Instruments (2010)Google Scholar
- 14.Maestre, E., Bonada, J., Blaauw, M., Pérez, A., Guaus, E.: Acquisition of violin instrumental gestures using a commercial EMF device. In: International Computer Music Conference, Copenhagen, Denmark (2007)Google Scholar
- 15.Norton, J.: Motion capture to build a foundation for a computer-controlled instrument by study of classical guitar performance. Ph.D. thesis, Department of Music, Stanford University (2008)Google Scholar
- 16.Penttinen, H., Välimäki, V.: A time-domain approach to estimating the plucking point of guitar tones obtained with an under-saddle pickup. Appl. Acoust. 65(12), 1207–1220 (2004). http://www.sciencedirect.com/science/article/pii/S0003682X04001057CrossRefGoogle Scholar
- 18.Pérez-Carrillo, A.: Enhancing spectral synthesis techniques with performance gestures using the violin as a case study. Ph.D. thesis, Universitat Pompeu Fabra, Barcelona, Spain (2009). www.mtg.upf.edu/static/media/Perez-Alfonso-PhD-2009.pdf
- 20.Reboursière, L., Lähdeoja, O., Drugman, T., Dupont, S., Picard-Limpens, C., Riche, N.: Left and right-hand guitar playing techniques detection. In: NIME. University of Michigan, Ann Arbor, 21–23 May 2012Google Scholar
- 21.Scherrer, B.: Physically-informed indirect acquisition of instrumental gestures on the classical guitar: extracting the angle of release. Ph.D. thesis, McGill University, Montréal, QC (2013)Google Scholar
- 22.Scherrer, B., Depalle, P.: Extracting the angle of release from guitar tones: preliminary results. In: Proceedings of Acoustics, Nantes, France (2012)Google Scholar
- 23.Schneider, J.: The Contemporary Guitar, New Instrumentation, vol. 5. University of California Press, Berkeley (1985)Google Scholar
- 24.Schoonderwaldt, E., Guettler, K., Askenfelt, A.: An empirical investigation of bow-force limits in the Schelleng diagram. AAA 94(4), 604–622 (2008)Google Scholar
- 25.Traube, C., Depalle, P.: Deriving the plucking point location along a guitar string from a least-square estimation of a comb filter delay. In: IEEE Canadian Conference on Electrical and Computer Engineering, vol. 3, pp. 2001–2004, May 2003Google Scholar
- 26.Traube, C., Smith, J.O.: Estimating the plucking point on a guitar string. In: Proceedings of COST-G6 Conference on Digital Audio Effects, Verona, Italy (2000). http://profs.sci.univr.it/dafx/DAFx-final-papers.html
- 28.Wanderley, M.M., Depalle, P.: Gestural control of sound synthesis. In: Proceedings of the IEEE, pp. 632–644 (2004)Google Scholar
- 29.Zhang, B., Wang, Y.: Automatic music transcription using audio-visual fusion for violin practice in home environment. Technical report, TRA7/09, Shool of Computing, National University of Singapore (2009)Google Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (http://creativecommons.org/licenses/by-nc/2.5/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.