Skip to main content

The Rach3 Dataset: Towards Data-Driven Analysis of Piano Performance Rehearsal

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2024)

Abstract

Musicians spend more time practicing than performing live, but the process of rehearsal has been understudied. This paper introduces a dataset for using AI and machine learning to address this gap. The project observes the progression of pianists learning new repertoire over long periods of time by recording their rehearsals, generating a comprehensive multimodal dataset, the Rach3 dataset, with video, audio, and MIDI for computational analysis. This dataset will help investigating the way in which advanced students and professional classical musicians, particularly pianists, learn new music and develop their own expressive interpretations of a piece.

This work is supported by the European Research Council (ERC) under the EU’s Horizon 2020 research and innovation programme, grant agreement No. 101019375 (Whither Music?).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://yousician.com.

  2. 2.

    https://www.hellosimply.com/simply-piano.

  3. 3.

    The Musical Instrument Digital Interface (MIDI) is a standardized protocol that enables digital musical instruments to communicate and record music.

  4. 4.

    The first author is a professionally trained musician with a degree in Piano Performance and 14 years of formal music education.

  5. 5.

    https://youtube.com/playlist?list=PLPUWmNCGflVOcjb5p4-ae3zFm0Z5l15RH.

  6. 6.

    https://dataset.rach3project.com/.

  7. 7.

    https://usa.yamaha.com/products/musical_instruments/pianos/silent_piano/index.html.

  8. 8.

    https://usa.yamaha.com/products/musical_instruments/pianos/disklavier/.

  9. 9.

    When recording rehearsals on silent pianos, it is possible to just get the output MIDI signal from the optical sensors built into the piano, without mechanically silencing the piano, which enables capturing MIDI and the real sound of the piano.

  10. 10.

    Early in the dataset, recordings were made at 30 fps, and a few at 120 fps. Higher frame rates caused the GoPro to shut down due to overheating, so 60 fps was chosen as a compromise between higher resolution and long recording times.

  11. 11.

    This period corresponds roughly to the Baroque, Classical, Romantic, and early 20th Century periods of Western Classical music.

  12. 12.

    https://imslp.org/wiki/Main_Page.

  13. 13.

    https://musescore.com.

  14. 14.

    https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english.

References

  1. Benetos, E., Dixon, S., Duan, Z., Ewert, S.: Automatic music transcription: an overview. IEEE Signal Process. Mag. 36(1), 20–30 (2019). https://doi.org/10.1109/MSP.2018.2869928

    Article  Google Scholar 

  2. Bishop, L., Bailes, F., Dean, R.T.: Performing musical dynamics. Music. Percept. 32(1), 51–66 (2014). https://doi.org/10.1525/mp.2014.32.1.51

    Article  Google Scholar 

  3. Cancino-Chacón, C.E., Grachten, M., Goebl, W., Widmer, G.: Computational models of expressive music performance: a comprehensive and critical review. Front. Digital Human. 5, 25 (2018). https://doi.org/10.3389/fdigh.2018.00025

    Article  Google Scholar 

  4. Chaffin, R., Imreh, G.: Practicing perfection: piano performance as expert memory. Psychol. Sci. 13(4), 342–349 (2005). https://doi.org/10.4324/9781410612373

    Article  Google Scholar 

  5. Chaffin, R., Lisboa, T., Logan, T., Begosh, K.T.: Preparing for memorized cello performance: the role of performance cues. Psychol. Music 38(1), 3–30 (2010). https://doi.org/10.1177/0305735608100377

    Article  Google Scholar 

  6. Cook, N.: Analysing performance and performing analysis. In: Cook, N., Everist, M. (eds.) Rethinking Music, pp. 239–261. Oxford University Press, Oxford (1999)

    Google Scholar 

  7. Cook, N., Johnson, P., Zender, H.: Theory into Practice: Composition, Performance And The Listening Experience. Leuven University Press (2021). https://doi.org/10.2307/j.ctv1rh36q7

  8. Dahl, et al.: Gestures in Performance. In: Musical Gestures: Sound, Movement, and Meaning, pp. 36–68. Routledge (2010)

    Google Scholar 

  9. Demos, A.P., Lisboa, T., Chaffin, R.: Flexibility of expressive timing in repeated musical performances. Front. Psychol. 7, 1490 (2016). https://doi.org/10.3389/fpsyg.2016.01490

    Article  Google Scholar 

  10. Ericsson, K.A., Krampe, R.T., Tesch-Romer, C.: The role of deliberate practice in the acquisition of expert performance. Psychol. Rev. 100(3), 364–403 (1993)

    Article  Google Scholar 

  11. Goebl, W.: Movement and touch in piano performance. In: Müller, B., Wolf, S.I., Brueggemann, G.P., Deng, Z., McIntosh, A., Miller, F., Selbie, W.S. (eds.) Handbook of Human Motion. Springer International Publishing, Cham (2017). https://doi.org/10.1007/978-3-319-30808-1

  12. Goebl, W., Dixon, S., Poli, G.D., Friberg, A., Widmer, G.: Sense in expressive music performance: data acquisition, computational studies, and models. In: Polotti, P., Rocchesso, D. (eds.) Sound to Sense - Sense to Sound: A State of the Art in Sound and Music Computing, pp. 195–242. Logos, Berlin (2008)

    Google Scholar 

  13. Hallam, S., Papageorgi, I., Varvarigou, M., Creech, A.: Relationships between practice, motivation, and examination outcomes. Psychol. Music 49(1), 3–20 (2021). https://doi.org/10.1177/0305735618816168

    Article  Google Scholar 

  14. Hawthorne, C., et al.: Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset. In: Proceedings of the International Conference on Learning Representation, New Orleans, USA (2019)

    Google Scholar 

  15. How, E.R., Tan, L., Miksza, P.: A PRISMA review of research on music practice. Musicae Scientiae 26(3), 455–697 (2022). https://doi-org.ezproxy.uio.no/10.1177/10298649211005531

  16. Jensenius, A.R., Wanderley, M.M., Godøy, R.I., Leman, M.: Musical gestures concepts and methods in research. In: Godøy, R.I., Leman, M. (eds.) Musical Gestures: Sound, Movement, and Meaning, pp. 12–35. Routledge (2010)

    Google Scholar 

  17. Juslin, P.N., Laukka, P.: Communication of emotions in vocal expression and music performance: different channels, same code? Psychol. Bull. 129(5), 770–814 (2003). https://doi.org/10.1037/0033-2909.129.5.770

    Article  Google Scholar 

  18. Lerch, A., Arthur, C., Pati, A., Gururani, S.: An interdisciplinary review of music performance analysis. Trans. Int. Soc. Music Inf. Retr. 3(1), 221–245 (2020). https://doi.org/10.5334/tismir.53

    Article  Google Scholar 

  19. Loureiro, D., Barbieri, F., Neves, L., Espinosa Anke, L., Camacho-Collados, J.: TimeLMs: diachronic language models from Twitter. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 251–260. Dublin, Ireland (2022). https://doi.org/10.18653/v1/2022.acl-demo.25

  20. Mathis, A., Schneider, S., Lauer, J., Mathis, M.W.: A primer on motion capture with deep learning: principles, pitfalls, and perspectives. Neuron 108(1), 44–65 (2020). https://doi.org/10.1016/j.neuron.2020.09.017

    Article  Google Scholar 

  21. Miksza, P.: A review of research on practicing: summary and synthesis of the extant research with implications for a new theoretical orientation. Bull. Counc. Res. Music. Educ. 190, 51–92 (2011). https://doi.org/10.5406/bulcouresmusedu.190.0051

    Article  Google Scholar 

  22. Palmer, C.: Music performance. Annu. Rev. Psychol. 48, 115–138 (1997)

    Article  Google Scholar 

  23. Peter, S.D., et al.: Automatic note-level score-to-performance alignments in the ASAP dataset. Trans. Int. Soc. Music Inf. Retr. 6(1), 27–42 (2023). https://doi.org/10.5334/tismir.149

    Article  MathSciNet  Google Scholar 

  24. Reid, S.: Preparing for performance. In: Rink, J. (ed.) Musical Performance, pp. 102–112. Cambridge University Press (2002). https://doi.org/10.1017/CBO9780511811739.008

  25. Steyer, R., Schwenkmezger, P., Notz, P., Eid, M.: Development of the Multidimensional Mood State Questionnaire (MDBF). Primary data. (Version 1.0.0) [Data and Documentation]. Tech. rep., Trier: Center for Research Data in Psychology: PsychData of the Leibniz Institute for Psychology ZPID (2004)

    Google Scholar 

  26. Verdugo, F., Pelletier, J., Michaud, B., Traube, C., Begon, M.: Effects of trunk motion, touch, and articulation on upper-limb velocities and on joint contribution to endpoint velocities during the production of loud piano tones. Front. Psychol. 11, 1159 (2020). https://doi.org/10.3389/fpsyg.2020.01159

    Article  Google Scholar 

  27. Visi, F.G., Östersjö, S., Ek, R., Röijezon, U.: Method development for multimodal data corpus analysis of expressive instrumental music performance. Front. Psychol. 11, 576751 (2020). https://doi.org/10.3389/fpsyg.2020.576751

    Article  Google Scholar 

  28. Walls, P.: Historical performance and the modern performer. In: Rink, J. (ed.) Musical Performance, pp. 17–34. Cambridge University Press (2002). https://doi.org/10.1017/CBO9780511811739.003

  29. Winges, S., Furuya, S.: Distinct digit kinematics by professional and amateur pianists. Neuroscience 284, 643–652 (2015). https://doi.org/10.1016/j.neuroscience.2014.10.041

    Article  Google Scholar 

  30. Zhang, F., et al.: MediaPipe hands: on-device real-time hand tracking. In: Proceedings of the CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, USA (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Carlos Eduardo Cancino-Chacón .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cancino-Chacón, C.E., Pilkov, I. (2024). The Rach3 Dataset: Towards Data-Driven Analysis of Piano Performance Rehearsal. In: Rudinac, S., et al. MultiMedia Modeling. MMM 2024. Lecture Notes in Computer Science, vol 14565. Springer, Cham. https://doi.org/10.1007/978-3-031-56435-2_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56435-2_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56434-5

  • Online ISBN: 978-3-031-56435-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics