Hybrid hidden Markov models and artificial neural networks for handwritten music recognition in mensural notation

  • Jorge Calvo-ZaragozaEmail author
  • Alejandro H. Toselli
  • Enrique Vidal
Short paper


In this paper, we present a hybrid approach using hidden Markov models (HMM) and artificial neural networks to deal with the task of handwritten Music Recognition in mensural notation. Previous works have shown that the task can be addressed with Gaussian density HMMs that can be trained and used in an end-to-end manner, that is, without prior segmentation of the symbols. However, the results achieved using that approach are not sufficiently accurate to be useful in practice. In this work, we hybridize HMMs with deep multilayer perceptrons (MLPs), which lead to remarkable improvements in optical symbol modeling. Moreover, this hybrid architecture maintains important advantages of HMMs such as the ability to properly model variable-length symbol sequences through segmentation-free training, and the simplicity and robustness of combining optical models with N-gram language models, which provide statistical a priori information about regularities in musical symbol concatenation observed in the training data. The results obtained with the proposed hybrid MLP-HMM approach outperform previous works by a wide margin, achieving symbol-level error rates around 26%, as compared with about 40% reported in previous works.


Handwritten music recognition Mensural notation Hidden Markov models Artificial neural networks N-gram Language Models 


Compliance with ethical standards

Conflict of interest

Authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.


  1. 1.
    Bainbridge D, Bell T (2001) The challenge of optical music recognition. Comput Humanit 35(2):95–121CrossRefGoogle Scholar
  2. 2.
    Bertolami R, Bunke H (2008) Hidden markov model-based ensemble methods for offline handwritten text line recognition. Pattern Recognit 41(11):3452–3460CrossRefzbMATHGoogle Scholar
  3. 3.
    Bicego M, Pekalska E, Tax DMJ, Duin RPW (2009) Component-based discriminative classification for hidden markov models. Pattern Recognit 42(11):2637–2648CrossRefzbMATHGoogle Scholar
  4. 4.
    Bosch V, Calvo-Zaragoza J, Toselli AH, Vidal-Ruiz E (2016) Sheet music statistical layout analysis. In: 15th International conference on frontiers in handwriting recognition, ICFHR 2016, Shenzhen, China, 23–26 Oct 2016, pp 313–318Google Scholar
  5. 5.
    Bourlard H, Wellekens C (1990) Links between markov models and multilayer perceptrons. IEEE Trans Pattern Anal Mach Intell 12(11):1167–1178CrossRefGoogle Scholar
  6. 6.
    Calvo-Zaragoza J, Barbancho I, Tardón LJ, Barbancho AM (2015) Avoiding staff removal stage in optical music recognition: application to scores written in white mensural notation. Pattern Anal Appl 18(4):933–943MathSciNetCrossRefGoogle Scholar
  7. 7.
    Calvo-Zaragoza J, Toselli AH, Vidal E (2016) Early handwritten music recognition with hidden markov models. In: 15th International conference on frontiers in handwriting recognition, ICFHR 2016, Shenzhen, China, 23–26 Oct 2016, pp 319–324Google Scholar
  8. 8.
    Calvo-Zaragoza J, Toselli AH, Vidal E (2017) Handwritten music recognition for mensural notation: formulation, data and baseline results. In: 14th International conference on document analysis and recognition, ICDAR 2017, Kyoto, Japan, 13–15 Aug 2017, pp 1081–1086Google Scholar
  9. 9.
    Cardoso JS, Capela A, Rebelo A, Guedes C, Pinto J (2009) Staff detection with stable paths. IEEE Trans Pattern Anal Mach Intell 31(6):1134–1139CrossRefGoogle Scholar
  10. 10.
    Espana-Boquera S, Castro-Bleda MJ, Gorbe-Moya J, Zamora-Martinez F (2011) Improving offline handwritten text recognition with hybrid hmm/ann models. IEEE Trans Pattern Anal Mach Intell 33(4):767–779CrossRefGoogle Scholar
  11. 11.
    Fujinaga I, Hankinson A, Cumming JE (2014) Introduction to SIMSSA (single interface for music score searching and analysis). In: Proceedings of the 1st international workshop on digital libraries for musicology, DLfM@JCDL 2014, London, UK, 12 Sept 2014, pp 1–3Google Scholar
  12. 12.
    Gallego A, Calvo-Zaragoza J (2017) Staff-line removal with selectional auto-encoders. Expert Syst Appl 89:138–148CrossRefGoogle Scholar
  13. 13.
    Goodfellow I, Bengio Y, Courville A (2016) Deep learning. The MIT Press, CambridgezbMATHGoogle Scholar
  14. 14.
    Günter S, Bunke H (2004) Hmm-based handwritten word recognition: on the optimization of the number of states, training iterations and gaussian components. Pattern Recognit 37(10):2069–2079CrossRefGoogle Scholar
  15. 15.
    Hankinson A, Burgoyne JA, Vigliensoni G, Fujinaga I (2012) Creating a large-scale searchable digital collection from printed music materials. In: Proceedings of the 21st world wide web conference, WWW 2012, Lyon, France, 16–20 April 2012 (Companion Volume), pp 903–908Google Scholar
  16. 16.
    Jelinek F (1998) Statistical methods for speech recognition. MIT Press, CambridgeGoogle Scholar
  17. 17.
    Kneser R, Ney H (1995) Improved backing-off for m-gram language modeling. In: International conference on acoustics, speech, and signal processing, ICASSP ’95, Detroit, Michigan, USA, 08–12 May 1995, pp 181–184Google Scholar
  18. 18.
    Lee S, Son SJ, Oh J, Kwak N (2016) Handwritten music symbol classification using deep convolutional neural networks. In: International conference on information science and security (ICISS), 2016. IEEE, pp 1–5Google Scholar
  19. 19.
    Ortmanns S, Ney H, Aubert X (1997) A word graph algorithm for large vocabulary continuous speech recognition. Comput Speech Lang 11(1):43–72CrossRefGoogle Scholar
  20. 20.
    OShaughnessy D (2008) Automatic speech recognition: History, methods and challenges. Pattern Recognit 41(10):2965–2979CrossRefzbMATHGoogle Scholar
  21. 21.
    Povey D (2003) Discriminative training for large vocabulary speech recognition. Ph.D. thesis, University of CambridgeGoogle Scholar
  22. 22.
    Pugin L (2006) Optical music recognition of early typographic prints using hidden markov models. In: Proceedings of the ISMIR 2006, 7th international conference on music information retrieval, Victoria, Canada, Oct 8–12, pp 53–56Google Scholar
  23. 23.
    Rabiner L, Juang BH (1993) Fundamentals of speech recognition. Prentice Hall, Upper Saddle RiverGoogle Scholar
  24. 24.
    Ramirez C, Ohya J (2014) Automatic recognition of square notation symbols in western plainchant manuscripts. J New Music Res 43(4):390–399CrossRefGoogle Scholar
  25. 25.
    Rebelo A, Fujinaga I, Paszkiewicz F, Marçal ARS, Guedes C, Cardoso JS (2012) Optical music recognition: state-of-the-art and open issues. Int J Multimed Inf Retrieval 1(3):173–190CrossRefGoogle Scholar
  26. 26.
    Toselli AH, Juan A, Vidal E (2004) Spontaneous handwriting recognition and classification. In: 17th International conference on pattern recognition, ICPR 2004, Cambridge, UK, 23–26 August 2004, pp 433–436Google Scholar
  27. 27.
    Toselli AH, Romero V, Pastor M, Vidal E (2010) Multimodal interactive transcription of text images. Pattern Recognit 43(5):1814–1825CrossRefzbMATHGoogle Scholar
  28. 28.
    Toselli AH, Romero V, Vidal E (2011) Alignment between text images and their transcripts for handwritten documents. Language Technology for Cultural Heritage, pp 23–37Google Scholar
  29. 29.
    Vidal E, Thollard F, De La Higuera C, Casacuberta F, Carrasco RC (2005) Probabilistic finite-state machines-part ii. IEEE Trans Pattern Anal Mach Intell 27(7):1026–1039CrossRefGoogle Scholar
  30. 30.
    Yang W, Tao J, Ye Z (2016) Continuous sign language recognition using level building based on fast hidden markov model. Pattern Recognit Lett 78:28–35CrossRefGoogle Scholar
  31. 31.
    Young S, Evermann G, Gales M, Hain T, Kershaw D, Liu X, Moore G, Odell J, Ollason D, Povey D, et al (2015) The HTK book, vol 3.5. Entropic Cambridge Research Laboratory, CambridgeGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.PRHLT Research CenterUniversitat Politècnica de ValènciaValenciaSpain

Personalised recommendations