Towards Polyphony Reconstruction Using Multidimensional Multiple Sequence Alignment

  • Dimitrios BountouridisEmail author
  • Frans Wiering
  • Dan Brown
  • Remco C. Veltkamp
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10198)


The digitization of printed music scores through the process of optical music recognition is imperfect. In polyphonic scores, with two or more simultaneous voices, errors of duration or position can lead to badly aligned and inharmonious digital transcriptions. We adapt biological sequence analysis tools as a post-processing step to correct the alignment of voices. Our multiple sequence alignment approach works on multiple musical dimensions and we investigate the contribution of each dimension to the correct alignment. Structural information, such musical phrase boundaries, is of major importance; therefore, we propose the use of the popular bioinformatics aligner Mafft which can incorporate such information while being robust to temporal noise. Our experiments show that a harmony-aware Mafft outperforms sophisticated, multidimensional alignment approaches and can achieve near-perfect polyphony reconstruction.


Multiple Sequence Alignment Dynamic Time Warping Segment Boundary Substitution Matrix Music Information Retrieval 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors would like to thank Meinard Müller and Hendrik Vincent Koops for comments that greatly improved the manuscript.


  1. 1.
    Blackburne, B.P., Whelan, S.: Measuring the distance between multiple sequence alignments. Bioinformatics 28(4), 495–502 (2012)CrossRefGoogle Scholar
  2. 2.
    Boulanger-Lewandowski, N., Bengio, Y., Vincent, P.: Modeling temporal dependencies in high-dimensional sequences: application to polyphonic music generation and transcription. arXiv preprint arXiv:1206.6392 (2012)
  3. 3.
    Bountouridis, D., Koops, H.V., Wiering, F., Veltkamp, R.C.: A data-driven approach to chord similarity and chord mutability. In: International Conference on Multimedia Big Data, pp. 275–278 (2016)Google Scholar
  4. 4.
    Cambouropoulos, E.: The local boundary detection model (LBDM) and its application in the study of expressive timing. In: International Computer Music Conference, pp. 17–22 (2001)Google Scholar
  5. 5.
    Carrillo, H., Lipman, D.: The multiple sequence alignment problem in biology. SIAM J. Appl. Math. 48(5), 1073–1082 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Carroll, H., Clement, M.J., Ridge, P., Snell, Q.O.: Effects of gap open and gap extension penalties. In: Biotechnology and Bioinformatics Symposium, pp. 19–23 (2006)Google Scholar
  7. 7.
    Dayhoff, M.O., Schwartz, R.M., Orcutt, B.C.: 22 a model of evolutionary change in proteins. In: Atlas of Protein Sequence and Structure, vol. 5, pp. 345–352. National Biomedical Research Foundation Silver Spring, MD (1978)Google Scholar
  8. 8.
    Durbin, R., Eddy, S.R., Krogh, A., Mitchison, G.: Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press, Cambridge (1998)CrossRefzbMATHGoogle Scholar
  9. 9.
    Feng, D.-F., Doolittle, R.F.: Progressive sequence alignment as a prerequisitetto correct phylogenetic trees. J. Mol. Evol. 25(4), 351–360 (1987)CrossRefGoogle Scholar
  10. 10.
    Henikoff, S., Henikoff, J.G.: Amino acid substitution matrices from protein blocks. Proc. Natl. Acad. Sci. 89(22), 10915–10919 (1992)CrossRefGoogle Scholar
  11. 11.
    Hogeweg, P., Hesper, B.: The alignment of sets of sequences and the construction of phyletic trees: an integrated method. J. Mol. Evol. 20(2), 175–186 (1984)CrossRefGoogle Scholar
  12. 12.
    Hudek, A.K.: Improvements in the accuracy of pairwise genomic alignment (2010)Google Scholar
  13. 13.
    Joh, C.-H., Arentze, T., Hofman, F., Timmermans, H.: Activity pattern similarity: a multidimensional sequence alignment method. Transp. Res. Part B Methodol. 36(5), 385–403 (2002)CrossRefGoogle Scholar
  14. 14.
    Katoh, K., Misawa, K., Kuma, K., Miyata, T.: Mafft: a novel method for rapid multiple sequence alignment based on fast fourier transform. Nucleic Acids Res. 30(14), 3059–3066 (2002)CrossRefGoogle Scholar
  15. 15.
    Lartillot, O., Toiviainen, P., Eerola, T.: A matlab toolbox for music information retrieval. In: Preisach, C., Burkhardt, H., Schmidt-Thieme, L., Decker, R. (eds.) Data Analysis, Machine Learning and Applications, pp. 261–268. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  16. 16.
    Lerdahl, F., Jackendoff, R.: An overview of hierarchical structure in music. Music Percept. Interdisc. J. 1(2), 229–252 (1983)CrossRefGoogle Scholar
  17. 17.
    Lyu, Q., Wu, Z., Zhu, J., Meng, H.: Modelling high-dimensional sequences with LSTM-RTRBM: application to polyphonic music generation. In: International Conference on Artificial Intelligence, pp. 4138–4139. AAAI Press (2015)Google Scholar
  18. 18.
    Morgenstern, B., Dress, A., Werner, T.: Multiple dna and protein sequence alignment based on segment-to-segment comparison. Proc. Natl. Acad. Sci. 93(22), 12098–12103 (1996)CrossRefzbMATHGoogle Scholar
  19. 19.
    Pugin, L., Crawford, T.: Evaluating omr on the early music online collection. In: International Society on Music, Information Retrieval, pp. 439–444 (2013)Google Scholar
  20. 20.
    Rebelo, A., Fujinaga, I., Paszkiewicz, F., Marcal, A.R.S., Guedes, C., Cardoso, J.S.: Optical music recognition: state-of-the-art and open issues. Int. J. Multimedia Inf. Retrieval 1(3), 173–190 (2012)CrossRefGoogle Scholar
  21. 21.
    Rodríguez López, M.E.: Automatic Melody Segmentation. Ph.D. thesis, Utrecht University (2016)Google Scholar
  22. 22.
    Sanguansat, P.: Multiple multidimensional sequence alignment using generalized dynamic time warping. WSEAS Trans. Math. 11(8), 668–678 (2012)Google Scholar
  23. 23.
    Tenney, J., Polansky, L.: Temporal gestalt perception in music. J. Music Theory 24(2), 205–241 (1980)CrossRefGoogle Scholar
  24. 24.
    Thompson, J.D., Higgins, D.G., Gibson, T.J.: Clustal w: improving the sensitivity of progressive multiple sequence alignment through sequence weighting, position-specific gap penalties and weight matrix choice. Nucleic Acids Res. 22(22), 4673–4680 (1994)CrossRefGoogle Scholar
  25. 25.
    van Kranenburg, P.: A computational approach to content-based retrieval of folk song melodies. Ph.D. thesis (2010)Google Scholar
  26. 26.
    Volk, A., Garbers, J., Van Kranenburg, P., Wiering, F., Veltkamp, R.C., Grijp, L.P.: Applying rhythmic similarity based on inner metric analysis to folksong research. In: International Society on Music Information Retrieval, pp. 293–296 (2007)Google Scholar
  27. 27.
    Wang, L., Jiang, T.: On the complexity of multiple sequence alignment. J. Comput. Biol. 1(4), 337–348 (1994)CrossRefGoogle Scholar
  28. 28.
    Wang, S., Ewert, S., Dixon, S.: Robust joint alignment of multiple versions of a piece of music, pp. 83–88 (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Dimitrios Bountouridis
    • 1
    Email author
  • Frans Wiering
    • 1
  • Dan Brown
    • 2
  • Remco C. Veltkamp
    • 1
  1. 1.Department of Information and Computing SciencesUtrecht UniversityUtrechtNetherlands
  2. 2.David R. Cheriton School of Computer ScienceUniversity of WaterlooWaterlooCanada

Personalised recommendations