Markov constraints: steerable generation of Markov sequences


Markov chains are a well known tool to model temporal properties of many phenomena, from text structure to fluctuations in economics. Because they are easy to generate, Markovian sequences, i.e. temporal sequences having the Markov property, are also used for content generation applications such as text or music generation that imitate a given style. However, Markov sequences are traditionally generated using greedy, left-to-right algorithms. While this approach is computationally cheap, it is fundamentally unsuited for interactive control. This paper addresses the issue of generating steerable Markovian sequences. We target interactive applications such as games, in which users want to control, through simple input devices, the way the system generates a Markovian sequence, such as a text, a musical sequence or a drawing. To this aim, we propose to revisit Markov sequence generation as a branch and bound constraint satisfaction problem (CSP). We propose a CSP formulation of the basic Markovian hypothesis as elementary Markov Constraints (EMC). We propose algorithms that achieve domain-consistency for the propagators of EMCs, in an event-based implementation of CSP. We show how EMCs can be combined to estimate the global Markovian probability of a whole sequence, and accommodate for different species of Markov generation such as fixed order, variable-order, or smoothing. Such a formulation, although more costly than traditional greedy generation algorithms, yields the immense advantage of being naturally steerable, since control specifications can be represented by arbitrary additional constraints, without any modification of the generation algorithm. We illustrate our approach on simple yet combinatorial chord sequence and melody generation problems and give some performance results.


  1. 1.

    Addessi, A.-R., & Pachet, F. (2005). Experiments with a musical machine: Musical style replication in 3/5 year old children. British Journal of Music Education, 22(1), 21–46.

    Article  Google Scholar 

  2. 2.

    Allauzen, C., Crochemore, M., & Raffinot, M. (1999). Factor oracle: A new structure for pattern matching. In J. Pavelka, G. Tel, & M. Bartosek (Eds.), Proceedings of the 26th conference on current trends in theory and practice of informatics (November 27–Dec. 4), LNCS (Vol. 1725, pp. 295–310). London: Springer.

    Google Scholar 

  3. 3.

    Anders, T., & Miranda, E. R. (2008). Constraint-based composition in real time. In Proc. of int. computer music conference, Belfast, UK.

    Google Scholar 

  4. 4.

    Anders, T., & Miranda, E. R. (2010). Constraint programming systems for modeling music theories and composition. ACM Computing Surveys (in press).

  5. 5.

    Assayag, G., Dubnov, S., & Delerue, O. (1999). Guessing the composer’s mind: Applying universal prediction to musical style. In Proc. of int. computer music conference, Bejing, China, ICMA.

    Google Scholar 

  6. 6.

    Bejerano, G. (2004). Algorithms for variable length Markov chain modeling. Bioinformatics, 20(5), 788–789.

    Article  Google Scholar 

  7. 7.

    Beldiceanu, N., Carlsson, M., & Demassey, S. (2007). Global constraint catalogue: Past, present and future. Constraints, 12(1), 32–62

    MathSciNet  Google Scholar 

  8. 8.

    Brooks, F. P., Hopkins, A. L., Neumann, P. G., & Wright, W. V. (1993). An experiment in musical composition. In S. M. Schwanauer & D. A. Levitt (Eds.), Machine models of music (pp. 23–40). Cambridge: MIT.

    Google Scholar 

  9. 9.

    Conklin, D. (2003). Music generation from statistical models. In Proc. of AISB 2003 symposium on artificial intelligence and creativity in the arts and sciences (pp. 30–35).

  10. 10.

    Conklin, D., & Witten, I. H. (1995). Multiple viewpoint systems for music prediction. Journal of New Music Research, 24(1), 51–73.

    Article  Google Scholar 

  11. 11.

    Cont, A., Dubnov, S., & Assayag, G. (2007). Anticipatory model of musical style imitation using collaborative and competitive reinforcement learning. In Anticipatory behavior in adaptive learning systems. LNCS (Vol. 4520/2007, pp. 285–306). Berlin: Springer.

    Google Scholar 

  12. 12.

    Demassey, S., Pesant, G., & Rousseau, L.-M. (2006). A cost-regular based hybrid column generation approach. Constraints, 11(4), 315–333.

    Article  MATH  MathSciNet  Google Scholar 

  13. 13.

    Di Meola, A., McLaughlin, J., & De Lucia, P. (1992). Friday night in San Francisco. Milwaukee: Hal Leonard Corporation.

    Google Scholar 

  14. 14.

    Eigenfield, A., & Pasquier, P. (2010). Realtime generation of harmonic progressions using controlled markov selection. In Proc. of the 1st international conference on computational creativity (ICCCX) (pp. 16–25). Lisbon: ACM.

    Google Scholar 

  15. 15.

    Farbood, M., & Schoner, B. (2001). Analysis and synthesis of palestrina-style counterpoint using Markov chains. In Proc. of international computer music conference, Cuba (pp. 471–474).

  16. 16.

    Ilog (2001) Ilog solver 5.2 reference manual. Available online at

  17. 17.

    Johnson, D. S. (1974). Approximation algorithms for combinatorial problems. Journal of Computer and System Sciences, 9(3), 256–278.

    Article  MATH  MathSciNet  Google Scholar 

  18. 18.

    Marinescu, R., & Dechter, R. (2005). Advances in AND/OR branch-and-bound search for constraint optimization. In The 7th international workshop on preferences and soft constraints of the eleventh international conference on principles and practice of constraint programming, CP’2005.

  19. 19.

    Markov Constraints (2010). Available online at Accessed September 2010.

  20. 20.

    Menana, J., & Demassey, S. (2009). Sequencing and counting with the multicost-regular constraint. In Lecture notes in computer science, 6th international conference on integration of AI and OR techniques in constraint programming for combinatorial optimization problems (CPAIOR’09), Pittsburgh, USA.

  21. 21.

    Nierhaus, G. (2009). Algorithmic composition—paradigms of automated music generation. Berlin: Springer.

    Google Scholar 

  22. 22.

    Pachet, F. (2002). The continuator: Musical interaction with style. In Proc. of international computer music conference (pp. 211–218). Goteborg, Sweden. September, Best Paper Award.

  23. 23.

    Pachet, F., Roy, P., & Cazaly, D. (2000). A combinatorial approach to content-based music selection. IEEE Multimedia, 7(1), 44–51.

    Article  Google Scholar 

  24. 24.

    Parker, C. (2001). Charlie Parker omnibook: For C instruments. Van Nuys: Alfred Publishing.

    Google Scholar 

  25. 25.

    Pearce, M., & Wiggins, G. (2004). Improved methods for statistical modeling of monophonic music. Journal of New Music Research, 33(4), 367–385.

    Article  Google Scholar 

  26. 26.

    Perrot, J.-F. (1977). Informatique et Algèbre, La théorie des codes à longueur variable. In Theoretical computer science, G. I. conference, lecture notes in computer science (3rd Ed., no. 48, pp. 27–44).

  27. 27.

    Pesant, G. (2004). A regular language membership constraint for finite sequences of variables. Lecture Notes in Computer Science, 32(58), 482–495.

    Article  Google Scholar 

  28. 28.

    Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286.

    Article  Google Scholar 

  29. 29.

    Radicioni, D. P., & Lombardo, V. (2007). A constraint-based approach for annotating music scores with gestural information. Constraints, 12(4), 405–428.

    Article  MATH  MathSciNet  Google Scholar 

  30. 30.

    Régin, J.-C. (1994). A filtering algorithm for constraints of difference in CSPs. In Proc. of the 12th national conference on artificial intelligence. AAAI 1994 (pp. 362–367).

  31. 31.

    Ron, D., Singer, Y., & Tishby, N. (1996). The power of amnesia: Learning probabilistic automata with variable memory length. Machine Learning, 25(2–3), 117–149.

    Article  MATH  Google Scholar 

  32. 32.

    Rossi, F., van Beek, P., & Walsh, T. (Eds.) (2006). Handbook of constraint programming. New York: Elsevier.

    Google Scholar 

  33. 33.

    Simhon, S., & Dudek, G. (2004). Sketch interpretation and refinement using statistical models. In H. W. Jensen, & A. Keller (Eds.), Eurographics symposium on rendering.

  34. 34.

    Steedman, M. J. (1984). A generative grammar for jazz chord sequences. Music Perception, 2(1), 52–77.

    Google Scholar 

  35. 35.

    van Hoeve, W. J., & Katriel, I. (2006). Global constraints. Handbook of constraint programming (chap. 6). New York: Elsevier Science Inc.

    Google Scholar 

  36. 36.

    van Hoeve, W.-J., Pesant, G., Rousseau, L.-M., & Sabharwal, A. (2006). Revisiting the sequence constraint. In Proc. of the 12th international conference on principles and practice of constraint programming (CP 2006). LNCS 4204 (pp. 620–634).

  37. 37.

    Witten, I. H., & Bell, T. C. (1991). The zero-frequency problem: Estimating the probabilities of novel events in adaptive text compression. IEEE Transaction on Information Theory, 37(4), 1085–1094.

    Article  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to François Pachet.

Rights and permissions

Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (, which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Reprints and Permissions

About this article

Cite this article

Pachet, F., Roy, P. Markov constraints: steerable generation of Markov sequences. Constraints 16, 148–172 (2011).

Download citation


  • CSP
  • Markov chains
  • Sequence constraints
  • Global constraints
  • Interactive applications