Interactive Generation of Calligraphic Trajectories from Gaussian Mixtures

  • Daniel BerioEmail author
  • Frederic Fol Leymarie
  • Sylvain Calinon
Part of the Unsupervised and Semi-Supervised Learning book series (UNSESUL)


The chapter presents an approach for the interactive definition of curves and motion paths based on Gaussian mixture model (GMM) and optimal control. The input of our method is a mixture of multivariate Gaussians defined by the user, whose centers define a sparse sequence of key-points, and whose covariances define the precision required to pass through these key-points. The output is a dynamical system generating curves that are natural looking and reflect the kinematics of a movement, similar to that produced by human drawing or writing. In particular, the stochastic nature of the GMM combined with optimal control is exploited to generate paths with natural variations, which are defined by the user within a simple interactive interface. Several properties of the Gaussian mixture are exploited in this application. First, there is a direct link between multivariate Gaussian distributions and optimal control formulations based on quadratic objective functions (linear quadratic tracking), which is exploited to extend the GMM representation to a controller. We then exploit the option of tying the covariances in the GMM to modulate the style of the calligraphic trajectories. The approach is tested to generate curves and traces that are geometrically and dynamically similar to the ones that can be seen in art forms such as calligraphy or graffiti.


Optimal control Model predictive control Stochastic dynamical system Non-photorealistic rendering Calligraphy and graffiti generation User interface 


  1. 1.
    AlMeraj, Z., Wyvill, B., Isenberg, T., Gooch, A., Guy, R.: Automatically mimicking unique hand-drawn pencil lines. Comput. Graph. 33(4), 496–508 (2009)CrossRefGoogle Scholar
  2. 2.
    Baran, I., Lehtinen, J., Popović, J.: Sketching clothoid splines using shortest paths. In: Computer Graphics Forum, vol. 29, pp. 655–664. Wiley, London (2010)CrossRefGoogle Scholar
  3. 3.
    Berio, D., Calinon, S., Fol Leymarie, F.: Learning dynamic graffiti strokes with a compliant robot. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3981–3986. IEEE, Piscataway (2016)Google Scholar
  4. 4.
    Berio, D., Calinon, S., Fol Leymarie, F.: Generating calligraphic trajectories with model predictive control. In: Proceedings of the 43rd Conference on Graphics Interface, pp. 132–139. Canadian Human-Computer Communications Society School of Computer Science, Waterloo (2017)Google Scholar
  5. 5.
    Berio, D., Leymarie, F.F., Plamondon, R.: Expressive curve editing with the sigma lognormal model. In: Diamanti, O., Vaxman, A. (eds.) EG 2018—Short Papers. The Eurographics Association (2018)Google Scholar
  6. 6.
    Bernstein, N.A., Latash, M.L., Turvey, M.: Dexterity and Its Development. Taylor & Francis, London (1996)Google Scholar
  7. 7.
    Calinon, S.: A tutorial on task-parameterized movement learning and retrieval. Intell. Serv. Robot. 9(1), 1–29 (2016)CrossRefGoogle Scholar
  8. 8.
    Cooper, M., Chalfant, H.: Subway Art. Rinehart and Winston, Holt (1984)Google Scholar
  9. 9.
    d’Avella, A., Saltiel, P., Bizzi, E.: Combinations of muscle synergies in the construction of a natural motor behavior. Nat. Neurosci. 6(3), 300–308 (2003)CrossRefGoogle Scholar
  10. 10.
    De Boor, C.: A Practical Guide to Splines, vol. 27. Springer, New York (1978)zbMATHCrossRefGoogle Scholar
  11. 11.
    Dooijes, E.: Analysis of handwriting movements. Acta Psychol. 54(1), 99–114 (1983)CrossRefGoogle Scholar
  12. 12.
    Edelman, S., Flash, T.: A model of handwriting. Biol. Cybern. 57(1–2), 25–36 (1987)CrossRefGoogle Scholar
  13. 13.
    Egerstedt, M., Martin, C.: Control Theoretic Splines: Optimal Control, Statistics, and Path Planning. Princeton University Press; Princeton Oxford, Princeton (2009)Google Scholar
  14. 14.
    Ferrer, M.A., Diaz, M., Carmona-Duarte, C., Morales, A.: A behavioral handwriting model for static and dynamic signature synthesis. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1041–1053 (2017)CrossRefGoogle Scholar
  15. 15.
    Flash, T., Handzel, A.: Affine differential geometry analysis of human arm movements. Biol. Cybern. 96(6), 577–601 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Flash, T., Henis, E.: Arm trajectory modifications during reaching towards visual targets. J. Cogn. Neurosci. 3(3), 220–230 (1991)CrossRefGoogle Scholar
  17. 17.
    Flash, T., Hochner, B.: Motor primitives in vertebrates and invertebrates. Curr. Opin. Neurobiol. 15(6), 660–666 (2005)CrossRefGoogle Scholar
  18. 18.
    Flash, T., Hogan, N.: The coordination of arm movements. J. Neurosci. 5(7), 1688–1703 (1985)CrossRefGoogle Scholar
  19. 19.
    Flash, T., Hogan, N.: Optimization principles in motor control. In: The Handbook of Brain Theory and Neural Networks, pp. 682–685. MIT Press, Cambridge, MA (1998)Google Scholar
  20. 20.
    Freedberg, D., Gallese, V.: Motion, emotion and empathy in esthetic experience. Trends Cogn. Sci. 11(5), 197–203 (2007)CrossRefGoogle Scholar
  21. 21.
    Freeman, F.: Experimental analysis of the writing movement. Psychol. Monogr. Gen. Appl. 17(4), 1–57 (1914)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Fujioka, H., Kano, H., Nakata, H., Shinoda, H.: Constructing and reconstructing characters, words, and sentences by synthesizing writing motions. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 36(4), 661–670 (2006)CrossRefGoogle Scholar
  23. 23.
    Haeberli, P.: Dynadraw: A dynamic drawing technique. (1989)
  24. 24.
    House, D., Singh, M.: Line drawing as a dynamic process. In: Proceedings of the 15th Pacific Conference on Computer Graphics and Applications, pp. 351–60. IEEE, Piscataway (2007)Google Scholar
  25. 25.
    Jordan, M., Wolpert, D.: Computational motor control. In: Gazzaniga, M. (ed.) The Cognitive Neurosciences, 2nd edn. MIT Press, Cambridge, MA (1999)Google Scholar
  26. 26.
    Kyprianidis, J., Collomosse, J., Wang, T., Isenberg, T.: State of the “art”: A taxonomy of artistic stylization techniques for images and video. IEEE Trans. Vis. Comput. Graph. 19(5), 866–885 (2013)CrossRefGoogle Scholar
  27. 27.
    Lacquaniti, F., Terzuolo, C., Viviani, P.: The law relating the kinematic and figural aspects of drawing movements. Acta Psychol. 54(1), 115–130 (1983)CrossRefGoogle Scholar
  28. 28.
    Leder, H., Bär, S., Topolinski, S.: Covert painting simulations influence aesthetic appreciation of artworks. Psychol. Sci. 23(12), 1479–1481 (2012)CrossRefGoogle Scholar
  29. 29.
    Longcamp, M., Anton, J.L., Roth, M., Velay, J.L.: Visual presentation of single letters activates a premotor area involved in writing. NeuroImage 19(4), 1492–1500 (2003)CrossRefGoogle Scholar
  30. 30.
    Lu, J., Yu, F., Finkelstein, A., DiVerdi, S.: Helpinghand: Example-based stroke stylization. ACM Trans. Graph. 31(4), 46 (2012)CrossRefGoogle Scholar
  31. 31.
    McCrae, J., Singh, K.: Sketching piecewise clothoid curves. Comput. Graph. 33(4), 452–461 (2009)CrossRefGoogle Scholar
  32. 32.
    Mediavilla, C., Marshall, A., van Stone, M., Xuriguera, G., Jackson, D.: Calligraphy: from calligraphy to abstract painting. Scirpus (1996)Google Scholar
  33. 33.
    Meirovitch, Y., Bennequin, D., Flash, T.: Geometrical invariance and smoothness maximization for task-space movement generation. IEEE Trans. Robot. 32(4), 837–853 (2016)CrossRefGoogle Scholar
  34. 34.
    Morasso, P.: Spatial control of arm movements. Exp. Brain Res. 42(2), 223–7 (1981)CrossRefGoogle Scholar
  35. 35.
    Pignocchi, A.: How the intentions of the draftsman shape perception of a drawing. Conscious. Cogn. 19(4), 887–898 (2010)CrossRefGoogle Scholar
  36. 36.
    Plamondon, R.: A kinematic theory of rapid human movements. Part I. Biol. Cybern. 72(4), 295–307 (1995)zbMATHCrossRefGoogle Scholar
  37. 37.
    Plamondon, R., O’Reilly, C., Galbally, J., Almaksour, A., Anquetil, É.: Recent developments in the study of rapid human movements with the kinematic theory. Pattern Recogn. Lett. 35, 225–35 (2014)CrossRefGoogle Scholar
  38. 38.
    Shoemake, K.: Arcball: A user interface for specifying three-dimensional orientation using a mouse. In: Graphics Interface, vol. 92, pp. 151–156 (1992)Google Scholar
  39. 39.
    Tanwani, A., Calinon, S.: Learning robot manipulation tasks with task-parameterized semitied hidden semi-Markov model. IEEE Robot. Autom. Lett. 1(1), 235–242 (2016)CrossRefGoogle Scholar
  40. 40.
    Thiel, Y., Singh, K., Balakrishnan, R.: Elasticurves: Exploiting stroke dynamics and inertia for the real-time neatening of sketched 2D curves. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pp. 383–392. ACM, New York (2011)Google Scholar
  41. 41.
    Todorov, E., Jordan, M.: Optimal feedback control as a theory of motor coordination. Nat. Neurosci. 5(11), 1226–1235 (2002)CrossRefGoogle Scholar
  42. 42.
    Todorov, E., Jordan, M.I.: Smoothness maximization along a predefined path accurately predicts the speed profiles of complex arm movements. J. Neurophysiol. 80(2), 696–714 (1998)CrossRefGoogle Scholar
  43. 43.
    Uno, Y., Kawato, M., Suzuki, R.: Formation and control of optimal trajectory in human multijoint arm movement. Biol. Cybern. 61(2), 89–101 (1989)CrossRefGoogle Scholar
  44. 44.
    Viviani, P., Schneider, R.: A developmental study of the relationship between geometry and kinematics in drawing movements. J. Exp. Psychol. Hum. Percept. Perform. 17(1), 198–218 (1991)CrossRefGoogle Scholar
  45. 45.
    Zeestraten, M., Calinon, S., Caldwell, D.G.: Variable duration movement encoding with minimal intervention control. In: Proceedings of the International Conference on Robotics and Automation (ICRA), pp. 497–503. IEEE, Stockholm, Sweden (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Daniel Berio
    • 1
    Email author
  • Frederic Fol Leymarie
    • 1
  • Sylvain Calinon
    • 2
  1. 1.GoldsmithsUniversity of LondonLondonUK
  2. 2.Idiap Research InstituteMartignySwitzerland

Personalised recommendations