The Visual Computer

, Volume 31, Issue 12, pp 1569–1586 | Cite as

Correlation-optimized time warping for motion

  • S. Ali Etemad
  • Ali Arya
Original Article


Retrieval and comparative editing/modeling of motion data require temporal alignment. In other words, for such processes to perform accurately, critical features of motion sequences need to occur simultaneously. In this paper, we propose correlation-optimized time warping (CoTW) for aligning motion data. CoTW utilizes a correlation-based objective function for characterizing alignment. The method solves an optimization problem to determine the optimum warping degree for different segments of the sequence. Using segment-wise interpolated warping, smooth motion trajectories are achieved that can be readily used for animation. Our method allows for manual tuning of the parameters, resulting in high customizability with respect to the number of actions in a single sequence as well as spatial regions of interest within the character model. Moreover, measures are taken to reduce distortion caused by over-warping. The framework also allows for automatic selection of an optimum reference when multiple sequences are available. Experimental results demonstrate the very accurate performance of CoTW compared to other techniques such as dynamic time warping, derivative dynamic time warping and canonical time warping. The mentioned customization capabilities are also illustrated.


Motion analysis Time warping  Temporal alignment  Correlation Optimization 



This work was supported in part by the Natural Sciences and Engineering Council of Canada (NSERC) and Ontario Centers of Excellence (OCE).

Supplementary material

ESM 1 (WMV 34,781 KB)

ESM 2 (WMV 4,513 KB)


  1. 1.
    Aggarwal, J.K., Ryoo, M.S.: Human activity analysis: a review. ACM Comput. Surv. 43(3), 16 (2011)CrossRefGoogle Scholar
  2. 2.
    Etemad, S.A., Arya, A.: Modeling and transformation of 3D human motion. In: Proceedings of 5th international conference on computer graphics theory and applications, pp. 307–315 (2010)Google Scholar
  3. 3.
    Amaya, K., Bruderlin, A., Calvert, T.: Emotion from motion. In: Proceedings of graphics interface, pp. 222–229 (1996)Google Scholar
  4. 4.
    Rose, C., Cohen, M.F., Bodenheimer, B.: Verbs and adverbs: multidimensional motion interpolation. IEEE Comput. Graph. Appl. 18(5), 32–40 (1998)CrossRefGoogle Scholar
  5. 5.
    Etemad, S.A., Arya, A.: Extracting movement, posture, and temporal style features from human motion. Biol. Inspir. Cognit. Archit. 7, 15–25 (2013)Google Scholar
  6. 6.
    Etemad, S.A., Arya, A.: Classification and translation of style and affect in human motion using RBF neural networks. Neurocomputing 129, 585–595 (2013)CrossRefGoogle Scholar
  7. 7.
    Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. Speech. Signal Process. 26(1), 43–49 (1978)CrossRefzbMATHGoogle Scholar
  8. 8.
    Zhou, F., De la Torre, F.L.: Canonical time warping for alignment of human behavior. In: Proceedings NIPS, pp. 1–9 (2009)Google Scholar
  9. 9.
    Hsu, E., Pulli, K., Popovic, F.: Style translation for human motion. ACM Trans. Graph. 24(3), 1082–1089 (2005)CrossRefGoogle Scholar
  10. 10.
    Vest Nielsen, N.-P., Carstensen, J.M., Smedsgaard, J.: Aligning of single and multiple wavelength chromatographic profiles for chemometric data analysis using correlation optimised warping. J. Chromatogr. A 805, 17–35 (1998)CrossRefGoogle Scholar
  11. 11.
    Etemad, S.A., Arya, A.: A customizable time warping method for motion alignment. In: Proceedings of the 7th IEEE international conference on semantic computing, pp. 387–388 (2013)Google Scholar
  12. 12.
    Etemad, S.A., Arya, A., Parush, A.: Additivity in perception of affect from limb motion. Neurosci. Lett. 558, 132–136 (2014)CrossRefGoogle Scholar
  13. 13.
    Fu, A.W.-C., Keogh, E., Yung Lau, L., Ratanamahatana, C.A., Chi-Wing Wong, R.: Scaling and time warping in time series querying. Very Large Data Bases J. 17(4), 899–921 (2008)CrossRefGoogle Scholar
  14. 14.
    Keogh, E.J., Pazzani, M.J.: Derivative dynamic time warping. In 1st SIAM international conference on data mining. (2001)Google Scholar
  15. 15.
    Rabiner, L., Juang, B.-H.: Fundamentals of speech recognition. Prentice Hall (1993)Google Scholar
  16. 16.
    Myers, C., Rabiner, L., Rosenberg, A.: Performance tradeoffs in dynamic time warping algorithms for isolated word recognition. IEEE Trans. Acoust. Speech Signal Process. 28(6), 623–635 (1980)CrossRefzbMATHGoogle Scholar
  17. 17.
    Lemire, D.: Faster retrieval with a two-pass dynamic-time-warping lower bound. Pattern Recognit. 42(9), 2169–2180 (2009)CrossRefzbMATHGoogle Scholar
  18. 18.
    Zhou, F., De la Torre, F.: Generalized time warping for multi-modal alignment of human motion. In IEEE conference on computer vision and pattern recognition, pp. 1282–1289 (2012)Google Scholar
  19. 19.
    Shapiro, A., Cao, Y., Faloutsos, P.: Style components. In graphics interface, pp. 33–39 (2006)Google Scholar
  20. 20.
    Liu, G., Pan, Z., Lin, Z.: Style subspaces for character animation. J. Vis. Comput. Anim. 19(3–4), 199–209 (2008)Google Scholar
  21. 21.
    Heloir, A., Courty, N., Gibet, S., Multon, F.: Temporal alignment of communicative gesture sequences. J. Vis. Comput. Anim. 17(3–4), 347–357 (2006)Google Scholar
  22. 22.
    Listgarten, J., Neal, R.M., Roweis, S.T., Emili, A.: Multiple alignment of continuous time series. In NIPS, 817–824 (2005)Google Scholar
  23. 23.
    Brand, M., Oliver, N., Pentland, A.: Coupled hidden Markov models for complex action recognition. In IEEE conference on computer vision and pattern recognition, pp. 994–999 (1997)Google Scholar
  24. 24.
    Brand, M., Hertzmann, A.: Style machines. In SIGGRAPH, pp. 183–192 (2000)Google Scholar
  25. 25.
    Taylor, G.W., Hinton, G.E., Roweis, S.T.: Modeling human motion using binary latent variables. In NIPS, pp. 1345 (2007)Google Scholar
  26. 26.
    Bruderlin, A., Williams, A.: Motion signal processing. In: Proceedings of the 22nd annual conference on computer graphics and interactive techniques, pp. 97–104 (1995)Google Scholar
  27. 27.
    Witkin, A., Popovic, Z.: Motion warping. In: Proceedings of the 22nd annual conference on computer graphics and interactive techniques, pp. 105–108 (1995)Google Scholar
  28. 28.
    Gleicher, M.: Retargetting motion to new characters. In: Proceedings of the 25th annual conference on computer graphics and interactive techniques, pp. 33–42 (1998)Google Scholar
  29. 29.
    Kovar, L., Gleicher, L.: Flexible automatic motion blending with registration curves. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on computer animation, pp. 214–224 (2003)Google Scholar
  30. 30.
    Kovar, L., Gleicher, M.: Automated extraction and parameterization of motions in large data sets. ACM Trans. Graph. 23(3), 559–568 (2004)CrossRefGoogle Scholar
  31. 31.
    Müller, M., Röder, T., Clausen, M.: Efficient content-based retrieval of motion capture data. ACM Trans. Graph. 24(3), 677–685 (2005)CrossRefGoogle Scholar
  32. 32.
    Müller, M., Röder, T.: Motion templates for automatic classification and retrieval of motion capture data. In: Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on computer animation, pp. 137–146 (2006)Google Scholar
  33. 33.
    Zhou, F., De la Torre, F., Hodgins, J.K.: Aligned cluster analysis for temporal segmentation of human motion. In 8th IEEE international conference on automatic face and gesture recognition, pp. 1–7 (2008)Google Scholar
  34. 34.
    Kim, M., Hyun, K., Kim, J., Lee, J.: Synchronized multi-character motion editing. ACM Trans. Graph. 28(3), 79 (2009)Google Scholar
  35. 35.
    Raptis, M., Kirovski, D., Hoppe, H.: Real-time classification of dance gestures from skeleton animation. In: Proceedings of the 2011 ACM SIGGRAPH/Eurographics symposium on computer animation, pp. 147–156 (2011)Google Scholar
  36. 36.
    Cimen, G., Ilhan, H., Capin, T., Gurcay, H.: Classification of human motion based on affective state descriptors. Comput. Anim. Virtual Worlds 24(3–4), 355–363 (2013)CrossRefGoogle Scholar
  37. 37.
    Hsu, E., da Silva, M., Popović, J.: Guided time warping for motion editing. In: Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on computer animation, pp. 45–52 (2007)Google Scholar
  38. 38.
    Caspi, Y., Irani, M.: Aligning non-overlapping sequences. Int. J. Comput. Vis. 48(1), 39–51 (2002)CrossRefzbMATHGoogle Scholar
  39. 39.
    Caspi, Y., Irani, M.: Spatio-temporal alignment of sequences. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1409–1424 (2002)CrossRefGoogle Scholar
  40. 40.
    Padua, F.L.C., Carceroni, R.L., Santos, G.A.M.R., Kutulakos, K.N.: Linear sequence-to-sequence alignment. IEEE Trans. Pattern Anal. Mach. Intell. 32(2), 304–320 (2010)CrossRefGoogle Scholar
  41. 41.
    Junejo, I.N., Dexter, E., Laptev, I., Perez, P.: View-independent action recognition from temporal self-similarities. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 172–185 (2011)CrossRefGoogle Scholar
  42. 42.
    Lu, C., Mandal, M.: A robust technique for motion-based video sequences temporal alignment. IEEE Trans. Multimed. 15(1), 70–82 (2013)CrossRefGoogle Scholar
  43. 43.
    Pravdova, V., Walczak, B., Massart, D.L.: A comparison of two algorithms for warping of analytical signals. Anal. Chim. Acta 456, 77–92 (2002)CrossRefGoogle Scholar
  44. 44.
    Tomasi, G., van den Berg, F., Andersson, C.: Correlation optimized warping and dynamic time warping as preprocessing methods for chromatographic data. J. Chem. 18(5), 231–241 (2004)CrossRefGoogle Scholar
  45. 45.
    Tomasi, G.; Practical and computational aspects in chemometric data analysis. Ph.D. Thesis, The Royal Veterinary and Agricultural University (2006)Google Scholar
  46. 46.
    Skov, T., van den Berg, F., Tomasi, G., Bro, R.: Automated alignment of chromatographic data. J. Chem. 20, 484–497 (2006)CrossRefGoogle Scholar
  47. 47.
    Wang, S., Yao, J., Liu, J., Petrick, N., Van Uitert, R.L., Periaswamy, S., Summers, R.M.: Registration of prone and supine CT colonography scans using correlation optimized warping and canonical correlation analysis. Med. Phys. 36(12), 5595–5603 (2009)CrossRefGoogle Scholar
  48. 48.
    Yu, T., Shen, X., Li, Q., Geng, W.: Motion retrieval based on movement notation language. Comp. Anim. Virtual Worlds 16(3–4), 273–282 (2005)CrossRefGoogle Scholar
  49. 49.
    Tang, J.K., Leung, H., Komura, T., Shum, H.P.: Emulating human perception of motion similarity. Comput. Anim. Virtual Worlds 19(3–4), 211–221 (2008)CrossRefGoogle Scholar
  50. 50.
    Gray, P.O.: Psychology, 5th edn. Worth, New York (2006)Google Scholar
  51. 51.
    Wolfe, J.M., Kluender, K.R., Levi, D.M., Bartoshuk, L.M., Herz, R.S., Klatzky, R.L., Lederman, S.J.: Gestalt Grouping Principles. Sensation and perception. 2nd ed., Sinauer Associates. pp. 78–80 (2008)Google Scholar
  52. 52.
    Wang, J., Yagi, Y., Makihara, Y.: People tracking and segmentation using efficient shape sequences matching. In: Zha, H., Taniguchi, R., Maybank, S. (eds) 9th Asian Conferenceon Computer Vision, vol. 5995, pp. 204–213. Springer, Berlin (2009)Google Scholar
  53. 53.
    Wang, J., Bodenheimer, B.: Synthesis and evaluation of linear motion transitions. ACM Trans. Graph. 27(1), 1 (2008)Google Scholar
  54. 54.
    Wang, Y.-S., Lin, H.-C., Sorkine, O., Lee, T.-Y.: Motion-based video retargeting with optimized crop-and-warp. ACM Trans. Graph. 29(4), 90 (2010)Google Scholar
  55. 55.
    Prazák, M., Hoyet, L., O’Sullivan, C., Perceptual evaluation of footskate cleanup. In symposium on computer animation, pp. 287–294 (2011)Google Scholar
  56. 56.
    Bouillaguet, C., Chen, H.C., Cheng, C.M., Chou, T., Niederhagen, R., Shamir, A., Yang, B.Y.: Fast exhaustive search for polynomial systems in F2. Cryptographic hardware and embedded systems, pp. 203–218. Springer, Berlin (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.School of Information TechnologyCarleton UniversityOttawaCanada

Personalised recommendations