Correlation-optimized time warping for motion


Retrieval and comparative editing/modeling of motion data require temporal alignment. In other words, for such processes to perform accurately, critical features of motion sequences need to occur simultaneously. In this paper, we propose correlation-optimized time warping (CoTW) for aligning motion data. CoTW utilizes a correlation-based objective function for characterizing alignment. The method solves an optimization problem to determine the optimum warping degree for different segments of the sequence. Using segment-wise interpolated warping, smooth motion trajectories are achieved that can be readily used for animation. Our method allows for manual tuning of the parameters, resulting in high customizability with respect to the number of actions in a single sequence as well as spatial regions of interest within the character model. Moreover, measures are taken to reduce distortion caused by over-warping. The framework also allows for automatic selection of an optimum reference when multiple sequences are available. Experimental results demonstrate the very accurate performance of CoTW compared to other techniques such as dynamic time warping, derivative dynamic time warping and canonical time warping. The mentioned customization capabilities are also illustrated.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17


  1. 1.

    Aggarwal, J.K., Ryoo, M.S.: Human activity analysis: a review. ACM Comput. Surv. 43(3), 16 (2011)

  2. 2.

    Etemad, S.A., Arya, A.: Modeling and transformation of 3D human motion. In: Proceedings of 5th international conference on computer graphics theory and applications, pp. 307–315 (2010)

  3. 3.

    Amaya, K., Bruderlin, A., Calvert, T.: Emotion from motion. In: Proceedings of graphics interface, pp. 222–229 (1996)

  4. 4.

    Rose, C., Cohen, M.F., Bodenheimer, B.: Verbs and adverbs: multidimensional motion interpolation. IEEE Comput. Graph. Appl. 18(5), 32–40 (1998)

  5. 5.

    Etemad, S.A., Arya, A.: Extracting movement, posture, and temporal style features from human motion. Biol. Inspir. Cognit. Archit. 7, 15–25 (2013)

  6. 6.

    Etemad, S.A., Arya, A.: Classification and translation of style and affect in human motion using RBF neural networks. Neurocomputing 129, 585–595 (2013)

  7. 7.

    Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. Speech. Signal Process. 26(1), 43–49 (1978)

  8. 8.

    Zhou, F., De la Torre, F.L.: Canonical time warping for alignment of human behavior. In: Proceedings NIPS, pp. 1–9 (2009)

  9. 9.

    Hsu, E., Pulli, K., Popovic, F.: Style translation for human motion. ACM Trans. Graph. 24(3), 1082–1089 (2005)

  10. 10.

    Vest Nielsen, N.-P., Carstensen, J.M., Smedsgaard, J.: Aligning of single and multiple wavelength chromatographic profiles for chemometric data analysis using correlation optimised warping. J. Chromatogr. A 805, 17–35 (1998)

  11. 11.

    Etemad, S.A., Arya, A.: A customizable time warping method for motion alignment. In: Proceedings of the 7th IEEE international conference on semantic computing, pp. 387–388 (2013)

  12. 12.

    Etemad, S.A., Arya, A., Parush, A.: Additivity in perception of affect from limb motion. Neurosci. Lett. 558, 132–136 (2014)

  13. 13.

    Fu, A.W.-C., Keogh, E., Yung Lau, L., Ratanamahatana, C.A., Chi-Wing Wong, R.: Scaling and time warping in time series querying. Very Large Data Bases J. 17(4), 899–921 (2008)

  14. 14.

    Keogh, E.J., Pazzani, M.J.: Derivative dynamic time warping. In 1st SIAM international conference on data mining. (2001)

  15. 15.

    Rabiner, L., Juang, B.-H.: Fundamentals of speech recognition. Prentice Hall (1993)

  16. 16.

    Myers, C., Rabiner, L., Rosenberg, A.: Performance tradeoffs in dynamic time warping algorithms for isolated word recognition. IEEE Trans. Acoust. Speech Signal Process. 28(6), 623–635 (1980)

  17. 17.

    Lemire, D.: Faster retrieval with a two-pass dynamic-time-warping lower bound. Pattern Recognit. 42(9), 2169–2180 (2009)

  18. 18.

    Zhou, F., De la Torre, F.: Generalized time warping for multi-modal alignment of human motion. In IEEE conference on computer vision and pattern recognition, pp. 1282–1289 (2012)

  19. 19.

    Shapiro, A., Cao, Y., Faloutsos, P.: Style components. In graphics interface, pp. 33–39 (2006)

  20. 20.

    Liu, G., Pan, Z., Lin, Z.: Style subspaces for character animation. J. Vis. Comput. Anim. 19(3–4), 199–209 (2008)

  21. 21.

    Heloir, A., Courty, N., Gibet, S., Multon, F.: Temporal alignment of communicative gesture sequences. J. Vis. Comput. Anim. 17(3–4), 347–357 (2006)

  22. 22.

    Listgarten, J., Neal, R.M., Roweis, S.T., Emili, A.: Multiple alignment of continuous time series. In NIPS, 817–824 (2005)

  23. 23.

    Brand, M., Oliver, N., Pentland, A.: Coupled hidden Markov models for complex action recognition. In IEEE conference on computer vision and pattern recognition, pp. 994–999 (1997)

  24. 24.

    Brand, M., Hertzmann, A.: Style machines. In SIGGRAPH, pp. 183–192 (2000)

  25. 25.

    Taylor, G.W., Hinton, G.E., Roweis, S.T.: Modeling human motion using binary latent variables. In NIPS, pp. 1345 (2007)

  26. 26.

    Bruderlin, A., Williams, A.: Motion signal processing. In: Proceedings of the 22nd annual conference on computer graphics and interactive techniques, pp. 97–104 (1995)

  27. 27.

    Witkin, A., Popovic, Z.: Motion warping. In: Proceedings of the 22nd annual conference on computer graphics and interactive techniques, pp. 105–108 (1995)

  28. 28.

    Gleicher, M.: Retargetting motion to new characters. In: Proceedings of the 25th annual conference on computer graphics and interactive techniques, pp. 33–42 (1998)

  29. 29.

    Kovar, L., Gleicher, L.: Flexible automatic motion blending with registration curves. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on computer animation, pp. 214–224 (2003)

  30. 30.

    Kovar, L., Gleicher, M.: Automated extraction and parameterization of motions in large data sets. ACM Trans. Graph. 23(3), 559–568 (2004)

  31. 31.

    Müller, M., Röder, T., Clausen, M.: Efficient content-based retrieval of motion capture data. ACM Trans. Graph. 24(3), 677–685 (2005)

  32. 32.

    Müller, M., Röder, T.: Motion templates for automatic classification and retrieval of motion capture data. In: Proceedings of the 2006 ACM SIGGRAPH/Eurographics symposium on computer animation, pp. 137–146 (2006)

  33. 33.

    Zhou, F., De la Torre, F., Hodgins, J.K.: Aligned cluster analysis for temporal segmentation of human motion. In 8th IEEE international conference on automatic face and gesture recognition, pp. 1–7 (2008)

  34. 34.

    Kim, M., Hyun, K., Kim, J., Lee, J.: Synchronized multi-character motion editing. ACM Trans. Graph. 28(3), 79 (2009)

  35. 35.

    Raptis, M., Kirovski, D., Hoppe, H.: Real-time classification of dance gestures from skeleton animation. In: Proceedings of the 2011 ACM SIGGRAPH/Eurographics symposium on computer animation, pp. 147–156 (2011)

  36. 36.

    Cimen, G., Ilhan, H., Capin, T., Gurcay, H.: Classification of human motion based on affective state descriptors. Comput. Anim. Virtual Worlds 24(3–4), 355–363 (2013)

  37. 37.

    Hsu, E., da Silva, M., Popović, J.: Guided time warping for motion editing. In: Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on computer animation, pp. 45–52 (2007)

  38. 38.

    Caspi, Y., Irani, M.: Aligning non-overlapping sequences. Int. J. Comput. Vis. 48(1), 39–51 (2002)

  39. 39.

    Caspi, Y., Irani, M.: Spatio-temporal alignment of sequences. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1409–1424 (2002)

  40. 40.

    Padua, F.L.C., Carceroni, R.L., Santos, G.A.M.R., Kutulakos, K.N.: Linear sequence-to-sequence alignment. IEEE Trans. Pattern Anal. Mach. Intell. 32(2), 304–320 (2010)

  41. 41.

    Junejo, I.N., Dexter, E., Laptev, I., Perez, P.: View-independent action recognition from temporal self-similarities. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 172–185 (2011)

  42. 42.

    Lu, C., Mandal, M.: A robust technique for motion-based video sequences temporal alignment. IEEE Trans. Multimed. 15(1), 70–82 (2013)

  43. 43.

    Pravdova, V., Walczak, B., Massart, D.L.: A comparison of two algorithms for warping of analytical signals. Anal. Chim. Acta 456, 77–92 (2002)

  44. 44.

    Tomasi, G., van den Berg, F., Andersson, C.: Correlation optimized warping and dynamic time warping as preprocessing methods for chromatographic data. J. Chem. 18(5), 231–241 (2004)

  45. 45.

    Tomasi, G.; Practical and computational aspects in chemometric data analysis. Ph.D. Thesis, The Royal Veterinary and Agricultural University (2006)

  46. 46.

    Skov, T., van den Berg, F., Tomasi, G., Bro, R.: Automated alignment of chromatographic data. J. Chem. 20, 484–497 (2006)

  47. 47.

    Wang, S., Yao, J., Liu, J., Petrick, N., Van Uitert, R.L., Periaswamy, S., Summers, R.M.: Registration of prone and supine CT colonography scans using correlation optimized warping and canonical correlation analysis. Med. Phys. 36(12), 5595–5603 (2009)

  48. 48.

    Yu, T., Shen, X., Li, Q., Geng, W.: Motion retrieval based on movement notation language. Comp. Anim. Virtual Worlds 16(3–4), 273–282 (2005)

  49. 49.

    Tang, J.K., Leung, H., Komura, T., Shum, H.P.: Emulating human perception of motion similarity. Comput. Anim. Virtual Worlds 19(3–4), 211–221 (2008)

  50. 50.

    Gray, P.O.: Psychology, 5th edn. Worth, New York (2006)

  51. 51.

    Wolfe, J.M., Kluender, K.R., Levi, D.M., Bartoshuk, L.M., Herz, R.S., Klatzky, R.L., Lederman, S.J.: Gestalt Grouping Principles. Sensation and perception. 2nd ed., Sinauer Associates. pp. 78–80 (2008)

  52. 52.

    Wang, J., Yagi, Y., Makihara, Y.: People tracking and segmentation using efficient shape sequences matching. In: Zha, H., Taniguchi, R., Maybank, S. (eds) 9th Asian Conferenceon Computer Vision, vol. 5995, pp. 204–213. Springer, Berlin (2009)

  53. 53.

    Wang, J., Bodenheimer, B.: Synthesis and evaluation of linear motion transitions. ACM Trans. Graph. 27(1), 1 (2008)

  54. 54.

    Wang, Y.-S., Lin, H.-C., Sorkine, O., Lee, T.-Y.: Motion-based video retargeting with optimized crop-and-warp. ACM Trans. Graph. 29(4), 90 (2010)

  55. 55.

    Prazák, M., Hoyet, L., O’Sullivan, C., Perceptual evaluation of footskate cleanup. In symposium on computer animation, pp. 287–294 (2011)

  56. 56.

    Bouillaguet, C., Chen, H.C., Cheng, C.M., Chou, T., Niederhagen, R., Shamir, A., Yang, B.Y.: Fast exhaustive search for polynomial systems in F2. Cryptographic hardware and embedded systems, pp. 203–218. Springer, Berlin (2010)

Download references


This work was supported in part by the Natural Sciences and Engineering Council of Canada (NSERC) and Ontario Centers of Excellence (OCE).

Author information

Correspondence to S. Ali Etemad.

Electronic supplementary material

Below is the link to the electronic supplementary material.

ESM 1 (WMV 34,781 KB)

ESM 2 (WMV 4,513 KB)

ESM 1 (WMV 34,781 KB)

ESM 2 (WMV 4,513 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Etemad, S.A., Arya, A. Correlation-optimized time warping for motion. Vis Comput 31, 1569–1586 (2015).

Download citation


  • Motion analysis
  • Time warping
  • Temporal alignment
  • Correlation
  • Optimization