Object Trajectory Analysis in Video Indexing and Retrieval Applications

  • Mattia Broilo
  • Nicola Piotto
  • Giulia Boato
  • Nicola Conci
  • Francesco G. B. De Natale
Part of the Studies in Computational Intelligence book series (SCI, volume 287)


The focus of this chapter is to present a survey on the most recent advances in representation and analysis of video object trajectories, with application to indexing and retrieval systems. We will review the main methodologies for the description of motion trajectories, as well as the indexing techniques and similarity metrics used in the retrieval process. Strengths and weaknesses of different solutions will be discussed through a comparative analysis, taking into account performance and implementation issues. In order to provide a deeper insight on the exploitation of these technologies in real world products, a selection of exampleswill be introduced and examined. The set of possible applications is very wide and includes (but it is not limited to) generic browsing of video databases, as well as more specific and context-dependent scenarios such as indexing and retrieval in visual surveillance, traffic monitoring, sport events analysis, video-on-demand, and video broadcasting.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Wang, Z., Lu, L., Bovik, A.C.: Video quality assessment based on structural distortion measurement. Signal Processing: Image Communication 19(2), 121–132 (2004)CrossRefGoogle Scholar
  2. 2.
    Mansouri, A.R., Mitiche, A., El Feghali, R.: Spatio-temporal motion segmentation via level set partial differential equation. In: Fifth IEEE Southwest Symposium on Image Analysis and Interpretation, p. 243. IEEE Computer Society, Los Alamitos (2002)CrossRefGoogle Scholar
  3. 3.
    Naftel, A., Khalid, S.: Classifying spatiotemporal object trajectories using unsupervised learning in the coefficient feature space. Multimedia Systems 12(3), 227–238 (2006)CrossRefGoogle Scholar
  4. 4.
    Min, J., Kasturi, R.: Activity recognition based on multiple motion trajectories. In: Int. Conf. on Pattern Recognition, vol. 4, pp. 199–202 (August 2004)Google Scholar
  5. 5.
    Bashir, F., Khokhar, A., Schonfeld, D.: Object trajectory-based activity classification and recognition using hidden Markov models. IEEE Trans. on Image Processing 16(7), 1912–1919 (2007)CrossRefMathSciNetGoogle Scholar
  6. 6.
    Morris, B.T., Trivedi, M.M.: A Survey of Vision-Based Trajectory Learning and Analysis for Surveillance. IEEE Trans. on Circuits and Systems for Video Tech. 18(8), 1114–1127 (2008)CrossRefGoogle Scholar
  7. 7.
    Oliver, N.M., Rosario, B., Pentland, A.P.: A bayesian computer vision system for modeling human interactions. IEEE Trans. on Pattern Analysis and Machine Intelligence 22(8), 831–843 (2000)CrossRefGoogle Scholar
  8. 8.
    Parameswaran, V., Chellappa, R.: View invariance for human action recognition. Int. J. Comput. Vision 66, 83–101 (2006)CrossRefGoogle Scholar
  9. 9.
    Ma, X., Bashir, F., Khokhar, A., Schonfeld, D.: Event analysis based on multiple interactive motion trajectories. IEEE Trans. Circuits Syst. Video Techn. 19(3), 397–406 (2009)CrossRefGoogle Scholar
  10. 10.
    Kaneko, T., Okudaira, M.: Encoding of arbitrary curves based on the chain code representation. IEEE Trans. on Communications 33(7), 697–707 (1985)CrossRefGoogle Scholar
  11. 11.
    Pavlidis, T.: Polygonal approximations by newton’s method. IEEE Trans. on Computing 26(8), 800–807 (1977)MATHCrossRefGoogle Scholar
  12. 12.
    Von Stryk, O., Bulirsch, R.: Direct and indirect methods for trajectory optimization. Annals of Operations Research 37(1), 357–373 (1992)MATHCrossRefMathSciNetGoogle Scholar
  13. 13.
    Medioni, G., Yasumoto, Y.: Corner detection and curve representation using cubic B-splines. In: IEEE Int. Conf. on Robotics and Automation, vol. 3 (1986)Google Scholar
  14. 14.
    Chen, X., Schonfeld, D., Khokhar, A.: Robust null space representation and sampling for view-invariant motion trajectory analysis. In: IEEE Conf. on Computer Vision and Pattern Recognition, pp. 1–6 (June 2008)Google Scholar
  15. 15.
    Rosin, P.L.: Techniques for assessing polygonal approximations of curves. IEEE Trans. on Pattern Analisys and Machine Intelligence 19(6), 659–666 (1997), doi:10.1109/34.601253CrossRefGoogle Scholar
  16. 16.
    Chan, W.S., Chin, F.: Approximation of polygonal curves with minimum number of line segments. LNCS, p. 378. Springer, Heidelberg (1992)Google Scholar
  17. 17.
    Sklansky, J., Gonzalez, V.: Fast polygonal approximation of digitized curves. Pattern Recognition 12(5), 327–331 (1980)CrossRefGoogle Scholar
  18. 18.
    Kurozumi, Y., Davis, W.A.: Polygonal approximation by the minimax method. Computer Graphics and Image Processing 19(3), 248–264 (1982)MATHCrossRefGoogle Scholar
  19. 19.
    Wall, K., Danielsson, P.E.: A fast sequential method for polygonal approximation of digitized curves. Computer Vision Graphics Image Processing 28(3), 220–227 (1984)CrossRefGoogle Scholar
  20. 20.
    Kumar Ray, B., Ray, K.S.: Determination of optimal polygon from digital curve using L1 norm. Pattern Recognition 26(4), 505–509 (1993)CrossRefGoogle Scholar
  21. 21.
    Douglas, D.H., Peucker, T.K.: Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: The Int. J. for Geographic Information and Geovisualization 10(2), 112–122 (1973)CrossRefGoogle Scholar
  22. 22.
    Ballard, D.H.: Strip trees: A hierarchical representation for curves. Communications of the ACM 24(5), 310–321 (1981)CrossRefGoogle Scholar
  23. 23.
    Duda, R.O., Hart, P.E.: Pattern classification and scene analysis, New York (1973)Google Scholar
  24. 24.
    Leu, J.G., Chen, L.: Polygonal approximation of 2-D shapes through boundary merging. Pattern Recognition Letters 7(4), 231–238 (1988)CrossRefGoogle Scholar
  25. 25.
    Ansari, N., Delp, E.J.: On detecting dominant points. Pattern Recognition 24(5), 441–451 (1991)CrossRefGoogle Scholar
  26. 26.
    Ray, B.K., Ray, K.S.: A new split-and-merge technique for polygonal approximation of chain coded curves. Pattern Recognition Letters 16(2), 161–169 (1995)CrossRefGoogle Scholar
  27. 27.
    Attneave, F.: Some informational aspects of visual perception. Psychological Review 61(3), 183–193 (1954)CrossRefGoogle Scholar
  28. 28.
    Teh, C.H., Chin, R.T.: On the detection of dominant points on digital curves. IEEE Trans. on Pattern Analysis and Machine Intelligence 11(8), 859–872 (1989)CrossRefGoogle Scholar
  29. 29.
    Held, A., Abe, K., Arcelli, C.: Towards a hierarchical contour description via dominant point detection. IEEE Trans. on Systems, Man and Cybernetics 24(6), 942–949 (1994)CrossRefGoogle Scholar
  30. 30.
    Zhu, P., Chirlian, P.M.: On critical point detection of digital shapes. IEEE Trans. on Pattern Analysis and Machine Intelligence 17(8), 737–748 (1995)CrossRefGoogle Scholar
  31. 31.
    Mikheev, A., Vincent, L., Faber, V., Inc, L.T., Seattle, W.A.: High-quality polygonal contour approximation based on relaxation. In: Int. Conf. on Document Analysis and Recognition, pp. 361–365 (2001)Google Scholar
  32. 32.
    Ho, S.Y., Chen, Y.C.: An efficient evolutionary algorithm for accurate polygonal approximation. Pattern Recognition 34(12), 2305–2317 (2001)MATHCrossRefGoogle Scholar
  33. 33.
    Yin, P.Y.: Ant colony search algorithms for optimal polygonal approximation of plane curves. Pattern Recognition 36(8), 1783–1797 (2003)MATHCrossRefGoogle Scholar
  34. 34.
    Yin, P.Y.: A discrete particle swarm algorithm for optimal polygonal approximation of digital curves. J. of Visual Communication and Image Representation 15(2), 241–260 (2004)CrossRefGoogle Scholar
  35. 35.
    Yin, P.Y.: A tabu search approach to polygonal approximation of digital curves. Int. J. of Pattern Recognition and Artificial Intelligence 14(2), 243–255 (2000)CrossRefGoogle Scholar
  36. 36.
    O’connell, K.J., Inc, M., Schaumburg, I.L.: Object-adaptive vertex-based shape coding method. IEEE Trans. on Circuits and Systems for Video Tech. 7(1), 251–255 (1997)CrossRefGoogle Scholar
  37. 37.
    Moore, B.: Principal component analysis in linear systems: Controllability, observability, and model reduction. IEEE Trans. on Automatic Control 26(1), 17–32 (1981)MATHCrossRefGoogle Scholar
  38. 38.
    Jolliffe, I.T.: Principal Component Analysis. Springer, New York (2002)MATHGoogle Scholar
  39. 39.
    Bashir, F., Khokhar, A., Schonfeld, D.: Segmented trajectory based indexing and retrieval of video data. In: IEEE Int. Conf. on Image Processing, vol. 2 (2003)Google Scholar
  40. 40.
    Needleman, S.B., Wunsch, C.D.: A general method applicable to the search for similarities in the amino acid sequence of two proteins. J. Molecular Biology 48(3), 443–453 (1970)CrossRefGoogle Scholar
  41. 41.
    Gdalyahu, Y., Weinshall, D.: Flexible syntactic matching of curves and its application to automatic hierarchical classification of silhouettes. IEEE Trans. on Pattern Analysis and Machine Intelligence 21(12), 1312–1328 (1999)CrossRefGoogle Scholar
  42. 42.
    Wu, G., Wu, Y., Jiao, L., Wang, Y.F., Chang, E.Y.: Multi-camera spatio-temporal fusion and biased sequence-data learning for security surveillance. In: ACM Int. Conf. on Multimedia, pp. 528–538. ACM, New York (2003)Google Scholar
  43. 43.
    Hsieh, J.W., Yu, S.L., Chen, Y.S.: Motion-based video retrieval by trajectory matching. IEEE Trans. on Circuits and Systems for Video Tech. 16(3), 396–409 (2006)CrossRefGoogle Scholar
  44. 44.
    Piotto, N., Conci, N., De Natale, F.G.B.: Syntactic matching of pedestrian trajectories for ambient intelligence applications. IEEE Trans. on Multimedia 11(7) (2009)Google Scholar
  45. 45.
    Ristad, E.S., Yianilos, P.N.: Learning string-edit distance. IEEE Trans. on Pattern Analysis and Machine Intelligence 20(5), 522–532 (1998)CrossRefGoogle Scholar
  46. 46.
    Schoenberg, I.J.: Contributions to the problem of approximation of equidistant data by analytic functions. Quarterly of Applied Mathematics 4, 45–99 (1946)MathSciNetGoogle Scholar
  47. 47.
    Loncaric, S.: A survey of shape analysis techniques. Pattern Recognition (1998)Google Scholar
  48. 48.
    Ikebe, Y., Miyamoto, S.: Shape design, representation, and restoration with splines. Picture Engineering, 75–95 (1982)Google Scholar
  49. 49.
    Chen, W., Chang, S.F.: Motion trajectory matching of video objects. In: IS&T/SPIE, San Jose, CA (2000)Google Scholar
  50. 50.
    Bashir, F.I., Khokhar, A.A., Schonfeld, D.: Real-time motion trajectory-based indexing and retrieval of video sequences. IEEE Trans. on Multimedia 9(1), 58–65 (2007)CrossRefGoogle Scholar
  51. 51.
    Idris, F., Panchanathan, S.: Review of image and video indexing techniques. J. of Visual Communication and Image Representation 8(2), 146–166 (1997)CrossRefGoogle Scholar
  52. 52.
    Dagtas, S., Al-Khatib, W., Ghafoor, A., Kashyap, R.L., Res, P., Manor, B.: Models for motion-based video indexing and retrieval. IEEE Trans. on Image Processing 9(1), 88–101 (2000)CrossRefGoogle Scholar
  53. 53.
    Sellis, T., Roussopoulos, N., Faloutsos, C.: The R-tree: A dynamic index for multi-dimensional objects. The VLDB Journal, 507–518 (1987)Google Scholar
  54. 54.
    Theoderidis, Y., Vazirgiannis, M., Sellis, T.: Spatio-temporal indexing for large multimedia applications. In: IEEE Int. Conf. on Multimedia Computing and Systems, pp. 441–448 (1996)Google Scholar
  55. 55.
    Nascimento, M.A., Silva, J.R.O.: Towards Historical R-trees. In: ACM Symposium on Applied Computing, pp. 235–240. ACM, New York (1998)Google Scholar
  56. 56.
    Nascimento, M.A., Silva, J.R.O., Theodoridis, Y.: Evaluation of access structures for discretely moving points. LNCS, pp. 171–188. Springer, Heidelberg (1999)Google Scholar
  57. 57.
    Tao, Y., Papadias, D.: Mv3r-tree: a spatio-temporal access method for timestamp and interval queries. In: Int. Conf. on Very Large Data Bases, pp. 431–440. Morgan Kaufmann Publishers Inc., San Francisco (2001)Google Scholar
  58. 58.
    Becker, B., Gschwind, S., Ohler, T., Seeger, B., Widmayer, P.: An asymptotically optimal multiversion B-tree. The Int. J. on Very Large Data Bases 5(4), 264–275 (1996)CrossRefGoogle Scholar
  59. 59.
    Pfoser, D., Jensen, C.S., Theodoridis, Y.: Novel approaches in query processing for moving object trajectories. In: Int. Conf. on Very Large Data Bases, pp. 395–406. Morgan Kaufmann Publishers Inc., San Francisco (2000)Google Scholar
  60. 60.
    Stollnitz, E.J., DeRose, T.D., Salesin, D.H.: Wavelets for computer graphics: theory and applications. Morgan Kaufmann, San Francisco (1996)Google Scholar
  61. 61.
    Akansu, A.N., Haddad, R.A.: Multiresolution signal decomposition. Academic Press, Boston (1992)MATHGoogle Scholar
  62. 62.
    Chan, K.P., Fu, A.W.C.: Efficient time series matching by wavelets. In: IEEE Int. Conf. on Data Engineering, pp. 126–135. Institute of Electrical and Electronics Engineers (1999)Google Scholar
  63. 63.
    Korn, F., Jagadish, H.V., Faloutsos, C.: Efficiently supporting ad hoc queries in large datasets of time sequences. In: ACM SIGMOD Int. Conf. on Management of Data, pp. 289–300. ACM, New York (1997)CrossRefGoogle Scholar
  64. 64.
    Berndt, D., Clifford, J.: Using dynamic time warping to find patterns in time series. In: AAAI 1994 Workshop on Knowledge Discovery and Databases, pp. 229–248 (1994)Google Scholar
  65. 65.
    Keogh, E.J., Pazzani, M.J.: Scaling up dynamic time warping for data mining application. In: ACM SIGKDD Int. Conf. on Knowledge discovery and data mining, pp. 285–289 (2000)Google Scholar
  66. 66.
    Hu, W., Tan, T., Wang, L., Maybank, S.: A survey on visual surveillance of object motion and behaviors. IEEE Trans. on Systems, Man and Cybernetics 34, 334–352 (2004)Google Scholar
  67. 67.
    Das, G., Gunopoulos, D., Mannila, H.: Finding similar time series. In: Komorowski, J., Żytkow, J.M. (eds.) PKDD 1997. LNCS, vol. 1263, pp. 88–100. Springer, Heidelberg (1997)Google Scholar
  68. 68.
    Vlachos, M., Hadjieleftheriou, M., Gunopoulos, D., Keogh, E.: Indexing multidimensional time-series with support for multiple distance measures. In: ACM SIGKDD, pp. 216–225 (2003)Google Scholar
  69. 69.
    Porikli, F.: Trajectory distance metric using hidden Markov model based representation. In: IEEE European Conf. on Computer Vision (2004)Google Scholar
  70. 70.
    Li, X., Hu, W., Hu, W.: A coarse-to-fine strategy for vehicle motion trajectory clustering. In: IEEE Int. Conf. on Pattern Recognition, vol. 1, pp. 591–594 (2006)Google Scholar
  71. 71.
    Anjum, N., Cavallaro, A.: Unsupervised fuzzy clustering for trajectory analysis. In: IEEE Int. Conf. on Image Processing, vol. 3, pp. 213–216 (2007)Google Scholar
  72. 72.
    Piciarelli, C., Foresti, G.L., Snidaro, L.: Trajectory clustering and its application for video surveillance. In: IEEE Conf. on Advanced Video and Signal Based Surveillance, pp. 40–45 (2005)Google Scholar
  73. 73.
    Johnson, N., Hogg, N.: Learning the distribution of object trajectories for event recognition. Image and Vision Computing 14, 609–615 (1996)CrossRefGoogle Scholar
  74. 74.
    Mecocci, A., Pannozzo, M.: A completely autonomous system that learns anomalous movements in advanced video surveillance applications. In: IEEE Int. Conf. on Image Processing, vol. 2, pp. 586–589 (2005)Google Scholar
  75. 75.
    Sumpter, N., Bulpitt, A.: Learning spatio-temporal patterns for predicting object behavior. Image and Visio Computing 18, 697–704 (2000)CrossRefGoogle Scholar
  76. 76.
    Owens, J., Hunter, A.: Application of the self-organizing map to trajectory classification. In: IEEE Int. Workshop Visual Surveillance, vol. 18, pp. 77–83 (2000)Google Scholar
  77. 77.
    Kohonen, T.: Self-organizing maps. Springer, Heidelberg (1995)Google Scholar
  78. 78.
    Hu, W.M., Xie, D., Tan, T.N.: A hierarchical self-organizing approach for learning the patterns of motion trajectories. IEEE Trans. on Neural Network 15, 135–144 (2004)CrossRefGoogle Scholar
  79. 79.
    Imran, N., Javed, O., Shah, M.: Multi feature path modeling for video surveillance. In: Int. Conf. on Pattern Recognition, pp. 716–719 (2004)Google Scholar
  80. 80.
    Marzal, A., Vidal, E.: Computation of normalized edit distance and applications. IEEE Trans. on Pattern Analysis and Machine Intelligence 15, 926–932 (1993)CrossRefGoogle Scholar
  81. 81.
    Chen, L., Otsu, M.T., Oria, V.: Symbolic representation and retrieval of moving object trajectories. In: ACM SIGMM Int. Workshop on Multimedia Information Retrieval, pp. 227–234 (2004)Google Scholar
  82. 82.
    Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10, 707–710 (1966)MathSciNetGoogle Scholar
  83. 83.
    Zheng, J.B., Feng, D.D., Zhao, R.C.: Trajectory Matching and Classification of Video Moving Objects. In: IEEE Workshop on Multimedia Signal Processing, vol. 10, pp. 1–4 (2005)Google Scholar
  84. 84.
    Hu, W., Xie, D., Fu, Z., Zeng, W., Maybank, S.: Semantic-based surveillance video retrieval. IEEE Trans. on Image Processing 16, 1168–1181 (2007)CrossRefMathSciNetGoogle Scholar
  85. 85.
    Calderara, S., Cucchiara, R., Prati, A.: A Dynamic programming technique for classifying trajectories. In: Int. Conf. on Image Analysis and Processing, pp. 137–142 (2007)Google Scholar
  86. 86.
    Piotto, N., Conci, N., De Natale, F.G.B.: Syntactic matching of pedestrian trajectories for behavioral analysis. In: Proc. of 10th IEEE Workshop on Multimedia Signal Processing, pp. 877–882 (2008)Google Scholar
  87. 87.
    Needleman, S.B., Wunsch, C.D.: A general method applicable to the search for similarities in the amino acid sequence of two proteins. J. of Molecular Biology 48, 443–453 (1970)CrossRefGoogle Scholar
  88. 88.
    Smith, T.F., Waterman, M.S.: Identification of Common Molecular Subsequences. J. of Molecular Biology 147, 195–197 (1981)CrossRefGoogle Scholar
  89. 89.
    Vlacos, M., Kollios, G., Gunopulos, D.: Discovering similar multidimensional trajectories. In: Int. Conf. on Data Engineering, pp. 673–684 (2002)Google Scholar
  90. 90.
    Shan, M.K., Lee, S.Y.: Content-based video retrieval via motion trajectories. In: Society of Photo-Optical Instrumentation Engineers (SPIE) Conf. Series, pp. 52–61 (1998)Google Scholar
  91. 91.
    Chang, S.F., Chen, W., Meng, H.J., Sundaram, H.: A fully automated content-based video search engine supportingspatiotemporal queries. IEEE Trans. on Circuits and Systems for Video Tech. 8(5), 602–615 (1998)CrossRefGoogle Scholar
  92. 92.
    Sahouria, E., Zakhor, A.: Motion indexing of video. In: IEEE Int. Conf. on Image Processing, vol. 2 (1997)Google Scholar
  93. 93.
    Jung, Y.K., Lee, K.W., Ho, Y.S.: Content-based event retrieval using semantic scene interpretation for automated traffic surveillance. IEEE Trans. on Intelligent Transportation Systems 2(3), 151–163 (2001)CrossRefGoogle Scholar
  94. 94.
    Basharat, A., Zhai, Y., Shah, M.: Content based video matching using spatiotemporal volumes. Computer Vision and Image Understanding 110(3), 360–377 (2008)CrossRefGoogle Scholar
  95. 95.
    Chang, S.F., Chen, W., Meng, H.J., Sundaram, H.: VideoQ: an automated content based video search system using visual cues. In: ACM Int. Conf. on Multimedia, pp. 313–324. ACM, New York (1997)CrossRefGoogle Scholar
  96. 96.
    Yoshitaka, A., Hosoda, Y., Yoshimitsu, M., Hirakawa, M., Ichikawa, T.: Violone: Video retrieval by motion example. J. of Visual Languages and Computing 7(4), 423–443 (1996)CrossRefGoogle Scholar
  97. 97.
    Aghbari, Z., Kaneko, K., Makinouchi, A.: Content-trajectory approach for searching video databases. IEEE Trans. on Multimedia 5(4), 516–531 (2003)CrossRefGoogle Scholar
  98. 98.
    Le, T., Boucher, A., Thonnat, M.: Subtrajectory-based video indexing and retrieval. In: Cham, T.-J., Cai, J., Dorai, C., Rajan, D., Chua, T.-S., Chia, L.-T. (eds.) MMM 2007. LNCS, vol. 4351, p. 418. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  99. 99.
    Buxton, H., Gong, S.: Visual surveillance in a dynamic and uncertain world. Artificial Intelligence 78(1-2), 431–459 (1995)CrossRefGoogle Scholar
  100. 100.
    Remagnino, P., Tan, T., Baker, K.: Multi-agent visual surveillance of dynamic scenes. Image and Vision Computing 16(8), 529–532 (1998)CrossRefGoogle Scholar
  101. 101.
    Kuijpers, B., Othman, W.: Trajectory databases: Data models, uncertainty and complete query languages. In: Schwentick, T., Suciu, D. (eds.) ICDT 2007. LNCS, vol. 4353, pp. 224–238. Springer, Heidelberg (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Mattia Broilo
    • 1
  • Nicola Piotto
    • 1
  • Giulia Boato
    • 1
  • Nicola Conci
    • 1
  • Francesco G. B. De Natale
    • 1
  1. 1.DISIUniversity of TrentoTrento(Italy)

Personalised recommendations