Kinect Sensing of Shopping Related Actions

  • Mirela Popa
  • Alper Kemal Koc
  • Leon J. M. Rothkrantz
  • Caifeng Shan
  • Pascal Wiggers
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 277)


Surveillance systems in shopping malls or supermarkets are usually used for detecting abnormal behavior. We used the distributed video cameras system to design digital shopping assistants which assess the behavior of customers while shopping, detect when they need assistance, and offer their support in case there is a selling opportunity.  In this paper we propose a system for analyzing human behavior patterns related to products interaction, such as browse through a set of products, examine, pick products, try on, interact with the shopping cart, and look for support by waiving one hand.  We used the Kinect sensor to detect the silhouettes of people and extracted discriminative features for basic action detection. Next we analyzed different classification methods, statistical and also spatio-temporal ones, which capture relations between frames, features, and basic actions. By employing feature level fusion of appearance and movement information we obtained an accuracy of 80% for the mentioned six basic actions.


Shopping Behavior Action Recognition Surveillance Kinect 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Popa, M.C., Rothkrantz, L.J.M., Yang, Z., Wiggers, P., Braspenning, R., Shan, C.: Analysis of Shopping Behavior based on Surveillance System. In: 2010 IEEE Int. Conf. on Systems, Man, and Cybernetics (SMC 2010), Istanbul, Turkey (2010)Google Scholar
  2. 2.
    Popa, M.C., Gritti, T., Rothkrantz, L.J.M., Shan, C., Wiggers, P.: Detecting Customers’ Buying Events on a Real-Life Database. In: Real, P., Diaz-Pernil, D., Molina-Abril, H., Berciano, A., Kropatsch, W. (eds.) CAIP 2011, Part I. LNCS, vol. 6854, pp. 17–25. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  3. 3.
    Microsoft Corp. Redmond WA. Kinect for Xbox 360Google Scholar
  4. 4.
    Moeslund, T.B., Hilton, A., Kruger, V.: A Survey of Advances in Vision-based Human Motion Capture and Analysis. Computer Vision and Image Understanding (2006)Google Scholar
  5. 5.
    Akita, K.: Image sequence analysis of real world human motion. Pattern Recognition 17(1), 73–83 (1984)CrossRefGoogle Scholar
  6. 6.
    Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-Time Human Pose Recognition in Parts from Single Depth Images. In: CVPR (2011)Google Scholar
  7. 7.
    Bobick, A.F., Davis, J.W.: The recognition of human movement using temporal templates. IEEE Trans. PAMI (2001)Google Scholar
  8. 8.
    Ekinci, M., Gedikli, E.: Silhouette Based Human Motion Detection and Analysis for Real-Time Automated Video Surveillance. Turkish Journal of Electrical Engineering and Computer Sciences 13(2), 199–230 (2005)Google Scholar
  9. 9.
    Jiang, H., Drew, M.S., Li, Z.-N.: Action Detection in Cluttered Video with Successive Convex Matching. IEEE Transactions on Circuits and Systems for Video Technology 20(1) (2010)Google Scholar
  10. 10.
    Haritaoglu, I., Cutler, R., Harwood, D., Davis, L.S.: Detection of People Carrying Objects Using Silhouettes. In: International Conference on Computer Vision, Corfu, Greece (1999)Google Scholar
  11. 11.
    Moore, D.J., Essa, I.A., Hayes, M.H.: Exploiting Human Actions and Object Context for Recognition Tasks. In: IEEE International Conference on Computer Vision, Corfu, Greece (1999)Google Scholar
  12. 12.
    Brand, M., Oliver, N., Pentland, A.: Coupled Hidden Markov Models for Complex Action Recognition. In: Proceedings of IEEE Computer Vision and Pattern Recognition (1996)Google Scholar
  13. 13.
    Yamato, J., Ohya, J., Ishii, K.: Recognizing Human Action in Time-Sequential Images using Hidden Markov Model. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 379–385 (1992)Google Scholar
  14. 14.
    Schindler, K., Van Gool, L.: Action Snippets: How Many Frames Does Human Action Recognition Require? In: IEEE Computer Society Conference on Computer Vision (2007)Google Scholar
  15. 15.
    Shüldt, C., Laptev, I., Caputo, B.: Recognizing Human Actions: A Local SVM Approach. In: Proceedings of the 17th International Conference on Pattern Recognition (2004)Google Scholar
  16. 16.
    Gorelick, L., Blank, M., Shechtman, E., Irani, M., Basri, R.: Actions as Space-Time Shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(12) (2007)Google Scholar
  17. 17.
    Weinland, D., Ronfard, R., Boyer, E.: Free Viewpoint Action Recognition using Motion History Volumes. Computer Vision Image Understanding 104(2), 249–257 (2006)CrossRefGoogle Scholar
  18. 18.
    Prismall, S.P.: Object reconstruction by moments extended to moving sequences, PhD thesis, Department Electronic and Computer Science, University of Southampton (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Mirela Popa
    • 1
    • 2
  • Alper Kemal Koc
    • 1
    • 2
  • Leon J. M. Rothkrantz
    • 1
    • 3
  • Caifeng Shan
    • 2
  • Pascal Wiggers
    • 1
  1. 1.Man-Machine Interaction group, Department of MediamaticsDelft University of TechnologyDelftThe Netherlands
  2. 2.Video and Image Processing Department, Philips Research, HTC 36EindhovenThe Netherlands
  3. 3.Sensor Technology, SEWACO DepartmentNetherlands Defence AcademyDen HelderThe Netherlands

Personalised recommendations