Abstract
For the effective operation of intelligent assistive systems working in real-world human environments, it is important to be able to recognise human activities and their intentions. In this paper we propose a novel approach to activity recognition from visual data. Our approach is based on qualitative and quantitative spatio-temporal features which encode the interactions between human subjects and objects in an efficient manner. Unlike the state of the art, our approach uses significantly fewer assumptions and does not require knowledge about object types, their affordances, or the sub-level activities that high-level activities consist of. We perform an automatic feature selection process which provides the most representative descriptions of the learnt activities. We validated the method using these descriptions on the CAD-120 benchmark dataset, consisting of video sequences showing humans performing daily real-world activities. The method is shown to outperform state of the art benchmarks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
Note that this is similar to the INDU calculus [34] which extends the interval calculus by discretising whether intervals in a before, meets or overlaps relationship are shorter, equal or longer than each other.
- 4.
References
Turaga, P., Chellappa, R., Subrahmanian, V., Udrea, O.: Machine recognition of human activities: a survey. IEEE Trans. Circ. Syst. Video Technol. 18, 1473–1488 (2008)
Poppe, R.: A survey on vision-based human action recognition. Image Vis. Comput. 28, 976–990 (2010)
Weinland, D., Ronfard, R., Boyer, E.: A survey of vision-based methods for action representation, segmentation and recognition. Comput. Vis. Image Underst. 115, 224–241 (2011)
Xu, X., Tang, J., Zhang, X., Liu, X., Zhang, H., Qiu, Y.: Exploring techniques for vision based human activity recognition: methods, systems, and evaluation. Sensors 13, 1635–1650 (2013). Basel, Switzerland
Collins, R., Lipton, A., Kanade, T.: Introduction to the special section on video surveillance. IEEE Trans. Pattern Anal. Mach. Intell. 22, 745–746 (2000)
Gowsikhaa, D., Abirami, S., Baskaran, R.: Automated human behavior analysis from surveillance videos: a survey. Artif. Intell. Rev., 1–19 (2012)
Ko, T.: A survey on behavior analysis in video surveillance for homeland security applications. In: 2008 37th IEEE Applied Imagery Pattern Recognition Workshop, pp. 1–8 IEEE (2008)
Chen, J., Cohn, A.G., Liu, D., Wang, S., Ouyang, J., Yu, Q.: A survey of qualitative spatial representations. Knowl. Eng. Rev. FirstView, 1–31 (2013)
Laptev, I.: On space-time interest points. Int. J. Comput. Vis. 64, 107–123 (2005)
Koppula, H., Saxena, A.: Learning spatio-temporal structure from RGB-D videos for human activity detection and anticipation. In: Proceedings of the International Conference on Machine Learning (ICML) (2013)
Koppula, H., Gupta, R., Saxena, A.: Learning human activities and object affordances from RGB-D videos. Int. J. Robot. Res. 32, 951–970 (2013)
Sridhar, M., Cohn, A.G., Hogg, D.C.: Learning functional object-categories from a relational spatio-temporal representation. In: European Conference on Artificial Intelligence (2008)
Sridhar, M., Cohn, A.G., Hogg, D.C.: Unsupervised learning of event classes from video. In: AAAI (2010)
Sridhar, M., Cohn, A.G., Hogg, D.C.: Discovering an event taxonomy from video using qualitative spatio-temporal graphs. In: European Conference on Artificial Intelligence (2010)
Randell, D., Zhan, C., Cohn, A.G.: A spatial logic based on regions and connection. In: Third International Conference on Knowledge Representation and Reasoning (1992)
Cohn, A.G., Hazarika, S.: Qualitative spatial representation and reasoning: an overview. Fundamenta Informaticae 46, 1–29 (2001)
Cohn, A.G., Renz, J.: Qualitative spatial representation and reasoning. In: van Harmelen, F., Lifschitz, V., Porter, B. (eds.) Handbook of Knowledge Representation, pp. 551–596. Elsevier B.V, Amsterdam (2008)
Allen, J.: Maintaining knowledge about temporal intervals. Commun. ACM 26, 832–843 (1983)
Dubba, K., Cohn, A.G., Hogg, D.C.: Event model learning from complex videos using ILP. In: European Conference on Artificial Intelligence, pp. 93–98 (2010)
Galata, A., Johnson, N., Hogg, D.: Learning behaviour models of human activities. In: British Machine Vision Conference (1999)
Galata, A., Johnson, N., Hogg, D.: Learning variable-length markov models of behavior. Comput. Vis. Image Underst. 81, 398–413 (2001)
Dollar, P., Rabaud, V., Cottrell, G., Belongie, S.: Behavior recognition via sparse spatio-temporal features. In: 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, pp. 65–72. IEEE (2005)
Xia, L., Aggarwal, J.: Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2834–2841. IEEE (2013)
Zhang, H., Parker, L.: 4-dimensional local spatio-temporal features for human activity recognition. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2044–2049. IEEE (2011)
Forbus, K.: Qualitative modeling. In: van Harmelen, F., Lifschitz, V., Porter, B. (eds.) Handbook of Knowledge Representation, pp. 361–393. Elsevier B.V, Amsterdam (2008)
Zhang, Y., Liu, X., Chang, M.-C., Ge, W., Chen, T.: Spatio-temporal phrases for activity recognition. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part III. LNCS, vol. 7574, pp. 707–721. Springer, Heidelberg (2012)
Taylor, G.W., Fergus, R., LeCun, Y., Bregler, C.: Convolutional learning of spatio-temporal features. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part VI. LNCS, vol. 6316, pp. 140–153. Springer, Heidelberg (2010)
Chen, B., Ting, J.A., Marlin, B., de Freitas, N.: Deep learning of invariant spatio-temporal features from video. In: NIPS 2010 Deep Learning and Unsupervised Feature Learning Workshop (2010)
Le, Q.V., Zou, W.Y., Yeung, S.Y., Ng, A.Y.: Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In: CVPR 2011, pp. 3361–3368. IEEE (2011)
Ryoo, M.S., Aggarwal, J.: Spatio-temporal relationship match: video structure comparison for recognition of complex human activities. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 1593–1600 (2009)
Brendel, W., Todorovic, S.: Learning spatiotemporal graphs of human activities. In: Proceedings of the 2011 International Conference on Computer Vision, ICCV 2011. IEEE Computer Society (2011)
Rybok, L., Schauerte, B., Al-Halah, Z., Stiefelhagen, R.: “Important Stuff, Everywhere!” activity recognition with salient proto-objects as context. In: IEEE Winter Conference on Applications of Computer Vision (WACV) (2014)
Peng, H., Long, F., Ding, C.: Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell. 27, 1226–1238 (2005)
Pujari, A., Vijaya Kumari, G., Sattar, A.: INDu: an interval & duration network. In: Foo, N. (ed) AI 1999. LNCS, vol. 1747, pp. 291–303. Springer, Heidelberg (1999)
Behera, A., Hogg, D.C., Cohn, A.G.: Egocentric activity monitoring and recovery. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012, Part III. LNCS, vol. 7726, pp. 519–532. Springer, Heidelberg (2013)
Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Tech. 2, 27:1–27:27 (2011)
Acknowledgement
The financial support of RACE (FP7-ICT-287752) and STRANDS (FP7-ICT-600623) projects is gratefully acknowledged.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Tayyub, J., Tavanai, A., Gatsoulis, Y., Cohn, A.G., Hogg, D.C. (2015). Qualitative and Quantitative Spatio-temporal Relations in Daily Living Activity Recognition. In: Cremers, D., Reid, I., Saito, H., Yang, MH. (eds) Computer Vision -- ACCV 2014. ACCV 2014. Lecture Notes in Computer Science(), vol 9007. Springer, Cham. https://doi.org/10.1007/978-3-319-16814-2_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-16814-2_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16813-5
Online ISBN: 978-3-319-16814-2
eBook Packages: Computer ScienceComputer Science (R0)