Understanding of Human Behavior with a Robotic Agent Through Daily Activity Analysis
- 39 Downloads
Abstract
Personal assistive robots to be realized in the near future should have the ability to seamlessly coexist with humans in unconstrained environments, with the robot’s capability to understand and interpret the human behavior during human–robot cohabitation significantly contributing towards this end. Still, the understanding of human behavior through a robot is a challenging task as it necessitates a comprehensive representation of the high-level structure of the human’s behavior from the robot’s low-level sensory input. The paper at hand tackles this problem by demonstrating a robotic agent capable of apprehending human daily activities through a method, the Interaction Unit analysis, that enables activities’ decomposition into a sequence of units, each one associated with a behavioral factor. The modelling of human behavior is addressed with a Dynamic Bayesian Network that operates on top of the Interaction Unit, offering quantification of the behavioral factors and the formulation of the human’s behavioral model. In addition, light-weight human action and object manipulation monitoring strategies have been developed, based on RGB-D and laser sensors, tailored for onboard robot operation. As a proof of concept, we used our robot to evaluate the ability of the method to differentiate among the examined human activities, as well as to assess the capability of behavior modeling of people with Mild Cognitive Impairment. Moreover, we deployed our robot in 12 real house environments with real users, showcasing the behavior understanding ability of our method in unconstrained realistic environments. The evaluation process revealed promising performance and demonstrated that human behavior can be automatically modeled through Interaction Unit analysis, directly from robotic agents.
Keywords
Human behavior understanding Daily activities interpretation Interaction Unit analysis Bayesian networks Mobile robotsNotes
Acknowledgements
We would like to thank Fundacio ACE Barcelona Alzheimer Institute & Research Centre as well as the Medical University of Lublin, Poland, for offering their expertise in the annotation of the behavioral factors in the IU analysis as well as for offering their premises for the dataset acquisition with RAMCIP robot.
Funding
This work has been supported by the EU Horizon 2020 funded project namely: “Robotic Assistant for MCI Patients at home (RAMCIP)” under the Grant Agreement with No. 643433.
Compliance with Ethical Standards
Conflict of interest
The authors declare that they have no conflict of interest.
References
- 1.Aggarwal JK, Ryoo MS (2011) Human activity analysis: a review. ACM Comput Surv (CSUR) 43(3):16CrossRefGoogle Scholar
- 2.Atkeson CG, Hale JG, Pollick F, Riley M, Kotosaka S, Schaul S, Shibata T, Tevatia G, Ude A, Vijayakumar S et al (2000) Using humanoid robots to study human behavior. IEEE Intell Syst Appl 15(4):46–56CrossRefGoogle Scholar
- 3.Baddeley AD, Baddeley H, Bucks R, Wilcock G (2001) Attentional control in Alzheimer’s disease. Brain 124(8):1492–1508CrossRefGoogle Scholar
- 4.Berelson B, Steiner GA (1964) Human behavior: an inventory of scientific findings. Trans-action 1:2–2Google Scholar
- 5.Bishop CM (2006) Pattern recognition and machine learning. Springer, New YorkzbMATHGoogle Scholar
- 6.Bloem BR, Valkenburg VV, Slabbekoorn M, Willemsen MD (2001) The multiple tasks test: development and normal strategies. Gait Posture 14(3):191–202CrossRefGoogle Scholar
- 7.Charalampous K, Kostavelis I, Gasteratos A (2016) Robot navigation in large-scale social maps: an action recognition approach. Expert Syst Appl 66:261–273CrossRefGoogle Scholar
- 8.Charalampous K, Kostavelis I, Gasteratos A (2017) Recent trends in social aware robot navigation: a survey. Robot Auton Syst 93:85–104CrossRefGoogle Scholar
- 9.Chrungoo A, Manimaran S, Ravindran B (2014) Activity recognition for natural human robot interaction. In: International conference on social robotics. Springer, pp 84–94Google Scholar
- 10.Coppola C, Cosar S, Faria DR, Bellotto N (2017) Automatic detection of human interactions from RGB-D data for social activity classification. In: 2017 26th IEEE international symposium on robot and human interactive communication (RO-MAN), pp 871–876Google Scholar
- 11.Doumanoglou A, Kouskouridas R, Malassiotis S, Kim TK (2016) Recovering 6D object pose and predicting next-best-view in the crowd. In: IEEE conference on computer vision and pattern recognition, pp 3583–3592Google Scholar
- 12.Faria DR, Premebida C, Nunes U (2014) A probabilistic approach for human everyday activities recognition using body motion from RGB-D images. In: The 23rd IEEE international symposium on robot and human interactive communication, pp 732–737Google Scholar
- 13.Foka AF, Trahanias PE (2010) Probabilistic autonomous robot navigation in dynamic environments with human motion prediction. Int J Soc Robot 2(1):79–94CrossRefGoogle Scholar
- 14.Garrell A, Villamizar M, Moreno-Noguer F, Sanfeliu A (2017) Teaching robots proactive behavior using human assistance. Int J Soc Robot 9(2):231–249CrossRefGoogle Scholar
- 15.Grześ M, Hoey J, Khan SS, Mihailidis A, Czarnuch S, Jackson D, Monk A (2014) Relational approach to knowledge engineering for POMDP-based assistance systems as a translation of a psychological model. Int J Approx Reason 55(1):36–58CrossRefGoogle Scholar
- 16.Hall ET (1966) The hidden dimension. Doubleday, New YorkGoogle Scholar
- 17.Han F, Reily B, Hoff W, Zhang H (2017) Space-time representation of people based on 3D skeletal data: a review. Comput Vis Image Underst 158:85–105CrossRefGoogle Scholar
- 18.Hoey J, Plötz T, Jackson D, Monk A, Pham C, Olivier P (2011) Rapid specification and automated generation of prompting systems to assist people with dementia. Pervasive Mobile Comput 7(3):299–318CrossRefGoogle Scholar
- 19.Karplus K, Brown M, Hughey R, Krogh A, Mian IS, Haussler D (1996) Dirichlet mixtures: a method for improving detection of weak but significant protein sequence homology. Comput Appl Biosci 12:327–345Google Scholar
- 20.Katz S, Downs TD, Cash HR, Grotz RC (1970) Progress in development of the index of ADL1. Gerontologist 10(1 Part 1):20–30CrossRefGoogle Scholar
- 21.Kim B, Pineau J (2016) Socially adaptive path planning in human environments using inverse reinforcement learning. Int J Soc Robot 8(1):51–66CrossRefGoogle Scholar
- 22.Koppula H, Saxena A (2013) Learning spatio-temporal structure from RGB-D videos for human activity detection and anticipation. In: International conference on machine learning, pp 792–800Google Scholar
- 23.Kostavelis I, Kargakos A, Giakoumis D, Tzovaras D (2017) Robots workspace enhancement with dynamic human presence for socially-aware navigation. In: International conference on computer vision systems. Springer, pp 279–288Google Scholar
- 24.Leigh A, Pineau J (2014) Laser-based person tracking for clinical locomotion analysis. In: IROS-rehabilitation and assistive roboticsGoogle Scholar
- 25.Leite I, Martinho C, Paiva A (2013) Social robots for long-term interaction: a survey. Int J Soc Robot 5(2):291–308CrossRefGoogle Scholar
- 26.Martinez-Contreras F, Orrite-Urunuela C, Herrero-Jaraba E, Ragheb H, Velastin SA (2009) Recognizing human actions using silhouette-based HMM. In: Sixth IEEE international conference on advanced video and signal based surveillance. IEEE, pp 43–48Google Scholar
- 27.Norman DA, Shallice T (1986) Attention to action. In: Davidson RJ, Schwartz GE, Shapiro D (eds) Consciousness and self-regulation. Springer, New York, pp 1–18Google Scholar
- 28.Oreifej O, Liu Z (2013) HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences. In: IEEE conference on computer vision and pattern recognition, pp 716–723Google Scholar
- 29.Panangadan A, Matarić M, Sukhatme GS (2010) Tracking and modeling of human activity using laser rangefinders. Int J Soc Robot 2(1):95–107CrossRefGoogle Scholar
- 30.Piyathilaka L, Kodagoda S (2015) Human activity recognition for domestic robots. In: Mejias L, Corke P, Roberts J (eds) Field and service robotics. Springer, Berlin, pp 395–408Google Scholar
- 31.Premebida C, Faria D, Souza F (2017) Dynamic Bayesian network for time-dependent classification problems in robotics. In: Prieto Tejedor J (ed) Bayesian inference, chapter 15. InTech, CroatiaGoogle Scholar
- 32.Rahmani H, Mahmood A, Huynh DQ, Mian A (2014) Real time action recognition using histograms of depth gradients and random decision forests. In: 2014 IEEE winter conference on applications of computer vision. IEEE, pp 626–633Google Scholar
- 33.Roitberg A, Perzylo A, Somani N, Giuliani M, Rickert M, Knoll A (2014) Human activity recognition in the context of industrial human–robot interaction. In: Asia-Pacific Signal and Information Processing Association. IEEE, pp 1–10Google Scholar
- 34.Rusu RB, Marton ZC, Blodow N, Dolha M, Beetz M (2008) Towards 3D point cloud based object maps for household environments. Robot Auton Syst 56(11):927–941CrossRefGoogle Scholar
- 35.Rybok L, Schauerte B, Al-Halah Z, Stiefelhagen R (2014) Important stuff, everywhere! activity recognition with salient proto-objects as context. In: IEEE winter conference on applications of computer vision. IEEE, pp 646–651Google Scholar
- 36.Ryu H, Monk A (2005) Will it be a capital letter: signalling case mode in mobile phones. Interact Comput 17(4):395–418CrossRefGoogle Scholar
- 37.Ryu H, Monk A (2009) Interaction unit analysis: a new interaction design framework. Hum Comput Interact 24(4):367–407CrossRefGoogle Scholar
- 38.Salah A, Ruiz-del Solar J, Mericli C, Oudeyer PY (2012) Human behavior understanding for robotics. In: Salah AA, Ruiz-del-Solar J, Meriçli Ç, Oudeyer P-Y (eds) Human behavior understanding. Springer, Berlin, pp 1–16CrossRefGoogle Scholar
- 39.Salah AA, Gevers T, Sebe N, Vinciarelli A et al (2010) Challenges of human behavior understanding. In: Salah AA, Ruiz-del-Solar J, Meriçli Ç, Oudeyer P-Y (eds) HBU. Springer, Berlin, pp 1–12Google Scholar
- 40.Salah AA, Lepri B, Pianesi F, Pentland AS (2011) Human behavior understanding for inducing behavioral change: application perspectives. In: International workshop on human behavior understanding. Springer, pp 1–15Google Scholar
- 41.Santos L, Khoshhal K, Dias J (2015) Trajectory-based human action segmentation. Pattern Recognit 48(2):568–579CrossRefGoogle Scholar
- 42.Schmidler SC, Liu JS, Brutlag DL (2000) Bayesian segmentation of protein secondary structure. J Comput Biol 7(1–2):233–248CrossRefGoogle Scholar
- 43.Schmidt T, Newcombe R, Fox D (2015) DART: dense articulated real-time tracking with consumer depth cameras. Auton Robots 39:239–258CrossRefGoogle Scholar
- 44.Schmidt-Rohr SR, Losch M, Dillmann R (2008) Human and robot behavior modeling for probabilistic cognition of an autonomous service robot. In: IEEE international symposium on robot and human interactive communication. IEEE, pp 635–640Google Scholar
- 45.Shan J, Akella S (2014) 3D human action segmentation and recognition using pose kinetic energy. In: IEEE workshop on advanced robotics and its social impacts. IEEE, pp 69–75Google Scholar
- 46.Shotton J, Sharp T, Kipman A, Fitzgibbon A, Finocchio M, Blake A, Cook M, Moore R (2013) Real-time human pose recognition in parts from single depth images. Commun ACM 56(1):116–124CrossRefGoogle Scholar
- 47.Skinner BF (1953) Science and human behavior. Simon and Schuster, New YorkGoogle Scholar
- 48.Sonn U (1996) Longitudinal studies of dependence in daily life activities among elderly persons. Scand J Rehabilit Med Suppl 34:1–35Google Scholar
- 49.Stavropoulos G, Giakoumis D, Moustakas K, Tzovaras D (2017) Automatic action recognition for assistive robots to support MCI patients at home. In: 10th international conference on pervasive technologies related to assistive environments. ACM, pp 366–371Google Scholar
- 50.Sung J, Ponce C, Selman B, Saxena A (2012) Unstructured human activity detection from RGBD images. In: IEEE international conference on robotics and automation. IEEE, pp 842–849Google Scholar
- 51.Takayama L, Pantofaru C (2009) Influences on proxemic behaviors in human–robot interaction. In: IEEE international conference on intelligent robots and systems. IEEE, pp 5495–5502Google Scholar
- 52.Taylor J, Shotton J, Sharp T, Fitzgibbon A (2012) The Vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 103–110Google Scholar
- 53.Tsai MJ, Wu CL, Pradhan SK, Xie Y, Li TY, Fu LC, Zeng YC (2016) Context-aware activity prediction using human behavior pattern in real smart home environments. In: 2016 IEEE international conference on automation science and engineering (CASE). IEEE, pp 168–173Google Scholar
- 54.Vasileiadis M, Malassiotis S, Giakoumis D, Bouganis CS, Tzovaras D (2017) Robust human pose tracking for realistic service robot applications. In: IEEE international conference on computer vision workshops, pp 1363–1372Google Scholar
- 55.Wang J, Liu Z, Wu Y (2014) Learning actionlet ensemble for 3D human action recognition. In: Wang J (ed) Human action recognition with depth cameras. Springer, Basel, pp 11–40CrossRefGoogle Scholar
- 56.Whiten C, Laganiere R, Bilodeau GA (2013) Efficient action recognition with MoFREAK. In: International conference on computer and robot vision. IEEE, pp 319–325Google Scholar
- 57.Wu J, Osuntogun A, Choudhury T, Philipose M, Rehg JM (2007) A scalable approach to activity recognition based on object use. In: IEEE international conference on computer vision. IEEE, pp 1–8Google Scholar
- 58.Yang X, Tian Y (2014) Effective 3D action recognition using eigenjoints. J Vis Commun Image Represent 25(1):2–11MathSciNetCrossRefGoogle Scholar
- 59.Zhu Y, Chen W, Guo G (2014) Evaluating spatiotemporal interest point features for depth-based action recognition. Image Vis Comput 32(8):453–464CrossRefGoogle Scholar
- 60.Ziaeefard M, Bergevin R (2015) Semantic human activity recognition: a literature review. Pattern Recognit 48(8):2329–2345CrossRefGoogle Scholar
- 61.Zipf GK (2016) Human behavior and the principle of least effort: an introduction to human ecology. Ravenio Books, CambridgeGoogle Scholar