Towards Multi-resident Activity Monitoring with Smarter Safer Home Platform

  • Son N. Tran
  • Qing ZhangEmail author
Part of the Computer Communications and Networks book series (CCN)


This chapter demonstrates a system that can turn a normal house to a smart house for daily activity monitoring with the use of ambient sensors. We first introduce our smarter safer home platform and its applications in supporting independent livings of seniors in their own home. Then we show a proof of concept which includes a novel activity annotation method through voice recording and deep learning techniques for automatic activity recognition. Our multi-resident activity recognition system (MRAR) is designed to support multiple occupants in a house with minimum impact on their living styles. We evaluate the system in a real house lived by a family of three. The experimental results show that it is promising to develop a smart home system for multiple residents which is portable and easy to deploy.


Multi-resident activity Smart homes Ambient intelligence Voice-based annotation 


  1. 1.
    Jiang L, Liu D-Y, Yang B (2004) Smart home research. In: Proceedings of International Conference on Machine Learning and Cybernetics (IEEE Cat. No.04EX826), vol 2, pp 659–663Google Scholar
  2. 2.
    Lutolf R (1992) Smart home concept and the integration of energy meters into a home based system. In: Proceedings of the 7th International Conference on Metering Apparatus and Tariffs for Electricity Supply, pp 277–278Google Scholar
  3. 3.
    Cook DJ, Das SK (2007) How smart are our environments? an updated look at the state of the art. Pervasive Mob Comput 3(2):53–73CrossRefGoogle Scholar
  4. 4.
    Cucchiara R, Grana C, Prati A, Vezzani R (2005) Computer vision system for in-house video surveillance. In: IEEE Proceedings-Vision, Image, and Signal Processing, 2005, pp 242–249CrossRefGoogle Scholar
  5. 5.
    Nguyen N, Venkatesh S, Bui H (2006) Recognising behaviours of multiple people with hierarchical probabilistic model and statistical data association. In: Proceedings of the 17th British Machine Vision Conference (BMVC’06). The British Machine Vision Association and Society for Pattern Recognition, pp 1239–1248Google Scholar
  6. 6.
    McCowan I, Gatica-Perez D, Bengio S, Lathoud G, Barnard M, Zhang D (2005) Automatic analysis of multimodal group actions in meetings. IEEE Trans Pattern Anal Mach Intell 27(3):305–317. Scholar
  7. 7.
    Du Y, Chen F, Xu W (2007) Human interaction representation and recognition through motion decomposition. IEEE Signal Process Lett 14(12):952–955. Scholar
  8. 8.
    Du Y, Chen F, Xu W, Li Y (2006) Recognizing interaction activities using dynamic Bayesian network. In: Proceedings of the 18th International Conference on Pattern Recognition (ICPR’06), pp 618–621Google Scholar
  9. 9.
    Jalal A, Uddin MZ, Kim T-S (2012) Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home. IEEE Trans Consum Electron 58(3):863–871CrossRefGoogle Scholar
  10. 10.
    Han J, Pauwels EJ, de Zeeuw PM, de With PHN (2012) Employing a RGB-D sensor for real-time tracking of humans across multiple re-entries in a smart environment. IEEE Trans Consum Electron 58(2):255–263Google Scholar
  11. 11.
    Yamamoto Y, Yoda I, Sakaue K (2004) Arm-pointing gesture interface using surrounded stereo cameras system. In: Proceedings of the 17th International Conference on Pattern Recognition, 2004, ICPR 2004, vol 4. Cambridge, pp 965–970Google Scholar
  12. 12.
    Liu L, Cheng L, Liu Y, Jia Y, Rosenblum D (2016) Recognizing complex activities by a probabilistic interval-based model. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI Press, pp 1266–1272Google Scholar
  13. 13.
    Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2activity: recognizing complex activities from sensor data. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI’15. AAAI Press, pp 1617–1623Google Scholar
  14. 14.
    Plötz T, Hammerla NY, Olivier P (2011) Feature learning for activity recognition in ubiquitous computing. In: Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, IJCAI’11. AAAI Press, pp 1729–1734Google Scholar
  15. 15.
    Alemdar H, Ertan H, Incel OD, Ersoy C (2013) Aras human activity datasets in multiple homes with multiple residents. In: Proceedings of the 7th International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth ’13, ICST, Brussels, Belgium, pp 232–235Google Scholar
  16. 16.
    Chen R, Tong Y (2014) A two-stage method for solving multi-resident activity recognition in smart environments. Entropy 16(4):2184CrossRefGoogle Scholar
  17. 17.
    Chiang Y-T, Hsu KC, Lu CH, Fu L-C, Hsu JY-J (2010) Interaction models for multiple-resident activity recognition in a smart home. In: IEEE/RSJ International Conference on IROS, pp 3753–3758Google Scholar
  18. 18.
    Cook DJ (2012) Learning setting-generalized activity models for smart spaces. IEEE Intell Syst 27:32–38CrossRefGoogle Scholar
  19. 19.
    Singla G, Cook DJ, Schmitter-Edgecombe M (2010) Recognizing independent and joint activities among multiple residents in smart environments. J Ambient Intell Humaniz Comput 1(1):57–63CrossRefGoogle Scholar
  20. 20.
    Tran SN, Zhang Q, Karunanithi M (2017) Improving multi-resident activity recognition for smarter homes. In: IJCAI WS on AI for IoTGoogle Scholar
  21. 21.
    Wang L, Tao G, Tao X, Chen H, Jian L (2011) Recognizing multi-user activities using wearable sensors in a smart home. Pervasive Mob Comput 7(3):287–298CrossRefGoogle Scholar
  22. 22.
    Khan SS, Karg ME, Hoey J, Kulic D (2012) Towards the detection of unusual temporal events during activities using HMMs. In: Proceedings of the Conference on Ubiquitous Computing (UbiComp’12). ACM, New York, pp 1075–1084.
  23. 23.
    Riboni D, Pareschi L, Radaelli L, Bettini C (2011) Is ontology-based activity recognition really effective? In: Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops’11), pp 427–431.
  24. 24.
    van Kasteren TLM, Englebienne G, Kröse BJA (2011) Hierarchical activity recognition using automatically clustered actions. In Proceedings of the 2nd International Conference on Ambient Intelligence (AmI’11), pp 82–91Google Scholar
  25. 25.
    van Kasteren TLM, Englebienne G, Krose BJA (2010) Human activity recognition from wireless sensor network data: benchmark and software. In: Chen L, Nugent CD, Biswas J, Hoey J (eds) Activity recognition in pervasive intelligent environments. Atlantis ambient and pervasive intelligence. Atlantis Press, pp 165–186Google Scholar
  26. 26.
    van Kasteren TLM, Noulas A, Englebienne G, Krose B (2008) Accurate activity recognition in a home setting. In: Proceedings of the 10th International Conference on Ubiquitous Computing (UbiComp’08). ACM, pp 1–9.
  27. 27.
    Benmansour A, Bouchachia A, Feham M (2015) Multioccupant activity recognition in pervasive smart home environments. ACM Comput Surv 48(3):34:1–34:36CrossRefGoogle Scholar
  28. 28.
    Liao L, Fox D, Kautz H (2005) Location-based activity recognition using relational Markov networks. In: Proceedings of the International Joint Conference on Artificial Intelligence, pp 773–778Google Scholar
  29. 29.
    Munguia-Tapia E, Intille SS, Larson K (2004) Activity recognition in the home using simple and ubiquitous sensors. In: Proceedings of Pervasive, pp 158–175Google Scholar
  30. 30.
    Philipose M, Fishkin K, Perkowitz M, Patterson D, Fox D, Kautz H, Hahnel D (2004) Inferring activities from interactions with objects. IEEE Pervasive Comput 3:50–57CrossRefGoogle Scholar
  31. 31.
    Szewcyzk S, Dwan K, Minor B, Swelove B, Cook D (2009) Annotating smart environment sensor data for activity learning. In: Technology and Health Care, special issue on Smart Environments: Technology to support health careGoogle Scholar
  32. 32.
    Chen C, Cook D (2012) Behavior-based home energy prediction. In: Proceedings of the International Conference on Intelligent EnvironmentsGoogle Scholar
  33. 33.
    Hsu K-C, Chiang Y-T, Lin G-Y, Lu C-H, Hsu JY-J, Fu L-C (2010) Strategies for inference mechanism of conditional random fields for multiple-resident activity recognition in a smart home. In: Proceedings of International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Springer, Berlin, pp 417–426CrossRefGoogle Scholar
  34. 34.
    Crandall AS, Cook DJ (2008) Resident and caregiver: handling multiple people in a smart carefacility. In: Proceedings of the AAAI Fall Symposium—AI in Eldercare: New Solutions to Old Problems, pp 39–47.
  35. 35.
    Crandall AS, Cook DJ (2008) Attributing events to individuals in multi-inhabitant environments. In Proceedings of the IET 4th International Conference on Intelligent Environments (August 2008), pp 1–8.
  36. 36.
    Tunca C, Alemdar H, Ertan H, Incel OD, Ersoy C (2014) Multimodal wireless sensor network-based ambient assisted living in real homes with multiple residents. SensorsGoogle Scholar
  37. 37.
    Chung J, Gülçehre Ç, Cho KH, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555Google Scholar
  38. 38.
    Just Checking (2016)
  39. 39.
    My Lively (2016)
  40. 40.
    Tunstall (2012) Solution sheet ADLife an activities of daily living (ADL) monitoring.
  41. 41.
    Telemedcare (2016)
  42. 42.
    Care Innovations (2016)
  43. 43.
    Bradford D, Freyne J, Karunanithi M (2013) Sensors on my bed: the ups and downs of in-home monitoring. In: Inclusive Society: Health and Well-being in the Community, and Care at Home. Proceedings of the 11th International Conference on Smart Homes and Health Telematics, ICOST 2013, Singapore, pp 10–18CrossRefGoogle Scholar
  44. 44.
    Chen L, Nugent CD, Wang H (2012) A knowledge-driven approach to activity recognition in smart homes. IEEE Trans Knowl Data Eng 24(6):961–974CrossRefGoogle Scholar
  45. 45.
    Nugent CD, Chen L, Okeyo G (2014) An ontology-based hybrid approach to activity modelling for smart homes. IEEE Trans Hum-Mach Syst 44(1):92–105Google Scholar
  46. 46.
    Alshammari NO et al (2017) OpenSHS: open smart home simulator. SensorsGoogle Scholar
  47. 47.
    WSU (2007) Washington State University Casas project.
  48. 48.
    Feuz KD, Cook DJ (2013) Real-time annotation tool (RAT). In: Proceedings of the AAAI Workshop on Activity Context-Aware System ArchitecturesGoogle Scholar
  49. 49.
    Karunanithi M, Zhang Q (2015) Objective assessment and scoring of activities of daily living. Pending PatentGoogle Scholar
  50. 50.
    Cook DJ, Crandall A, Singla G, Thomas B (2010) Detection of social interaction in smart spaces. Cybern Syst 41(2):90–104CrossRefGoogle Scholar
  51. 51.
    Sutton C, McCallum A, Rohanimanesh K (2007) Dynamic conditional random fields: factorized probabilistic models for labeling and segmenting sequence data. J Mach Learn Res 8:693–723zbMATHGoogle Scholar
  52. 52.
    Prossegger M, Bouchachia A (2014) Multi-resident activity recognition using incremental decision trees. In: Adaptive and Intelligent Systems—Third International Conference, ICAIS 2014, Bournemouth, UK, pp 182–191Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.ICT DisciplineThe University of TasmaniaLauncestonAustralia
  2. 2.The Australian E-Health Research Centre, CSIROHerstonAustralia

Personalised recommendations