Ambient and smartphone sensor assisted ADL recognition in multi-inhabitant smart environments

  • Nirmalya Roy
  • Archan Misra
  • Diane Cook
Original Research


Activity recognition in smart environments is an evolving research problem due to the advancement and proliferation of sensing, monitoring and actuation technologies to make it possible for large scale and real deployment. While activities in smart home are interleaved, complex and volatile; the number of inhabitants in the environment is also dynamic. A key challenge in designing robust smart home activity recognition approaches is to exploit the users’ spatiotemporal behavior and location, focus on the availability of multitude of devices capable of providing different dimensions of information and fulfill the underpinning needs for scaling the system beyond a single user or a home environment. In this paper, we propose a hybrid approach for recognizing complex activities of daily living (ADL), that lie in between the two extremes of intensive use of body-worn sensors and the use of ambient sensors. Our approach harnesses the power of simple ambient sensors (e.g., motion sensors) to provide additional ‘hidden’ context (e.g., room-level location) of an individual, and then combines this context with smartphone-based sensing of micro-level postural/locomotive states. The major novelty is our focus on multi-inhabitant environments, where we show how the use of spatiotemporal constraints along with multitude of data sources can be used to significantly improve the accuracy and computational overhead of traditional activity recognition based approaches such as coupled-hidden Markov models. Experimental results on two separate smart home datasets demonstrate that this approach improves the accuracy of complex ADL classification by over 30 %, compared to pure smartphone-based solutions.


Activity Recognition Smart Home Viterbi Algorithm Smart Environment Smart Home Environment 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The work of Nirmalya Roy is partially supported by the National Science Foundation Award \(\#1344990\) and Constellation \(E^2\): Energy to Educate Grant. The work of Archan Misra is partially supported by the Singapore Ministry of Education Academic Research Fund Tier 2 under research Grant MOE2011-T2-1-001. The work of Diane Cook is partially supported by NSF Grants 1064628, 0852172, CNS-1255965, and NIH Grant R01EB009675.


  1. Acampora G, Cook D, Rashidi P, Vasilakos A (2013) A survey on ambient intelligence in healthcare. Proc IEEE 101(12):2470–2494CrossRefGoogle Scholar
  2. Activity Recognition Challenge (2013). Accessed June 2013
  3. Activity Recognition Code (2014). Accessed Jan 2014
  4. Alam M, Pathak N, Roy N (2015) Mobeacon: an iBeacon-assisted smartphone-based real time activity recognition framework. In: Proceedings of the 12th international conference on mobile and ubiquitous systems: computing, networking and services (in press)Google Scholar
  5. Alam M, Roy N (2014) Gesmart: a gestural activity recognition model for predicting behavioral health. In: Proceeding of the IEEE international conference on smart computingGoogle Scholar
  6. Almashaqbeh G, Hayajneh T, Vasilakos A, Mohd B (2014) QoS-aware health monitoring system using cloud-based WBANs. J Med Syst 38(10):121CrossRefGoogle Scholar
  7. Android Wear: Information that Moves with You (2015). Accessed Jan 2015
  8. Bergmann J, McGregor A (2011) Body-worn sensor design: what do patients and clinicians want? Ann Biomed Eng 39(9):2299–2312CrossRefGoogle Scholar
  9. Brand M (1996) Coupled hidden Markov models for modeling interacting processes. Technical report 405, MIT Lab for Perceptual ComputingGoogle Scholar
  10. Chavarriaga R, Sagha H, Calatroni A, Digumarti S, Trster G, Milln J, Roggen D (2013) The opportunity challenge: a benchmark database for on-body sensor-based activity recognition. Pattern Recognit Lett 34(15):2033–2042CrossRefGoogle Scholar
  11. Chen L, Hoey J, Nugent C, Cook D, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man Cybern-Part C 42(6):790–808CrossRefGoogle Scholar
  12. Chen M, Gonzalez S, Vasilakos A, Cao H, Leung V (2011) Body area networks: a survey. MONET 16(2):171–193Google Scholar
  13. Clarkson B, Mase K, Pentland A (2000) Recognizing user context via wearable sensors. In: Proceedings of the 4th international symposium on wearable computersGoogle Scholar
  14. Dernbach S, Das B, Krishnan N, Thomas B, Cook D (2012) Simple and complex acitivity recognition through smart phones. In: Proceedings of the international conference on intelligent environmentsGoogle Scholar
  15. Feng Z, Zhu Y, Zhang Q, Ni L, Vasilakos A (2014) Trac: truthful auction for location-aware collaborative sensing in mobile crowdsourcing. INFOCOM, 1231–1239Google Scholar
  16. Fortino G, Fatta G, Pathan M, Vasilakos A (2014) Cloud-assisted body area networks: state-of-the-art and future challenges. Wirel Netw 20(7):1925–1938CrossRefGoogle Scholar
  17. Gong S, Xiang T (2003) Recognition of group activities using dynamic probabilistic networks. In: Proceedings of international conference on computer visionGoogle Scholar
  18. Gyorbiro N, Fabian A, Homanyi G (2008) An activity recognition system for mobile phones. Mob Netw Appl 14(1):82–91CrossRefGoogle Scholar
  19. Hayajneh T, Almashaqbeh G, Ullah S, Vasilakos A (2014) A survey of wireless technologies coexistence in wban: analysis and open research issues. Wirel Netw 20(8):2165–2199CrossRefGoogle Scholar
  20. Hossain H, Roy N, Khan M (2015) Sleep well: a sound sleep monitoring framework for community scaling. In: Proceeding of the IEEE international conference on mobile data managementGoogle Scholar
  21. Huawei Smart Bracelet (2015). Accessed Feb 2015
  22. Huynh T, Blanke U, Schiele B (2007) Scalable recognition of daily activities from wearable sensors. In: LNCS LoCA, vol 4718Google Scholar
  23. Intel Make it Wearable (2014). Accessed Apr 2014
  24. Intille S, Larson K, Tapia E, Beaudin J, Kaushik P, Nawyn J, Rockinson R (2006) Using a live-in laboratory for ubiquitous computing research. In: Proceedings of 4th international conference on pervasive computing, vol 3968Google Scholar
  25. Kasteren T, Noulas A, Englebienne G, Krose B (2008) Accurate activity recognition in a home setting. In: Proceedings of the 10th international conference on ubiquitous computing, vol 3968Google Scholar
  26. Khan M, Hossain H, Roy N (2015a) Infrastructure-less occupancy detection and semantic localization in smart environments. In Proceedings of the 12th international conference on mobile and ubiquitous systems: computing, networking and services (in press)Google Scholar
  27. Khan M, Hossain H, Roy N (2015b) Sensepresence: infrastructure-less occupancy detection for opportunistic sensing applications. In: IEEE international conference on mobile data management (in press)Google Scholar
  28. Khan M, Lu S, Roy N, Pathak N (2015c) Demo abstract: a microphone sensor based system for green building applications. In: IEEE international conference on pervasive computing and communications (PerCom)Google Scholar
  29. Kwapisz J, Weiss G, Moore S (2010) Activity recognition using cell phone accelerometers. In: International workshop on knowledge discovery from sensor dataGoogle Scholar
  30. Lee C, Hsu C, Lai Y, Vasilakos A (2013) An enhanced mobile-healthcare emergency system based on extended chaotic maps. J Med Syst 37(5):9973CrossRefGoogle Scholar
  31. Lester J, Choudhury T, Borriello G (2006) A practical approach to recognizing physical activities. In: PERVASIVE LNCS, vol 3968Google Scholar
  32. Lin D, Labeau F, Vasilakos A (2015a) QoE-based optimal resource allocation in wireless healthcare networks: opportunities and challenges. Wirel NetwGoogle Scholar
  33. Lin D, Wu X, Labeau F, Vasilakos A (2015b) Internet of vehicles for e-health applications in view of EMI on medical sensors. J SensGoogle Scholar
  34. Logan B, Healey J, Philipose M, Tapia E, Intille S (2007) A long-term evaluation of sensing modalities for activity recognition. In: UbiComp LNCS, vol 4717Google Scholar
  35. Oliver N, Rosario B, Pentland A (2000) A bayesian computer vision system for modeling human interactions. IEEE Trans Pattern Anal Mach Intell 22(8):831–843CrossRefGoogle Scholar
  36. Pathak N, Khan M, Roy N (2015) Acoustic based appliance state identifications for fine grained energy analytics. In: IEEE international conference on pervasive computing and communications (PerCom)Google Scholar
  37. Philipose M, Fishkin K, Perkowitz M, Patterson D, Hahnel D, Fox D, Kautz H (2004) Inferring activities from interactions with objects. IEEE Pervasive Comput 3(4):50–57CrossRefGoogle Scholar
  38. Plotz T, Flink G (2004) Accelerating the evaluation of profile hmms by pruning techniques. Report 2004-03. In: Tech rep., Faculty of Technology, University of BielefeldGoogle Scholar
  39. Rabiner L (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77(2):257–285CrossRefGoogle Scholar
  40. Rahimi MR, Ren J, Liu C, Vasilakos A, Venkatasubramanian N (2014) Mobile cloud computing: a survey, state of art and future directions. MONET 19(2):133–143Google Scholar
  41. Rahimi MR, Venkatasubramanian N, Mehrotra S, Vasilakos A (2012) Mapcloud: mobile applications on an elastic and scalable 2-tier cloud architecture. In: IEEE/ACM UCCGoogle Scholar
  42. Roy N, Das SK, Julien C (2012) Resource-optimized quality-assured ambiguous context mediation in pervasive environments. IEEE Trans Mob Comput 11(2):218–229CrossRefGoogle Scholar
  43. Roy N, Julien C (2014) Immersive physiotherapy: challenges for smart living environments and inclusive communities. In: Proceeding of the 12th international conference on smart homes and health telematicsGoogle Scholar
  44. Roy N, Kindle B (2014) Monitoring patient recovery using wireless physiotherapy devices. In: Proceeding of the 12th international conference on smart homes and health telematicsGoogle Scholar
  45. Roy N, Misra A, Cook D (2013) Infrastructure-assisted smartphone-based adl recognition in multi-inhabitant smart environments. In: Percom, pp 38–46Google Scholar
  46. Roy N, Misra A, Das SK, Julien C (2009) Quality-of-inference (qoinf)-aware context determination in assisted living environments. In: ACM SIGMOBILE workshop on medical-grade wireless networksGoogle Scholar
  47. Roy N, Misra A, Julien C, Das SK, Biswas J (2011) An energy efficient quality adaptive multi-modal sensor framework for context recognition. In: Percom, pp 63–73Google Scholar
  48. Roy N, Pathak N, Misra A (2015) Aarpa: combining pervasive and power-line sensing for fine-grained appliance usage and energy monitoring. In: IEEE international conference on mobile data management (in press)Google Scholar
  49. Roy N, Roy A, Das S (2006) Context-aware resource management in multi-inhabitant smart homes: a nash h-learning based approach. In: Proceedings of IEEE international conference on pervasive computing and communications (PerCom), pp 372–404Google Scholar
  50. Sheng Z, Yang S, Yu Y, Vasilakos A, McCann J, Leung K (2014) A survey on the ietf protocol suite for the internet of things: standards, challenges, and opportunities. IEEE Wirel Commun 20(6):91–98CrossRefGoogle Scholar
  51. Wang L, Gu T, Tao X, Chen H, Lu J (2011) Recognizing multi-user activities using wearable sensors in a smart home. Pervasive Mob Comput 7(3):287–298CrossRefGoogle Scholar
  52. Wilson D, Atkeson C (2005) Simultaneous tracking and activity recognition (STAR) using many anonymous, binary sensors. Pervasive Comput, 3468:62–79CrossRefGoogle Scholar
  53. Witten L, Frank E (1999) Data mining: practicial machine learning tools and techniques with Java implementations. Morgan Kaufmann, San FranciscoGoogle Scholar
  54. Yan Z, Chakraborty D, Misra A, Jeung H, Aberer K (2012) Sammple: detecting semantic indoor activities in practical settings using locomotive signatures. In: International symposium on wearable computersGoogle Scholar
  55. Yan Z, Zhang P, Vasilakos A (2014) A survey on trust management for internet of things. J Netw Comput Appl 42:37–40CrossRefGoogle Scholar
  56. Yi-Ting C, Kuo-Chung H, Ching-Hu L, Li-Chen F, John H (2010) Interaction models for multiple-resident activity recognition in a smart home. IROS, 3753–3758Google Scholar
  57. Zhang Z, Wang H, Vasilakos A, Fang H (2012) ECG-cryptography and authentication in body area networks. IEEE Trans Inf Technol Biomed 16(6):1070–1078CrossRefGoogle Scholar
  58. Zheng Y, Li D, Vasilakos A (2013) Real-time data report and task execution in wireless sensor and actuator networks using self-aware mobile actuators. Comput Commun 36(9):988–997CrossRefGoogle Scholar
  59. Zhou L, Xiong N, Shu L, Vasilakos A, Yeo S (2010) Context-aware middleware for multimedia services in heterogeneous networks. IEEE Intell Syst 25(2):40–47CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Department of Information SystemsUniversity of Maryland, Baltimore CountyBaltimoreUSA
  2. 2.School of Information SystemsSingapore Management UniversitySingaporeSingapore
  3. 3.School of Electrical Engineering and Computer ScienceWashington State UniversityPullmanUSA

Personalised recommendations