Skip to main content
SpringerLink
Log in

Ambient and smartphone sensor assisted ADL recognition in multi-inhabitant smart environments

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Activity recognition in smart environments is an evolving research problem due to the advancement and proliferation of sensing, monitoring and actuation technologies to make it possible for large scale and real deployment. While activities in smart home are interleaved, complex and volatile; the number of inhabitants in the environment is also dynamic. A key challenge in designing robust smart home activity recognition approaches is to exploit the users’ spatiotemporal behavior and location, focus on the availability of multitude of devices capable of providing different dimensions of information and fulfill the underpinning needs for scaling the system beyond a single user or a home environment. In this paper, we propose a hybrid approach for recognizing complex activities of daily living (ADL), that lie in between the two extremes of intensive use of body-worn sensors and the use of ambient sensors. Our approach harnesses the power of simple ambient sensors (e.g., motion sensors) to provide additional ‘hidden’ context (e.g., room-level location) of an individual, and then combines this context with smartphone-based sensing of micro-level postural/locomotive states. The major novelty is our focus on multi-inhabitant environments, where we show how the use of spatiotemporal constraints along with multitude of data sources can be used to significantly improve the accuracy and computational overhead of traditional activity recognition based approaches such as coupled-hidden Markov models. Experimental results on two separate smart home datasets demonstrate that this approach improves the accuracy of complex ADL classification by over 30 %, compared to pure smartphone-based solutions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

Notes

  1. We interchangeably use Context as a state s in our HMM model. For brevity we denote \(Context^i(t)=s_t\) and \(Context^j(t) = s_t^{'}\) in equations.

References

  • Acampora G, Cook D, Rashidi P, Vasilakos A (2013) A survey on ambient intelligence in healthcare. Proc IEEE 101(12):2470–2494

    Article  Google Scholar 

  • Activity Recognition Challenge (2013). http://www.opportunity-project.eu/challengeDataset. Accessed June 2013

  • Activity Recognition Code (2014). http://ailab.wsu.edu/casas/ar/. Accessed Jan 2014

  • Alam M, Pathak N, Roy N (2015) Mobeacon: an iBeacon-assisted smartphone-based real time activity recognition framework. In: Proceedings of the 12th international conference on mobile and ubiquitous systems: computing, networking and services (in press)

  • Alam M, Roy N (2014) Gesmart: a gestural activity recognition model for predicting behavioral health. In: Proceeding of the IEEE international conference on smart computing

  • Almashaqbeh G, Hayajneh T, Vasilakos A, Mohd B (2014) QoS-aware health monitoring system using cloud-based WBANs. J Med Syst 38(10):121

    Article  Google Scholar 

  • Android Wear: Information that Moves with You (2015). http://googleblog.blogspot.co.uk/2014/03/sharing-whats-up-our-sleeve-android.html. Accessed Jan 2015

  • Bergmann J, McGregor A (2011) Body-worn sensor design: what do patients and clinicians want? Ann Biomed Eng 39(9):2299–2312

    Article  Google Scholar 

  • Brand M (1996) Coupled hidden Markov models for modeling interacting processes. Technical report 405, MIT Lab for Perceptual Computing

  • Chavarriaga R, Sagha H, Calatroni A, Digumarti S, Trster G, Milln J, Roggen D (2013) The opportunity challenge: a benchmark database for on-body sensor-based activity recognition. Pattern Recognit Lett 34(15):2033–2042

    Article  Google Scholar 

  • Chen L, Hoey J, Nugent C, Cook D, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man Cybern-Part C 42(6):790–808

    Article  Google Scholar 

  • Chen M, Gonzalez S, Vasilakos A, Cao H, Leung V (2011) Body area networks: a survey. MONET 16(2):171–193

    Google Scholar 

  • Clarkson B, Mase K, Pentland A (2000) Recognizing user context via wearable sensors. In: Proceedings of the 4th international symposium on wearable computers

  • Dernbach S, Das B, Krishnan N, Thomas B, Cook D (2012) Simple and complex acitivity recognition through smart phones. In: Proceedings of the international conference on intelligent environments

  • Feng Z, Zhu Y, Zhang Q, Ni L, Vasilakos A (2014) Trac: truthful auction for location-aware collaborative sensing in mobile crowdsourcing. INFOCOM, 1231–1239

  • Fortino G, Fatta G, Pathan M, Vasilakos A (2014) Cloud-assisted body area networks: state-of-the-art and future challenges. Wirel Netw 20(7):1925–1938

    Article  Google Scholar 

  • Gong S, Xiang T (2003) Recognition of group activities using dynamic probabilistic networks. In: Proceedings of international conference on computer vision

  • Gyorbiro N, Fabian A, Homanyi G (2008) An activity recognition system for mobile phones. Mob Netw Appl 14(1):82–91

    Article  Google Scholar 

  • Hayajneh T, Almashaqbeh G, Ullah S, Vasilakos A (2014) A survey of wireless technologies coexistence in wban: analysis and open research issues. Wirel Netw 20(8):2165–2199

    Article  Google Scholar 

  • Hossain H, Roy N, Khan M (2015) Sleep well: a sound sleep monitoring framework for community scaling. In: Proceeding of the IEEE international conference on mobile data management

  • Huawei Smart Bracelet (2015). http://www.huawei.com/us/index.htm. Accessed Feb 2015

  • Huynh T, Blanke U, Schiele B (2007) Scalable recognition of daily activities from wearable sensors. In: LNCS LoCA, vol 4718

  • Intel Make it Wearable (2014). https://makeit.intel.com/. Accessed Apr 2014

  • Intille S, Larson K, Tapia E, Beaudin J, Kaushik P, Nawyn J, Rockinson R (2006) Using a live-in laboratory for ubiquitous computing research. In: Proceedings of 4th international conference on pervasive computing, vol 3968

  • Kasteren T, Noulas A, Englebienne G, Krose B (2008) Accurate activity recognition in a home setting. In: Proceedings of the 10th international conference on ubiquitous computing, vol 3968

  • Khan M, Hossain H, Roy N (2015a) Infrastructure-less occupancy detection and semantic localization in smart environments. In Proceedings of the 12th international conference on mobile and ubiquitous systems: computing, networking and services (in press)

  • Khan M, Hossain H, Roy N (2015b) Sensepresence: infrastructure-less occupancy detection for opportunistic sensing applications. In: IEEE international conference on mobile data management (in press)

  • Khan M, Lu S, Roy N, Pathak N (2015c) Demo abstract: a microphone sensor based system for green building applications. In: IEEE international conference on pervasive computing and communications (PerCom)

  • Kwapisz J, Weiss G, Moore S (2010) Activity recognition using cell phone accelerometers. In: International workshop on knowledge discovery from sensor data

  • Lee C, Hsu C, Lai Y, Vasilakos A (2013) An enhanced mobile-healthcare emergency system based on extended chaotic maps. J Med Syst 37(5):9973

    Article  Google Scholar 

  • Lester J, Choudhury T, Borriello G (2006) A practical approach to recognizing physical activities. In: PERVASIVE LNCS, vol 3968

  • Lin D, Labeau F, Vasilakos A (2015a) QoE-based optimal resource allocation in wireless healthcare networks: opportunities and challenges. Wirel Netw

  • Lin D, Wu X, Labeau F, Vasilakos A (2015b) Internet of vehicles for e-health applications in view of EMI on medical sensors. J Sens

  • Logan B, Healey J, Philipose M, Tapia E, Intille S (2007) A long-term evaluation of sensing modalities for activity recognition. In: UbiComp LNCS, vol 4717

  • Oliver N, Rosario B, Pentland A (2000) A bayesian computer vision system for modeling human interactions. IEEE Trans Pattern Anal Mach Intell 22(8):831–843

    Article  Google Scholar 

  • Pathak N, Khan M, Roy N (2015) Acoustic based appliance state identifications for fine grained energy analytics. In: IEEE international conference on pervasive computing and communications (PerCom)

  • Philipose M, Fishkin K, Perkowitz M, Patterson D, Hahnel D, Fox D, Kautz H (2004) Inferring activities from interactions with objects. IEEE Pervasive Comput 3(4):50–57

    Article  Google Scholar 

  • Plotz T, Flink G (2004) Accelerating the evaluation of profile hmms by pruning techniques. Report 2004-03. In: Tech rep., Faculty of Technology, University of Bielefeld

  • Rabiner L (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77(2):257–285

    Article  Google Scholar 

  • Rahimi MR, Ren J, Liu C, Vasilakos A, Venkatasubramanian N (2014) Mobile cloud computing: a survey, state of art and future directions. MONET 19(2):133–143

  • Rahimi MR, Venkatasubramanian N, Mehrotra S, Vasilakos A (2012) Mapcloud: mobile applications on an elastic and scalable 2-tier cloud architecture. In: IEEE/ACM UCC

  • Roy N, Das SK, Julien C (2012) Resource-optimized quality-assured ambiguous context mediation in pervasive environments. IEEE Trans Mob Comput 11(2):218–229

    Article  Google Scholar 

  • Roy N, Julien C (2014) Immersive physiotherapy: challenges for smart living environments and inclusive communities. In: Proceeding of the 12th international conference on smart homes and health telematics

  • Roy N, Kindle B (2014) Monitoring patient recovery using wireless physiotherapy devices. In: Proceeding of the 12th international conference on smart homes and health telematics

  • Roy N, Misra A, Cook D (2013) Infrastructure-assisted smartphone-based adl recognition in multi-inhabitant smart environments. In: Percom, pp 38–46

  • Roy N, Misra A, Das SK, Julien C (2009) Quality-of-inference (qoinf)-aware context determination in assisted living environments. In: ACM SIGMOBILE workshop on medical-grade wireless networks

  • Roy N, Misra A, Julien C, Das SK, Biswas J (2011) An energy efficient quality adaptive multi-modal sensor framework for context recognition. In: Percom, pp 63–73

  • Roy N, Pathak N, Misra A (2015) Aarpa: combining pervasive and power-line sensing for fine-grained appliance usage and energy monitoring. In: IEEE international conference on mobile data management (in press)

  • Roy N, Roy A, Das S (2006) Context-aware resource management in multi-inhabitant smart homes: a nash h-learning based approach. In: Proceedings of IEEE international conference on pervasive computing and communications (PerCom), pp 372–404

  • Sheng Z, Yang S, Yu Y, Vasilakos A, McCann J, Leung K (2014) A survey on the ietf protocol suite for the internet of things: standards, challenges, and opportunities. IEEE Wirel Commun 20(6):91–98

    Article  Google Scholar 

  • Wang L, Gu T, Tao X, Chen H, Lu J (2011) Recognizing multi-user activities using wearable sensors in a smart home. Pervasive Mob Comput 7(3):287–298

    Article  Google Scholar 

  • Wilson D, Atkeson C (2005) Simultaneous tracking and activity recognition (STAR) using many anonymous, binary sensors. Pervasive Comput, 3468:62–79

    Article  Google Scholar 

  • Witten L, Frank E (1999) Data mining: practicial machine learning tools and techniques with Java implementations. Morgan Kaufmann, San Francisco

  • Yan Z, Chakraborty D, Misra A, Jeung H, Aberer K (2012) Sammple: detecting semantic indoor activities in practical settings using locomotive signatures. In: International symposium on wearable computers

  • Yan Z, Zhang P, Vasilakos A (2014) A survey on trust management for internet of things. J Netw Comput Appl 42:37–40

    Article  Google Scholar 

  • Yi-Ting C, Kuo-Chung H, Ching-Hu L, Li-Chen F, John H (2010) Interaction models for multiple-resident activity recognition in a smart home. IROS, 3753–3758

  • Zhang Z, Wang H, Vasilakos A, Fang H (2012) ECG-cryptography and authentication in body area networks. IEEE Trans Inf Technol Biomed 16(6):1070–1078

    Article  Google Scholar 

  • Zheng Y, Li D, Vasilakos A (2013) Real-time data report and task execution in wireless sensor and actuator networks using self-aware mobile actuators. Comput Commun 36(9):988–997

    Article  Google Scholar 

  • Zhou L, Xiong N, Shu L, Vasilakos A, Yeo S (2010) Context-aware middleware for multimedia services in heterogeneous networks. IEEE Intell Syst 25(2):40–47

    Article  Google Scholar 

Download references

Acknowledgments

The work of Nirmalya Roy is partially supported by the National Science Foundation Award \(\#1344990\) and Constellation \(E^2\): Energy to Educate Grant. The work of Archan Misra is partially supported by the Singapore Ministry of Education Academic Research Fund Tier 2 under research Grant MOE2011-T2-1-001. The work of Diane Cook is partially supported by NSF Grants 1064628, 0852172, CNS-1255965, and NIH Grant R01EB009675.

Author information

Authors and Affiliations

  1. Department of Information Systems, University of Maryland, Baltimore County, Baltimore, USA

    Nirmalya Roy

  2. School of Information Systems, Singapore Management University, Singapore, Singapore

    Archan Misra

  3. School of Electrical Engineering and Computer Science, Washington State University, Pullman, USA

    Diane Cook

Authors
  1. Nirmalya Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Archan Misra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Diane Cook
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nirmalya Roy.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Roy, N., Misra, A. & Cook, D. Ambient and smartphone sensor assisted ADL recognition in multi-inhabitant smart environments. J Ambient Intell Human Comput 7, 1–19 (2016). https://doi.org/10.1007/s12652-015-0294-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-015-0294-7

Keywords

Search

Navigation