Modeling, learning, and simulating human activities of daily living with behavior trees

Abstract

Autonomy is a key factor in the quality of life of a person. With the aging of the population, an increasing number of people suffers from a reduced level of autonomy. That compromises their capacity of performing their daily activities and causes safety issues. The new concept of ambient assisted living (AAL), and more specifically its application in smart homes for supporting elderly people, constitutes a great avenue of the solution. However, to be able to automatically assist a user carrying out is activities, researchers and engineers face three main challenges in the development of smart homes: (i) how to represent the activity models, (ii) how to automatically construct theses models based on historical data and (iii) how to be able to simulate the user behavior for tests and calibration purpose. Most of recent works addressing these challenges exploit simple models of activity with no semantic, or use logically complex ones or else use probabilistically rigid representations. In this paper, we propose a global approach to address the three challenges. We introduce a new way of modeling human activities in smart homes based on behavior trees which are used in the video game industry. We then present an algorithmic way to automatically learn these models with sensors logs. We use a simulator that we have developed to validate our approach.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Notes

  1. 1.

    https://github.com/Iannyck/shima.

  2. 2.

    https://github.com/Iannyck/shima.

References

  1. 1.

    Al-Shaqi R, Mourshed M, Rezgui Y (2016) Progress in ambient assisted systems for independent living by the elderly. SpringerPlus 5(1):624

    Article  Google Scholar 

  2. 2.

    Alshammari N, Alshammari T, Sedky M, Champion J, Bauer C (2017) OpenSHS: open smart home simulator. Sensors 17(5):1003

    Article  Google Scholar 

  3. 3.

    Ariani A, Redmond SJ, Chang D, Lovell NH (2013) Simulation of a smart home environment. In: 3rd International conference on instrumentation, communications, information technology, and biomedical engineering (ICICI-BME). IEEE, pp 27–32

  4. 4.

    Artikis A, Sergot M, Paliouras G (2010) A logic programming approach to activity recognition. In: Proceedings of the 2nd ACM international workshop on events in multimedia. ACM, pp 3–8

  5. 5.

    Baader F, Calvanese D, McGuinness DL, Nardi D, Patel-Schneider PF (eds) (2003) The description logic handbook: theory, implementation, and applications. Cambridge University Press, New York

    MATH  Google Scholar 

  6. 6.

    Barwise J, Perry J (1981) Situations and attitudes. J Philos 78(11):668–691

    MATH  Article  Google Scholar 

  7. 7.

    Belley C, Gaboury S, Bouchard B, Bouzouane A (2015) Nonintrusive system for assistance and guidance in smart homes based on electrical devices identification. Expert Syst Appl 42(19):6552–6577

    Article  Google Scholar 

  8. 8.

    Bergeron F, Giroux S, Bouchard K, Gaboury, S (2017) RFID based activities of daily living recognition. In: IEEE SmartWorld, ubiquitous intelligence computing, advanced trusted computed, scalable computing communications, cloud big data computing, internet of people and smart city innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), pp 1–5. https://doi.org/10.1109/UIC-ATC.2017.8397548

  9. 9.

    Blackman S, Matlo C, Bobrovitskiy C, Waldoch A, Fang ML, Jackson P, Mihailidis A, Nygård L, Astell A, Sixsmith A (2016) Ambient assisted living technologies for aging well: a scoping review. J Intell Syst 25(1):55–69

    Article  Google Scholar 

  10. 10.

    Botía JA, Campillo P, Campuzano F, Serrano E. UbikSim website. https://github.com/emilioserra/UbikSim/wiki. Accessed 28 May 2020

  11. 11.

    Bouchard B (2017) Smart technologies in healthcare. CRC Press, Boca Raton

    Book  Google Scholar 

  12. 12.

    Bouchard B, Gaboury S, Bouchard K, Francillette Y (2018) Modeling human activities using behaviour trees in smart homes. In: Proceedings of the 11th PErvasive technologies related to assistive environments conference. ACM, pp 67–74

  13. 13.

    Bouchard B, Giroux S, Bouzouane A (2007) A keyhole plan recognition model for alzheimer’s patients: first results. Appl Artif Intell 21(7):623–658

    Article  Google Scholar 

  14. 14.

    Bouchard K, Ajroud A, Bouchard B, Bouzouane A (2012) Simact: a 3d open source smart home simulator for activity recognition with open database and visual editor. Int J Hybrid Inf Technol 5(3):13–32

    Google Scholar 

  15. 15.

    Bouchard K, Bouchard B, Bouzouanea A (2017) Practical guidelines to build smart homes: lessons learned. In: Hasan SF (ed) Opportunistic networking: vehicular, D2D and cognitive radio networks. CRC Press, Boca Raton, pp 206–234

    Google Scholar 

  16. 16.

    Bouchard K, Fortin-Simard D, Gaboury S, Bouchard B, Bouzouane A (2013) Accurate rfid trilateration to learn and recognize spatial activities in smart environment. Int J Distrib Sens Netw 9(6):936816

    Article  Google Scholar 

  17. 17.

    Camilleri G (1999) A generic formal plan recognition theory. In: International conference on information intelligence and systems. Proceedings. IEEE, pp 540–547

  18. 18.

    Carberry S (2001) Techniques for plan recognition. User Model User Adap Interact 11(1–2):31–48

    MATH  Article  Google Scholar 

  19. 19.

    Charniak E, Goldman RP (1993) A bayesian model of plan recognition. Artif Intell 64(1):53–79

    Article  Google Scholar 

  20. 20.

    Chen L, Hoey J, Nugent CD, Cook DJ, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man Cybern Part C (Appl Rev) 42(6):790–808

    Article  Google Scholar 

  21. 21.

    Chen W, Augusto JC, Seoane F (2015) Recent advances in ambient assisted living-bridging assistive technologies, e-health and personalized health care, vol 20. IOS Press, Amsterdam

    Google Scholar 

  22. 22.

    Cilla R, Patricio MA, García J, Berlanga A, Molina JM (2009) Recognizing human activities from sensors using hidden markov models constructed by feature selection techniques. Algorithms 2(1):282–300

    MATH  Article  Google Scholar 

  23. 23.

    Fahad LG, Ali A, Rajarajan M (2015) Learning models for activity recognition in smart homes. In: Kim KJ (ed) Information science and applications. Springer, Berlin, pp 819–826

    Chapter  Google Scholar 

  24. 24.

    Fleury A, Vacher M, Noury N (2010) Svm-based multimodal classification of activities of daily living in health smart homes: sensors, algorithms, and first experimental results. IEEE Trans Inf Technol Biomed 14(2):274–283

    Article  Google Scholar 

  25. 25.

    Fortin-Simard D, Bilodeau JS, Bouchard K, Gaboury S, Bouchard B, Bouzouane A (2015) Exploiting passive RFID technology for activity recognition in smart homes. IEEE Intell Syst 30(4):7–15

    Article  Google Scholar 

  26. 26.

    Fortin-Simard D, Bilodeau JS, Gaboury S, Bouchard B, Bouzouane A (2015) Method of recognition and assistance combining passive RFID and electrical load analysis that handles cognitive errors. Int J Distrib Sens Netw 11(10):643273

    Google Scholar 

  27. 27.

    Francillette Y, Boucher E, Bouzouane A, Gaboury S (2017) The virtual environment for rapid prototyping of the intelligent environment. Sensors 17(11):2562

    Article  Google Scholar 

  28. 28.

    Gadelhag Mohmed Ahmad Lotfi CLAP (2018) Unsupervised learning fuzzy finite state machine for human activities recognition. In: Conference: the 11th PErvasive technologies related to assistive environments conference, pp 1–8

  29. 29.

    Geib CW, Goldman RP (2005) Partial observability and probabilistic plan/goal recognition. In: Proceedings of the international workshop on modeling other agents from observations (MOO-05), vol 8

  30. 30.

    Giegerich R, Kurtz S (1997) From ukkonen to mccreight and weiner: a unifying view of linear-time suffix tree construction. Algorithmica 19(3):331–353

    MathSciNet  MATH  Article  Google Scholar 

  31. 31.

    Giroux S, Pigot H (2013) Smart homes for people suffering from cognitive disorders. In: Computer science and ambient intelligence, pp 225–262

  32. 32.

    Goodfellow I, Bengio Y, Courville A (2016) Deep learning. Adaptive computation and machine learning series. MIT Press, Cambridge, p 800

    MATH  Google Scholar 

  33. 33.

    Grześ M, Hoey J, Khan SS, Mihailidis A, Czarnuch S, Jackson D, Monk A (2014) Relational approach to knowledge engineering for POMDP-based assistance systems as a translation of a psychological model. Int J Approx Reason 55(1):36–58

    Article  Google Scholar 

  34. 34.

    Helal A, Cho K, Lee W, Sung Y, Lee J, Kim E (2012) 3d modeling and simulation of human activities in smart spaces. In: 9th International conference on ubiquitous intelligence and computing and 9th international conference on autonomic and trusted computing (UIC/ATC). IEEE, pp 112–119

  35. 35.

    Ho B, Vogts D, Wesson J (2019) A smart home simulation tool to support the recognition of activities of daily living. Proc S Afr Inst Comput Sci Inf Technol 2019:1–10

    Google Scholar 

  36. 36.

    Humphreys G, Forde E (1998) Disordered action schema and action disorganisation syndrome. Cogn Neuropsychol 15(6):771–812

    Google Scholar 

  37. 37.

    Kautz HA (1991) A formal theory of plan recognition and its implementation. In: Reasoning about plans. Morgan Kaufmann Publishers Inc., San Francisco, pp 69–124. http://dl.acm.org/citation.cfm?id=117019.117021. Accessed 28 May 2020

  38. 38.

    Kingston A, Collerton J, Davies K, Bond J, Robinson L, Jagger C (2012) Losing the ability in activities of daily living in the oldest old: a hierarchic disability scale from the newcastle 85+ study. PLoS ONE 7(2):e31665

    Article  Google Scholar 

  39. 39.

    Krishnan NC, Cook DJ (2014) Activity recognition on streaming sensor data. Pervasive Mob Comput 10:138–154

    Article  Google Scholar 

  40. 40.

    Luke S, Cioffi-Revilla C, Panait L, Sullivan K, Balan G (2005) MASON: a multiagent simulation environment. Simulation 81(7):517–527

    Article  Google Scholar 

  41. 41.

    Lundström J, Synnott J, Järpe E, Nugent CD (2015) Smart home simulation using avatar control and probabilistic sampling. In: IEEE international conference on pervasive computing and communication workshops (PerCom workshops). IEEE, pp 336–341

  42. 42.

    Marcotte R, Hamilton HJ (2017) Behavior trees for modelling artificial intelligence in games: a tutorial. Comput Games J 6(3):171–184. https://doi.org/10.1007/s40869-017-0040-9

    Article  Google Scholar 

  43. 43.

    Mojidra HS, Borisagar VH (2012) A literature survey on human activity recognition via hidden Markov model. In: IJCA proceedings on international conference on recent trends in information technology and computer science, pp 1–5

  44. 44.

    Nazerfard E, Cook DJ (2012) Bayesian networks structure learning for activity prediction in smart homes. In: 8th International conference on intelligent environments (IE). IEEE, pp 50–56

  45. 45.

    Nazerfard E, Das B, Holder LB, Cook DJ (2010) Conditional random fields for activity recognition in smart environments. In: Proceedings of the 1st ACM international health informatics symposium. ACM, pp 282–286

  46. 46.

    Ni Q, Pau de la Cruz I, García Hernando AB (2016) A foundational ontology-based model for human activity representation in smart homes. J Ambient Intell Smart Environ 8(1):47–61

    Article  Google Scholar 

  47. 47.

    Park B, Min H, Bang G, Ko I (2015) The user activity reasoning model in a virtual living space simulator. Int J Softw Eng Appl 9(6):53–62

    Google Scholar 

  48. 48.

    Paternò F (2004) ConcurTaskTrees: an engineered notation for task models. The handbook of task analysis for human–computer interaction, pp 483–503

  49. 49.

    Paternò F, Mancini C, Meniconi S (1997) ConcurTaskTrees: a diagrammatic notation for specifying task models. In: Human–computer interaction INTERACT’97. Springer, pp 362–369

  50. 50.

    Patterson DJ, Fox D, Kautz H, Philipose M (2005) Fine-grained activity recognition by aggregating abstract object usage. In: Ninth IEEE international symposium on wearable computers. Proceedings. IEEE, pp 44–51

  51. 51.

    Puybaret E (2016) Sweet home 3d. https://sourceforge.net/projects/sweethome3d/. Accessed 28 May 2020

  52. 52.

    Ramos C, Augusto JC, Shapiro D (2008) Ambient intelligence-the next step for artificial intelligence. IEEE Intell Syst 23(2):15–18

    Article  Google Scholar 

  53. 53.

    Rook A, Knauss A, Damian D, Thomo A (2014) A case study of applying data mining to sensor data for contextual requirements analysis. In: IEEE 1st international workshop on artificial intelligence for requirements engineering (AIRE). IEEE, pp 43–50

  54. 54.

    Roy PC, Abidi SR, Abidi SS (2017) Possibilistic activity recognition with uncertain observations to support medication adherence in an assisted ambient living setting. Knowl Based Syst 133:156–173

    Article  Google Scholar 

  55. 55.

    Roy PC, Bouchard B, Bouzouane A, Giroux S (2013) Ambient activity recognition in smart environments for cognitive assistance. Int J Robot Appl Technol (IJRAT) 1(1):29–56

    Google Scholar 

  56. 56.

    Serrano E, Botia J (2013) Validating ambient intelligence based ubiquitous computing systems by means of artificial societies. Inf Sci 222:3–24. https://doi.org/10.1016/j.ins.2010.11.012

    Article  Google Scholar 

  57. 57.

    Synnott J, Chen L, Nugent C, Moore G (2014) The creation of simulated activity datasets using a graphical intelligent environment simulation tool. In: 36th Annual international conference of the IEEE on engineering in medicine and biology society (EMBC). IEEE, pp 4143–4146

  58. 58.

    Technologies U (2016) Unity-game engine. https://unity3d.com. Accessed 28 May 2020

  59. 59.

    United Nations D.o.E., Social Affairs, PD (2019) World population ageing 2019, p 46

  60. 60.

    Van Viamen AE (2018) Person-environment fit: a review of its basic tenets. Ann Rev Organ Psychol Organ Behav 5:75–101

    Article  Google Scholar 

  61. 61.

    Wang J, Chen Y, Hao S, Peng X, Hu L (2018) Deep learning for sensor-based activity recognition: a survey. Pattern Recognit Lett 119:3–11

    Article  Google Scholar 

  62. 62.

    Wang WY, Cohen WW (2016) Learning first-order logic embeddings via matrix factorization. In: IJCAI, pp 2132–2138

  63. 63.

    Witten IH, Frank E, Hall MA, Pal CJ (2016) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, Burlington

    Google Scholar 

  64. 64.

    Wobcke W (2002) Two logical theories of plan recognition. J Logic Comput 12(3):371–412

    MathSciNet  MATH  Article  Google Scholar 

  65. 65.

    Ziaeefard M, Bergevin R (2015) Semantic human activity recognition: a literature review. Pattern Recognit 48(8):2329–2345

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Yannick Francillette.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Francillette, Y., Bouchard, B., Bouchard, K. et al. Modeling, learning, and simulating human activities of daily living with behavior trees. Knowl Inf Syst 62, 3881–3910 (2020). https://doi.org/10.1007/s10115-020-01476-x

Download citation

Keywords

  • Behavior tree
  • Machine learning
  • Visualization
  • Human activity modeling