Skip to main content
SpringerLink
Log in

Generative replay for multi-class modeling of human activities via sensor data from in-home robotic companion pets

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Deploying socially assistive robots (SARs) at home, such as robotic companion pets, can be useful for tracking behavioral and health-related changes in humans during lifestyle fluctuations over time, like those experienced during CoVID-19. However, a fundamental problem required when deploying autonomous agents such as SARs in people’s everyday living spaces is understanding how users interact with those robots when not observed by researchers. One way to address that is to utilize novel modeling methods based on the robot’s sensor data, combined with newer types of interaction evaluation such as ecological momentary assessment (EMA), to recognize behavior modalities. This paper presents such a study of human-specific behavior classification based on data collected through EMA and sensors attached onboard a SAR, which was deployed in user homes. Classification was conducted using generative replay models, which attempt to use encoding/decoding methods to emulate how human dreaming is thought to create perturbations of the same experience in order to learn more efficiently from less data. Both multi-class and binary classification were explored for comparison, using several types of generative replay (variational autoencoders, generative adversarial networks, semi-supervised GANs). The highest-performing binary model showed approximately 79% accuracy (AUC 0.83), though multi-class classification across all modalities only attained 33% accuracy (AUC 0.62, F1 0.25), despite various attempts to improve it. The paper here highlights the strengths and weaknesses of using generative replay for modeling during human–robot interaction in the real world and also suggests a number of research paths for future improvement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

De-identified data may be made available from the corresponding author upon reasonable request.

References

  1. Bennett CC, Sabanovic S, Piatt JA, Nagata S, Eldridge L, Randall N (2017) A robot a day keeps the blues away. In: 2017 IEEE international conference on healthcare informatics (ICHI). IEEE, pp 536–540

  2. Bennett CC (2021) Evoking an intentional stance during human-agent social interaction: appearances can be deceiving. In: 2021 30th IEEE international conference on robot & human interactive communication (RO-MAN). IEEE, pp 362–368

  3. Pu L, Moyle W, Jones C, Todorovic M (2019) The effectiveness of social robots for older adults: a systematic review and meta-analysis of randomized controlled studies. Gerontologist 59(1):37–51

    Article  Google Scholar 

  4. Randall N, Bennett CC, Šabanović S, Nagata S, Eldridge L, Collins S, Piatt JA (2019) More than just friends: in-home use and design recommendations for sensing socially assistive robots (SARs) by older adults with depression. Paladyn J Behav Robot 10(1):237–255

    Article  Google Scholar 

  5. Robinson H, MacDonald B, Broadbent E (2014) The role of healthcare robots for older people at home: a review. Int J Soc Robot 6(4):575–591

    Article  Google Scholar 

  6. Lindsay S, Jackson D, Schofield G, Olivier P (2012) Engaging older people using participatory design. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 1199–1208

  7. Shiffman S, Stone AA, Hufford MR (2008) Ecological momentary assessment. Annu Rev Clin Psychol 4:1–32

    Article  Google Scholar 

  8. Vesel C, Rashidisabet H, Zulueta J, Stange JP, Duffecy J, Hussain F, Piscitello A, Bark J, Langenecker SA, Young S et al (2020) Effects of mood and aging on keystroke dynamics metadata and their diurnal patterns in a large open-science sample: A biaffect ios study. J Am Med Inform Assoc 27(7):1007–1018

    Article  Google Scholar 

  9. Zulueta J, Piscitello A, Rasic M, Easter R, Babu P, Langenecker SA, McInnis M, Ajilore O, Nelson PC, Ryan K et al (2018) Predicting mood disturbance severity with mobile phone keystroke metadata: a biaffect digital phenotyping study. J Med Internet Res 20(7):9775

    Article  Google Scholar 

  10. Bennett CC, Stanojević, Č, Šabanović SA, Piatt J, Kim S (2021) When no one is watching: ecological momentary assessment to understand situated social robot use in healthcare. In: Proceedings of the 9th international conference on human–agent interaction, pp 245–251

  11. Stanojević Č, Bennett CC, Šabanović S, Piatt JA, Kim S, Lee J (2022) Utilization of ema and the transtheoretical model of behavior change to prevent relapse during long-term socially assistive robot based interventions. In: Workshop on longitudinal social impacts of HRI over long-term deployments at the 2022 ACM/IEEE international conference on human robot interaction (LSI-HRI)

  12. Bennett CC, Šabanović S, Kim SA, Piatt J, Lee J, Yu J, Oh J (in press) Comparison of in-home robotic companion pet use in South Korea and the united states: a case study. In: 9th IEEE international conference on biomedical robotics & biomechatronics (BIOROB)

  13. Xia K, Huang J, Wang H (2020) LSTM-CNN architecture for human activity recognition. IEEE Access 8:56855–56866

    Article  Google Scholar 

  14. Zhou X, Liang W, Kevin I, Wang K, Wang H, Yang LT, Jin Q (2020) Deep-learning-enhanced human activity recognition for internet of healthcare things. IEEE Internet Things J 7(7):6429–6438

    Article  Google Scholar 

  15. Stanojevic C, Bennett CC, Sabanovic S, Collins S, Henkel KB, Henkel Z, Piatt JA (2023) Conceptualizing socially-assistive robots as a digital therapeutic tool in healthcare. Front Digit Health 5:15

    Article  Google Scholar 

  16. van de Ven GM, Siegelmann HT, Tolias AS (2020) Brain-inspired replay for continual learning with artificial neural networks. Nat Commun 11(1):1–14

    Google Scholar 

  17. Eldridge L, Nagata S, Piatt J, Stanojevic C, Šabanović S, Bennett CC, Randall N et al (2020) Utilization of socially assistive robots in recreational therapy. Am J Recreat Ther 19(2):35–45

    Google Scholar 

  18. Vesel C, Rashidisabet H, Zulueta J, Stange JP, Duffecy J, Hussain F, Piscitello A, Bark J, Langenecker SA, Young S et al (2020) Effects of mood and aging on keystroke dynamics metadata and their diurnal patterns in a large open-science sample: A biaffect ios study. J Am Med Inform Assoc 27(7):1007–1018

    Article  Google Scholar 

  19. Zulueta J, Piscitello A, Rasic M, Easter R, Babu P, Langenecker SA, McInnis M, Ajilore O, Nelson PC, Ryan K et al (2018) Predicting mood disturbance severity with mobile phone keystroke metadata: a biaffect digital phenotyping study. J Med Internet Res 20(7):9775

    Article  Google Scholar 

  20. Huckins JF, Wang W, Hedlund E, Rogers C, Nepal SK, Wu J, Obuchi M, Murphy EI, Meyer ML, Wagner DD et al (2020) Mental health and behavior of college students during the early phases of the covid-19 pandemic: longitudinal smartphone and ecological momentary assessment study. J Med Internet Res 22(6):20185

    Article  Google Scholar 

  21. Bennett CC, Ross MK, Baek E, Kim D, Leow AD (2022) Predicting clinically relevant changes in bipolar disorder outside the clinic walls based on pervasive technology interactions via smartphone typing dynamics. Pervasive Mob Comput 83:101598

    Article  Google Scholar 

  22. Chen L, Hoey J, Nugent CD, Cook DJ, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man Cybern Part C (Appl Rev) 42(6):790–808

    Article  Google Scholar 

  23. Liu L, Dugas D, Cesari G, Siegwart R, Dubé R (2020) Robot navigation in crowded environments using deep reinforcement learning. In: 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 5671–5677

  24. Papagiannaki A, Zacharaki EI, Kalouris G, Kalogiannis S, Deltouzos K, Ellul J, Megalooikonomou V (2019) Recognizing physical activity of older people from wearable sensors and inconsistent data. Sensors 19(4):880

    Article  Google Scholar 

  25. Jiang J, Pozza R, Gunnarsdóttir K, Gilbert N, Moessner K (2017) Using sensors to study home activities. J Sens Actuator Netw 6(4):32

    Article  Google Scholar 

  26. Caine K, Sabanovic S, Carter M (2012) The effect of monitoring by cameras and robots on the privacy enhancing behaviors of older adults. In: Proceedings of the seventh annual ACM/IEEE international conference on human–robot interaction, pp 343–350

  27. Schulz TW, Herstad J (2018) Walking away from the robot: negotiating privacy with a robot. In: Electronic workshops in computing (eWiC), 2018. British Computer Society (BCS)

  28. Bertz JW, Epstein DH, Preston KL (2018) Combining ecological momentary assessment with objective, ambulatory measures of behavior and physiology in substance-use research. Addict Behav 83:5–17

    Article  Google Scholar 

  29. Huckins JF, Wang W, Hedlund E, Rogers C, Nepal SK, Wu J, Obuchi M, Murphy EI, Meyer ML, Wagner DD et al (2020) Mental health and behavior of college students during the early phases of the covid-19 pandemic: longitudinal smartphone and ecological momentary assessment study. J Med Internet Res 22(6):20185

    Article  Google Scholar 

  30. Lee K, Kwan M-P (2018) Physical activity classification in free-living conditions using smartphone accelerometer data and exploration of predicted results. Comput Environ Urban Syst 67:124–131

    Article  Google Scholar 

  31. Banos O, Galvez J-M, Damas M, Pomares H, Rojas I (2014) Window size impact in human activity recognition. Sensors 14(4):6474–6499

    Article  Google Scholar 

  32. Niazi AH, Yazdansepas D, Gay JL, Maier FW, Ramaswamy L, Rasheed K, Buman MP (2017) Statistical analysis of window sizes and sampling rates in human activity recognition. In: HEALTHINF, pp 319–325

  33. Sheng Z, Hailong C, Chuan J, Shaojun Z (2015) An adaptive time window method for human activity recognition. In: 2015 IEEE 28th Canadian conference on electrical and computer engineering (CCECE). IEEE, pp 1188–1192

  34. Ma C, Li W, Cao J, Du J, Li Q, Gravina R (2020) Adaptive sliding window based activity recognition for assisted livings. Inf. Fus. 53:55–65

    Article  Google Scholar 

  35. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) Smote: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357

    Article  Google Scholar 

  36. Lee K, Kwan M-P (2018) Physical activity classification in free-living conditions using smartphone accelerometer data and exploration of predicted results. Comput Environ Urban Syst 67:124–131

  37. Shin H, Lee JK, Kim J, Kim J (2017) Continual learning with deep generative replay. Adv Neural Inf Process Syst 30

  38. Kingma DP, Welling M (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

  39. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Syst 27

  40. Odena A (2016) Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv:1606.01583

  41. Tanha J, Abdi Y, Samadi N, Razzaghi N, Asadpour M (2020) Boosting methods for multi-class imbalanced data classification: an experimental review. J Big Data 7(1):1–47

    Article  Google Scholar 

  42. Jung M, Hinds P (2018) Robots in the wild: a time for more robust theories of human–robot interaction. ACM, New York

    Book  Google Scholar 

  43. Brinck I, Balkenius C (2020) Mutual recognition in human–robot interaction: a deflationary account. Philos Technol 33(1):53–70

    Article  Google Scholar 

  44. Zhou X, Liang W, Kevin I, Wang K, Wang H, Yang LT, Jin Q (2020) Deep-learning-enhanced human activity recognition for internet of healthcare things. IEEE Internet Things J 7(7):6429–6438

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the research fund of Hanyang University (HY-2020) in Korea, as well as the National Science Foundation in the United States (Grant# IIS-1900683). We would also like to thank the numerous graduate and undergraduate research assistants who contributed to this work (Janghoon Yu, Jiyeong Oh, Zach Kauffman, Jonathan Gerth, Sawyer Collins).

Funding

Funding was provided by Hanyang University (HY-2020) and National Science Foundation (IIS-1900683).

Author information

Authors and Affiliations

  1. Department of Intelligence Computing, Hanyang University, Seoul, Korea

    Seongcheol Kim, Casey C. Bennett & Jinjae Lee

  2. College of Computing & Digital Media, DePaul University, Chicago, IL, USA

    Casey C. Bennett

  3. Department of Computer Science and Engineering, Mississipi State University, Starkville, MS, USA

    Zachary Henkel, Kenna Baugus & Cindy L. Bethel

  4. Department of Parks, Recreation, and Tourism Management, Clemson University, Clemson, SC, USA

    Cedomir Stanojevic

  5. School of Public Health, Indiana University, Bloomington, IN, USA

    Jennifer A. Piatt

  6. School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN, USA

    Selma Šabanović

Authors
  1. Seongcheol Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Casey C. Bennett
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zachary Henkel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinjae Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cedomir Stanojevic
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kenna Baugus
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Cindy L. Bethel
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jennifer A. Piatt
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Selma Šabanović
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Seongcheol Kim or Casey C. Bennett.

Ethics declarations

Conflict of interest

The authors have no conflicts or competing interests to declare related to this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, S., Bennett, C.C., Henkel, Z. et al. Generative replay for multi-class modeling of human activities via sensor data from in-home robotic companion pets. Intel Serv Robotics 17, 277–287 (2024). https://doi.org/10.1007/s11370-023-00496-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-023-00496-0

Keywords

Search

Navigation