Skip to main content

Advertisement

SpringerLink
Log in

Affect Estimation with Wearable Sensors

  • Research Article
  • Published:
Journal of Healthcare Informatics Research Aims and scope Submit manuscript

Abstract

Affective states are associated with people’s mental health status and have profound impact on daily life, thus unobtrusively understanding and estimating affects have been brought to the public attention. The pervasiveness of wearable sensors makes it possible to build automatic systems for affect tracking. However, constructing such systems is a challenging task due to the complexity of human behaviors. In this work, we focus on the problem of estimating daily self-reported affects from sensor-generated data. We first analyze the intra- and inter-subject differences of self-reported affect labels. Second, we explore different machine learning models as well as label transformation techniques to overcome the individual differences in self-reported responses estimation. We conceptualize three experimental settings including long-term and short-term estimation scenarios. Our experimental results show that the mixed effects model and label transformation can yield better estimation of individual daily affect. This work poses the basis for future sensor-based individualized and real-time affective digital and/or clinical interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. OMsignal: https://omsignal.com/technology/

  2. Fitbit Charge 2: https://www.fitbit.com/charge2

References

  1. Aharony N, Pan W, Ip C, Khayal I, Pentland A (2011) Social fmri: investigating and shaping social mechanisms in the real world. Pervasive Mob Comput 7(6):643–659

    Article  Google Scholar 

  2. Aichele S, Rabbitt P, Ghisletta P (2016) Think fast, feel fine, live long: a 29-year study of cognition, health, and survival in middle-aged and older adults. Psychol Sci 27(4):518–529

    Article  Google Scholar 

  3. Appelhans BM, Luecken LJ (2006) Heart rate variability as an index of regulated emotional responding. Rev Gen Psychol 10(3):229

    Article  Google Scholar 

  4. Baker J, Cameron M (1996) The effects of the service environment on affect and consumer perception of waiting time: an integrative review and research propositions. J Acad Market Sci 24(4):338

    Article  Google Scholar 

  5. Beattie Z, Oyang Y, Statan A, Ghoreyshi A, Pantelopoulos A, Russell A, Heneghan C (2017) Estimation of sleep stages in a healthy adult population from optical plethysmography and accelerometer signals. Physiol Measur 38(11):1968

    Article  Google Scholar 

  6. Bhat N, Bhat K (1999) Anger control using biofeedback: A clinical model for heart patients. Biofeedback Newsmagazine 27(4):15–17

    Google Scholar 

  7. Bogomolov A, Lepri B, Ferron M, Pianesi F, Pentland A (2014) Daily stress recognition from mobile phone data, weather conditions and individual traits. In: Proceedings of the 22nd ACM international conference on Multimedia. ACM, pp 477–486

  8. Booth BM, Mundnich K, Feng T, Nadarajan A, Falk T, Villatte JL, Ferrara E, Narayanan S (2019) Multimodal human and environmental sensing for longitudinal behavioral studies in naturalistic settings: framework for sensor selection, deployment, and management. J Med Internet Res 21(8):e12,832

    Article  Google Scholar 

  9. Canzian L, Musolesi M (2015) Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. In: Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. ACM, pp 1293–1304

  10. Chen T, Guestrin C (2016) Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. ACM, pp 785–794

  11. Cheng H, Furnham A (2003) Personality, self-esteem, and demographic predictions of happiness and depression. Person Individual differ 34(6):921–942

    Article  Google Scholar 

  12. Dalal RS, Lam H, Weiss HM, Welch ER, Hulin CL (2009) A within-person approach to work behavior and performance: concurrent and lagged citizenship-counterproductivity associations, and dynamic relationships with affect and overall job performance. Acad Manag J 52(5):1051–1066

    Article  Google Scholar 

  13. Dunn OJ (1964) Multiple comparisons using rank sums. Technometrics 6 (3):241–252

    Article  Google Scholar 

  14. Eagle N, Pentland A (2006) Reality mining: sensing complex social systems. Person Ubiquit Comput 10(4):255–268

    Article  Google Scholar 

  15. Erdoğan S, İnan E (2018) Burnout level of nurses working in a hospital. Age 20(29):85

    Google Scholar 

  16. Fox KR (1999) The influence of physical activity on mental well-being. Publ Health Nutrition 2(3a):411–418

    Article  Google Scholar 

  17. Ganz FD (2012) Sleep and immune function. Crit Care Nurse 32(2):e19–e25

    Article  Google Scholar 

  18. Ghandeharioun A, Fedor S, Sangermano L, Ionescu D, Alpert J, Dale C, Sontag D, Picard R (2017) Objective assessment of depressive symptoms with machine learning and wearable sensors data. In: 2017 Seventh international conference on affective computing and intelligent interaction (ACII). IEEE, pp 325–332

  19. Hajjem A, Bellavance F, Larocque D (2014) Mixed-effects random forest for clustered data. J Stat Comput Simul 84(6):1313–1328

    Article  MathSciNet  MATH  Google Scholar 

  20. Jaques N, Taylor S, Azaria A, Ghandeharioun A, Sano A, Picard R (2015) Predicting students’ happiness from physiology, phone, mobility, and behavioral data. In: 2015 International conference on affective computing and intelligent interaction. IEEE, pp 222–228

  21. Jaques N, Taylor S, Sano A, Picard R, et al. (2017) Predicting tomorrow’s mood, health, and stress level using personalized multitask learning and domain adaptation. In: IJCAI 2017 Workshop on artificial intelligence in affective computing, pp 17–33

  22. L’Hommedieu M, L’Hommedieu J, Begay C, Schenone A, Dimitropoulou L, Margolin G, Falk T, Ferrara E, Lerman K, Narayanan S (2019) Lessons learned: recommendations for implementing a longitudinal study using wearable and environmental sensors in a healthcare organization. Journal of Medical Internet Research

  23. LiKamWa R, Liu Y, Lane ND, Zhong L (2013) Moodscope: building a mood sensor from smartphone usage patterns. In: Proceeding of the 11th annual international conference on mobile systems, applications, and services. ACM, pp 389–402

  24. Mackinnon A, Jorm AF, Christensen H, Korten AE, Jacomb PA, Rodgers B (1999) A short form of the positive and negative affect schedule: evaluation of factorial validity and invariance across demographic variables in a community sample. Person Individual Differ 27(3):405–416

    Article  Google Scholar 

  25. Mazur A, Goś A, Humeniuk E (2016) Existential attitudes and occupational burnout syndrome in nurses. Pielegniarstwo XXI wieku/Nurs 21st Cent 15(3):5–10

    Article  Google Scholar 

  26. Philippot P, Chapelle G, Blairy S (2002) Respiratory feedback in the generation of emotion. Cogn Emot 16(5):605–627

    Article  Google Scholar 

  27. Pollak JP, Adams P, Gay G (2011) Pam: a photographic affect meter for frequent, in situ measurement of affect. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, pp 725–734

  28. Rasch B, Born J (2013) About sleep’s role in memory. Physiol Rev 93 (2):681–766

    Article  Google Scholar 

  29. Rozet A, Kronish IM, Schwartz JE, Davidson KW (2019) Using machine learning to derive just-in-time and personalized predictors of stress: observational study bridging the gap between nomothetic and ideographic approaches. J Med Internet Res 21(4):e12,910

    Article  Google Scholar 

  30. Sanchez W, Martinez A, Gonzalez M (2017) Towards job stress recognition based on behavior and physiological features. In: International conference on ubiquitous computing and ambient intelligence. Springer, pp 311–322

  31. Sano A, Amy ZY, McHill AW, Phillips AJ, Taylor S, Jaques N, Klerman EB, Picard RW (2015) Prediction of happy-sad mood from daily behaviors and previous sleep history. In: Engineering in medicine and biology society (EMBC), 2015 37th annual international conference of the IEEE. IEEE, pp 6796–6799

  32. Sano A, Phillips AJ, Amy ZY, McHill AW, Taylor S, Jaques N, Czeisler CA, Klerman EB, Picard RW (2015) Recognizing academic performance, sleep quality, stress level, and mental health using personality traits, wearable sensors and mobile phones. In: 2015 IEEE 12Th international conference on wearable and implantable body sensor networks, pp 1–6

  33. Stephens T (1988) Physical activity and mental health in the United States and canada: evidence from four population surveys. Prevent Med 17(1):35–47

    Article  Google Scholar 

  34. Wang R, Chen F, Chen Z, Li T, Harari G, Tignor S, Zhou X, Ben-Zeev D, Campbell AT (2014) Studentlife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In: Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing. ACM, pp 3–14

  35. Wang R, Wang W, daSilva A, Huckins JF, Kelley WM, Heatherton TF, Campbell AT (2018) Tracking depression dynamics in college students using mobile phone and wearable sensing. Proceedings of the ACM on Interactive, Mobile. Wearab Ubiquit Technol 2(1):43

    Google Scholar 

  36. Westefeld JS, Furr SR (1987) Suicide and depression among college students. Prof Psychol Res Pract 18(2):119

    Article  Google Scholar 

  37. Yan S, Hosseinmardi H, Kao HT, Narayanan S, Lerman K, Ferrara E (2019) Estimating individualized daily self-reported affect with wearable sensors. In: IEEE International conference on healthcare informatics 2019 (ICHI 2019)

Download references

Acknowledgments

The authors are grateful to the TILES team for the efforts in study design, data collection, and sharing, that enable this work.

Funding

This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No 2017-17042800005.

Author information

Authors and Affiliations

  1. University of Southern California, Information Sciences Institute, Los Angeles, CA, 90089, USA

    Shen Yan, Homa Hosseinmardi, Hsien-Te Kao, Shrikanth Narayanan, Kristina Lerman & Emilio Ferrara

Authors
  1. Shen Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Homa Hosseinmardi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hsien-Te Kao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shrikanth Narayanan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kristina Lerman
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Emilio Ferrara
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shen Yan.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 10 Extracted features
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, S., Hosseinmardi, H., Kao, HT. et al. Affect Estimation with Wearable Sensors. J Healthc Inform Res 4, 261–294 (2020). https://doi.org/10.1007/s41666-019-00066-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41666-019-00066-z

Keywords

Search

Navigation