Skip to main content
SpringerLink
Log in

Passive Sensing of Affective and Cognitive Functioning in Mood Disorders by Analyzing Keystroke Kinematics and Speech Dynamics

  • Chapter
  • First Online:
Digital Phenotyping and Mobile Sensing

Abstract

Mood disorders can be difficult to diagnose, evaluate, and treat. They involve affective and cognitive components, both of which need to be closely monitored over the course of the illness. Current methods like interviews and rating scales can be cumbersome, sometimes ineffective, and oftentimes infrequently administered. Even ecological momentary assessments, when used alone, are susceptible to many of the same limitations and still require active participation from the subject. Passive, continuous, frictionless, and ubiquitous means of recording and analyzing mood and cognition obviate the need for more frequent and lengthier doctor’s visits, can help identify misdiagnoses, and would potentially serve as an early warning system to better manage medication adherence and prevent hospitalizations. Activity trackers and smartwatches have long provided exactly such a tool for evaluating physical fitness. What if smartphones, voice assistants, and eventually Internet of Things devices and ambient computing systems could similarly serve as fitness trackers for the brain, without imposing any additional burden on the user? In this chapter, we explore two such early approaches—an in-depth analytical technique based on examining meta-features of virtual keyboard usage and corresponding typing kinematics, and another method which analyzes the acoustic features of recorded speech—to passively and unobtrusively understand mood and cognition in people with bipolar disorder. We review innovative studies that have used these methods to build mathematical models and machine learning frameworks that can provide deep insights into users’ mood and cognitive states. We then outline future research considerations and close by discussing the opportunities and challenges afforded by these modes of researching mood disorders and passive sensing approaches in general.

Life is too sweet and too short to express our affection with just our thumbs. Touch is meant for more than a keyboard.

—Kristin Armstrong, Olympic cyclist

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Ajilore O (2018) A voice-enabled diabetes self-management program that addresses mood—the DiaBetty experience. In: American Diabetes Association’s 78th Scientific Sessions, Orlando, FL, USA

    Google Scholar 

  • Ajilore O, Vizueta N, Walshaw P et al (2015) Connectome signatures of neurocognitive abnormalities in euthymic bipolar I disorder. J Psychiatr Res 68:37–44

    Article  Google Scholar 

  • American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Publishing, Arlington, VA, USA

    Book  Google Scholar 

  • Anderson K, Burford O, Emmerton L (2016) Mobile health apps to facilitate self-care: a qualitative study of user experiences. PLoS ONE 11(5):e0156164

    Article  Google Scholar 

  • Andreassen O, Houenou J, Duchesnay E et al (2018) 121. Biological insight from large-scale studies of bipolar disorder with multi-modal imaging and genomics. Biol Psychiat 83(9):S49–S50

    Google Scholar 

  • Asselbergs J, Ruwaard J, Ejdys M et al (2016) Mobile phone-based unobtrusive ecological momentary assessment of day-to-day mood: an explorative study. J Med Internet Res 18(3):e72

    Article  Google Scholar 

  • Avunjian N (2018) ‘Westworld’ cognition cowboy hats are a step up from a real science tool (inverse). USC Leonard Davis School of Gerontology. http://gero.usc.edu/2018/06/20/westworld-cognition-cowboy-hats-are-a-step-up-from-a-real-science-tool-inverse/

  • Balthazar P, Harri P, Prater A et al (2018) Protecting your patients’ interests in the era of big data, artificial intelligence, and predictive analytics. J Am Coll Radiol 15(3, Part B):580–586

    Google Scholar 

  • Banks IM (2002) Look to windward. Simon and Schuster

    Google Scholar 

  • Banks IM (2010) Surface detail. Orbit

    Google Scholar 

  • Bourne C, Aydemir Ö, Balanzá-Martínez V et al (2013) Neuropsychological testing of cognitive impairment in euthymic bipolar disorder: an individual patient data meta-analysis. Acta Psychiat Scand 128(3):149–162

    Article  CAS  Google Scholar 

  • Canhoto AI, Arp S (2017) Exploring the factors that support adoption and sustained use of health and fitness wearables. J Mark Manag 33(1–2):32–60

    Article  Google Scholar 

  • Cao B, Zheng L, Zhang C et al (2017) Deepmood: modeling mobile phone typing dynamics for mood detection. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, pp 747–755

    Google Scholar 

  • Cho K, van Merrienboer B, Gulcehre C et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. eprint arXiv:1406.1078

  • Chung JE, Joo HR, Fan JL et al (2018) High-density, long-lasting, and multi-region electrophysiological recordings using polymer electrode arrays. bioRxiv:242693

    Google Scholar 

  • Clifford C (2017) This former Google[X] exec is building a high-tech hat that she says will make telepathy possible in 8 years. This former Google[X] exec is building a high-tech hat that she says will make telepathy possible in 8 years. https://www.cnbc.com/2017/07/07/this-inventor-is-developing-technology-that-could-enable-telepathy.html

  • Cummings N, Schuller BW (2019) Advances in computational speech analysis for mobile sensing. In: Baumeister H, Montag C (eds) Mobile sensing and psychoinformatics. Berlin, Springer, pp 141–159

    Google Scholar 

  • Dixon-Román E (2016) Algo-Ritmo: more-than-human performative acts and the racializing assemblages of algorithmic architectures. Cult Studies? Crit Methodol 16(5):482–490

    Google Scholar 

  • Durstewitz D, Koppe G, Meyer-Lindenberg A (2019) Deep neural networks in psychiatry. Mol Psychiatry

    Google Scholar 

  • Ebner-Priemer UW, Eid M, Kleindienst N et al (2009) Analytic strategies for understanding affective (in)stability and other dynamic processes in psychopathology. J Abnorm Psychol 118(1):195–202

    Article  Google Scholar 

  • Ebner-Priemer UW, Trull TJ (2009) Ecological momentary assessment of mood disorders and mood dysregulation. Psychol Assess 21(4):463–475

    Article  Google Scholar 

  • Feng CH (2018) How a smartwatch literally saved this man’s life and why he wants more people to wear one. South China Morning Post. https://www.scmp.com/lifestyle/health-wellness/article/2145681/how-apple-watch-literally-saved-mans-life-and-why-he-wants

  • Fu T-M, Hong G, Zhou T et al (2016) Stable long-term chronic brain mapping at the single-neuron level. Nat Methods 13:875

    Article  CAS  Google Scholar 

  • Gideon J, Provost EM, McInnis M (2016) Mood state prediction from speech of varying acoustic quality for individuals with bipolar disorder. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), 20–25 March 2016, pp 2359–2363

    Google Scholar 

  • Global Burden of Disease Collaborative Network (2017) Global Burden of Disease study 2016 (GBD 2016) results. Institute for Health Metrics and Evaluation (IHME) Seattle, United States

    Google Scholar 

  • Hou L, Bergen SE, Akula N et al (2016) Genome-wide association study of 40,000 individuals identifies two novel loci associated with bipolar disorder. Hum Mol Genet 25(15):3383–3394

    Article  CAS  Google Scholar 

  • Huang H, Cao B, Yu PS et al (2018) dpMood: exploiting local and periodic typing dynamics for personalized mood prediction. Paper presented at the IEEE International Conference on Data Mining

    Google Scholar 

  • Ikeda M, Takahashi A, Kamatani Y et al (2017) A genome-wide association study identifies two novel susceptibility loci and trans population polygenicity associated with bipolar disorder. Mol Psychiatr 23:639

    Article  Google Scholar 

  • Jepsen ML (2017) Open Water Internet Inc. Optical imaging of diffuse medium. U.S. Patent No. 9,730,649,

    Google Scholar 

  • Karam ZN, Provost EM, Singh S et al (2014) Ecologically valid long-term mood monitoring of individuals with bipolar disorder using speech. In: 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), 4–9 May 2014, pp 4858–4862

    Google Scholar 

  • Khorram S, Gideon J, McInnis MG et al (2016) Recognition of depression in bipolar disorder: leveraging cohort and person-specific knowledge. In: INTERSPEECH

    Google Scholar 

  • Khorram S, Jaiswal M, Gideon J et al (2018) The PRIORI emotion dataset: linking mood to emotion detected in-the-wild. ArXiv e-prints

    Google Scholar 

  • Kubiak T, Smyth JM (2019) Connecting domains—ecological momentary assessment in a mobile sensing framework. In: Baumeister H, Montag C (eds) Mobile sensing and psychoinformatics. Berlin, Springer, pp 201–207

    Google Scholar 

  • Leow A, Ajilore O, Zhan L et al (2013) Impaired inter-hemispheric integration in bipolar disorder revealed with brain network analyses. Biol Psychiat 73(2):183–193

    Article  Google Scholar 

  • Lovatt M, Holmes J (2017) Digital phenotyping and sociological perspectives in a Brave New World. Addiction (Abingdon, England) 112(7):1286–1289

    Article  Google Scholar 

  • Martinez-Martin N, Kreitmair K (2018) Ethical issues for direct-to-consumer digital psychotherapy apps: addressing accountability, data protection, and consent. JMIR Ment Health 5(2):e32–e32

    Article  Google Scholar 

  • McInnis M, Gideon J, Mower Provost E (2017) Digital Phenotyping in bipolar disorder. Eur Neuropsychopharm 27:S440

    Article  Google Scholar 

  • Messner E-M, Probst T, O’Rourke T et al (2019) mHealth applications: potentials, limitations, current quality and future directions. In Baumeister H, Montag C (eds) Mobile sensing and psychoinformatics. Berlin, Springer

    Google Scholar 

  • Montag C, Markowetz A, Blaszkiewicz K et al (2017) Facebook usage on smartphones and gray matter volume of the nucleus accumbens. Behav Brain Res 329:221–228

    Google Scholar 

  • Muthukrishna M, Henrich J (2019) A problem in theory. Nat Hum Behav

    Google Scholar 

  • National Collaborating Centre for Mental Health (2018) Bipolar disorder: the NICE guideline on the assessment and management of bipolar disorder in adults, children and young people in primary and secondary care. In: British Psychological Society, pp 39–40

    Google Scholar 

  • Perlow J (2018) How Apple watch saved my life. ZDNet. https://www.zdnet.com/article/how-apple-watch-saved-my-life/

  • Phillips ML, Kupfer DJ (2013) Bipolar disorder diagnosis: challenges and future directions. Lancet 381(9878):1663–1671

    Article  Google Scholar 

  • Phillips ML, Ladouceur CD, Drevets WC (2008) A neural model of voluntary and automatic emotion regulation: implications for understanding the pathophysiology and neurodevelopment of bipolar disorder. Mol Psychiatr 13:833

    Article  Google Scholar 

  • Rabbi M, Klasnja P, Choudhury T et al (2019) Optimizing mHealth interventions with a bandit. In: Baumeister H, Montag C (eds) Mobile sensing and psychoinformatics. Berlin, Springer, pp 277–291

    Google Scholar 

  • Samzelius J (2016) Neurametrix Inc. System and method for continuous monitoring of central nervous system diseases. U.S. Patent No. 15,166,064,

    Google Scholar 

  • Sanford K (2018) Will this “neural lace” brain implant help us compete with AI? http://nautil.us/blog/-will-this-neural-lace-brain-implant-help-us-compete-with-ai

  • Sariyska R, Rathner E-M, Baumeister H et al (2018) Feasibility of linking molecular genetic markers to real-world social network size tracked on smartphones. Front Neurosci 12(945)

    Google Scholar 

  • Shropshire C (2015) Americans prefer texting to talking, report says. Chicago Tribune. http://www.chicagotribune.com/business/ct-americans-texting-00327-biz-20150326-story.html

  • Stange JP, Zulueta J, Langenecker SA et al (2018) Let your fingers do the talking: passive typing instability predicts future mood outcomes. Bipolar Disord 20(3):285–288

    Article  Google Scholar 

  • Steel Z, Marnane C, Iranpour C et al (2014) The global prevalence of common mental disorders: a systematic review and meta-analysis 1980–2013. 43(2):476–493

    Google Scholar 

  • Sun L, Wang Y, Cao B et al (2017) Sequential keystroke behavioral biometrics for mobile user identification via multi-view deep learning. Paper presented at the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 01 November 2017

    Google Scholar 

  • Turakhia MP (2018) Moving from big data to deep learning—the case of atrial fibrillation. JAMA Cardiol 3(5):371–372

    Article  Google Scholar 

  • Turakhia MP, Desai M, Hedlin H et al (2019) Rationale and design of a large-scale, app-based study to identify cardiac arrhythmias using a smartwatch: the Apple heart study. Am Heart J 207:66–75

    Article  Google Scholar 

  • Wolkenstein L, Bruchmuller K, Schmid P et al (2011) Misdiagnosing bipolar disorder—do clinicians show heuristic biases? J Affect Disorders 130(3):405–412

    Article  Google Scholar 

  • Zulueta J, Piscitello A, Rasic M et al (2018) Predicting mood disturbance severity with mobile phone keystroke metadata: a biaffect digital phenotyping study. J Med Internet Res 20(7):e241

    Article  Google Scholar 

Download references

Acknowledgements

We are grateful to the Robert Wood Johnson Foundation, the Prechter Bipolar Research Fund, Apple, Luminary Labs, and Sage Bionetworks, all of whom have helped enable much of the research discussed in this chapter. Jonathan P. Stange was supported by grant K23MH112769 from NIMH.

Author information

Authors and Affiliations

  1. Collaborative Neuroimaging Environment for Connectomics, University of Illinois, Chicago, USA

    Faraz Hussain, John Zulueta, Olusola A. Ajilore & Alex Leow

  2. Cognition and Affect Regulation Lab, University of Illinois, Chicago, USA

    Jonathan P. Stange

  3. University Neuropsychiatric Institute, University of Utah, Salt Lake City, UT, USA

    Scott A. Langenecker

  4. Heinz C. Prechter Bipolar Research Program, University of Michigan, Ann Arbor, MI, USA

    Melvin G. McInnis

  5. Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy

    Andrea Piscitello

  6. Video Understanding Team, Applied Machine Learning, Facebook, Menlo Park, CA, USA

    Bokai Cao

  7. Department of Computer Science, University of Illinois, Chicago, USA

    He Huang & Philip S. Yu

  8. College of Engineering, University of Illinois, Chicago, USA

    Peter Nelson

Authors
  1. Faraz Hussain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan P. Stange
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Scott A. Langenecker
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Melvin G. McInnis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. John Zulueta
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Andrea Piscitello
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Bokai Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. He Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Philip S. Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Peter Nelson
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Olusola A. Ajilore
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Alex Leow
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jonathan P. Stange .

Editor information

Editors and Affiliations

  1. Department of Clinical Psychology and Psychotherapy, Institute of Psychology and Education, Ulm University, Ulm, Germany

    Harald Baumeister

  2. Department of Molecular Psychology, Institute of Psychology and Education, Ulm University, Ulm, Germany

    Christian Montag

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Hussain, F. et al. (2019). Passive Sensing of Affective and Cognitive Functioning in Mood Disorders by Analyzing Keystroke Kinematics and Speech Dynamics. In: Baumeister, H., Montag, C. (eds) Digital Phenotyping and Mobile Sensing. Studies in Neuroscience, Psychology and Behavioral Economics. Springer, Cham. https://doi.org/10.1007/978-3-030-31620-4_10

Download citation

Publish with us

Policies and ethics

Search

Navigation