Digital Phenotyping Using Multimodal Data


Purpose of Review

Digital phenotyping involves the quantification of in situ phenotypes using personal digital devices and holds the potential to dramatically reshape how serious mental illnesses (SMI) assessment is conducted. Despite promise, few, if any, digital phenotyping efforts for SMI have garnered the support necessary for clinical implementation.

Recent Findings

In this paper, we highlight how digital phenotyping efforts can be improved by integrating data from multiple channels (i.e., “multimodal” data integration). We begin by arguing that “multimodal” integration is critical for digital phenotyping of many, possibly most, SMI symptoms. Next, we consider computational approaches that can accommodate multimodal data.


We conclude by considering next steps for multimodal data for research and clinical applications. We punctuate this paper with examples of multimodal digital phenotyping using natural language processing (NLP) to measure speech disorganization in psychosis.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3


Papers of particular interest, published recently, have been highlighted as: • Of importance

  1. 1.

    Insel TR. Assessing the economic costs of serious mental illness. Am J Psychiatry. 2008;165:663–5.

    PubMed  Article  PubMed Central  Google Scholar 

  2. 2.

    Meyer GJ, Finn SE, Eyde LD, Kay GG, Moreland KL, Dies RR, et al. Psychological testing and psychological assessment. A review of evidence and issues. Am Psychol. 2001;56:128–65.

  3. 3.

    Rush AJJ, First MB, Blacker D. Handbook of psychiatric measures, second edition: American Psychiatric Publishing, Inc.; 2008.

  4. 4.

    Kessler RC, Aguilar-Gaxiola S, Alonso J, Chatterji S, Lee S, Ormel J, et al. The global burden of mental disorders: an update from the WHO World Mental Health (WMH) surveys. Epidemiol Psichiatr Soc. NIH Public Access. 2009;18:23–33.

  5. 5.

    Onnela JP, Rauch SL. Harnessing smartphone-based digital phenotyping to enhance behavioral and mental health. Neuropsychopharmacology. 2016;41:1691–6.

    PubMed  PubMed Central  Article  Google Scholar 

  6. 6.

    Insel T, Cuthbert B. Brain disorders? Precisely. Precision medicine comes to psychiatry. Science. 2015;348:499–500.

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  7. 7.

    Cohen AS, Elvevåg B. Automated computerized analysis of speech in psychiatric disorders. Curr Opin Psychiatry. NIH Public Access. 2014;27:203–9.

    Article  Google Scholar 

  8. 8.

    Cohen AS. Advancing ambulatory biobehavioral technologies beyond proof of concept: introduction to the special section. Psychol Assess. 2019;31:277–84.

    PubMed  Article  PubMed Central  Google Scholar 

  9. 9.

    • Cohen AS, Schwartz EK, Le TP, Fedechko T, Kirkpatrick B, Strauss GP, et al. Using biobehavioral technologies to effectively advance research on negative symptoms. World Psychiatry. 2019;18:103–4 This reference provides an important summary of the application of biobehavioral technology to negative symptoms, a critical component of SMI.

    PubMed  PubMed Central  Article  Google Scholar 

  10. 10.

    Trull TJ, Ebner-Priemer U. The role of ambulatory assessment in psychological science. Curr Dir Psychol Sci. 2014;23:466–70.

    PubMed  PubMed Central  Article  Google Scholar 

  11. 11.

    Wright AGC, Zimmermann J. Applied ambulatory assessment: integrating idiographic and nomothetic principles of measurement. Psychol Assess. 2019;31:1467–80.

    PubMed  Article  Google Scholar 

  12. 12.

    • Hsin H, Fromer M, Peterson B, Walter C, Fleck M, Campbell A, et al. Transforming psychiatry into data-driven medicine with digital measurement tools. npj Digit Med. 2018;1:1–4 This reference is a necessary grounding on the importance of data-driven measurement.

    Article  Google Scholar 

  13. 13.

    Ben-Zeev D, Scherer EA, Wang R, Xie H, Campbell AT. Next-generation psychiatric assessment: using smartphone sensors to monitor behavior and mental health. Psychiatr Rehabil J. 2015;38:218–26.

    PubMed  PubMed Central  Article  Google Scholar 

  14. 14.

    Fisher AJ, Medaglia JD, Jeronimus BF. Lack of group-to-individual generalizability is a threat to human subjects research. Proc Natl Acad Sci. 2018;115:E6106–15.

    CAS  Article  Google Scholar 

  15. 15.

    Insel TR. The NIMH research domain criteria (RDoC) project: precision medicine for psychiatry. Am J Psychiatry. 2014;171:395–397.

  16. 16.

    Mulder R, Newton-Howes G, Coid JW. The futility of risk prediction in psychiatry. Br J Psychiatry. 2016;209:271–2.

    PubMed  Article  PubMed Central  Google Scholar 

  17. 17.

    American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5). Am. Psychiatr. Publ. Arlington, VA; 2013.

  18. 18.

    Bedi G, Carillo F, Cecchi GA, Slezak DF, Sigman M, Mota NB, et al. Automated analysis of disorganized communication predicts transition to psychosis among clinical high risk patients. Neuropsychopharmacology. 2015;30:S436–7.

    Google Scholar 

  19. 19.

    Corcoran CM, Carrillo F, Fernández-Slezak D, Bedi G, Klim C, Javitt DC, et al. Prediction of psychosis across protocols and risk cohorts using automated language analysis. World Psychiatry. 2018;17:67–75.

  20. 20.

    • Holmlund TB, Foltz PW, Cohen AS, Johansen HD, Sigurdsen R, Fugelli P, et al. Moving psychological assessment out of the controlled laboratory setting and into the hands of the individual: practical challenges. Psychological Assessment. 2019;31(3):292–303. This reference addresses some of the real-world considerations of digital phenotyping.

  21. 21.

    Foltz PW, Rosenstein M, Elvevåg B. Detecting clinically significant events through automated language analysis: quo imus? Nat Partn Journals Schizophr. Nat Publ Group; 2016;2:15054.

  22. 22.

    Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25:44–56.

    CAS  PubMed  Article  Google Scholar 

  23. 23.

    Hinton G. Deep learning-a technology with the potential to transform health care. JAMA - J. Am. Med. Assoc. 2018;320:1101–2.

    Article  Google Scholar 

  24. 24.

    Raugh IM, Chapman HC, Bartolomeo LA, Gonzalez C, Strauss GP. A comprehensive review of psychophysiological applications for ecological momentary assessment in psychiatric populations. Psychol Assess. 2019;31:304–17.

    PubMed  Article  Google Scholar 

  25. 25.

    Cohen AS, Mitchell KR, Elvevåg B. What do we really know about blunted vocal affect and alogia? A meta-analysis of objective assessments. Schizophr Res. NIH Public Access. 2014;159:533–8.

    Article  Google Scholar 

  26. 26.

    Smoller JW. The use of electronic health records for psychiatric phenotyping and genomics. Am J Med Genet Part B Neuropsychiatr Genet. 2018;177:601–12.

    Article  Google Scholar 

  27. 27.

    Liang Y, Zheng X, Zeng DD. A survey on big data-driven digital phenotyping of mental health. Inf Fusion. 2019;52:290–307.

    Article  Google Scholar 

  28. 28.

    Dagum P. Digital biomarkers of cognitive function. npj Digit Med. 2018;1:10.

    PubMed  PubMed Central  Article  Google Scholar 

  29. 29.

    Ben-Zeev D, Brian R, Wang R, Wang W, Campbell AT, Aung MSH, et al. CrossCheck: integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatr Rehabil J. 2017;40:266–75.

  30. 30.

    Garcia-Ceja E, Riegler M, Nordgreen T, Jakobsen P, Oedegaard KJ, Tørresen J. Mental health monitoring with multimodal sensing and machine learning: a survey. Pervasive Mob Comput. 2018;51:1–26.

    Article  Google Scholar 

  31. 31.

    • Torous J, Onnela JP, Keshavan M. New dimensions and new tools to realize the potential of RDoC: digital phenotyping via smartphones and connected devices. Transl Psychiatry. 2017;7:e1053. This reference is a very important discussion of the ways to leverage digital phenotyping beyond SMI.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  32. 32.

    Torous J, Baker JT. Why psychiatry needs data science and data science needs psychiatry: Connecting with technology. JAMA Psychiatry. 2016;73:3–4.

  33. 33.

    Eyben F, Scherer KR, Schuller BW, Sundberg J, Andre E, Busso C, et al. The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for voice research and affective computing. IEEE Trans Affect Comput. 2016;7:190–202.

  34. 34.

    Insel TR, Cuthbert B, Garvey M, Heinssen R, Pine DS, Quinn K, et al. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am J Psychiatry. 2010;167:748–51.

  35. 35.

    Jahshan C, Heaton RK, Golshan S, Cadenhead KS. Course of neurocognitive deficits in the prodrome and first episode of schizophrenia. Neuropsychology. 2010;24:109–20.

    PubMed  PubMed Central  Article  Google Scholar 

  36. 36.

    Fusar-Poli P, Borgwardt S, Bechdolf A, Addington J, Riecher-Rössler A, Schultze-Lutter F, et al. The psychosis high-risk state: a comprehensive state-of-the-art review. Arch Gen Psychiatry. 2013;70:107–20.

    Google Scholar 

  37. 37.

    van Eeden WA, van Hemert AM, Carlier IVE, Penninx BW, Giltay EJ. Severity, course trajectory, and within-person variability of individual symptoms in patients with major depressive disorder. Acta Psychiatr Scand. 2019;139:194–205.

    PubMed  Article  Google Scholar 

  38. 38.

    Pogue-Geile MF, Harrow M. Negative symptoms in schizophrenia: their longitudinal course and prognostic importance. Schizophr Bull. 1985;11:427–39.

    CAS  PubMed  Article  Google Scholar 

  39. 39.

    Findling RL, Jo B, Frazier TW, Youngstrom EA, Demeter CA, Fristad MA, et al. The 24-month course of manic symptoms in children. Bipolar Disord. 2013;15:669–79.

  40. 40.

    Wass SV, de Barbaro K, Clackson K. Tonic and phasic co-variation of peripheral arousal indices in infants. Biol Psychol. 2015;111:26–39.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  41. 41.

    Pole N. The psychophysiology of posttraumatic stress disorder: a meta-analysis. Psychol Bull. 2007;133:725–46.

    PubMed  Article  PubMed Central  Google Scholar 

  42. 42.

    Aue T, Hoeppli ME, Piguet C. The sensitivity of physiological measures to phobic and nonphobic fear intensity. J Psychophysiol. 2012;24:156–67.

    Google Scholar 

  43. 43.

    Nagai Y, Critchley HD, Featherstone E, Trimble MR, Dolan RJ. Activity in ventromedial prefrontal cortex covaries with sympathetic skin conductance level: a physiological account of a “default mode” of brain function. Neuroimage. 2004;22:243–51.

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  44. 44.

    Berntson GG, Cacioppo JT, Quigly KS, Fabro VT. Autonomic space and psychophysiological response. Psychophysiology. 1994;31:44–61.

    CAS  PubMed  Article  PubMed Central  Google Scholar 

  45. 45.

    Kimhy D, Myin-Germeys I, Palmier-Claus J, Swendsen J. Mobile assessment guide for research in schizophrenia and severe mental disorders. Schizophr Bull. 2012;38:386–95.

    PubMed  PubMed Central  Article  Google Scholar 

  46. 46.

    Schwartz S, Schultz S, Reider A, Saunders EFH. Daily mood monitoring of symptoms using smartphones in bipolar disorder: a pilot study assessing the feasibility of ecological momentary assessment. J Affect Disord. 2016;191:88–93.

    PubMed  Article  PubMed Central  Google Scholar 

  47. 47.

    Trull TJ, Solhan MB, Tragesser SL, Jahng S, Wood PK, Piasecki TM, et al. Affective instability: measuring a core feature of borderline personality disorder with ecological momentary assessment. J Abnorm Psychol. 2008;117:647–61.

  48. 48.

    Kwapil TR, Silvia PJ, Myin-Germeys I, Anderson AJ, Coates SA, Brown LH. The social world of the socially anhedonic: exploring the daily ecology of asociality. J Res Pers. 2009;43:103–6.

    Article  Google Scholar 

  49. 49.

    Gore WL, Widiger TA. Fluctuation between grandiose and vulnerable narcissism. Personal Disord Theory Res Treat. 2016;7:363–71.

    Article  Google Scholar 

  50. 50.

    De Ridder J, Pihet S, Suter M, Caldara R. Empathy in institutionalized adolescents with callous-unemotional traits: an ecological momentary assessment study of emotion recognition. Crim Justice Behav. 2016;43:653–69.

    Article  Google Scholar 

  51. 51.

    Trull TJ, Ebner-Priemer UW. Using experience sampling methods/ecological momentary assessment (ESM/EMA) in clinical assessment and clinical research: introduction to the special section. Psychol Assess. 2009;21:457–62.

    PubMed  PubMed Central  Article  Google Scholar 

  52. 52.

    Adam EK, Quinn ME, Tavernier R, McQuillan MT, Dahlke KA, Gilbert KE. Diurnal cortisol slopes and mental and physical health outcomes: a systematic review and meta-analysis. Psychoneuroendocrinology. 2017;83:25–41.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  53. 53.

    Nestler EJ, Peña CJ, Kundakovic M, Mitchell A, Akbarian S. Epigenetic basis of mental illness. Neuroscientist. 2016;22:447–63.

    CAS  PubMed  Article  Google Scholar 

  54. 54.

    Elvevåg B, Foltz PW, Weinberger DR, Goldberg TE. Quantifying incoherence in speech: an automated methodology and novel application to schizophrenia. Schizophr Res. 2007;93:304–16.

    PubMed  PubMed Central  Article  Google Scholar 

  55. 55.

    Andreasen NC, Grove WM. Thought, language, and communication in schizophrenia: diagnosis and prognosis. Schizophr Bull. 1986;12:348–59.

    CAS  PubMed  Article  Google Scholar 

  56. 56.

    Kuperberg GR. Language in schizophrenia part 1: an introduction. Lang Linguist Compass. 2010;4:576–89.

    PubMed  PubMed Central  Article  Google Scholar 

  57. 57.

    Bedi G, Carrillo F, Cecchi GA, Slezak DF, Sigman M, Mota NB, et al. Automated analysis of free speech predicts psychosis onset in high-risk youths. NPJ Schizophr. 2015;1:15030.

  58. 58.

    Taylor WL. “Cloze procedure”: a new tool for measuring readability. Journal Bull. 1953;30:415–33.

    Google Scholar 

  59. 59.

    Elvevåg B, Fisher JE, Gurd JM, Goldberg TE. Semantic clustering in verbal fluency: schizophrenic patients versus control participants. Psychol Med. 2002;32:909–17.

    PubMed  Article  PubMed Central  Google Scholar 

  60. 60.

    Docherty NM, Rhinewine JP, Nienow TM, Cohen AS. Affective reactivity of language symptoms, startle responding, and inhibition in schizophrenia. J Abnorm Psychol. 2001;110:194–8.

    CAS  PubMed  Article  Google Scholar 

  61. 61.

    Le TP, Najolia GM, Minor KS, Cohen AS. The effect of limited cognitive resources on communication disturbances in serious mental illness. Psychiatry Res. 2017;248:98–104.

    PubMed  Article  Google Scholar 

  62. 62.

    Loevinger J. Objective tests as instruments of psychological theory. Psychol Rep. 1957;3:635–94.

    Google Scholar 

  63. 63.

    Krueger RF, Eaton NR. Structural validity and the classification of mental disorders. In: Kendler KS, Parnas J, editors. Philosophical Issues in Psychiatry II: Nosology. Oxford; 2012.

  64. 64.

    Rouse WB, Morris NM. On looking into the black box: Prospects and limits in the search for mental models. Psychol Bull. 1986;100:349–363.

  65. 65.

    • Cohen AS, Schwartz E, Le T, Cowan T, Cox C, Tucker R, et al. Validating digital phenotyping technologies for clinical use: the critical importance of “resolution.”. World Psychiatry. 2019;19:114–5 This paper introduces the concept of resolution which is a critical strength of digital phenotyping.

    Article  Google Scholar 

  66. 66.

    Cohen, A S, Mitchell K, Schwartz E, Le T, Cox C, Tucker RP, et al. Psychometrics for the 21st century: resolution as a key to circumventing the “psychiatric miasma.”

  67. 67.

    Pestian JP, Sorter M, Connolly B, Bretonnel Cohen K, McCullumsmith C, Gee JT, et al. A machine learning approach to identifying the thought markers of suicidal subjects: a prospective multicenter trial. Suicide Life-Threatening Behav. 2017;47:112–21.

  68. 68.

    Okser S, Pahikkala T, Airola A, Salakoski T, Ripatti S, Aittokallio T. Regularized machine learning in the genetic prediction of complex traits. PLoS Genet. 2014;10:e1004754.

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  69. 69.

    Oswal U, Cox C, Ralph MAL, Rogers T, Nowak R. Representational similarity learning with application to brain networks. In: 33rd International Conference on Machine Learning ICML; 2016.

    Google Scholar 

  70. 70.

    Raoy NS, Cox CR, Nowaky RD, Rogers TT. Sparse overlapping sets lasso for multitask learning and its application to fMRI analysis. In: Advances in Neural Information Processing Systems 2013;2202–2210.

  71. 71.

    Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB, editors. Bayesian data analysis, third edition. Taylor & Francis: CRC Press; 2013.

    Google Scholar 

  72. 72.

    Janssen RJ, Mourão-Miranda J, Schnack HG. Making individual prognoses in psychiatry using neuroimaging and machine learning. Biol Psychiatry Cogn Neurosci Neuroimaging. 2018;3:798–808.

    PubMed  PubMed Central  Google Scholar 

  73. 73.

    Yang TH, Wu CH, Huang KY, Su MH. Detection of mood disorder using speech emotion profiles and LSTM. In: 10th International Symposium on Chinese Spoken Language Processing ISCSLP; 2016.

  74. 74.

    Durstewitz D, Koppe G, Meyer-Lindenberg A. Deep neural networks in psychiatry. Mol Psychiatry. 2019;24:1583–98.

    PubMed  Article  PubMed Central  Google Scholar 

  75. 75.

    Wu JL, He Y, Yu LC, Robert LK. Identifying emotion labels from psychiatric social texts using a bi-directional LSTM-CNN model. IEEE Access. 2020;8:66638–46.

    Article  Google Scholar 

  76. 76.

    Shim K, Lee M, Choi I, Boo Y, Sung W. SVD-softmax: fast softmax approximation on large vocabulary neural networks. In: Advances in Neural Information Processing Systems; 2017.

    Google Scholar 

  77. 77.

    • Ramachandram D, Taylor GW. Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process Mag. 2017;34:96–108 This source is an important reference on the computational possibilities for multimodal digital phenotyping.

    Article  Google Scholar 

  78. 78.

    Baltrusaitis T, Ahuja C, Morency LP. Multimodal machine learning: a survey and taxonomy. IEEE Trans Pattern Anal Mach Intell. 2019;41:423–43.

    PubMed  Article  Google Scholar 

  79. 79.

    Garagić D, Peskoe J, Liu F, Claffey MS, Bendich P, Hineman J, et al. Upstream fusion of multiple sensing modalities using machine learning and topological analysis: an initial exploration. IEEE Aerospace Conference Proceedings: In; 2018.

    Google Scholar 

  80. 80.

    Cohen AS, Cowan T, Le TP, Schwartz EK, Kirkpatrick B, Raugh IM, et al. Ambulatory digital phenotyping of blunted affect and alogia using objective facial and vocal analysis: Proof of concept. Schizophr Res. 2020;In Press.

  81. 81.

    • Cohen AS, Schwartz EK, Le T, Cowan TM, Kirkpatrick B, Raugh IM, et al. Digital phenotyping of negative symptoms: the relationship to clinician ratings. Schizophr Bull. 2020; In Press. This reference provides evidence for external validity and the challenges in establishing external validity in digital phenotyping.

  82. 82.

    Torous J, Staples P, Onnela JP. Realizing the potential of mobile mental health: new methods for new data in psychiatry. Curr Psychiatry Rep. 2015;17:602.

    PubMed  PubMed Central  Article  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Alex S. Cohen.

Ethics declarations

Conflict of Interest

Terje B. Holmlund has received research funding through a grant from the Research Council of Norway, but all other authors have no potential conflicts of interest to disclose.

Human and Animal Rights

All reported studies/experiments with human or animal subjects performed by the authors have been previously published and complied with all applicable ethical standards (including the Helsinki declaration and its amendments, institutional/national research committee standards, and international/national/institutional guidelines).

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the Topical Collection on Psychosis

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Cohen, A.S., Cox, C.R., Masucci, M.D. et al. Digital Phenotyping Using Multimodal Data. Curr Behav Neurosci Rep (2020).

Download citation


  • Digital phenotyping
  • Multimodal
  • Machine learning
  • Serious mental illness
  • Ambulatory