Purpose of Review
Digital phenotyping involves the quantification of in situ phenotypes using personal digital devices and holds the potential to dramatically reshape how serious mental illnesses (SMI) assessment is conducted. Despite promise, few, if any, digital phenotyping efforts for SMI have garnered the support necessary for clinical implementation.
In this paper, we highlight how digital phenotyping efforts can be improved by integrating data from multiple channels (i.e., “multimodal” data integration). We begin by arguing that “multimodal” integration is critical for digital phenotyping of many, possibly most, SMI symptoms. Next, we consider computational approaches that can accommodate multimodal data.
We conclude by considering next steps for multimodal data for research and clinical applications. We punctuate this paper with examples of multimodal digital phenotyping using natural language processing (NLP) to measure speech disorganization in psychosis.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Papers of particular interest, published recently, have been highlighted as: • Of importance
Insel TR. Assessing the economic costs of serious mental illness. Am J Psychiatry. 2008;165:663–5.
Meyer GJ, Finn SE, Eyde LD, Kay GG, Moreland KL, Dies RR, et al. Psychological testing and psychological assessment. A review of evidence and issues. Am Psychol. 2001;56:128–65.
Rush AJJ, First MB, Blacker D. Handbook of psychiatric measures, second edition: American Psychiatric Publishing, Inc.; 2008.
Kessler RC, Aguilar-Gaxiola S, Alonso J, Chatterji S, Lee S, Ormel J, et al. The global burden of mental disorders: an update from the WHO World Mental Health (WMH) surveys. Epidemiol Psichiatr Soc. NIH Public Access. 2009;18:23–33.
Onnela JP, Rauch SL. Harnessing smartphone-based digital phenotyping to enhance behavioral and mental health. Neuropsychopharmacology. 2016;41:1691–6.
Insel T, Cuthbert B. Brain disorders? Precisely. Precision medicine comes to psychiatry. Science. 2015;348:499–500.
Cohen AS, Elvevåg B. Automated computerized analysis of speech in psychiatric disorders. Curr Opin Psychiatry. NIH Public Access. 2014;27:203–9.
Cohen AS. Advancing ambulatory biobehavioral technologies beyond proof of concept: introduction to the special section. Psychol Assess. 2019;31:277–84.
• Cohen AS, Schwartz EK, Le TP, Fedechko T, Kirkpatrick B, Strauss GP, et al. Using biobehavioral technologies to effectively advance research on negative symptoms. World Psychiatry. 2019;18:103–4 This reference provides an important summary of the application of biobehavioral technology to negative symptoms, a critical component of SMI.
Trull TJ, Ebner-Priemer U. The role of ambulatory assessment in psychological science. Curr Dir Psychol Sci. 2014;23:466–70.
Wright AGC, Zimmermann J. Applied ambulatory assessment: integrating idiographic and nomothetic principles of measurement. Psychol Assess. 2019;31:1467–80.
• Hsin H, Fromer M, Peterson B, Walter C, Fleck M, Campbell A, et al. Transforming psychiatry into data-driven medicine with digital measurement tools. npj Digit Med. 2018;1:1–4 This reference is a necessary grounding on the importance of data-driven measurement.
Ben-Zeev D, Scherer EA, Wang R, Xie H, Campbell AT. Next-generation psychiatric assessment: using smartphone sensors to monitor behavior and mental health. Psychiatr Rehabil J. 2015;38:218–26.
Fisher AJ, Medaglia JD, Jeronimus BF. Lack of group-to-individual generalizability is a threat to human subjects research. Proc Natl Acad Sci. 2018;115:E6106–15.
Insel TR. The NIMH research domain criteria (RDoC) project: precision medicine for psychiatry. Am J Psychiatry. 2014;171:395–397.
Mulder R, Newton-Howes G, Coid JW. The futility of risk prediction in psychiatry. Br J Psychiatry. 2016;209:271–2.
American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5). Am. Psychiatr. Publ. Arlington, VA; 2013.
Bedi G, Carillo F, Cecchi GA, Slezak DF, Sigman M, Mota NB, et al. Automated analysis of disorganized communication predicts transition to psychosis among clinical high risk patients. Neuropsychopharmacology. 2015;30:S436–7.
Corcoran CM, Carrillo F, Fernández-Slezak D, Bedi G, Klim C, Javitt DC, et al. Prediction of psychosis across protocols and risk cohorts using automated language analysis. World Psychiatry. 2018;17:67–75.
• Holmlund TB, Foltz PW, Cohen AS, Johansen HD, Sigurdsen R, Fugelli P, et al. Moving psychological assessment out of the controlled laboratory setting and into the hands of the individual: practical challenges. Psychological Assessment. 2019;31(3):292–303. https://doi.org/10.1037/pas0000647. This reference addresses some of the real-world considerations of digital phenotyping.
Foltz PW, Rosenstein M, Elvevåg B. Detecting clinically significant events through automated language analysis: quo imus? Nat Partn Journals Schizophr. Nat Publ Group; 2016;2:15054.
Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25:44–56.
Hinton G. Deep learning-a technology with the potential to transform health care. JAMA - J. Am. Med. Assoc. 2018;320:1101–2.
Raugh IM, Chapman HC, Bartolomeo LA, Gonzalez C, Strauss GP. A comprehensive review of psychophysiological applications for ecological momentary assessment in psychiatric populations. Psychol Assess. 2019;31:304–17.
Cohen AS, Mitchell KR, Elvevåg B. What do we really know about blunted vocal affect and alogia? A meta-analysis of objective assessments. Schizophr Res. NIH Public Access. 2014;159:533–8.
Smoller JW. The use of electronic health records for psychiatric phenotyping and genomics. Am J Med Genet Part B Neuropsychiatr Genet. 2018;177:601–12.
Liang Y, Zheng X, Zeng DD. A survey on big data-driven digital phenotyping of mental health. Inf Fusion. 2019;52:290–307.
Dagum P. Digital biomarkers of cognitive function. npj Digit Med. 2018;1:10.
Ben-Zeev D, Brian R, Wang R, Wang W, Campbell AT, Aung MSH, et al. CrossCheck: integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatr Rehabil J. 2017;40:266–75.
Garcia-Ceja E, Riegler M, Nordgreen T, Jakobsen P, Oedegaard KJ, Tørresen J. Mental health monitoring with multimodal sensing and machine learning: a survey. Pervasive Mob Comput. 2018;51:1–26.
• Torous J, Onnela JP, Keshavan M. New dimensions and new tools to realize the potential of RDoC: digital phenotyping via smartphones and connected devices. Transl Psychiatry. 2017;7:e1053. This reference is a very important discussion of the ways to leverage digital phenotyping beyond SMI.
Torous J, Baker JT. Why psychiatry needs data science and data science needs psychiatry: Connecting with technology. JAMA Psychiatry. 2016;73:3–4.
Eyben F, Scherer KR, Schuller BW, Sundberg J, Andre E, Busso C, et al. The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for voice research and affective computing. IEEE Trans Affect Comput. 2016;7:190–202.
Insel TR, Cuthbert B, Garvey M, Heinssen R, Pine DS, Quinn K, et al. Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. Am J Psychiatry. 2010;167:748–51.
Jahshan C, Heaton RK, Golshan S, Cadenhead KS. Course of neurocognitive deficits in the prodrome and first episode of schizophrenia. Neuropsychology. 2010;24:109–20.
Fusar-Poli P, Borgwardt S, Bechdolf A, Addington J, Riecher-Rössler A, Schultze-Lutter F, et al. The psychosis high-risk state: a comprehensive state-of-the-art review. Arch Gen Psychiatry. 2013;70:107–20.
van Eeden WA, van Hemert AM, Carlier IVE, Penninx BW, Giltay EJ. Severity, course trajectory, and within-person variability of individual symptoms in patients with major depressive disorder. Acta Psychiatr Scand. 2019;139:194–205.
Pogue-Geile MF, Harrow M. Negative symptoms in schizophrenia: their longitudinal course and prognostic importance. Schizophr Bull. 1985;11:427–39.
Findling RL, Jo B, Frazier TW, Youngstrom EA, Demeter CA, Fristad MA, et al. The 24-month course of manic symptoms in children. Bipolar Disord. 2013;15:669–79.
Wass SV, de Barbaro K, Clackson K. Tonic and phasic co-variation of peripheral arousal indices in infants. Biol Psychol. 2015;111:26–39.
Pole N. The psychophysiology of posttraumatic stress disorder: a meta-analysis. Psychol Bull. 2007;133:725–46.
Aue T, Hoeppli ME, Piguet C. The sensitivity of physiological measures to phobic and nonphobic fear intensity. J Psychophysiol. 2012;24:156–67.
Nagai Y, Critchley HD, Featherstone E, Trimble MR, Dolan RJ. Activity in ventromedial prefrontal cortex covaries with sympathetic skin conductance level: a physiological account of a “default mode” of brain function. Neuroimage. 2004;22:243–51.
Berntson GG, Cacioppo JT, Quigly KS, Fabro VT. Autonomic space and psychophysiological response. Psychophysiology. 1994;31:44–61.
Kimhy D, Myin-Germeys I, Palmier-Claus J, Swendsen J. Mobile assessment guide for research in schizophrenia and severe mental disorders. Schizophr Bull. 2012;38:386–95.
Schwartz S, Schultz S, Reider A, Saunders EFH. Daily mood monitoring of symptoms using smartphones in bipolar disorder: a pilot study assessing the feasibility of ecological momentary assessment. J Affect Disord. 2016;191:88–93.
Trull TJ, Solhan MB, Tragesser SL, Jahng S, Wood PK, Piasecki TM, et al. Affective instability: measuring a core feature of borderline personality disorder with ecological momentary assessment. J Abnorm Psychol. 2008;117:647–61.
Kwapil TR, Silvia PJ, Myin-Germeys I, Anderson AJ, Coates SA, Brown LH. The social world of the socially anhedonic: exploring the daily ecology of asociality. J Res Pers. 2009;43:103–6.
Gore WL, Widiger TA. Fluctuation between grandiose and vulnerable narcissism. Personal Disord Theory Res Treat. 2016;7:363–71.
De Ridder J, Pihet S, Suter M, Caldara R. Empathy in institutionalized adolescents with callous-unemotional traits: an ecological momentary assessment study of emotion recognition. Crim Justice Behav. 2016;43:653–69.
Trull TJ, Ebner-Priemer UW. Using experience sampling methods/ecological momentary assessment (ESM/EMA) in clinical assessment and clinical research: introduction to the special section. Psychol Assess. 2009;21:457–62.
Adam EK, Quinn ME, Tavernier R, McQuillan MT, Dahlke KA, Gilbert KE. Diurnal cortisol slopes and mental and physical health outcomes: a systematic review and meta-analysis. Psychoneuroendocrinology. 2017;83:25–41.
Nestler EJ, Peña CJ, Kundakovic M, Mitchell A, Akbarian S. Epigenetic basis of mental illness. Neuroscientist. 2016;22:447–63.
Elvevåg B, Foltz PW, Weinberger DR, Goldberg TE. Quantifying incoherence in speech: an automated methodology and novel application to schizophrenia. Schizophr Res. 2007;93:304–16.
Andreasen NC, Grove WM. Thought, language, and communication in schizophrenia: diagnosis and prognosis. Schizophr Bull. 1986;12:348–59.
Kuperberg GR. Language in schizophrenia part 1: an introduction. Lang Linguist Compass. 2010;4:576–89.
Bedi G, Carrillo F, Cecchi GA, Slezak DF, Sigman M, Mota NB, et al. Automated analysis of free speech predicts psychosis onset in high-risk youths. NPJ Schizophr. 2015;1:15030.
Taylor WL. “Cloze procedure”: a new tool for measuring readability. Journal Bull. 1953;30:415–33.
Elvevåg B, Fisher JE, Gurd JM, Goldberg TE. Semantic clustering in verbal fluency: schizophrenic patients versus control participants. Psychol Med. 2002;32:909–17.
Docherty NM, Rhinewine JP, Nienow TM, Cohen AS. Affective reactivity of language symptoms, startle responding, and inhibition in schizophrenia. J Abnorm Psychol. 2001;110:194–8.
Le TP, Najolia GM, Minor KS, Cohen AS. The effect of limited cognitive resources on communication disturbances in serious mental illness. Psychiatry Res. 2017;248:98–104.
Loevinger J. Objective tests as instruments of psychological theory. Psychol Rep. 1957;3:635–94.
Krueger RF, Eaton NR. Structural validity and the classification of mental disorders. In: Kendler KS, Parnas J, editors. Philosophical Issues in Psychiatry II: Nosology. Oxford; 2012. https://doi.org/10.1093/med/9780199642205.001.0001199.
Rouse WB, Morris NM. On looking into the black box: Prospects and limits in the search for mental models. Psychol Bull. 1986;100:349–363.
• Cohen AS, Schwartz E, Le T, Cowan T, Cox C, Tucker R, et al. Validating digital phenotyping technologies for clinical use: the critical importance of “resolution.”. World Psychiatry. 2019;19:114–5 This paper introduces the concept of resolution which is a critical strength of digital phenotyping.
Cohen, A S, Mitchell K, Schwartz E, Le T, Cox C, Tucker RP, et al. Psychometrics for the 21st century: resolution as a key to circumventing the “psychiatric miasma.”
Pestian JP, Sorter M, Connolly B, Bretonnel Cohen K, McCullumsmith C, Gee JT, et al. A machine learning approach to identifying the thought markers of suicidal subjects: a prospective multicenter trial. Suicide Life-Threatening Behav. 2017;47:112–21.
Okser S, Pahikkala T, Airola A, Salakoski T, Ripatti S, Aittokallio T. Regularized machine learning in the genetic prediction of complex traits. PLoS Genet. 2014;10:e1004754.
Oswal U, Cox C, Ralph MAL, Rogers T, Nowak R. Representational similarity learning with application to brain networks. In: 33rd International Conference on Machine Learning ICML; 2016.
Raoy NS, Cox CR, Nowaky RD, Rogers TT. Sparse overlapping sets lasso for multitask learning and its application to fMRI analysis. In: Advances in Neural Information Processing Systems 2013;2202–2210.
Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB, editors. Bayesian data analysis, third edition. Taylor & Francis: CRC Press; 2013.
Janssen RJ, Mourão-Miranda J, Schnack HG. Making individual prognoses in psychiatry using neuroimaging and machine learning. Biol Psychiatry Cogn Neurosci Neuroimaging. 2018;3:798–808.
Yang TH, Wu CH, Huang KY, Su MH. Detection of mood disorder using speech emotion profiles and LSTM. In: 10th International Symposium on Chinese Spoken Language Processing ISCSLP; 2016.
Durstewitz D, Koppe G, Meyer-Lindenberg A. Deep neural networks in psychiatry. Mol Psychiatry. 2019;24:1583–98.
Wu JL, He Y, Yu LC, Robert LK. Identifying emotion labels from psychiatric social texts using a bi-directional LSTM-CNN model. IEEE Access. 2020;8:66638–46.
Shim K, Lee M, Choi I, Boo Y, Sung W. SVD-softmax: fast softmax approximation on large vocabulary neural networks. In: Advances in Neural Information Processing Systems; 2017.
• Ramachandram D, Taylor GW. Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process Mag. 2017;34:96–108 This source is an important reference on the computational possibilities for multimodal digital phenotyping.
Baltrusaitis T, Ahuja C, Morency LP. Multimodal machine learning: a survey and taxonomy. IEEE Trans Pattern Anal Mach Intell. 2019;41:423–43.
Garagić D, Peskoe J, Liu F, Claffey MS, Bendich P, Hineman J, et al. Upstream fusion of multiple sensing modalities using machine learning and topological analysis: an initial exploration. IEEE Aerospace Conference Proceedings: In; 2018.
Cohen AS, Cowan T, Le TP, Schwartz EK, Kirkpatrick B, Raugh IM, et al. Ambulatory digital phenotyping of blunted affect and alogia using objective facial and vocal analysis: Proof of concept. Schizophr Res. 2020;In Press.
• Cohen AS, Schwartz EK, Le T, Cowan TM, Kirkpatrick B, Raugh IM, et al. Digital phenotyping of negative symptoms: the relationship to clinician ratings. Schizophr Bull. 2020; In Press. This reference provides evidence for external validity and the challenges in establishing external validity in digital phenotyping.
Torous J, Staples P, Onnela JP. Realizing the potential of mobile mental health: new methods for new data in psychiatry. Curr Psychiatry Rep. 2015;17:602.
Conflict of Interest
Terje B. Holmlund has received research funding through a grant from the Research Council of Norway, but all other authors have no potential conflicts of interest to disclose.
Human and Animal Rights
All reported studies/experiments with human or animal subjects performed by the authors have been previously published and complied with all applicable ethical standards (including the Helsinki declaration and its amendments, institutional/national research committee standards, and international/national/institutional guidelines).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article is part of the Topical Collection on Psychosis
About this article
Cite this article
Cohen, A.S., Cox, C.R., Masucci, M.D. et al. Digital Phenotyping Using Multimodal Data. Curr Behav Neurosci Rep (2020). https://doi.org/10.1007/s40473-020-00215-4
- Digital phenotyping
- Machine learning
- Serious mental illness