1 Introduction

The indicators concerning substance use and mental health in the United States account for an unseen number of cases since 2017 on psychological and behavioral disorders. An estimated 47.6 million adults aged 18 or older had any mental illness, in which 11.4 million adults presented episodes that substantially interfered with one or more daily activities (SAMHSA 2019). Not enough, the world has faced the pandemic of Covid-19, which induces the alertness of possibilities to aggravate these episodes (Huang and Zhao 2020). Nevertheless, this situation also encourages a better understanding of the broad spectrum of disorders, searching for particular prevention, early diagnosis, treatments, and prognosis.

Interdisciplinary efforts concerning human phenotypes have contributed to this understanding. Human phenotype regards individuals’ intrinsic and observable characteristics, and phenotyping fosters these characteristics’ employment toward valuable information to clinical and biomedical research. In turn, deep phenotyping focuses on individuals’ biological specificity derived from novel types of data through algorithms to identify potential connections between phenotypes, genetic variations and biological data, and disease subtypes (Delude 2015). In this sense, opportunities due to the extensive amount of information source provided by the usage of mobile and wearable devices (Editorial 2015), and the continuous spread of unobtrusive sensors, actuators, computing, and interactive interfaces (Amft et al. 2020), also account for improvements in the precision medicine (Torous et al. 2016).

In this context, related studies have been conducted to elucidate literature’s advancements, mainly regarding mobile and wearable devices. Benoit et al. (2020) focused on machine-learning techniques applied to smartphone data to assess symptoms of psychosis spectrum illnesses. Montag et al. (2021) reviewed mobile sensing to neuroscientific phenotyping research to outline the potential of this combination. Saccaro et al. (2021) provided the application of portable or wearable digital tools for bipolar diagnosis, describing population, technologies, and computational approaches. Mendes et al. (2022) systematically identified sensing applications and data sets for digital phenotyping of mental health. Finally, Tomicic et al. (2022) explored ethical, legal, and social challenges relevant to the implementation of digital phenotyping technologies in healthcare.

Contributing to these researched areas and aligned with big data magnitude, Ubiquitous Computing (UbiComp) encourages phenotyping research given its inherent features, such as computational artifacts physically embedded into the natural environment, both indoor and outdoor (Bardram and Matic 2020). This provides a relevant aspect regarding additional information from individuals’ ecological experience about biological, psychological, and social phenomena (Borsboom 2017; Pierleoni et al. 2021). These phenomena comprise the multitude of psychopathology symptoms and provide daily analytical measurements to complement medical information recorded in clinics. Different from related literature, this study discusses the extension and complement of phenotyping concerning the combination of intelligent environments and personal mobile devices.

Therefore, this article first describes human phenotype through the lens of computational range, presenting an approach that considers environmental, social, and psychophysiological dimensions capable of being reached by UbiComp. Furthermore, the article reviews the state-of-the-art of computational phenotyping, revealing the course in which computing has approached the characterization of phenotype. Besides, this study emphasizes the role of UbiComp in light of clinical and biomedical research. Afterward, the study addresses research opportunities that originated from technical, managerial, and ethical challenges.

The remainder of this article is organized as follows. Section 2 details dimensions around human phenotype and how they are related to ubiquitous technology. Section 3 reviews computational phenotyping, and Sect. 4 discusses the relevance of ubiquitous computing for phenotyping. Section 5 presents challenges and opportunities regarding the combination of these research fields. Finally, Sect. 6 approaches the final remarks of this work.

2 Human phenotype

Encyclopedia of Biodiversity presents a historical perspective of phenotype, where the author refers to the term as observable characteristics being a product of the interaction between the inherited material, that is, genetics, and the environment (Berry 2001). These characteristics denominate phenotypic traits, and they encompass an organism’s physical appearance, function, or behavior, including descriptions of non-visual properties such as physiology. The usage of human phenotype in clinical and biomedical research contributes to the understanding of phenotypic abnormalities associated with diseases with a focus on prevention, treatment, and prognosis (Robinson 2012).

In light of clinical psychology and psychiatry literature, studies indicated that biological, psychological, and social factors affect the development of psychological and behavioral disorders (Borsboom 2017). Figure 1 depicts a reinterpretation of the complex interaction between these factors encompassing physical and virtual experiences. This reinterpretation employs a bidirectional interrelationship of social, environmental, and psychophysiological dimensions in respect to the biological. The interrelationship introduces the environmental factor and proposes a synergy among biological and psychological dimensions through the inclusion of psychophysiology. The biological dimension (A) concerns individuals expressing their intrinsic characteristics guided by genetics, that is, their genotypes. These characteristics are observed from situations in light of the environmental (B) and social factors (C), which may stimulate psychophysiological responses.

Fig. 1
figure 1

Interrelationship of biological (A), environmental (B), social (C), and psychophysiological (D) dimensions. Biological concerns individuals with their genotypes and developmental form. Environmental and Social consider virtual and physical experience where biological reveals itself. Psychophysiological features the reflexes on mind and body unleashed by the three first, but also characterizes the way individuals affect those factors. In combination, these factors reveal the portrayal of possible phenotypic traits relevant to characterize ecological phenotypes

Full size image

The physical environment surrounds the most diverse indoor and outdoor circumstances, encompassing frequented places and movements from one to the other. The virtual environment represents interactions with mobile applications due to the widespread adoption of digital devices, particularly smartphones and smartwatches.

The social factor related to the physical environment regards personal relationships and how individuals behave in front of people in everyday environments. The virtual social factor comprises relationships on social media and virtual communities, and the respective social behavior considering the digital immersion, hence, adherent to a virtual environment.

Psychophysiological dimension (D) depicts the individuals’ psychological interpretation of life situations according to genotype and developmental information, their emotions and cognition, and the respective reflexes on the human body. Hence, this dimension leverages the usage of psychological (E) and physiological (F) responses, which are the underlying phenomena of psychophysiology.

Psychophysiology is a discipline concerned with the scientific pursuit of understanding human processes derived from the study of social, psychological, and behavioral phenomena related to and revealed through physiological principles (Cacioppo et al. 2001). In this sense, psychophysiology intersects smoothly with the human phenotype research since they meet in the biological and behavioral systematic observations concerning advances in the understanding of diseases. Besides, this discipline brings to light cognitive processes relevant to psychological and behavioral research. Hence, this intersection may provide directions into the study of the course of the genotype-phenotype correlation and how this course bidirectionally affects the environment. These directions emphasize recent literature gaps regarding physiology when mentioned the need to incorporate novel knowledge into the phenotyping (Weng et al. 2020).

In turn, phenotyping organizes phenotypes’ characterization, where interrelationship of social, environmental, and psychophysiological constitutes a holistic perspective of daily phenotypic traits. In this respect, the interrelationship approaches the phenotype in the reach of being gathered by computational methods, and capable of contributing to psychological and behavioral research. For example, researches have investigated physiological measurements as markers of psychological stress using smartphones and wearables (Smets et al. 2019). This investigation emphasizes the relevance of conducting studies in front of ecological situations, which indirectly approach a multitude of environmental and social involvement. Hence, this kind of study meets the inherent characteristics of phenotypes gathered by computational methods.

3 Computational phenotyping

Deep Phenotyping constitutes a broad research area regarding characterization and interpretation of phenotypes’ components, focusing on genotype and phenotype correlations according to individual characteristics (Robinson 2012; Delude 2015). Computational Phenotyping, in turn, provides algorithmic capabilities to deal with semantic representation, integration, and processing of phenotype information, which assists in the deep phenotyping research. In this context, two special issues have reviewed computational capabilities toward the advances in precision medicine concerning phenotyping. The first highlights the privacy preservation of personal genotype and phenotype, in addition to algorithms development for phenotyping, disease recognition and classification, and drug repositioning (Shen et al. 2019). The second features natural language processing for richer phenotypes, standardization beyond terminologies, the temporal trajectory of a phenotypes and sub-typing, besides identifying connections between diseases and the common phenotypic traits, and novel data for phenotyping (Weng et al. 2020). Regarding this latter, the reviewed studies originated from a diverse available information source, such as electronic health records, clinical and medical data, pathology reports, questionnaires, cardiovascular, and voice measurements.

Additionally, deep phenotyping has received two similar research disciplines derived from personal mobile devices and the respective information source, namely mobile sensors and smartphone usage patterns. The first discipline is named Digital Phenotyping, referring to the moment-by-moment quantification of the individual-level human phenotype in everyday life, using data from personal mobile devices (Torous et al. 2016). Similarly, Digital Phenotype refers to the redefinition of disease expression regarding individuals’ lived experience regarding wearable and mobile devices. This latter proposes that health-related data from digital devices and social media can shape human illness assessment. It can also expand the ability to understand diseases, using phenotypes as an underlying aspect (Jain et al. 2015).

Regardless of disciplines’ nomenclature, aforementioned works (Torous et al. 2016; Jain et al. 2015) leverage personal mobile devices as an information source for phenotyping since these devices present a high potential of providing longitudinal reflexes of ecological traits. Furthermore, mobile devices provide objective and subjective measurements of the experienced situation (Torous et al. 2016). The objective regards direct measures from the device’s sensors and occurs passively, characterizing unobtrusive gatherings concerning individuals. The subjective considers an active way to retrieve personal perspectives on ecological situations, typically through questionnaires. These measurements benefit information gathering since they are employed as close as possible to real-life contexts.

During the last decade, studies have employed these objective and subjective methods contributing to psychological and behavioral research (Bardram and Matic 2020). These researches reiterate where benefits from psychophysiology may be employed. This adoption considers the usage of wearable devices for physiological measurements primarily, hence, promoting opportunities to describe phenotypic traits regarding the psychophysiological dimension. For example, correlations between psychological states and physiological arousal have been employed supporting psychological and behavioral research (Bavaresco et al. 2020; Aranda et al. 2021).

More specifically, a review with a focus on sensing and computing techniques for digital phenotyping has approached challenges and opportunities in the concern of mental health (Liang et al. 2019). Throughout the review, the authors discuss large-scale data sensing, cognitive, emotional, and sentiment analysis, behavioral anomaly detection, social interaction, and biomarker analysis. The study also provides opportunities for knowledge representation, feature engineering, usage of advanced computational learning, non-intrusive human-computer interfaces, besides the accounting of missing applications for clinical diagnosis and treatment. In addition to these concerns, a recent viewpoint characterizes the combination of deep phenotyping and digital phenotyping as a promising proposal to understand diseases. This is because it benefits large clinical applications through advanced algorithms and data integration about behavioral and human-computer interactions (Fagherazzi 2020). Contributing to this integration, UbiComp has a prominent role in retrieving and gathering phenotypic traits due to computational capabilities concerning ecological observations. This role leverages ubiquitous computing resources to understand psychological and behavioral aspects that influence the course of health and disease.

4 Ubiquitous phenotyping

Individuals’ phenotypic traits are observed ubiquitously by nature, where virtual and physical environments encompass the experience of situations. Likewise, psychological and behavioral disorders manifest in most diverse forms according to social and environmental factors besides the individuals’ ability to cope with situations. For example, chronic stress or uncontrolled anxiety associated with social relationships do not rely only on physical experiences, such as conflicts at the workplace or home. These situations unleash from a combination of experiences and even manifest themselves in virtual communities.

This reveals the relevance of using ubiquitous technologies to describe phenotypic traits extending information gathering close to the natural experience. Therefore, ubiquitous phenotyping provides a view boosting phenotyping through opportunities from emerging fields. This harnesses the fact that UbiComp features invisible computing as an inherent step of artifacts physically embedded into the natural environment. Although invisible from a human perspective, these artifacts accompany individuals during real-life where the retrieval of phenotypic traits may be seamlessly employed to contribute to a series of biological and medical research.

In this sense, Fig. 2 depicts ubiquitous technologies characterized by outdoor and indoor observations that, in combination, provide opportunities to retrieve daily situations. More importantly, these technologies allow that systems become aware of physiological and psychological responses. Thus, the figure emphasizes personal mobile devices and intelligent environments as two research areas capable of providing information on social, environmental, and psychophysiological dimensions. In this figure, the biological dimension depicted in Fig. 1 is analogously reiterated through the individuals’ image into the circles.

Fig. 2
figure 2

The combination of personal mobile devices and intelligent environments in the complement of high-dimensional and heterogeneous ecological observations. Personal mobile devices are ubiquitously employed during outdoor activities, whereas intelligent environments provide indoor measurement capability, harnessing smart home, smart hospital, and smart factory research. This encourages the ubiquitous application of phenotyping research

Full size image

According to the field of digital phenotyping (Torous et al. 2016), personal mobile devices represent a significant amount of computational devices employed outdoors that may accompany individuals wherever they are. This is because mobile devices are suitable to be always along with the person. For example, smartphones, tablets, smartwatches, or smart bands, usually are non-transferable from one person to another, and they rely on multimodal capabilities as an information source. In particular, sounds and voices, images, texts, locations, activities, screen gestures, scrolling, and applications’ usage patterns. Moreover, smartphones, tablets, and smart wrist devices are commercially available, whether compared to other specific wearables or physiological devices. In other words, smart glasses, smart t-shirts, and other smaller usable devices, for example, speakers or smart earplugs, currently foster research and market competitiveness (Brush et al. 2020).

In this respect, wearable devices leverage psychophysiological research due to the possibility to incorporate physiological measurements as an information source, such as the electrocardiogram (ECG), electrodermal activity (EDA), electromyogram (EMG), and electroencephalogram (EEG). Although most wrist devices have already incorporated heart-rate sensors, the remainder of physiological possibilities is upon research for large-scale natural usage. In other words, laboratory studies have advanced in the employment of, for instance, electrodermal activity and electromyogram (Smets et al. 2019), and according to the miniaturization and unobtrusiveness of them, researches will benefit from their daily usage (Pierleoni et al. 2021).

In addition to outdoor technologies, ubiquitous phenotyping looks to the advances in the field of intelligent environments. This field features ubiquitous sensors, actuators, and wireless communication networks to provide smart capabilities and amenities around individuals (Acampora et al. 2013; Röcker et al. 2014). For these reasons, UbiComp is a representative part of this field alongside artificial intelligence, and three examples of application are capabilities employed in smart homes, smart hospitals, and smart factories. Consequently, these capabilities meet personal mobile devices in the benefit of phenotyping regarding the limitations that mobile devices present in front of indoor situations’ characteristics. For instance, when situations hamper mobile devices’ use due to working tasks or by the use of another kind of device. Besides, activities that require putting mobile devices standing near, and problems in tracking activities using these devices due to high-intensity movements or missing connection characterize limitations. Moreover, indoor technology also allows this analysis of movements through body gestures, audio, and current location, providing patterns and traits of daily actions such as sociability, engagement, loneliness, aggressiveness, stressful or pleasant (Acampora et al. 2013). As more devices and machines become digitally involved and more innovative, the operational activities, gestures, movements, and usage patterns of these devices also may foster a similar analysis employed with smartphone applications (Altilio et al. 2021).

Additionally, Fig. 2 outlines the capabilities of intelligent environments regarding the unobtrusiveness that these technologies present. In particular, the figure highlights passive measurements from environmental sensors, cameras and microphones, and devices for wireless monitoring. Besides, active measurements are represented by human–machine interfaces, such as the usage of smart devices and smart machinery. These capabilities produce opportunities considering novel dimensions of information retrieval to complement digital phenotyping and clinical data.

The adoption of these different but complementary fields enhances deep phenotyping with a holistic perspective of ecological phenotypes. This is relevant in the study of diseases since normal and abnormal behavioral, psychological, and physiological responses propagate themselves naturally in the most diverse environments (Ebner-Priemer and Santangelo 2020). For example, ubiquitous phenotyping may investigate different environments during adolescents’ development prone to anxiety or major depression. More specifically, this investigation may indicate where symptomatology interferes with daily activities, characterizing a possible distinct type of disorder. In this respect, accompanying individuals during work environments may help discover traits associated with occupational diseases and work-related health problems.

In summary, Fig. 3 depicts an overview of topics discussed in the reviewed literature in light of three main aspects that characterized this study. These aspects are Ubiquitous Phenotyping (A), Computational Phenotyping (B), and Phenotyping and Deep Phenotyping (C). The figure details each aspect according to diverse topics that they comprise through a taxonomic organization. The first (A) and the second (B) intersect smoothly since both derive from computer science and engineering. The third (C) encompasses psychological and behavioral besides biomedical research, which originate subjective, objective, and biological knowledge to characterize phenotypes. Together, these aspects constitute an interdisciplinary research pathway derived from ubiquitous phenotyping.

Fig. 3
figure 3

Taxonomic organization of the interdisciplinary research pathway of ubiquitous phenotyping. The Ubiquitous Phenotyping (A) comprises technological and clinical approaches for gathering information. Computational Phenotyping (B) presents research trends derived from the literature in computational capabilities using data from (A) in support of Phenotyping and Deep Phenotyping (C). In turn, this latter encompasses high-level information focusing on psychological and behavioral research and low-level information regarding biomedical research

Full size image

5 Challenges and opportunities

Efforts in the employment of UbiComp in social, environmental, and psychophysiological dimensions to characterize phenotypes, although promising, present technical, managerial, and ethical challenges. These efforts naturally involve interdisciplinary research, emphasizing the need to interchange knowledge between researchers from the biomedical area, engineering, and computer scientists (Fig. 3). This is because biological and medical are prevalent in the underlying aspects that will shape computational investigation to employ ecological phenotypes in prevention, diagnosis, and prognosis (Ebner-Priemer and Santangelo 2020).

In respect of this investigation, a technical challenge involves defining and representing the discovered information of phenotypic traits computationally and link this information with knowledge already existent in biomedical science. More specifically, this regards meaningfully understanding a person’s traits and their particularities in the most diverse physical and virtual environments and the correlations with possible diseases. Also, the extensions of these correlations may help discover cause and effect derived from phenotypic traits to disease. This is also a challenge because individuals differ in interpreting situations, physiological responses, and associated behavior. For example, the cause and effect of behavioral and physiological phenomena concentrate on understanding how and why specific environmental conditions unleash a particular psychological response. On the other hand, this also leads to an opportunity because although individuals are different in their traits but considering they inherit genetic material, a potential future scope may regard whether behavioral and physiological traits collected by ubiquitous resources are hierarchically and genetically propagated. Thus, this may produce potential insights regarding progeny diseases’ predisposition, facilitating possible inferences using computing for diagnoses and prognosis.

Another technical challenge comprises the intersection of information between personal mobile devices and intelligent environments. The intersection regards indoor and outdoor patterns that, in combination, assist in the holistic perception of situations and understanding of diseases. This combination encounters similar challenges of the high-dimensional and heterogeneous information source concerning the employment of digital phenotyping (Liang et al. 2019). For instance, the correlation of smartphone usage patterns or physiological measurements in respect of images captured from indoor cameras presents the need for timing and synchronization. Besides, ecological observations are noisy and may present missing information due to the sensors’ inaccuracy (Bardram and Matic 2020). Nevertheless, technical opportunities arise from these sources of information considering artificial intelligence methods and techniques. In particular, computational learning and reasoning, pattern analysis, and predictions may be employed for personalized behavior understanding, psychological and physiological correlations, and to identify environmental and social conditions.

Managerial challenges refer to technology’s employment in the everyday workplace environment, such as hospitals, factories, or even homes. Particularly in industries, despite adopting the industrial internet of things, there is a gap in considering sensing technologies to benefit workers’ health. Managers should be aware that observations during work activities, such as physiology and behavior, may help understand occupational and work-related diseases. This potentially presents opportunities for short- and long-term improvements regarding individuals’ health and productivity. Therefore, investments in this kind of observation are valuable both for individuals and companies. Furthermore, in the context of indoor monitoring, even with the widespread adoption of intelligent devices, this segment also accounts for research challenges, according to discussions about the usage of physiological devices, comfort, and privacy in monitoring (Brush et al. 2020).

In light of medical sciences, the research about ubiquitous phenotyping faces ethical challenges (Muller et al. 2021). This is because individuals’ ecological observations enter on particular aspects of privacy and confidentiality. The first regards authorization of personal information collection and usage, both indoors and outdoors. The second refers to having access to these data, which considers any individual or system. Moreover, if on the one hand, novel mobile and wearable technologies have been created to decrease device obtrusiveness and produce user acceptance regarding data sources, on the other, systems and applications may become invasive as long as computing artifacts begin to understand individual traits. This understanding, primarily leveraged by large-scale adoption of artificial intelligence on ubiquitous devices, also accounts for potential novel and unforeseen threats. This is why the interdisciplinary research community and practitioners must ensure some requirements in front of information collection and usage, for instance, safety, traceability, transparency, and validity (Holzinger et al. 2021).

In this sense, different data sources produce various degrees of obtrusiveness that instigates discussions. Retrieving objective and straightforward measurements such as physiological activation or intensity of movements seems less obtrusive than understanding behavioral habits as frequented places and daily routine. Because the first is less expressive and semantic than the second, and this second is also human-readable information. Therefore, beginning researches by those with less intrusiveness may become individuals adherent to the research, providing confidence to them as long as research advances. Besides, measurements collected for the specific purpose of an individual’s well-being may potentially be uncomplicated for users to consent whether compared to more general proposals.

Regardless of the obtrusive degree or purpose, any data collection and processing must have consent. The request for consent preceded by a didactic explanation may favor individuals to accept the collection of information. Explanations about the research’s final objective and the pathways to this become the process transparent, favoring awareness and acceptance by the individual. Further, it may be interesting for the individual to control when to enable and disable data collection at a system level. However, this generates a computational problem of frequent missing data, which also leads to technical challenges.

Finally, the storage of daily observation is also a challenge. Both individuals or research practitioners may own this information. The first provides a natural pathway, and it is positive since the information originated from individuals themselves, whereas the second requires additional consent due to confidentiality. Also, in the scenario of a workplace environment, companies may desire to have ownership of this knowledge, which may be explicit between the involved. These possibilities should consider storage strategies since they impact information retrieval and processing for indoor and outdoor data sources.

6 Conclusion

In view of the interdisciplinary clinical and biomedical research, computing allows sensing from a holistic perspective regarding indoors and outdoors, which present a relevant role to characterize phenotypes. With adequate conception and interpretation of phenotypic traits, this perspective will promote the consequent understanding of diseases. In addition to individuals’ historical information leveraged by computational analysis, this understanding advances toward possible correlations of phenotypes and the genetic predisposition related to health or disease.

According to phenotype characterization and interpretation advancements, computational alongside biological research potentially leverage the focus on genetics approaches concerning the understanding of genotype-phenotype linkage. This reflects an interest in genes’ roles on individual differences and similarities underlying health or disease, in light of the holistic perspective of social, environmental, and psychophysiological dimensions. Hence, the branch of psychology and psychiatry particularly will receive benefits from ubiquitous computing.