Background

Mental disorders frequently originate in early periods of life [1], and are associated with significant burden and personal suffering [2]. Anxiety, depression and posttraumatic stress disorder are prevalent in children and adolescents, with increasing incidence rates worldwide [3,4,5]. Without effective and timely treatments, these common mental disorders tend to continue into adulthood, with augmented risks to develop further mental disorders and other detrimental outcomes in the long-term [e.g., 6].

Empirically-supported treatment options for common mental disorders in youth (0–18 years) embrace psychotherapy conveyed face-to-face [e.g., 7] and pharmacotherapy [e.g., 8]. In particular, cognitive-behavioral therapy (CBT) has comprehensively documented its effectiveness and acceptability in the treatment of anxiety disorders [9], depression [10, 11] and PTSD in children and adolescents [12]. However, the availability and uptake rates of conventional evidence-based psychotherapies remain low [13], due to different individual and structural barriers, like shortage of mental healthcare supplies, long-waiting times, limitations in mobility, and high treatment costs or fear of stigmatization [14, 15]. Thus, innovative and scalable solutions to lessen this treatment gap are urgently needed; and these novel interventions have to prove their empirical support in terms of various aspects of evidence-based medicine/psychotherapy, such as efficacy and effectiveness, quality or intervention safety.

Internet- and mobile-based interventions (IMIs) hold the potential to lessen existing gaps in mental healthcare [13, 16], as they feature several advantages that might help to overcome some of these obstructions to treatment. First, structural barriers might be reduced, as IMIs can be implemented irrespective of the restraints of space and time [17, 18]. Second, IMIs are deemed to be cost-efficient, as they might be scalable for large numbers of patients [19, 20]. Third, IMIs might also diminish some individual barriers, because these interventions can reduce stigma threat by preserving anonymity, and allow patients to integrate them flexibly into their daily lives at their own pace and individual progress [13, 21]. Furthermore, the digital delivery mode of these novel interventions might be especially appealing for youth, who are frequently familiar and acquainted with the Internet and its applications on mobile devices from an early age [13, 22].

The evidence-base for the efficacy of internet-based psychotherapeutic interventions for some common mental disorders in children and adolescents is documented by several meta-analyses so far [13]. The strongest empirical support can be found for anxiety disorders in youth with effect sizes ranging from 0.30 to 0.77 (standardized mean differences (SMD); 95% CI − 0.53 to 1.45) derived from five meta-analyses [23,24,25,26,27]. Furthermore, the efficacy of internet-based interventions for depression in youth is documented by four meta-analyses [23, 24, 27, 28], with effect sizes in the range from 0.16 to 0.76 (SMD; 95% CI − 0.12 to 1.12). Aside from this meta-analytical evidence, there is are also preliminary indications that internet-based interventions might be acceptable [29] and cost-effective [30], and hold the potential to extend service supplies within a framework of stepped mental healthcare [29, 31].

Given the efficacy of internet-based psychotherapeutic interventions and the worldwide dispersal of smartphones, mobile health applications (MHA) become more and more appealing as a way to complement or deliver psychotherapeutic support [19]. Recent meta-analyses indicate that also app-based psychotherapeutic interventions are efficacious [32, 33]: For instance, for depressive symptoms (SMD = 0.28, 95% CI 0.21 to 0.36), generalized anxiety symptoms (SMD = 0.30, 95% CI 0.20 to 0.40) [32] or smoking behavior (SMD = 0.39, 95% CI 0.21 to 0.57) [33]. However, studies provide mainly evidence for adults. Of the k = 66 [32] and k = 19 [33] included studies in the meta-analyses on the efficacy of app-based psychotherapeutic interventions, only one focused on youth [34].

The limited evidence on mobile health apps for children and adolescents seems like the more important, as there is already a vast (evidence-free) commercial app market accessible in routine care: In an international cooperation study, the analysis of N = 1.299 commercially available MHA yielded, that 94.8% were not evidence-based [35]. Besides major limitations towards the evidence-base in the app markets, systematic evaluations about the quality of available MHA highlight a high heterogeneity and weak to moderate quality of MHA, especially regarding the engagement of users and information quality [16, 35,36,37,38]. Moreover, privacy and security features of MHA available in the app markets have been identified as crucial issues for patients’ safety [39, 40].

Hence, it is of utmost importance to evaluate the apps available in the commercial app market to highlight the few high-quality, secure and scientifically evaluated MHA to inform and guide users as well as healthcare professionals. Standardized and reliable instruments like the Mobile Application Rating Scale (MARS) [35, 41, 42] or the ENLIGHT instrument [43] that offer an assessment for quality, privacy features, and evidence-base enable such evaluations. Although previous reviews have systematically evaluated health apps for depression [16] and PTSD [36] in adults, to our knowledge, no such study was conducted relating to mobile-based interventions targeted for common mental disorders in children and adolescents to date. Therefore, this study aimed to systematically evaluate the quality of existing apps that were explicitly designated to address symptoms of anxiety, depression and PTSD in youth, by deploying two established psychometric app rating scales [41,42,43]. Specifically, we aimed to:

  1. 1)

    systematically evaluate the quality of eligible apps in the European Google Play Store and Apple App Store regarding user engagement, functionality, aesthetics, and information content;

  2. 2)

    assess privacy and security features;

  3. 3)

    identify theoretical foundations of apps and their respective intervention components;

  4. 4)

    assess the concordance of user ratings and expert ratings.

Methods

Search strategy and inclusion criteria

A systematic search for MHA targeting anxiety, depression and posttraumatic stress disorder (PTSD) in children and adolescents was conducted by using an automatic search engine (webcrawler) that was developed within the mobile health app database project (MHAD; http://mhad.science/ [16, 36]. The webcrawler is a program that searches the app stores for given search terms and automatically extracts the provided information (e.g., app name, app description, user ratings, etc.) from the stores. The functionality and validity of the program has been proven in previous studies [44,45,46]. For further technical details please see [47].

In the present study, three sets of search terms were used to identify MHA anxiety, depression and PTSD in youth (age range: 0–18 years; Additional file 1). The search terms were derived by a team of psychologists (ASE, MD, YT), among them a licensed psychotherapist for children and adolescents (MD) and an expert for MHA quality (YT; [16, 35, 36]). The systematic searches (both, in English and German) were conducted from April 24th to April 26th, 2019.

After the app identification, a two-step inclusion process was conducted. First all identified apps were screened based on the app title and description regarding the following inclusion criteria: (a) the app was developed to address at least one of the targeted mental disorders, (b) the app was focusing on children and/or their caregivers, (c) the app was available in a language spoken by the authors (German or English), (d) the app was available for download. Second, all remaining apps were downloaded and tested against the criteria a-c again and e) the app was operative. All eligible MHA were included in the current analysis.

Assessment

The Mobile Application Rating Scale (MARS German version [42]) and the ENLIGHT are two standardized and reliable scales to assess the contents and quality of MHA. In the present analysis app characteristics (e.g., app name, price, user-rating) were assessed using the classification site of the standardized MARS [41, 42]. In addition, the classification site was used to assess methods (e.g., mindfulness exercises) and the therapeutic background. Based on previous categorizations [16, 48] therapeutic background was rated as cognitive behavioral therapy (CBT), psychodynamic psychotherapy, behavioral therapy, systemic therapy, third-wave CBT, humanistic therapy, integrative therapy, other, not applicable (N.A.; e.g., mainly psychoeducational content), or mixed (e.g., elements from multiple therapeutic backgrounds and theoretical orientations).

The MARS offers several items to assess privacy and security features (e.g., password protection) on a descriptive level. All included apps were rated by these items. In addition, the ENLIGHT instrument contains four checklists (i.e., privacy, security, credibility, evidence-based score), providing further in-depth assessment domains. For privacy and security, explicit requirements were stated, which are either rated as fulfilled or unfilled. Credibility and evidence-base is rated from very poor to excellent. For further information see [41].

Quality rating

The MARS [41, 42] was used for the quality assessment. With the MARS, MHA quality is rated on a 5-point scale ranging from 1 “inadequate” to 5 “excellent”. A total of 19 items on four sub-dimensions were included: (A) engagement (5 items: fun, interest, individual adaptability, interactivity, target group), (B) functionality (4 items: performance, usability, navigation, gestural design), (C) aesthetics (3 items: layout, graphics, visual appeal), and (D) information quality (7 items: accuracy of app description, goals, quality of information, the quantity of information, quality of visual information, credibility, evidence base). The psychometric quality of the MARS is excellent [35, 41, 42]. To assess the evidence base of the included apps, each reviewer conducted a search using Google and Google Scholar (search terms: app name, efficacy, effectiveness, observation study, study, evaluation, usability) and searched on the website of the app and app provider for information on conducted studies. Based on the search results the evidence items was rated as NA (“The app has not been trialled/tested”), 1 (“The evidence suggests the app does not work”), 2 (“App has been trialled (e.g., acceptability, usability, satisfaction ratings) and has partially positive outcomes in studies that are not randomised controlled trials (RCTs), or there is little or no contradictory evidence”), 3 (“App has been trialled (e.g., acceptability, usability, satisfaction ratings) and has positive outcomes in studies that are not RCTs, and there is no contradictory evidence“), 4 (“App has been trialled and outcome tested in 1–2 RCTs indicating positive results”), or 5 (“App has been trialled and outcome tested in > 3 high quality RCTs indicating positive results”) [41]. In addition to the four objective quality scales, the MARS contains two subjective subscales: (E) subjective quality (4 items: recommendation, frequency of use, willingness to pay, overall star-rating) and (F) perceived impact (6 items: awareness, knowledge, attitudes, intention to change, help-seeking, behavioural change).

In addition to the MARS, the ENLIGHT instrument was applied, since it provides information on additional dimensions (e.g., therapeutic alliance). ENLIGHT’s items are rated on a 5-point scale ranging from 1 “very poor” to 5 “very good”. In total ENLIGHT covers 28 items on seven sub-dimensions: (a) usability (3 items), (b) visual design (3 items), (c) user engagement (5 items), (d) content (4 items), (e) therapeutic persuasiveness (7 items), (f) therapeutic alliance (3 items), and (g) general subjective evaluation (3 items). The internal consistency of ENLIGHT is excellent [43].

Rater training

The quality rating was conducted by two independent reviewers (ASE, SE). All reviewers completed online training for the MARS before the quality assessment (https://www.youtube.com/watch?v=5vwMiCWC0Sc; last updated on September 6, 2021). For the ENLIGHT instrument, no free online training is available. Instead, reviewers were trained in using the ENLIGHT by an researcher with prior experience and expertise in app quality assessment (YT; [16, 35, 36]. After the instruction, several test apps were rated by both reviewers (ASE, SE) and ratings were discussed between the reviewers and instructor (YT) to complete the training phase. The excellent inter-rater reliability between the trained reviewers for the MARS (ICC = 0.87, 95% CI 0.84 to 0.90) and the inter-rater reliability for the ENLIGHT (ICC = 0.75, 95% CI 0.65 to 0.83) highlight that the training was successful.

Data analyses

MHA characteristics, therapeutic backgrounds, methods, and privacy and security features were analyzed descriptively. As a measure of agreement between the trained reviewers, intra-class correlation (ICC) was calculated for the dimensional quality assessment [49]. An ICC above 0.75 was defined as a satisfactory agreement between the reviewers [50]. The ratings of both reviewers were pooled. After pooling, means and standard deviations were calculated for all quality dimensions of the MARS and ENLIGHT. The associations between user ratings (star-rating in the stores: 5-point rating from 1 star to 5 stars) and experts’ quality ratings with the MARS and ENLIGHT were investigated by correlation analysis. For correlation analysis, an alpha level of 5% was defined. Holm correction for multiple testing was applied [51]. Missing values were excluded list-wise.

Results

Search and app characteristics

In total, 3806 MHA were identified (Apple App Store = 2658, Play Store = 1148). After the eligibility check, 15 MHA were included. All MHA were developed for Apple devices and n = 7 MHA were also available on Android. The inclusion process is summarized in Fig. 1.

Fig. 1
figure 1

Flow-chart of the in- and exclusion process

Full size image

Most MHA (n = 9, 60%) were from the category Health and Fitness, followed by n = 6 (40%) from the category Medicine. Lifestyle (n = 3), Education (n = 2) and social networks (n = 1) were represented categories as well. Of the 15 included MHA, two targeted depression, four anxiety, three PTSD. The remaining six targeted mixed symptoms of anxiety, depression and PTSD. Most MHA were freely available (n = 9, 60%). Prices of the six fee-based MHA (40%) ranged from 1.09 EUR to 10.99 EUR. On average, users rated the included MHA moderate to good (M = 3.6; SD = 1.04); three MHA were not rated by users. MHA characteristics are summarized in Table 1.

Table 1 App characteristics
Full size table

The investigation of methods provided by the included MHA yielded 18 different features. Most frequently, MHA provided information (n = 7) and offered psycho-educational advice (n = 7). A total of n = 6 MHA (40%) included a monitoring function. MHA also included therapeutic functions. For an aggregated overview of all functions, see Fig. 2.

Fig. 2
figure 2

Methods and functions of included MHA

Full size image

In four apps the used functions and exercises were taken from multiple therapeutic backgrounds. A clear CBT focus was observed in two MHA, while at least some CBT elements (including third-wave CBT) were present in six of the MHA. Interestingly, a total of three apps incorporated the contact to a therapist or coach, which may lead to different therapeutic backgrounds and applied techniques depending on the assigned therapist or coach. An MHA-specific summary of the provided functions as well as the therapeutic background of the MHA is provided in Table 2.

Table 2 Content summary
Full size table

Privacy, security and evidence

Of all 15 included MHA, n = 6 (40%) offered password protection, n = 4 (27%) required a log-in, n = 7 (47%) had a visible privacy policy, and n = 7 (47%) had an imprint. Consent to data collection was required in an active form by n = 5 (33%) MHA and in a passive form by n = 8 (53%), n = 1 informed about conflicts of interest/financial background. An emergency function was provided by n = 2 (13%) MHA. The ENLIGHT checklist evaluation of privacy, security, credibility, and evidence-base is summarized in Table 3.

In addition to the ENLIGHT checklist, evidence-base was assessed by MARS with the item “Has the app been trialled/tested?”. The assessment revealed, that only a single MHA (“iChill”) has been evaluated and outcome tested in an RCT.

Table 3 Checklist summary ENLIGHT
Full size table

Quality rating

MARS

The overall average quality of MHA was M = 3.59 (SD = 0.50). Quality ratings of the four sub-dimensions resulted in: engagement M = 3.34 (SD = 0.76), functionality M = 3.81 (SD = 0.49), aesthetics M = 3.78 (SD = 0.63), information quality M = 3.42 (SD = 0.68). The subjective quality was M = 3.13 (SD = 0.62) and the perceived impact M = 2.80 (SD = 0.75). Detailed information can be obtained from Table 4.

Table 4 MARS rating
Full size table

ENLIGHT

The overall quality was M = 3.22 (SD = 0.73). Quality ratings of the sub-dimensions yielded: usability M = 3.84 (SD = 0.39), visual design M = 3.72 (SD = 0.55), user engagement M = 3.27 (SD = 0.80), content M = 3.34 (SD = 0.72), therapeutic persuasiveness M = 2.34 (SD = 0.78), therapeutic alliance M = 2.56 (SD = 1.08), general subjective evaluation M = 3.34 (SD = 0.83). For detailed information see Table 1.

Table 5 ENLIGHT rating
Full size table

Associations between measures

Correlation between the overall MARS and ENLIGHT were high (r = .936, p < .001). No significant correlations between either MARS or ENLIGHT with user star rating were found (p > .050).

Discussion

To the best of our knowledge, this is the first study that systematically evaluated the quality of available apps that were denoted to tackle symptoms of anxiety, depression and PTSD in youth. Although our systematic searches yielded in a high number of initial hits, only a very small fraction of apps (0.39%) fulfilled our eligibility criteria and were independently assessed with two established app rating scales. Overall, the results point to a mediocre quality of the 15 included apps, and most importantly, to a widespread absence of rigorous scientific evaluations of apps in RCT-studies.

Our quality ratings are in line with the results of previous systematic evaluations deploying the same rating scales on apps for various health conditions and application areas (e.g., depression, PTSD, pain, rheumatism, physical activity, weight management [16, 36, 38, 44,45,46, 52,53,54]), all indicating medium overall app qualities with the MARS and ENLIGHT scales. In our study, the categories with the highest ratings were aesthetics and functionality (M = 3.78; M = 3.81), followed by information quality (M = 3.42) and engagement (M = 3.34). Also corresponding to previous studies [16, 36], the most often deployed therapeutic background for health apps was CBT, with various intervention components applied such as psychoeducation, relaxation techniques, mindfulness, exposure or monitoring/tracking.

However, and most importantly, a common deficiency of MHA is the absence of empirical investigations on their efficacy [16, 36, 55], which represents a major limitation and confines further quality aspects in our view. Considering this lack of empirical support, it seems like a minor issue that youth, caregivers and mental health professionals are hardly able to disentangle the overabundance of available apps and to distinguish therapeutic relevant apps from those that are irrelevant for their health-related intentions [56]. Moreover, our results suggest that app users might not be able to sufficiently appraise the quality of available apps, given the discrepancy of expert and user ratings, which is consistent with findings from other studies [16]. Thus, it seems necessary that objective and scientifically-based expert ratings are publicly available, in order to enable patients to make informed and guided decisions about MHA in particular, and psychotherapeutic interventions in general, advocating evidence-based mental healthcare.

Next to several strengths (including the resort to established scientifically validated app rating scales, the focus on prevalent mental disorders and comprehensive scope on apps with divergent theoretical backgrounds), there are several limitations that ought to be considered when interpreting the findings of this current study. First, the search strategy embraced only available apps in the European Google Play and Apple App stores, and therefore the results might not be representative for other outlets and app stores. Second, the rapidly evolving dynamics of the app market [16, 36, 57] might have led to the omission of novel apps as well as updates of included apps, and that some of the apps might not be accessible anymore. Third, since the two rating scales were not designed for the specificities of child and adolescent psychotherapy (i.e., addressing relevant developmental issues), ratings of youth MHA should be interpreted with caution—especially in regard to the domains of functionality and information content. Therefore, it would be a significant scientific advancement, if specific MHA rating scales were available and psychometrically evaluated for mobile-based interventions for common mental disorders in youth. Fourth and foremost, quality ratings can never be sufficient to inform clinical practice and treatment approaches alone, since efficacy and effectiveness studies are the core and gold-standard of evidence-based mental healthcare.

Accordingly, the most important future direction and imperative necessity for the advancement of MHA, is their methodological sound and rigorous evaluation in randomized controlled trials (or novel alternative evaluation frameworks to MHA)—since only one app was subject to an efficacy study, with sobering non-significant findings [58]. Without comprehensive knowledge on their (non-)efficacy and intervention safety, these novel applications ought not to be part in routine mental healthcare. However, some stakeholders might compromise some established standards of evidence-based medicine and advocate the evaluation of MHA in routine clinical care – without prior information on their efficacy and harmlessness. This debatable procedure can be observed in the health sector in Germany, where clinicians and healthcare providers are currently entitled to prescribe apps, with the intention of the German health ministry and certifying agency (i.e., BfArM) to gain an evaluation of their effectiveness post-hoc (with app providers contributing these data themselves). As such, this development might bring several risks for individual patients (e.g., non-detected and unknown side effects, adverse events, or deterioration) and might contribute to an erosion of well-established standards of evidence-based medicine in general. This concern is also nourished by the massive commercialization of the app market and limited resources in healthcare globally, raising further methodological and ethical questions [59, 60]. Thus, the adherence to well-established standards of good clinical practices as well as scientific-based intervention evaluation should be urgently applied for MHA [61], similar as it is largely the case for internet-based interventions and conventional psychotherapies offered face-to-face. Recommendations for quality criteria and standards for MHA are available, and can complement efficacy-/effectiveness studies on MHA in RCTs [62, 63]. Furthermore, the development of MHA needs to get theoretically and empirically informed, since dismantling and additive design studies are largely pending in mobile-based interventions [18], and the resort to theoretically deduced and evidence-based intervention components is a critical step to build efficacious interventions that induce essential mechanisms of therapeutic change enabling improved outcomes [64].

Taken together, our study indicates that MHA for youth possess a rather small quality and, more importantly, are insufficiently evaluated concerning their efficacy, impeding widespread conclusions and recommendations for clinical practice and healthcare policies so far. Therefore, it is of utmost importance that patients and stakeholders are thoroughly and scientifically informed about this limited evidence base and the current deficiencies and restraints of MHA for youth. Information platforms offered by (non-profit) organizations, like www.mhad.science (last updated September 6, 2021) or https://onemindpsyberguide.org/ (last updated September 6, 2021), that gather and provide scientific knowledge on central aspects of digital health interventions (e.g., effectiveness, acceptability, quality, safety, or data security) in a balanced and unbiased way, are one important contribution for the evidence-based information and empowerment of patients seeking advice and help for their mental health resolutions.

Conclusions

This study systematically evaluated the quality of MHA for anxiety, depression and PTSD in youth. The results revealed a moderate overall app quality and point to a significant lack of empirically-informed intervention development, as well as an absence of rigorous studies to investigate the efficacy of apps that are currently offered in major marketplaces. Hence, future high-quality research is urgently needed, to advance and evaluate MHA, all contributing to the augmentation and evidence-based utilization of digital health interventions. Furthermore, non-profit provisions of scientifically-based information on essential aspects of apps are urgently required to enable informed and balanced treatment choices for patients, caregivers and stakeholders in mental health.