Efficacy of a Mindfulness-Based Mobile Application: a Randomized Waiting-List Controlled Trial

Participants

Participants were recruited through a Facebook advertisement of the study by a social media agency, targeted at people who were known to have an interest in mindfulness and spirituality. The advertisement was directed away from people who had previously liked VGZ’s Facebook page, to avoid the inclusion of participants who were already familiar with the VGZ Mindfulness Coach. Eligibility criteria included (a) being 18 years or older, (b) having sufficient fluency in Dutch to complete the research procedures and use the VGZ Mindfulness Coach, and (c) being willing to provide written informed consent prior to their inclusion in the study. The study procedures were approved by the institutional review board of the Faculty of Social and Behavioural Sciences of the University of Amsterdam, and the study has been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and its later amendments. The study protocol was preregistered at the Netherlands Trial Register (www.trialregister.nl, TC=5001).

For this study, we wanted to have sufficient power (0.80) for detecting a small effect (f ² = 0.10). Based on a repeated measures ANOVA test, this would require a total sample size of at least 200 participants (p < 0.05; two-tailed). We anticipated a large dropout rate of 50% for this online study, and therefore aimed to include 400 participants. We were able to include a slightly lower number of participants (n = 377), but, since the dropout rate was lower than expected, nevertheless retained enough posttest completers (n = 221).

Procedure

Participants who provided informed consent and met the eligibility criteria were included in the study and were randomly allocated to the experimental or WLC condition. Randomization followed a computer-generated block randomization (blocks of 50) and was carried out by an independent research assistant who was not involved in the statistical analysis or in writing up the study. Participants and researchers were not blinded to the allocated condition.

In the experimental condition, participants were directed to a website where they could download the VGZ Mindfulness Coach from the App Store (https://itunes.apple.com/nl/app/vgz-mindfulness-coach/id779531500?mt=8) or Google Play Store (https://play.google.com/store/apps/details?id=nl.vgz.mindfulness). The study measures were administered before randomization (baseline), 8 weeks after baseline (posttest), and 20 weeks after baseline (follow-up). The 8-week interval between baseline and posttest allowed participants in the experimental condition ample time to complete the 5-week program (see Intervention). Since participants in the WLC condition were offered the VGZ Mindfulness Coach immediately upon completion of the posttest, the follow-up was only conducted in the experimental condition. Participants received no financial or other forms of compensation.

The VGZ Mindfulness Coach is a self-help intervention without any form of automated or therapist-provided guidance or feedback. In our study, participants in the experimental condition were gently encouraged to decide upon the optional 5-week program, and received weekly automated, nonindividualized e-mails to promote their use of the app. The app offers information on how to navigate the various exercises and options of the app (e.g., the filters for length, aim, and setting), but other than the suggestion to follow the 5-week program and the standardized e-mail reminders, participants received no instructions on how the app should be used. The study was conducted completely online, and except for a few e-mails concerning technical difficulties of some participants, there was no contact with participants outside the pre-planned, standardized research procedures.

Measures

Data Analyses

Data were analyzed using IBM SPSS Statistics, version 22 for Windows and MLwin, version 2.30. Since we did not collect adherence data, we were unable to distinguish between study dropouts and intervention dropouts. Participants were therefore classified as dropouts if they failed to complete our primary outcome measure (FFMQ) at posttest or follow-up. All statistical tests were two-tailed.

First, chi-squares were used to determine whether there was a significant association between study condition and dropout rate at posttest. Second, chi-squares and independent-samples t tests were used to evaluate differences between completers and dropouts at posttest (in each study condition) or follow-up (in the experimental condition) on the demographic, meditation (currently practicing mindfulness or other forms of meditation), and outcome variables at baseline. Third, chi-squares and independent-samples t tests were used to evaluate differences between study conditions on these variables at baseline. Fourth, we used multilevel regression analysis to test for within-group (time) and between-group (time × condition) effects. Multilevel regression analysis was used because this is an intention-to-treat analysis in which all the data can remain in the analyses (Hox 2002). Variables that predicted dropout or showed between-group differences at baseline were used as covariates in the analyses. We ran the analyses on the intention-to-treat and on the completer sample.

Fifth, because attrition could have influenced the posttest scores and in turn our estimates of Cohen’s d, we imputed the missing values with a predictive mean matching procedure based on the missing at random assumption (Sterne et al. 2009). Using multiple imputation, we imputed all cases that did not respond to the posttest or follow-up by generating ten separate datasets and replaced the missing values with the mean of these ten sets. Sixth, within-group Cohen’s d effect sizes were calculated with (M _post − M _pre)/SD _pooled. Between-group Cohen’s d effect sizes were calculated from the difference in change scores between the study conditions divided by the pooled standard deviation of these change scores. In the intention-to-treat sample, these Cohen’s d effect sizes were estimated on the imputed sample.

Dropout

Of the 377 participants who met our eligibility criteria and completed the baseline assessment, 191 (50.7%) were allocated to the experimental condition and 186 (49.3%) were allocated to the WLC condition. Across study conditions, 221 (58.6%) completed the posttest assessment. A significant association was found between study condition and dropout rate at posttest (χ ²(1) = 44.702, p < 0.001), with relatively more participants completing posttest in the WLC condition (141, 75.8%) than in the experimental condition (80, 41.9%). In the experimental condition, a significantly greater proportion of posttest completers (69, 86.3%) than dropouts (77, 69.4%) were in a stable relationship (χ ²(1) = 7.356, p = 0.007). In the control condition, a significantly greater proportion of posttest completers (45, 31.9%) than dropouts (7, 15.6%) were already practicing mindfulness (χ ²(1) = 4.533, p = 0.033), and posttest completers (M = 45.32, SD = 10.26) had a significantly higher average age than dropouts (M = 38.98, SD = 9.81) (t[184] = −3.646, p < 0.001). There were no significant associations or differences between posttest completers and dropouts in either condition on the other study variables at baseline.

Within the experimental condition, 50 (26.2%) of the 191 participants that were originally allocated to this condition completed the 3-month follow-up. Follow-up completers (M = 48.99, SD = 8.35) had a significantly higher average age than dropouts (M = 44.44, SD = 9.07) (t[189] = −3.106, p = 0.002). There were no significant associations or differences between follow-up completers and dropouts in this condition on the other study variables at baseline. Figure 1 displays the flow of participants through the study.

Open image in new window

Baseline Equivalence

Outcome

Table 3 shows the imputed mean scores and corresponding Cohen’s d effect sizes for the pre- and posttest measurements. Online Resource 1 shows the observed (e.g., nonimputed) scores, which do not meaningfully differ from the imputed scores. On our primary outcome measure, we observed a significant interaction effect for the FFMQ total score, indicating superior performance of the VGZ Mindfulness Coach over the WLC condition, b = 11.89, SE = 1.94, p < 0.001. In addition, significant interaction effects in the same direction were observed for the FFMQ subscale scores (FFMQ-Observing, b = 3.27, SE = 0.54, p < 0.001; FFMQ-Describing, b = 1.26, SE = 0.59, p = 0.03; FFMQ-Acting with awareness, b = 2.95, SE = 0.59, p < 0.001; FFMQ-Nonjudging, b = 2.19, SE = 0.71, p < 0.01; FFMQ-Nonreactivity, b = 2.16, SE = 0.49, p < 0.001).

On our secondary outcome measures, we observed significant interaction effects for general psychiatric symptomatology (GHQ), b = 4.24, SE = 0.81, p < 0.001; psychological quality of life (WHOQOL-Psychological health), b = 1.03, SE = 0.35, p < 0.01; social quality of life (WHOQOL-Social relationships), b = 0.55, SE = 0.21, p < 0.01; and environmental quality of life (WHOQOL-Environment), b = 0.90, SE = 0.38, p < 0.05. For the following variables, we did not observe significant interaction effects: physical quality of life (WHOQOL-Physical), b = 0.71, SE = 0.49, p = 0.16, and self-actualization (SISA), b = 1.35, SE = 0.74, p = 0.07. We also ran the analyses based on the completer sample at posttest, and found that the same interaction effects reached significance with roughly the same between-group effect sizes (see Online Resource 2).

When correcting for multiple comparisons using the Bonferroni correction (dividing p = 0.05 by the number of outcome variables, i.e., p _corrected = 0.05/12 = 0.004), all significant interaction effects remained significant except for one subscale on the primary outcome measure (FFMQ-Describing, p = 0.03) and one subscale on one of the secondary outcome measures (WHOQOL-Social Relationships, p = 0.009). Of note, the interaction effects on the WHOQOL-Physical Health subscale (p = 0.16) and SISA (p = 0.07) were not significant in the original uncorrected analysis and are therefore not significant in the Bonferroni corrected analysis as well.

Table 2 also shows the imputed mean scores and corresponding Cohen’s d effect sizes for the follow-up measurement in the experimental condition. Compared to baseline, the magnitude of most effect sizes was smaller at follow-up than at posttest, except for environmental quality of life (WHOQOL-Environment) which showed a larger effect size at follow-up. Most within-group treatment effects were still significant at follow-up however. Specifically, on our primary outcome measure, we observed significant within-group effects for the FFMQ total score, b = 13.63, SE = 2.61, p < 0.001; FFMQ-Observing, b = 3.82, SE = 0.68, p < 0.001; FFMQ-Describing, b = 1.32, SE = 0.66, p < 0.05; FFMQ-Acting with awareness, b = 2.56, SE = 0.70, p < 0.001; FFMQ-Nonjudging, b = 2.68, SE = 0.76, p < 0.001; and FFMQ-Nonreactivity b = 3.03, SE = 0.60, p < 0.001.

With regard to our secondary outcome measures, within-group effects were significant at follow-up for general psychiatric symptomatology (GHQ), b = −6.26, SE = 1.00, p < 0.001; psychological quality of life (WHOQOL-Psychological health), b = 1.12, SE = 0.33, p < 0.001; environmental quality of life (WHOQOL-Environment), b = 1.88, SE = 0.43, p < 0.01; physical quality of life (WHOQOL-Physical health), b = 1.33, SE = 0.41, p < 0.01; and self-actualization (SISA), b = 2.85, SE = 0.79; p < 0.001. The only exception was that the follow-up treatment effect on social quality of life (WHOQOL-Social relationships) was not sustained at follow-up, b = 0.34, SE = 0.24, p = 0.15. Again, the same pattern was found for the completer sample at follow-up (see Online Resource 3).

Satisfaction and Frequency of Use

Of the 50 participants in the experimental condition who completed the follow-up, 16 (32.0%) reported to still use the app at follow-up. Of these, seven participants (43.8%) reported to use it several times a week, followed by four participants (25.0%) who used it weekly, occasionally (n = 4, 25.0%), or “monthly” (n = 1, 6.3%).

None of these frequency-of-use indices was significantly related to changes of the FFMQ total or subscale scores at posttest or follow-up, while controlling for baseline and posttest scores, respectively (lowest p = 0.08; details available from the authors).

Limitations

In designing this study, we placed much value on its external validity. Specifically, we tried to keep our screening and measurements as brief as possible to minimize the risk of measurement effects. Moreover, our eligibility criteria were limited to a few formal requirements that are not likely to have led to a selection bias. Inevitably, this focus on external validity contributed to a number of limitations that should be taken into account when interpreting our findings. First, our measurements relied on only self-report questionnaires, and while we controlled for prior experience with the practice of mindfulness and other forms of meditation, these potentially relevant background variables were measured in a limited way.

Second, the present app did not collect tracking data on how participants used and navigated through the app. We therefore had little control over our experimental manipulation and had to rely on self-reported data on frequency of use, which showed no relationships to changes in mindfulness. Of note, more objective adherence data did show such relationships in a recent meta-analysis of e-therapy studies. Specifically, this study found logins to be related to outcomes of physical health interventions and module completion to outcomes of psychological health interventions (Donkin et al. 2011).

Third, we had relatively high dropout rates and have no data on why participants discontinued their use of the app or their participation in the study. Such information might help future researchers to promote continued use of the app and to reduce study dropout. Of note is that our results were robust across the intention-to-treat versus completer analyses, suggesting that dropout did not meaningfully affect the present findings. Also, high dropout rates are not uncommon in e-health research and have even been described as “a natural and typical feature” rather than a limitation of such research (Eysenbach 2005, p. 1).

Fourth, the WHOQOL-BREF scales in our study showed poor (0.6 > α ≥ 0.5 for Psychological health) or questionable (0.7 > α ≥ 0.6 for Physical health, Social relationships, and Environment) internal consistency (George and Mallery 2003), which may have negatively affected our measurement of these quality-of-life aspects.

Fifth, although our study aimed to recruit a nonclinical population, the average GHQ-12 baseline scores were relatively high. To the extent that this limits the generalizability of our findings to healthy populations, it also underscores the potential of the VGZ Mindfulness Coach to reduce clinical symptoms or to prevent sub-clinical symptoms from reaching clinical thresholds. Evidence to date suggests, however, that MBIs and other meditation programs are not a panacea for all clinical outcome domains and may not exceed the efficacy of other active interventions (e.g., Goyal et al. 2014), and it is an empirical question whether the current results will generalize to different clinical samples.

Sixth, our sample was predominantly female. Although this appears to be the rule rather than the exception in similar studies (e.g., Demarzo et al. 2015; Howells et al. 2016; Ly et al. 2014) and in more or less adjoining research areas (e.g., Bower et al. 2013; Cuijpers et al. 2008), it limits the generalizability of our findings to male populations. The same is true for the comparatively high educational levels of our participants.

Seventh, our research team was not formally blinded to group allocation. Although we are confident that this has not influenced our analysis or findings, future studies should rule out this possibility by formally blinding the statistical analysis to group allocation.

Eighth, the study design does not allow us to disentangle a possible effect of the weekly e-mail reminders on frequency of app use or study adherence. For that, replication studies without these reminders are needed or studies with a comparison condition in which participants receive the app but not the reminders. Also, future studies should include placebo control conditions to rule out nonspecific effects as alternative explanations for the app’s efficacy.

In sum, despite these limitations, this study shows that it is possible to achieve durable positive effects on mindfulness, general psychiatric symptoms, and several aspects of quality of life at low costs with self-help MBIs such as the VGZ Mindfulness Coach app. We suggest that future studies remedy the above limitations (by, e.g., ensuring more equal sample distributions of sex and educational level, collecting objective adherence data, and including placebo control conditions) and explore the usefulness of MBI apps in specific clinical populations, for instance, as an adjunct to face-to-face treatments or as a means to prevent relapse after successful psychotherapies. Also, future research might now investigate the mechanisms that underlie the effects of the app, such as the possibility that improvements of psychiatric symptoms and quality of life are mediated by increased mindfulness. Of note, such research should take into account relationships of mindfulness with these and other psychosocial outcome variables may themselves be mediated by, e.g., prior meditation experience (Baer et al. 2008) or other variables.