Background

Youth, referring to a period of transitioning from childhood to adulthood spanning from 12 to 25 years of age, is a crucial time regarding the development of psychosocial capacities and the onset of mental disorders [1]. Specifically, 62.5% of individuals report an onset of a mental disorder before the age of 25 [2]. Thus, it is important to intervene in youth, however, a gap exists between their needs and modus operandi, and available care [3,4,5,6]. The importance of overcoming this gap has extensively been underscored in clinical work as well as research and has led to efforts for international reform of youth mental health [7]. In recent years, mHealth (i.e. mobile health, interventions making use of mobile devices) increasingly provides solutions to bridge this gap. Technology can make psychological interventions more accessible for youth, who are ‘digital natives’ [8]. An abundance of apps that promise to relieve psychological stress and enhance mental well-being are available in major app stores. But only a small number of applications are tested for efficacy [9] and most are not grounded in sound psychological theories, such as cognitive behavioral interventions [10].

Ecological momentary interventions (EMI) are typically digital interventions [11, 12]. However, the development of EMIs was not triggered by the digital transformation, it is rooted in ecological psychology, for which digital advances (e.g. with smartphones) provide new assessment methods and treatment delivery channels. Ecological psychology aims to study experience and behavior in the context of normal daily life. Therefore it uses Ecological Momentary Assessment (EMA) [11]. EMIs take this a step further and assume that experience and behavior are not only situated in, but also most amenable to change in momentary daily life situations [13]. EMIs are designed to intervene in daily life, herewith enhancing the generalizability and ecological validity of the learning situation, and focus interventions to target moments [14]. Moreover, EMIs can prompt behaviour, experiences, and assignments in real life, independant of contact with a therapist.

EMIs were developed for different mental health problems such as mood [15, 16], anxiety [17, 18], and substance use disorders [19,20,21]. In addition, EMIs can focus on transdiagnostic intervention elements [22, 23]. Unfortunately, still a limited number of studies rigorously research the effect of EMIs and several major (transdiagnostic) psychological domains remain unaddressed [24,25,26].

The term ‘just-in-time adaptive intervention’ (JITAI) has also been used in literature to describe an intervention delivered in daily life that is customized to an individual’s state aiming to provide the right type of support varying over time, using information assessed through e.g. EMA [27]. Even though there are similarities between EMI and JITAI (i.e. intervening in the moment when needed and prompted), JITAI emphasizes the specific element of adaptation over time (not inherent to EMI perse) and, hence, may be termed as a subclass of EMI [24]. However, from the outset, EMIs used adaptive strategies, e.g., referred to as interactive tasks in recent studies [28, 29], and as such adaptation of the intervention is inherent to EMIs. For this paper, with the EMI under study delivering interactive tasks, the term EMI will be used.

EMI research

To enhance the field of EMI research an important challenge is the need for high-quality trials [25, 30, 31]. The SELFIE trial targeting self-esteem, and the EMI compass trial targeting resilience through a compassion-focused intervention, are particular examples for targeting mechanisms that may confer the development of mental disorders at a later age [23, 29, 31]. Research moves from knowing that psychological interventions work to understanding how they work [32]. This requires understanding the ‘mechanism’, defined as the roots of the effect, i.e., the processes of events that are responsible for the change; the reasons why change occurred or how change came about” [33]. Thus, e.g. self-esteem may be coined as an ‘active ingredient’ for psychological interventions [34,35,36], and could be targeted as a mechanism of influence in the development and maintenance of psychological disorders [37]. Such a mechanism is hereafter referred to as ‘target mechanism’. Specifically in EMI research, the feasibility and effectiveness of intervening in target mechanisms in daily life (a core element of EMI) is researched. The testing of ecological interventionist causal models [12], offers an addition to existing research in testing whether an EMI can modify a target mechanism in daily life, and secondly test whether producing changes in the mechanism in daily life leads to changes in intended psychological outcomes. Nonetheless, how the target mechanism is being influenced remains unanswered. This influencing of the target mechanism may be seen as a mechanism in itself, defined as an explanatory theory of what elements of (the mode of delivery of) an intervention causes changes in individual behavior relating to the target mechanism [38], and will hereafter be referred to as ‘working mechanism’. Current research should pay attention to the target mechanisms as well as the working mechanisms of the intervention itself to improve the use of EMIs in clinical care [11].

Various working mechanisms have been pointed out in previous work. For example, EMIs are available at any given time and in every given context. Individuals may feel more equipped to apply new behaviours and skills in their actual experience in real life with the extra support of an EMI [14, 39]. Additionally, based on assessment responses, EMIs can tailor (the content of) the intervention to specific needs at specific times, based on, for example, need of additional support, motivation or readiness to change [14]. EMIs can also prompt contextual reminders to trigger specific behaviours in specific contexts [24, 40, 41]. Another established working mechanism is self-monitoring, individuals can track their behaviours and become more aware of them, which is necessary for change [41, 42]. In some EMIs individuals can connect with peers or friends for social support [14, 41]. Furthermore, personalized feedback on app progress can enhance self-awareness and motivation [15, 24, 40, 42], and goal-setting can increase motivation [14, 40]. Further, research has shown that guided self-help and face-to-face treatments may have comparable effects [43], and specifically for EMIs, guidance was proposed as an important working mechanism [10, 44]. Yet, limited research is available on how and for whom this mechanism works. Knowledge of working mechanisms is an important aspect to take into account when further implementing EMIs.

To understand the working mechanisms of an EMI, a realist evaluation method is a fitting approach because it develops a theory on how a program works, for whom, and under what circumstances [45]. Therefore, applying realist evaluation methodology within a EMI trial helps to advance current research. It also provides information to customize interventions [30]. Gaining insights into how and why an intervention works, would support future development and implementation of EMIs.

SELFIE intervention

The current paper reports on a realist evaluation within the SELFIE trial (Netherlands Trial Register NL7129 (NTR7475)) of which details are available elsewhere [29, 31]. The SELFIE trial aimed to investigate the efficacy and clinical feasibility of SELFIE, which is a smartphone-based guided self-help intervention for improving self-esteem in youth exposed to childhood trauma, in a multi-center, parallel-group, assessor-blind randomized controlled trial (RCT). Their findings show improvement in the primary outcome of self-esteem at postpost-intervention and 6-month follow-up, and small to moderate effect sizes point towards beneficial effects on some secondary outcomes such as general psychopathology and quality of life [31]. The EMI under study in the SELFIE trial aimed to enhance self-esteem in youth (12–25 years) who have experienced childhood adversity (i.e., abuse, neglect, bullying, and/or household discord). A 6-week manualized intervention is delivered by trained SELFIE therapists, consisting of three face-to-face sessions, three standardized e-mail contacts, and an EMI administered through a smartphone-based app (i.e., the PsyMate® app), supporting the adaptive real-time and real-world transfer of intervention components tailored to moment, person, and context. By providing an ecologically valid, accessible, and personalized intervention, SELFIE aimed to tailor interventions to the needs of youth [11, 12, 14]. The preventive content, principles and techniques of the transdiagnostic SELFIE intervention have been based on the CBT model and interventions [46] and the self-help manual by de Neef [47].

Aims

The aim of the present study was to investigate, within the SELFIE intervention, mechanisms of EMI and under what circumstances these mechanisms do or do not come into play.

Methods

Study design

A realist evaluation methodology was adopted to gain insight into EMI working mechanisms and consisted of three phases: (1) developing an initial programme theory (IPT), based on literature and two expert interviews and one focus group with stakeholders; (2) testing the IPT by comparing and explaining data offered by fifteen semi-structured interviews with youth who had received the SELFIE intervention, one focus-group with SELFIE therapists, and a debriefing questionnaire (n = 61); (3) refining the IPT. The guidelines offered by the RAMESES II reporting standards for realist evaluation [48] were followed, thereby adhering to the Standards for Reporting Qualitative Research (SRQR) [49].

Researchers undertaking this realist evaluation also partly played a role in developing (MP) and delivering the SELFIE intervention (MP, KS, SV). To minimize interpretation bias and influences due to a therapeutic relationship after delivering the intervention, researchers did not interview participants to whom they had previously delivered the SELFIE intervention. Neutrality in the process of data collection and analysis was ensured by reflexivity, i.e. documenting the progress, decisions, and motives of the researchers, aimed at supporting self-awareness concerning their role and impact on the research environment. In addition, frequent meetings with independent researchers (LD, MS, and a researcher conducting a realist evaluation in a different research trial) were held to discuss the researchers’ role.

The current realist evaluation was set in the context of the SELFIE trial [23, 29], and follows a previously undertaken realist evaluation of the SELFIE intervention [50]. This previously undertaken realist evaluation was a result of the known relevance of interventions aimed at low self-esteem in youth, and due to the complex nature of self-esteem and its targeting by an intervention, it was considered important to focus the analysis of the available qualitative data solely on characteristics and delivery of self-esteem interventions. The present realist evaluation analyses the same qualitative data, however, the analysis is aimed to investigate mechanisms of EMI, and under what circumstances these mechanisms do or do not come into play. The methods of the present realist evaluation, therefore, resemble the methods of the previous realist evaluation within the SELFIE trial but focus on the distinct aspect of EMI within the SELFIE intervention. A detailed description of the methods is shown in Table 1.

Table 1 Methods of recruitment, data collection, and analyses for phases 1 and 2
Full size table

Participants

Phase 1 (the development of the IPTs) was based on literature study, qualitative data from 2 expert interviews, and a focus group with stakeholders. Subsequently, for phase 2 (testing the IPTs) iterative data collection took place through 15 individual interviews with SELFIE participants (within six months after finishing the intervention), analyzing data from a debriefing questionnaire (n = 61), and a focus group with 4 SELFIE therapists. As stated, more details of the methods used in recruitment, data collection, and analyses are shown in Table 1.

Analyses

As stated above and shown in Table 1, data collection and analyses took place in an iterative process during the different phases of this realist evaluation leading to the development of IPTs after phase 1 and formulated context-mechanism-outcome configurations (CMOCs) after phase 2. In parallel, a debriefing questionnaire (participants filled this in directly after receiving the SELFIE intervention, and it inquired about participants’ experience with the app, the use of the exercises, satisfaction, and acceptance of the SELFIE intervention) was addressed in testing the IPTs. In phase 3 these CMOCs were synthesized back into the IPTs. MP conducted this analysis individually, and subsequently discussed it within the research team. The process of analysis was inspired by the approach described by Gilmore et al. [51]. Namely, IPTs were coded as nodes, to which quotations were assigned illustrative for IPT-specific CMOCs and their implications for the IPTs (refute, refine, or accept). To ensure transparency, the process of reasoning was explicitly described in research memos.

Results

Stakeholders and participants

For phase 1, two experts were contacted and found willing to participate. Further, four stakeholders took part in a focus group. For phase 2, firstly 23 participants in the SELFIE trial were contacted of whom 15 took part in semi-structured online (due to COVID-19 restrictions) interviews.

Table 2 Participant characteristics of the interview sample of this RE and the participant characteristics of the full sample of participants in the experimental condition of the SELFIE trial
Full size table

Table 2 shows the characteristics of this interview sample alongside the characteristics of the full sample of participants in the experimental condition of the SELFIE trial [31]. Not having sufficient time (n = 6) and mentally not feeling well enough to participate (n = 2) were reported as reasons not to participate. The average age of the participants was 21 years; 12 participants identified as women, and three as men. Secondly, six therapists who delivered the SELFIE intervention were contacted of whom four participated in a focus group. Non-participation was due to agenda restrictions (n = 2).

Main findings phase 1 (developing the IPTs)

Three pillars of the SELFIE intervention were defined based on its program architecture. First, it is delivered as an EMI, second, the intervention aims to target self-esteem, and last, it is offered as a guided self-help intervention. As described before, the working mechanisms that underpin EMIs are not fully understood, and this study seeks to obtain data that will help address this research gap by focusing the present analysis solely on the pillar regarding EMI. The other pillars are discussed in a separate paper [50].

Data collected during Phase 1 was described and interpreted in the context of pre-existing literature and theories. A detailed description of the process of identifying the IPTs can be found in Supplementary Material 6.

The three key IPTs related to EMI that were identified, tested against our data, and refined, were:

  • If participants experience the intervention as personalized, anonymous, and easily accessible, then they will be more comfortable with their input and can participate in the SELFIE intervention without feelings of shame.’ (IPT 1).

  • If a change in momentary self-esteem is established repeatedly and under different circumstances through the use of a smartphone application, then this will support the generalizability of this effect and support change in general self-esteem.’ (IPT 2).

  • If a technical malfunction is present or the reminder beeps are being perceived negatively, then the effect of the intervention will be limited and drop-out may occur.’ (IPT 3).

Table 3 shows the initial program theories, with supportive literature.

Table 3 Initial Program Theories on how the SELFIE intervention as an EMI may exert its effect on most users
Full size table

Main findings phase 2 and 3 (testing and refining IPTs)

The IPTs as shown in Table 3 were tested (phase 2) and refined (phase 3). Our main findings will be described below per IPT in the form of CMOCs as the analytical template of realist evaluation, followed by the refined IPT.

Testing IPT 1

A reported response to the intervention being offered repeatedly in daily life and being constantly available was to experience the intervention as accessible and feasible due to a lack of travel time to appointments, short exercises, reminders on the phone, and the ability to practice an exercise when there is time and need. The following quote illustrates the availability of the smartphone application being offered on their personal phone (important context vs. receiving a study phone).

“Ehm, well yes, it is nice that you can fill it in wherever, it does not matter where you are. I mean, yes ok, you could also bring a booklet anywhere but that would be something you might forget now and again. But you will not forget your phone as easily. You would go back home for it if you would forget it. So ehm, if you are on the train, at work, or you have a short break, then you can just work on it [the SELFIE intervention]. – Source: PI_3 (356–360)

Offering the intervention on their personal smartphone enhanced active participation, as illustrated in the following quote, supporting intervention outcomes.

“Yes, well it is just easy because you always carry your phone with you. So you don’t have to make a great effort (…) if I would have to do it [the exercises] on the computer, which I don’t carry with through the entire day, then you have to especially sit down to do it, making time for it, I feel. And with the app, you can note things down in between moments when you have some time left. (…) Yes, I think [that if I would have to use a computer] I would have been less engaged with the intervention. Because of the app, it just offers you easy access and so on, and that made me participate more actively than if I would have done it on a computer.” – Source: PI_6 (196–205)

In testing IPT 1, data supports the notion of ‘easily accessible’ as stated in IPT 1. Furthermore, it offers more detailed insight into why it is experienced as accessible. Having an intervention on your smartphone helps youngsters to stay committed due to the constant availability of the intervention, aiding flexible use of the exercises and making it more feasible to integrate it into their daily life since the expense in time is limited due to a lack of travel time to appointments for example. Supportive data was extracted from a debriefing questionnaire in that participants report to have experienced that the application helped them to practice the exercises in daily life, and that during the day they repetitively were aware of their context, feelings, thoughts, and behavior (as can be seen in Table 4).

Table 4 User experience and acceptance of the SELFIE intervention (n = 61)
Full size table

Regarding the sense of anonymity as phrased in IPT 1, it was found that the exercises were offered individually and privately, which offers a sense of privacy and anonymity without the interference of (possible) responses of others (response). The following quote from a participant illustrates this experience.

“By writing it down yourself in an app dares me to write down more that when I would have to say it (…) Yes, the idea that you are more anonymous, that people will not directly see or read everything or, well yes if you are in a conversation with someone else, I would experience that stronger. Now it was more the idea of writing it for yourself, rather than for someone else” - Source: PI_6 (182–195)

The sense of privacy and anonymity led to participants being less hesitant to fill in exercises in an honest and ‘unfiltered’ manner.

Reflecting on IPT 1, elements of ‘anonymity’ and ‘feeling more comfortable with their input’ are supported by our data. However, regarding the experience of participants that others will not interfere with their input, the word ‘privacy’ seems to be a better fit for this experience than ‘anonymity’ (as was used in IPT1). Participants were aware that their SELFIE therapist was able to partly read along on a reporting page, thus it wasn’t completely anonymous in that sense. In summary, a sense of privacy adds to the openness of participants when engaging in the intervention. They are less hesitant to fill in answers due to a lack of response by another person (for example, questioning their input further, or having an opinion about their input). It was inferred that open and active participation in the intervention is supportive for the intervention outcomes relating to self-esteem.

Refined IPT 1

The aforementioned findings in testing IPT 1 have been framed as CMOCs, as depicted in Table 5. These CMOCs have informed the following refinement of IPT 1:

  • The SELFIE intervention is offered through a smartphone application (C) enabling constant availability of the intervention and thereby increasing accessibility and feasibility (M), and, when offered on their personal smartphone (C) this enhanced a sense of privacy ensuring less hesitance in engaging with the app (M), leading to more open and active participation (O).

Table 5 CMOCs relating to IPT 1
Full size table

Testing IPT 2

Generalizability of effect is reported, however, emphasis is placed on acquiring a non-situation-specific skill (not dependent on situation-specific characteristics), and thereby able to lead to the generalizability of effect. The following quote refers to an exercise aimed at coping with receiving criticism, in which the participant previously mentioned that due to the exercise she gained a new perspective:

Interviewer: (…) What happens consequently when you look at things differently, does that change something in your thoughts or your feelings? What is its effect?

Participant: Mainly in my thoughts, that I think ‘ok’, well if I take the example again of that exercise on criticism, then, yes, how do you say that?… All of a sudden you are not so afraid of receiving feedback anymore or criticism or something like that, because I…, it has never been proven but I do think that I have a fear of failing. So I always found it very scary to make mistakes. And if it happens now I think ‘Yes, but actually, there is nothing wrong’. Because of that I then feel a lot more confident in my actions and experiences. Not just at work, but also in daily life. Well… my feeling related to that, that could make me very sad, that I felt that way, I could also be very anxious when I received criticism, which really frightened me. But, that is now a lot less.” - Source: PI_15

Participants rated the offering of prompting beeps for and the constant availability of the exercises as more than moderately effective in aiding change in self-esteem (also shown in Table 4). The sessions with the SELFIE therapist were rated slightly higher in supporting change in self-esteem, possibly implying the importance of guidance when delivering an EMI.

Synthesizing the above findings back into the IPT, we inferred that for the SELFIE intervention repetition in daily life is key in creating the circumstance of being more prone to cognitive restructuring, and since this surpasses situation-specific characteristics, with additionally the availability of certain tailored situation-specific exercises (e.g. criticism exercise), it could be argued that the effect is generalized over different situations in daily life. Thus generalizability is enhanced through the offering of the intervention in daily life as well as the availability of tailored interventions (as stated in IPT 2), however, this is not through the mechanism of changing momentary self-esteem in different situations, rather the mechanism seems to be that cognitive restructuring is more prominent and not dependent on situation-specific characteristics whereby effects are generalized.

Refined IPT 2

The above findings are phrased as a CMOC (presented in Table 6), and led to the following refined IPT 2:

  • Offering the intervention through the use of a smartphone application (C), facilitates the practice of skills that are not dependent on situation-specific characteristics in daily life, supporting repeated practice in different situations (M), leading to the generalizability of the effect (O).

Table 6 CMOC relating to IPT 2
Full size table

Testing IPT 3

Reminder beeps for the intervention exercises were offered through the smartphone application, popping up at random moments with an alerting sound. In some cases, as for the participant sharing the below-placed quote, this was experienced as loud and disruptive and/or for the participant at inconvenient timing.

“Ehm, yes I think the beeps are not so pleasant, I sometimes experienced them as rather annoying actually. (…) it beeped at inconvenient times, and the sound was also quite loud. Yes, it startled me sometimes .” - Source: PI_4 (317–321)

Thus, reminder beeps may sometimes be perceived as annoying or intrusive by sound or timing. Our data showed this could lead to negative sentiment and actions, such as turning off notifications and sound, with subsequently less active participation. It seems to be a balancing act between the intensity (repetition and duration) of the intervention being sufficient to reach positive outcomes (our data showed that in most cases, the reminder beeps were perceived as a positive element in supporting active participation), and the intensity being experienced as too much of a burden. Personal circumstances (well-being, energy, focus) seem to be an important context regarding assessing burden by capacity.

In the context of the intervention being delivered in the form of a smartphone application, technical malfunction may arise to which participants may respond by experiencing ‘hassle’ accompanied by irritation and a decrease in motivation. The following quote from a SELFIE participant who received the intervention illustrates discontinuity.

“It just made me postpone when it didn’t function to log in with the code, and yes, when I would receive a new code I would be in school for example, and with getting the email [with the new code] I would think ‘yes, I will do that this evening’, or something. (…) that made that actually one day or so was ‘lost” – Source: PI_1 (514–518)

In testing IPT 3, our data support the notion of a technical malfunction being perceived negatively and leading to less effective participation or drop-out. When, as part of the debriefing questionnaire, participants where asked about any problems they faced when using the SELFIE intervention, or whether they experienced it as annoying, rated as rarely (find participants’ ratings in Table 4), they mainly reported technical issues to explain their rating. In some cases, it was explained by their experience of not having enough space in the text boxes to write their answers. Technical malfunction is mainly experienced as a burden since it encompasses extra moments of contact or actions, both participants and SELFIE therapists have reported this. If needed, technical assistance was mentioned by participants to be readily available through, among more, WhatsApp, which was experienced as a low threshold, which is an important context for IPT 3. This buffered against the negative effects of technical malfunction. Further, as is shown in Table 4, ratings on the debriefing questionnaire prove very good satisfaction with and acceptance of the SELFIE intervention when also taking into account the mode of delivery.

Refined IPT 3

The aforementioned reflections have led to developing CMOCs relating to IPT 3, which can be found in Table 7. Subsequently, these CMOCs have informed the following refinement of IPT 3:

  • The use of a smartphone application to deliver the SELFIE intervention (C) may encompass technical malfunction and accompanied irritation and demotivation (M), leading to less active or delayed participation (O).

  • Furthermore, the reminder beeps (C) are activated at random moments and with an alerting sound which can be experienced as loud and disruptive (M), leading to decreased motivation and less active participation by e.g., turning off the notification or sound (O).

Table 7 CMOCs relating to IPT 3
Full size table

Discussion

Summary of findings

The aim of the present study was to investigate, within the SELFIE intervention, working mechanisms of EMI, and under what circumstances these working mechanisms do or do not come into play. The current research led to a revision of three IPTs. First, “The SELFIE intervention is a blended therapy using a smartphone application (C) enabling constant availability of the intervention and thereby increasing accessibility and feasibility (M), and, when offered on their personal smartphone (C) this enhanced a sense of privacy ensuring less hesitance in engaging with the app (M), leading to more open and active participation, thereby supporting intervention outcomes (O).” This refined programme theory illustrates that a smartphone application supports the mechanism of constant availability and consequently increases accessibility and feasibility. A supporting context for this mechanism is the use of their personal smartphone (rather than receiving a study phone), enhancing active participation and responding to the easily accessible intervention. Second, “Offering the intervention through the use of a smartphone application (C), facilitates the practice of skills (i.e. cognitive restructuring) that are not dependent on situation-specific characteristics in daily life, supporting repeated practice in different situations (M), leading to the generalizability of the effect (O).”. This refined programme theory stresses the importance of intervention exercises matching a variety of situations, meaning that if an intervention is offered repeatedly and in different situations, it will not be effective if the exercise offered is not applicable in a given situation. For the SELFIE study, it was found that practicing a relevant skill was not dependent on situation-specific characteristics and thereby effective to be offered repeatedly over a variety of situations. In addition, situation-specific exercises (such as how to deal with criticism) could be offered when permitting availability on demand. Third, “The use of a smartphone application to deliver the SELFIE intervention (C) may encompass technical malfunction and accompanied irritation which may induce demotivation (M), leading to less active or delayed participation (O).” Furthermore, “the reminder beeps (C) are activated at random moments and with a loud alerting sound which can be experienced as disruptive (M), leading to decreased motivation and less active participation by e.g., turning off the notification or sound (O).” These refined programme theories show that technical malfunction and disruptiveness of beeps do decrease motivation and engagement with the intervention. Since these ‘hurdles’ will not completely cease to exist, it is important to ‘buffer’ against the negative effects of this when delivering an EMI. We found that technical assistance being readily available and matching the participants’ needs (in this case youngsters who responded well to WhatsApp), may buffer against the negative effects of technical malfunction.

Comparison with existing literature

Delivering the SELFIE intervention through a personal smartphone app was found to be supportive of the outcome (refined IPT 1), which is in line with previous findings that the use of a smartphone seems to be associated with higher rates of adherence [52]. The use of one’s smartphone and not requiring multiple devices is often described as supporting the accessibility and usability of an EMI [53]. The current study sheds more light on the experience of youth in using their smartphone, not only supporting usability through convenience as previously reported but also revealing an underlying sense of privacy which seems to make participants less hesitant to engage with the app and thereby supporting open and active participation. Furthermore, the mentioning of a sense of privacy by participants links to research on therapeutic relationships in digital mental health delivery. A process proposed to enhance the openness of patients has been described as disinhibition, whereby clients were more prone to disclosure than in regular face-to-face contacts [54]. This is consistent with research on assessment methods, with computer-administered assessment methods obtaining more honest, open responses of personal information [55]. Furthermore, contact with a ‘virtual human’ controlled by a computer seemed to support participants’ openness and self-disclosure due to the experience that their responses were not being judged by another human, with the underlying assumption that experiencing fear of judgment will activate impression management resulting in withholding information that might threaten their reputation [55]. In contrast, even though quantitative research on the effect of coaching within an app-based intervention is limited and somewhat diffuse [56], evidence does seem to point towards human guidance within an app-based intervention to be positively associated with engagement, completion rates, and treatment outcomes [10, 57,58,59], and offering smartphone apps as standalone psychological interventions is not advisable due to the low level of current evidence on this [35]. These findings corroborate the results of the previous realist evaluation within the SELFIE trial, researching the element of guided self-help [50]. Thus, clinical judgment regarding the intention of an EMI could inform decision-making in EMI development on the degree of human interaction. The afore-described illustrates that an EMI with solely digital components is likely to differ substantially from an EMI with additional guidance by a therapist, and thus the findings from the current research should be placed within the context of guided EMIs.

The current results in the form of refined IPTs tentatively support the ecological interventionist causal model [12] in that participants report having practiced the intervention in their daily lives and that in their perception it leads to changes regarding the target mechanism of self-esteem. This is supportive of the main findings from the SELFIE study showing improvement in the primary outcome of self-esteem at post-intervention and 6-month follow-up [23]. As mentioned in the introduction, EMIs can prompt contextual reminders to trigger specific behaviours in specific contexts [24, 40, 41] aiding change in the aspired outcome of an intervention. In addition to this existing literature, our findings suggest exercises not only to be customized at the moment (and thus being context-specific) but also offer exercises to facilitate the practice of skills that are not dependent on situation-specific characteristics in daily life. By supporting repeated practice in different situations an EMI may broaden its relevance for the user.

In accordance with the current refined IPTs, previous studies have demonstrated that technical malfunction may be a negative influential factor regarding adherence and satisfaction when using app-based interventions [10, 60]. Furthermore, the finding that electronic prompts could be experienced as disturbing broadly supports the work of other studies where electronic prompts were experienced as a burden when they requested assessments of length, and a higher number of missing answers were reported when participants were prompted 8 times a day or more [52]. Thus, the findings replicate existing knowledge on prompts and technical issues with EMIs, however, the notion of a ‘buffer’ against possible negative effects by specifically adhering to the preferences of participants (in the current study the participants reported WhatsApp to be a low-threshold way of communicating, in contrast to email) is novel and further research may serve effective implementation and adherence to an EMI. User-experience outcomes could aid this type of research [61].

Strengths, limitations, and future directions

The method of realist evaluation made apparent possible working mechanisms of EMI which, even though researched within the context of the SELFIE intervention, seem to be relevant for other forms of (guided self-help) EMI. This is of crucial importance in the current state of research in the field of EMI, since developing EMIs in itself should be driven more by existing knowledge and theory of the working mechanisms to provide enhancement of the field. We would suggest that every trial that tests the efficacy of an EMI adds a process evaluation to interpret the main findings of the efficacy trial as well as to build knowledge on the working mechanisms of EMIs for further development and implementation. The current research offers insight into underlying processes that may exert an effect on the main outcome findings of the SELFIE trial. It should be noted, however, that the interactive element within the SELFIE intervention has not been addressed sufficiently concerning its importance within EMIs in the current paper. We would therefore suggest future research on this particular aspect. Further, synthesizing findings on efficacy in the SELFIE trial with findings from the realist evaluation would be of added value to deepen our understanding of interventions as well as target mechanisms as in this case self-esteem, and therefore, such a mixed-methods design would be strongly suggested for future research.

A future recommendation for EMI may be to allow participants to choose the notification sound and allow more flexibility in choosing time blocks. Studies using a pre-fixed sampling scheme (the prompts were programmed at certain times per day, and if multiple prompts were programmed, a fixed time interval was used) reported higher rates of adherence in (EMA) studies [52], however these studies offered very little prompts per day and did not aim to capture variability over the course of a day. The use of a pre-fixed sampling scheme is of course a trade-off with the effects due to the random occurrence of beeps and would not fit the aims of the SELFIE intervention since it would impair representative characterization of experience and, hence, limit EMIs substantially in tailoring/adapting to moments when help is most needed. Therefore, for the SELFIE intervention, we would recommend to offer flexibility in choosing the time blocks of a random sampling schedule within set blocks of time.

Furthermore, findings from feedback from users in an EMI for major depressive disorder reported three EMI features to have been highly appreciated: the possibility of receiving visual feedback about daily assessments, and consequently self-monitoring of daily patterns; the availability of psychoeducational material on depression and its mechanisms; and the opportunity to have continuous or periodic communication with a trained clinician [30]. Considering that the SELFIE intervention provides communication with a trained clinician, as well as psychoeducational material on self-esteem as session content, a new implication of the afore mentioned findings of Colombo, Fernández-Álvarez [30] is that the SELFIE intervention may benefit from offering participants visual feedback. In further development and implementation of the SELFIE intervention, the actual end users should continue to be involved in the design and evaluation of the technology for it to match their needs and to ensure that engagement is maximized [62].

Regarding methodological concerns, along with her role as researcher, MP was involved in developing and delivering the SELFIE intervention, which may lead to bias in carrying out the realist evaluation. To ensure reflexivity toward her role as a researcher, regular meetings with the research team were held, thereby focusing on the quality criteria of neutrality. Furthermore, the consensus in decision-making during the research process as well as the analysis was fostered through assistance by two other researchers (SV and KS) employing investigator triangulation over the study period. In addition, researchers not directly involved in the SELFIE trial (LD and MS) were regularly consulted. In contrast, the role of MP as being involved in both delivering and researching the SELFIE intervention may have supported the quality principle of applicability of evidence through transferability, i.e. the extent to which the current findings can be transferred or applied in different clinically relevant settings [63]. Second, despite applying a maximum variation sampling method, as described in Table 1 under ‘phase 2’, selection bias may have occurred in that individuals with particular characteristics might have agreed to participate in the study (e.g., participants experiencing enthusiasm regarding the SELFIE intervention may be more willing to share their experiences). For the focus group with SELFIE therapists, 4 out of 6 invited therapists were able to join the focus group. The two who were not able to attend reported this was due to agenda constraints and no particular characteristics of these two are thought to have had a bearing on the focus group data. Third, the timing of interviewing the participants was within six months after finishing the intervention to ensure a sufficient amount of interviews. It should be noted that the ability to recall the exact exercises and personal experiences is limited over such a period and should be taken into account when interpreting the findings. Future research should aim to interview participants closer to the date of finishing the intervention. Lastly, even though it is expected that certain findings are generalizable for EMIs not targeting self-esteem, it should be taken into account that self-esteem in itself may be a very important context for certain EMI working mechanisms to exert its effect. Illustrative of this is that our data support the notion that EMIs are particularly supportive of skill development. Furthermore, the context of the present study was a guided self-help intervention, this may limit the generalizability of our findings to EMIs in general. Therefore, the earlier-mentioned suggestion to promote research on the working mechanisms of EMI could aid in identifying certain combinations of context and working mechanisms to be effective.

Conclusions

The current research addressed working mechanisms of an EMI targeting self-esteem, such as the constant availability supporting increased accessibility and feasibility, for which the use of the personal smartphone was experienced as a facilitating context. Further, findings stressed the importance of delivering non situation-specific exercises (to support cognitive restructuring), or having situation-specific exercises available on demand (to aid in practicing adaptive copingstrategies in a given situation), to facilitate the mechanism of generalizability and ecological validity as an appreciated aspect of EMI. Lastly, technical malfunction and the burden of electronic prompts were experienced, reflecting previous literature, indicating the need for further research on ‘buffering’ possible negative effects of technical malfunction when delivering EMIs. Interestingly, within the SELFIE intervention, the context of accessible technical assistance showed indeed to be such a buffer against the negative effects.

The present findings can help us to understand possible working mechanisms and their contexts of EMIs, contributing to relatively limited research in this field. For the field to move forward, a better understanding of the working mechanisms of EMI needs to be developed, and it is strongly recommended that alongside efficacy trials of an EMI on specific target mechanisms, a process evaluation is conducted investigating the working mechanisms.