Background

Various psychotherapeutic treatments are established as evidence-based “best practices” for the individual treatment of people with eating disorders (EDs) [1]. Foremost among these are cognitive-behavioural therapy (CBT) for bulimia nervosa (BN) [2], and enhanced CBT (CBT-E) for BN with comorbid psychopathology and anorexia nervosa (AN) [3, 4]. After these comes dialectical behaviour therapy (DBT) for BN, BN with borderline personality disorder, binge-eating disorder (BED), and AN [5, 6], suitable when there are problems of impulse- or affect regulation. The preceding treatments have a known probability of achieving positive short-term responses in many individuals with EDs [1, 7, 8].

Many people with an eating disorder respond well to the evidence-based interventions described above. However, outcome remains limited and variable, with available data indicating that only about half of people will achieve symptom abstinence by the end of a manualized, time-limited treatment regimen (For full reviews see [9,10,11,12,13,14]. Limited access to informed treatment adds a further complication—as most people affected by an ED never obtain a recommended, evidence-based treatment or, if they do, it will generally be after an inordinately long wait [14, 15].

One proposed solution to the access problem has been the development and implementation of interactive self-help applications requiring minimal guidance from health-care professionals, that can be made accessible via internet-based or other digital platforms. Aside from practical advantages related to access, self-help interventions have been thought to also help improve people’s preparation and motivation for change, initiate symptom reduction or, at least, prevent symptom progression. In line with the preceding, several available studies have tested the effectiveness of digital self-help interventions for eating disorders [16,17,18,19,20,21]. Although studies differ substantially as to design and methodology, there is accumulating evidence to suggest that self-help applications can be effective–especially for individuals with milder, or less-entrenched ED variants.

Whereas various studies have evaluated ED self-help applications in community samples, relatively few have evaluated the feasibility and effectiveness of digital interventions in individuals when offered as an adjunct to treatment-as-usual. Mobile apps involving food records or thought-monitoring (like Recovery Record) have been used to help patients practice CBT techniques between therapy sessions. However, the empirical evidence for added benefits of such practices is mixed [20]. One research group has developed and tested a video imagery-based intervention for AN that includes cognitive, motivational and behavioural components to address symptoms (vodcast) [22]. The intervention was shown to be acceptable, to enhance motivation, and to reduce ED symptoms after 3–6 weeks of regular use [23,24,25]. The same research group developed another digital intervention for people with AN (called RecoveryMANTRA). The intervention consists of a 6-week online guided intervention, including online self-help materials combined with weekly one-hour, individual, text-based chat sessions with a mentor (graduate psychology students) and people with lived experiences, focusing on motivation and confidence to change [26, 27]. Findings have indicated that adding RecoveryMANTRA to treatment-as-usual increased patient confidence to recover and therapeutic alliance [26, 27].

To summarize, studies have thus far shown some benefits of digital interventions for treatment of EDs, with a small number of studies focusing on digital interventions used in specialized ED treatment settings. However, we are unaware of any digital intervention that are applicable in specialized ED treatment settings, fully self-guided, transdiagnostic, and that focused explicitly on building motivation for change. The latter was of particular interest, given a demonstrated, positive relationship between pre-treatment motivation for change and later outcome (see [28,29,30]).

This paper describes the development and psychometric properties of the application in question, called “Auto-Soins pour les Troubles de l’Alimentation” (ASTrA) (French for “self-help for eating disorders”). We examined face validity, content validity, and reliability. We also examined patients' and clinicians’ ratings of the application’s acceptability, and the extent to which they perceived the application to be suitable for use in clinical practice.

Methods

Development of the digital application ASTrA

We established a working group consisting of clinicians working in our specialized program for ED treatment (Eating Disorders Continuum, Douglas Mental Health University Institute), collaborating researchers affiliated with the Research Centre of the Douglas Mental Health University Institute, and a group of people with lived experience with an ED. Most of the clinicians involved also had experience in research on EDs. The working group was led by a senior clinician-researcher with more than 35 years of experience in clinical and research activity in the EDs (HS) and included a clinician-researcher and research methodologist with extensive research expertise in EDs (LB), four clinician-researchers from the EDC team, three team clinicians, a psychiatry fellow, a researcher with expertise in the development of digital interventions for mental disorders, one support staff member working in a digital health studio, and three individuals with lived experiences. Following a review of the scientific literature and first discussions by HS with international colleagues who had experience with developing digital eating-disorder intervention applications, members of the working group met regularly (1–2 times per month between August 31 2021 and June 13 2022), to discuss the development of the application. The group subsequently reached consensus about the specific modality of the application (i.e., an application with interactive exercises), relevant theoretical frameworks (self-determination theory, transtheoretical model of change and cognitive-behavioural model of eating disorders), specific eating-disorder symptom dimensions to assess (preoccupation with shape and weight), and response format (combination of multiple-choice and constructed-response format) of the application. Clinician members of the team subsequently drafted scripts for the interactive exercises (“items”), addressing common beliefs about eating, body image and weight control. HS further reviewed and edited the team members’ contributions, in collaboration with other members of the team. After a group voting process and further discussion, twelve scenarios were retained, which were in turn further revised by team members (both clinicians and individuals with lived experiences) for clarity, familiarity, and fidelity with the intended theoretical construct. Bilingual members of the team then translated each of the scenarios, originally written in English, to French and then back again to English, using a back-translation procedure. Next, the instructions and interactive items of the ASTrA application were programmed. The working group agreed on the final design and look of the application and performed usability testing to debug the digital tool.

The application was centered around four core clinical features relevant for most types of eating disorders: (i) fear of gaining weight (ii) obsessions around weight, shape and eating, (iii) ambivalence around change and (iv) binge-eating. Addressing the preceding core symptoms is in line with evidence-informed approaches in the treatment of different types of eating disorders [31, 32].

Each of the 12 items of the ASTrA application started with a description of a maladaptive eating-disorder-related belief/cognition, followed by one or more questions designed to promote reflection and re-evaluation of the belief (called “feedback for reflection”). For most of the items, the reflection question was followed by either a follow-up reflection question or a text providing additional psychoeducational information. If the participant’s answer to the reflection question suggested the presence of a maladaptive cognition around shape, weight or eating, the follow-up reflection question was designed to further challenge the maladaptive cognition. If the participant’s answer to the reflection question indicated a healthy cognition around shape, weight or eating, a confirmative response appeared on the screen (e.g., “indeed”, “good thinking”), followed by a brief psychoeducational description of why the chosen answer was correct. Next, the following item appeared on the screen until all 12 items were addressed. Participants could go back at any time to any of the content of an item by pressing a backward navigation button on the screen. The ASTrA application was programmed in a way that participants were not able to fully skip any item. See Table 1 for a summary of each of the scenarios.

Table 1 Themes of the items of the ASTrA application and a description of the content
Full size table

Participants

Following development and programming, the digital self-help application ASTrA was assessed quantitatively in an independent sample consisting of adult patients who were receiving treatment at the Eating Disorders Continuum and clinicians (permanent staff or graduate-level trained clinical interns) working in our specialized eating-disorder program.

All patient participants were diagnosed with AN, BN or other specified feeding or eating disorder (OSFED) as defined by DSM-5 criteria. Diagnosis was determined by experienced clinicians following semi-structured interviews and confirmed by multidisciplinary team consensus. Since the number of individuals offered treatment services in our specialized program with BED is rather small, our sample did not include individuals with BED. We also excluded people with avoidant/restrictive food intake disorder (ARFID), as our application was not designed for individuals with this diagnosis.

All participants provided written informed consent prior to participation. The study was approved by the Research Ethics Board of the Douglas Mental Health University Institute (REB: 2023-633).

Measures

Technical stability

Each participant was asked to test out certain technical features of ASTrA such as back and forward navigation, clickable fields, loading of webpages, and was invited to provide comments on each of the preceding technical features.

Face and content validity

The items to assess the validity of ASTrA were constructed using guidelines of the consensus-based standards for the selection of health measurement instruments (COSMIN) for evaluating content validity [33]. Participants were asked to rate on a 4-point scale for each item comprehensibility (3 items, including clarity, easy to understand and free of technical language), presence/absence of objective errors, relevance, as well as presence/absence of typos. Each item consisted of a 4-point scale, where 1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree.

Acceptability, appropriateness, and perceived feasibility

Participants completed the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and the Feasibility of Intervention Measure (FIM) [34, 35]. The AIM, IAM and FIM scales each consist of 4 items, rated on a 5-point scale (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree), designed to assess the acceptability, appropriateness, and feasibility of an intervention. The AIM, IAM and FIM measures are based on the theoretical framework by Procter et al. (2011) [36], postulating that clinicians’ perceived acceptability, appropriateness, and feasibility of a new intervention are key indicators of actual implementation success. The AIM, IAM and FIM developed by Weiner et al. (2017) showed good–excellent psychometric properties [35].

Statistics

Descriptive statistics were used to examine mean ratings on the outcome measures. Mann–Whitney U tests (two-tailed) were used to test for differences in mean ratings of each of the outcome measures between patients and clinicians or between subgroups of patients. To quantify the content validity of the individual items of the digital application, the item-level content validity index (i-CVI) was calculated, using the relevance rating for each of the 12 items of the application, measured on a 4-point scale. i-CVI was computed by measuring the number of times participants rated the relevance of a particular item as a “3” (agree) or “4” (strongly agree), divided by the total number of participants who rated the item [37]. According to the criteria of Polit et al. (2007) [37], in the case of three of more raters, an i-CVI larger than 0.78 is considered excellent [37]. Content validity for the overall application (sum of the content validity index, s-CVI) was computed by taking the average i-CVI scores across the 12 individual items of the application [37].

To examine the reliability, Cronbach’s alpha coefficients were calculated to assess the level of consistency across the 12 items in terms of comprehensibility, absence of objective errors and relevance of the ASTrA application. A Cronbach’s alpha higher than 0.80 is typically indicative of good internal consistency [38].

Power analysis

There is no consensus about the number of raters required to perform a content validity evaluation [33, 39, 40]. Although the specific number of raters depends on the required expertise and the range of knowledge among evaluators, a sample size between 3–10 for each group of experts (professionals and lay experts) is considered acceptable [39].

To support Cronbach’s alpha coefficients, we based ourselves on the formula of Bonett et al. [41] and followed the recommendations of Bujang et al. [42]. Specifically, with the null hypothesis of a coefficient of Cronbach’s alpha set at 0.50 and the alternative hypothesis of a Cronbach’s alpha set at 0.80, the minimum sample size to determine the internal consistency of the ratings for the 12 items is 22 participants, with a power of 0.80. We therefore considered the present sample (15 individuals with eating disorders and 13 clinicians working with individuals with eating disorders) sufficient to examine the research questions of the present study.

Results

Participants

Fifteen patients and 14 clinicians (9 professionals, 5 graduate trainees) were recruited for this study. One participant (a health professional) did not complete 3 items and his/her data were therefore excluded from analyses. Two other participants (one health professional, one patient) missed one item and were retained in the analyses, resulting in a final sample of 15 patients and 13 clinicians. Patients were diagnosed with AN-Restrictive type (AN-R, n = 6), AN-Binge-eating/purging type (AN-BP, n = 6), BN (n = 2), or OSFED-purging disorder (n = 1). Clinicians had a background in psychology (n = 5), social work (n = 3), psycho-education (n = 3) or medicine/nursing (n = 2), with an average experience of 6.38 years (SD = 10.9 years, range 0–38 years) in working with people with eating disorders. All participants were females.

Technical stability

All individuals indicated that the ASTrA application was easy to navigate, and that all fields and web pages loaded properly.

Face and content validity

Participants agreed/strongly agreed that the items were comprehensible (clear, easy to understand, free of technical knowledge), free of objective errors, and relevant (Table 2). Some suggestions were also given to improve the grammar. There were overall no differences in any of the item ratings between clinicians and patients, except that patients found item 2 (“does losing weight solve my problem?”) easier to understand than clinicians (Z = − 2.03, p = 0.04). There were also no significant differences in ratings between patients with a restrictive spectrum ED (AN-R, n = 6) and patients who presented with a bulimic spectrum ED (AN-BP, BN, or OSFED, purging disorder, n = 9).

Table 2 Descriptive statistics of the items of the application
Full size table

Item-CVI for each of the 12 items ranged from 0.86 to 0.96 (see Table 2), whereas the s-CVI was 0.93, indicating excellent content validity. Content validity was also considered excellent when analyzing the CVI indices separately for patients and clinicians (i-CVI range for patients: 0.80–0.93, s-CVI = 0.88; range i-CVI clinicians: 0.92–1, s-CVI: 0.99).

Reliability. The overall Cronbach’s alpha across all ratings of the 12 items was excellent (alpha = 0.97, 95% CI 0.95–0.98). The internal consistency was also considered very good to excellent for each of the individual rating scales, including clarity (alpha = 0.88, 95% CI 0.80–0.94), understandability (alpha = 0.89, 95% CI 0.82–0.94), free of technical language (alpha = 0.93, 95% CI 0.89–0.97), absence of objective errors (alpha = 0.94, 95% CI 0.90–0.97), and relevance (alpha = 0.96, 95% CI 0.93–0.98).

Acceptability, appropriateness, and perceived feasibility

Cronbach’s alpha across the 12 AIM, IAM and FIM items in the present study was excellent (alpha = 0.93).

Both patients and clinicians agreed – completely agreed that the ASTrA application (mean ± SD) was acceptable (4.26 ± 0.5), appropriate (4.37 ± 0.6) and feasible (4.54 ± 0.5). There were no differences in ratings between patients and clinicians on AIM, IAM and FIM total scores (p > 0.14). There were also no differences as to the acceptability, appropriateness, and perceived feasibility between patients with a restrictive spectrum ED (AN-R, n = 6) and patients who presented with a bulimic spectrum ED (AN-BP, BN, or OSFED, purging disorder, n = 9). At the individual item level, clinicians rated ASTrA as slightly more appealing than patients (Z = − 2.36, p = 0.02) and liked ASTrA slightly more than patients (Z = − 2.05, p = 0.04). Figure 1 displays the means (SD) of each of the individual AIM, IAM and FIM questions, stratified by group.

Fig. 1
figure 1

Acceptance, appropriateness and feasibility ratings of ASTrA, from a patient and clinician perspective. Ratings vary from 1 = completely disagree to 5 = completely agree. Values represent Means ± SD. *p < .05 (2-tailed). AIM = Acceptability of Intervention Measure; IAM = Intervention Appropriateness Measure; FIM = Feasibility of Intervention Measure

Full size image

Discussion

This paper describes the development of a digital self-help application (ASTrA) designed to promote re-evaluation of beliefs associated with ED symptoms and to enhance motivation for change, and to test the psychometric properties of the application and examine patients' and clinicians’ satisfaction with the application and their ratings of its suitability for implementation in clinical practice.

The ASTrA application had excellent psychometric properties; internal reliability was high, and the application showed excellent face and content validity. Furthermore, using standardized questionnaires on knowledge implementation feasibility, patients and clinicians were satisfied with the application and perceived it as relevant, feasible and acceptable to implement in clinical practice.

Over the years, numerous digital interventions for the treatment of eating disorders have been developed, and many of them have shown effectiveness in reducing eating-disorder symptomatology [16,17,18,19,20,21]. However, most applications are designed for people with a specific ED diagnosis (e.g., only AN or only BN), or assume participants' readiness to change. The few digital eating-disorder applications that include motivational components commonly require guidance from a therapist or peer mentor, or are designed solely for people with AN. To our knowledge, our ASTrA application is a first that is fully self-guided, includes both motivational and cognitive components and is suitable for different types of eating disorders. The use of such application in the context of specialized eating-disorder treatment is relevant at the present times, considering the increased number of people on the waitlist for eating-disorder treatment (e.g. [43, 44], and the shortage of mental health professionals [45].

The present study has some limitations: Whereas the content of ASTrA is relevant for any eating disorder in which shape and weight concerns are central (i.e., AN, BN, and certain forms of OSFED), most items are not relevant for ARFID. It is also important to mention that the patient sample in the present study consisted primarily of people diagnosed with AN. Although there were no significant differences in ratings between patients with a bulimic-spectrum eating-disorder presentation (AN-BP, BN, purging disorder) and patients with a restrictive-spectrum presentation (AN-R), few participants had a diagnosis of BN or OSFED. The preceding means that future studies examining the relevance and psychometric properties of the ASTrA application should include samples implicating more varied ED diagnoses, so as to ensure generalizability. Additionally, whereas the present study only described users’ experiences with the application, further research is needed to evaluate the application’s clinical effectiveness. Such work is currently underway in our eating-disorder program. Finally, the psychometric study was conducted with a non-random sample. As in most studies on digital applications, the preceding may imply that our study may have been subject to selection bias, which could have impacted the ratings. We aimed to minimize bias by explicitly stating in the instructions that we were interested in their perception and appreciation of the interactive digital application. In addition, each item was followed by an invitation to provide constructive comments, such as flagging sentences and text that are unclear or potentially missing portions of the content, and to report any other issues.

Strengths of the study are that the ASTrA application was based on key theoretical motivational frameworks relevant to eating disorders, designed according to best practices of test construction, and a result of a collaborative effort between clinicians, individuals with lived experiences and researchers, each being involved in all steps of the application’s development.

Our next step is to conduct a randomized controlled trial testing the effectiveness of ASTrA to enhance motivation and reduce symptom severity among individuals waitlisted for specialized ED treatment. If effective, implementing such application in clinical practice would be an essential step forward in improving access to and optimizing specialized eating-disorder care.

Conclusion

Findings suggest that our transdiagnostic interactive digital application ASTrA has excellent psychometric properties. Additionally, patients and clinicians were positive about the use of the application in clinical practice. The next step will be to investigate the application's effectiveness as an intervention among people on the waitlist for specialized ED treatment.