Residential care is among the most intensive forms of treatment in youth care. It serves youths with severe behavioral problems and is primarily focused on targeting externalizing problems. Despite best efforts, effect sizes remain moderate, which may be due to the disregarding of internalizing symptoms – in particular anxiety - and to limitations regarding the delivery model of interventions. This initial randomized controlled trial (n = 37) aimed to examine the effectiveness of a biofeedback videogame intervention (Dojo) as an addition to treatment as usual for youths with and without intellectual disability (ID) in residential care with clinical levels of anxiety and externalizing problems. Dojo targets both anxiety and externalizing problems, and incorporates the principles of conventional treatment, while addressing its limitations. Youths were randomly assigned to play Dojo (eight 30-min gameplay sessions) or to treatment as usual (TAU). Measurements of anxiety and externalizing problems were conducted at baseline, posttreatment, and 4-months follow-up through youths’ self-report and mentor-report. Completers-only analyses revealed decreases in self-reported anxiety and externalizing problems, and mentor-reported anxiety at posttreatment for participants in the Dojo condition compared to the control condition. Only mentor-reported anxiety was maintained at follow-up. No effect was found for mentor-reported externalizing problems. These findings provided preliminary evidence that Dojo is a promising, innovative intervention that engages high-risk youths. Practical implications are discussed.
The most intensive form of interventions for youth is residential care, which is often seen as a last resort solution for youths who have not responded well to previous treatment programs. Residential care is an out-of-home placement that typically provides 24-h care and offers mental health services with the goal of preparing youths to re-enter society (Whittaker et al. 2015). Such an invasive intervention is restricted to serve only those most in need, which includes youths with severe, complex behavioral and emotional problems. Their problem behavior is often combined with psychiatric disorders and/or intellectual disabilities (ID; Frensch and Cameron 2002), and up to 90% has been exposed to traumatic experiences such as neglect and/or abuse (Briggs et al. 2012). Thus, youths in residential care often show many risk factors and few protective factors (e.g., supportive caregivers, structured home settings) for behavioral and emotional problems, which contributes to the development and maintenance of these problems (Pollard and Hawkins 1999; Steinberg and Avenevoli 2000).
Residential care has a twofold purpose: first, to provide a safe and structured living environment for the youths in their group homes within the institution, and second, to offer intensive treatment to target problem behavior. At the group homes, group home workers are substitute caregivers who model appropriate behavior, provide support, and encourage youths to use the strategies they have learned in therapy. Treatment usually consists of evidence-based interventions to reduce problem behavior and teach youths adaptive alternative behaviors (McCurdy and McIntyre 2004; Kok et al. 1991). However, despite all efforts, residential treatment remains only moderately effective and approximately 25% of the youths leave care prematurely (Harder et al. 2006). A meta-analysis on the effectiveness of residential treatment showed an average effect size of d = .36 in the reduction of behavior problems for residential care with evidence-based treatment compared to standard residential care (group home care without specific treatment; de Swart et al. 2012). That meta-analysis highlights the importance of providing youths with treatment during their stay in the institution, but despite these promising outcomes, the average effect size of evidence-based treatment in residential care remains modest (Cohen 1988).
There are in particular two concomitant factors that may impede treatment effectiveness in residential care. The first one is that residential treatment focuses mainly on externalizing problems and tends to overlook internalizing symptoms, while most youths also exhibit co-occurring internalizing symptoms (Granic 2014). Comorbidity rates are in particular high for anxiety – approximately half of all youths in residential care shows clinically elevated levels of anxiety (Connor et al. 2004). These anxious feelings direct youths’ focus towards potential threatening stimuli, which may result in direct, impulsive aggression as a defense to these – real or perceived – threats (Vitaro et al. 2006). Even when youths do not show direct aggression, they tend to amplify their anxious feelings and maintain their state of arousal, rather than applying successful emotion-regulation strategies. This sustained attention to potential threats exhausts youths, and may eventually lead to externalizing behavior through an indirect route – the loss of inhibitory control (Granic 2014). Notably, for both the direct and the indirect route to aggression, anxiety is hypothesized as the eliciting mechanism that leads to youths’ inability to regulate their emotions. Thus, by ignoring anxiety in residential treatment, we may treat the symptoms while avoiding the causes of the behavior problems.
Another factor that may account for the modest effect sizes is that residential treatment usually consists of interventions based on cognitive-behavioral therapy (CBT) principles (de Lange et al. 2013). Although CBT is among the most effective forms of treatment in residential care (Garrido and Morales 2007; de Swart et al. 2012), it has some overarching limitations. These are limitations regarding the delivery model of CBT, not the principles themselves (Kazdin and Blase 2011; Kazdin and Rabbitt 2013). Not only should residential treatment focus on both internalizing and externalizing problems to improve its effectiveness, it should also be delivered in a way that is targeted towards youths’ needs.
Youths in residential care are often characterized by a lack of motivation to change their behavior (van Binsbergen 2003). Poor motivation is the key predictor for both low treatment effectiveness (Harder et al. 2012) and treatment dropout (Harder et al. 2011). In order to ensure that these youths do not leave care prematurely or drop out of intervention programs, these programs need to be engaging for youths. A new approach and potential solution for the delivery of youth interventions is the use of videogames. Whereas CBT depends largely upon imparting psychoeducational information, a didactic style of learning that contains few elements that are intrinsically motivating, videogames are able to deliver evidence-based techniques in an appealing context and make use of youths’ intrinsic motivation to engage them into treatment.
Also, psychoeducational CBT offers knowledge with limited opportunities to practice. Due to this gap between knowledge and behavior, the generalizability of CBT is limited. Youths usually know about appropriate, prosocial behavior, but in their everyday lives they often act impulsively and based on their emotions. Although CBT often incorporates exercises such as role-playing (Kendall et al. 2003), this rarely manages to provoke genuine emotions. Videogames, however, provide youths with the opportunity to learn by doing instead of memorizing (Vygotsky 1978), and are better able to elicit authentically emotional experiences. Youths are provided with an in-game environment where acquired techniques and strategies can be practiced until they are automatized and ideally can be generalized outside the game (Granic et al. 2014). The repetitive nature of gameplay fosters long-term learning (e.g., Rosas et al. 2003). In particular games that implement biofeedback may promote self-regulation skills and foster generalization of learned behaviors to youths’ daily lives (Yucha and Mongomery 2008). A videogame named RAGE-Control has successfully been integrated into a traditional CBT-based intervention to improve emotion-regulation among youths in residential care. RAGE-Control was effectively used to practice and strengthen the techniques learned during therapy sessions with a therapist (Ducharme et al. 2012; Kahn et al. 2013).
Another factor that impedes the effectiveness of CBT is that 20–25% of all youths in residential care is diagnosed with ID (van Nieuwenhuijzen 2010). These youths have limited social, emotional, and cognitive capacities (Magiati et al. 2014), while learning about cognitive biases and the links between feelings, thoughts, and behavior requires high-level processing. Playing a videogame, on the other hand, usually requires less cognitive load. Youths with ID might benefit more from experience-based interventions such as videogames compared to verbally based interventions such as CBT (Association for the Treatment of Sexual Abusers 2015). Treatment should be tailored to the specific needs of these youths and requires more simplistic language, smaller learning steps, and more emphasis on generalization to their real lives than conventional programs offer (Didden 2006). At the time, there is little evidence that conventional CBT alone has any beneficial effects on youths with ID (Sturney and Hamelin 2014; Taylor 2002). Although traditional CBT may require some adaptations for youths with ID, its principles have successfully been adjusted for their treatment. CBT components were made more concrete by implementing exposure and using relaxation techniques (i.e., deep breathing, muscle relaxation) to promote self-regulation among youths with ID (Shenk and Brown 2007).
The present study tested the initial effectiveness of Dojo, a biofeedback videogame intervention (developed by GameDesk, Los Angeles, CA), which targets the emotion-regulation problems that are hypothesized to underlie both anxiety and externalizing problems. The game consists of three in-game rooms (fear, frustration, and anger), each with one or two tutorials and an emotion-evoking mini game. The tutorials teach CBT-based relaxation techniques as deep-breathing techniques, progressive muscle relaxation, positive thinking, and guided imagery (Albano and Kendall 2002; Glancy and Saini 2005; Rapee et al. 2000; Sukhodolsky and Scahill 2012; Weisz and Kazdin 2010). The mini games are designed to trigger the emotion in question and challenge the youths to practice the newly acquired relaxation strategies in the in-game environment. This way, youths are playfully trained how to cope with their emotions. While playing the games, the players heart rate is monitored through biofeedback hardware and displayed on the screen, thus providing the player with real-time feedback on their stress levels. Controlling physiological reactions is required for success in the game, which encourages players to effectively regulate their emotions by using players’ desire to perform well in the game. For a more detailed description of the game, see Schuurmans et al. (2015). A recent pilot study demonstrated the feasibility and potential of Dojo as an intervention for a high-risk adolescent target population (Schuurmans et al. 2015). User evaluations and self-reported compliance for the tutorials were high, and initial outcome results on reductions in anxiety and externalizing problems were promising.
The present study was designed as an initial randomized controlled trial (RCT) to test the effectiveness of Dojo as an intervention for youths with clinical anxiety and externalizing problems in residential care. We hypothesized that participants who played Dojo as an addition to their treatment as usual (TAU) would show reduced levels of anxiety and externalizing problems compared to participants who received TAU alone. First, we focused on the main outcomes of the trial, the immediate posttreatment effects on symptoms of anxiety and externalizing problems. Then, we examined intervention outcomes at 4-months follow-up.
Materials and Method
Design and Procedure
The present study utilized a RCT design with two parallel conditions (Dojo versus TAU) and was conducted in residential institutions that provide (secure) youth care for youths with and without ID. In these institutions, youths live in group homes consisting of six to ten youths, with group home workers as substitute care givers. Participants were recruited by clinicians. Inclusion criteria consisted of clinically elevated levels of both anxiety and externalizing problems, based on clinician assessment. Participants were excluded if they were diagnosed with an Autism Spectrum Disorder or exhibited psychotic symptoms, and in one case, based on the clinician advising against study participation. This participant showed severe anxiety specifically for ghosts, which made the clinician fear that participation would have a negative effect. Next, participants were invited for an individual meeting during which they were informed about the study and were asked for their written consent. It was explained that they could quit the study at any time and that all information would be treated confidentially. For participants younger than the age of 16, their legal guardians were also informed and asked for written consent. All participants received ten euros for their participation. Ethical review and approval were provided by the Faculty of Social Sciences, Radboud University Nijmegen (ECSW2013–1811-154) and the procedure was registered in the Trial Register for RCTs (www.trialregister.nl; Trial ID: NTR4477).
Assessments were conducted for both conditions at three time points: week 1 – prior to the intervention (i.e., baseline), week 5 – immediately following the intervention (i.e., posttreatment), and at 4-months follow-up. Measurements consisted of participants’ self-report and mentor-report (the group home worker with whom they had the most contact). The self-report measures were completed in an interview format to ensure comprehension (in particular participants with ID had difficulties with reading). The interviews took 15 to 20 min and were conducted by the first author or a research assistant.
Experimental Condition (Dojo)
Participants in the experimental condition received the Dojo intervention as an addition to their usual treatment program. The intervention consisted of eight 30-min sessions during which participants played Dojo on a laptop. The sessions took place twice a week for four consecutive weeks in an office at the group homes or in a therapist office located on the campus of the residential institution. The game sessions were led by the first author and two research assistants who were trained to explain the game to participants and guide them through the tutorials and challenges according to a standardized protocol. Critically, in each session, participants were instructed to complete the tutorial – thus, to practice the relaxation technique – before they were allowed to start with the matching mini game.
Control Condition (TAU)
The TAU condition was designed to reflect standard practice. Participants in both conditions received TAU; treatment as recommended by their clinicians regardless of this study. There were no restrictions for the type of interventions participants received, we only kept track of it. Individual therapy (e.g., CBT) and/or medication (e.g., Ritalin) were the most received interventions. Some participants received group therapy (e.g., social skills training) and/or family therapy (e.g., multisystematic therapy; see Table 1).
Randomization of the participants to one of the two conditions (Dojo versus TAU) was stratified by gender and intellectual disability level (none/mild/moderate) to ensure equal ratios of participants in both conditions. The first author randomly assigned participants to the conditions using a computer-generated list of random numbers. Randomization was executed before we contacted the youths to ask whether they were willing to participate in the study. Participants were not informed about their condition before they decided whether they wanted to take part in the study. Then, participants in the Dojo condition were told that Dojo was designed to help them regulate their emotions. Participants in the control condition were told that they could play Dojo for treatment purposes after the follow-up measurement.
G*Power 3 (Faul et al. 2007) was used to estimate our targeted sample size, based on a small-to-medium effect of d = 0.36 (de Swart et al. 2012), an alpha of 0.05, a power of 0.80, and an estimated correlation of 0.7. A pre-post (within) by two groups (between) ANOVA would require a total sample of 40 participants.
Participants were recruited at two residential institutions from March 2014 to June 2014. We recruited 51 participants, of which ten were excluded before randomization because they did not meet the inclusion criteria or the participant/guardian declined to participate. At baseline, our sample consisted of 41 participants. We compared participants’ self-reported baseline scores on anxiety and externalizing problems with the self-reported scores of healthy Dutch adolescents within a similar age range (Muris et al. 2002; van Widenfelt et al. 2003). The mean total scores on anxiety were 19.33 for the complete sample, 18.13 for males, and 24.00 for females. In the sample of Muris et al. (2002), these scores were respectively 16.9, 12.7, and 20.4. It was noted that the males in our study report high anxiety scores in particular. The mean total scores on externalizing problems were 10.78 for the total sample, 10.90 for males, and 10.17 for females. Van Widenfelt et al. (2003) reported respectively 7.65, 8.0, and 7.3. Attrition was low with only four participants withdrawing from the study during the intervention (see Flow diagram, Fig. 1). Baseline demographic characteristics are shown in Table 1. No statistical differences were found between the experimental and control condition on any of these baseline factors, including sex, age, comorbid diagnoses, intellectual disability, previous gaming experience, and the type of interventions participants received during the study (all p > .10).
Self-reported and mentor-reported anxiety was measured using the total scores of the Dutch version of the Spence Children’s Anxiety Scale (SCAS; Spence 1998). The SCAS has 45 four-point items (e.g., I worry about things, I am scared of the dark) and is composed of five subscales: ‘separation anxiety’, ‘social phobia’, ‘obsessive–compulsive disorder’, ‘fears of physical injury’, and ‘generalized anxiety’. Cronbach’s alpha of the SCAS measurements were .88, .92, and .87 (self-report), and .88, .89, and .92 (mentor-report) for the baseline, posttreatment, and follow-up measurement, respectively. The SCAS has good reliability and validity and shows high correlations with other anxiety questionnaires (Muris et al. 2002). The SCAS has not yet been tested among youths with ID but is suitable for children as young as 8 years old (Spence 1998) so no problems were expected in a sample of adolescents with ID.
Self-reported and mentor-reported externalizing problems were measured using the Dutch version of the Strengths and Difficulties Questionnaire (SDQ; Goodman 1997; van Widenfelt et al. 2003). We used the externalizing subscales ‘conduct problems’ (e.g., I fight a lot), ‘hyperactivity-inattention’ (e.g., I am easily distracted), and ‘peer problems’ (e.g., I am usually on my own), each consisting of five three-point items. We calculated a total score of externalizing problems by summing up these three subscales. Cronbach’s alpha of this externalizing problems score were .81, .83, and .68 (self-report), and .75, .69, and .66 (mentor-report) for the baseline, posttreatment, and follow-up measurement, respectively. The SDQ was successfully used in a sample of youths with ID (Kaptein et al. 2008), has good validity, and correlates highly with other established questionnaires that measure externalizing problems (Muris et al. 2003).
To measure the appeal of the videogame intervention, we asked participants in the Dojo condition to rate statements regarding their satisfaction with the game on a 5-point scale (e.g., I liked playing Dojo). Participants’ compliance was assessed by self-report on a 7-point scale (e.g., How would you rate your compliance during the muscle relaxation training?). Finally, we asked participants which of the relaxation techniques they used most in their daily lives (participants were allowed to choose more than one answer).
At baseline, participants were asked for their previous gaming experience. A pilot study (Schuurmans et al. 2015) showed that it was difficult for participants with ID to come up with an estimation of the average hours per week they play videogames, so this measurement was divided in average amount of hours they played videogames: (1) on week days at the residential center, (2) during the weekend at the residential center, and (3) during the weekend when participants left the residential center and stayed at home. Based on these three estimations, we calculated the average hours of gameplay a week for each participant.
Our main outcomes were immediate intervention effects on (1) anxiety, and (2) externalizing problems. Additionally, we examined intervention outcomes at 4-months follow-up. We performed completers-only analyses, i.e. involving only those participants who completed the measurements. Outcome data were missing for 12.2% of the posttreatment measurements (both self-report and mentor-report), 12.2% of the self-reports at follow-up, and 34.1% of the mentor-reports at follow-up. Although missing data usually are imputed for analyses, imputation of missing data that are not missing at random may lead to misleading results and an even bigger bias than the analyses of complete cases only (Sterne et al. 2009). Our missing data were not missing at random, in particular the mentor-reports at follow-up: the missing cases were either participants who improved in their behavior and returned home, or participants whose behavior deteriorated and who were replaced to a secured institution.
We calculated descriptive statistics (means and standard deviations) for participants’ baseline characteristics, anxiety and externalizing problems separately for both conditions, and for user evaluations for participants in the Dojo condition. Also, correlations, independent t-tests and chi square tests to test for differences between conditions (Dojo versus TAU) at baseline were conducted. For the main effect analyses (anxiety and externalizing problems) we used a mixed-model repeated measures analysis of variance (ANOVA) with time as within-groups factor and condition as between-groups factor. Analyses were conducted comparing change from baseline to posttreatment, and from baseline to follow-up for the two conditions. The partial eta squared (η2 p) was derived in order to estimate the magnitude of the difference between groups and the effect size of the intervention. Cohen’s d effect size values (see Cohen 1988) were calculated for the within-groups effect sizes for change from baseline to posttreatment and from baseline to follow-up.
Main Outcomes at Posttreatment and Follow-Up
Baseline, posttreatment, and follow-up scores on self-reported and mentor-reported anxiety and externalizing problems are presented in Table 2. There was no significant difference on baseline scores (all p > .35) between the conditions. Table 2 also includes the statistics of the main outcome analyses. Significant interactions were reported for self-reported anxiety and externalizing problems at posttreatment, and mentor-reported anxiety at both posttreatment and follow-up. Effect sizes for these interactions ranged from small to medium (Cohen 1988). After the main analyses, we conducted additional analyses where we controlled for ID. There was no significant effect of ID on the intervention outcomes (all p < .10).
All participants in the Dojo condition attended the eight scheduled gameplay sessions, but for four participants it took 5 weeks to complete the sessions due to scheduling problems. Participants reported high satisfaction with Dojo: evaluation scores were 4.53 out of 5 for ‘liked playing Dojo’ (SD = .62), 4.00 for ‘thinks other youths will like playing Dojo’ (SD = .92), 3.88 for ‘liked Dojo being a videogame intervention’ (SD = 1.22), and 4.53 for ‘Dojo is useful in daily life (SD = 1.07). Participants also reported high compliance during the relaxation tutorials. The mean scores for self-reported effort is 5.76 out of 7 for positive self-talk, (SD = 1.15), 6.12 for muscle relaxation (SD = .99), 5.95 for guided imagery (SD = 2.00), and 6.06 for deep-breathing techniques (SD = 1.09). The relaxation techniques that were rated as most used in participants’ daily lives are deep-breathing relaxation (64.7%) and positive thinking (47.1%).
The current study utilized an initial RCT to test the effectiveness of Dojo as an intervention for high-risk youths with clinical anxiety and externalizing problems in residential institutions. We expected that eight sessions of Dojo gameplay would lead to reduced levels of anxiety and externalizing problems; these hypotheses were partly supported. Youths who played Dojo, compared with youths in the control condition, showed reductions in self-reported anxiety and externalizing problems at posttreatment, and mentor-reported anxiety at both posttreatment and follow-up. Contrary to our expectations, results showed no differences between conditions in self-reported anxiety and externalizing problems at follow-up, and mentor-reported externalizing problems at both posttreatment and follow-up. These findings indicate that the intervention resulted in an immediate reduction of anxiety, and from youths’ own perspective, also externalizing problems.
We have to be cautious with our conclusions regarding these outcomes, given the small sample size and subsequent low power in this study may have resulted in less robust results. The mentor-reported follow-up results have to be interpreted cautiously due to the high attrition rates. This was due to participants leaving the institutions. Participants were either discharged from the institutions to return home, because they showed improvements in their behavior (n = 3), they were replaced to other, secured institutions, because their behavior problems deteriorated (n = 6), or they refused further treatment and left the institutions without being discharged (n = 1). Whereas all participants themselves were willing to complete the last interview for this study, even when they left our institutions, we were unable to obtain mentor-reports for these participants. This non-random missing mentor-report data at follow-up may have resulted in an unrepresentative sample (Graham 2009). Nevertheless, we decided to report these results, but we have to refrain from drawing firm conclusions from this measurement.
Although youths themselves report a decrease in externalizing behavior at posttreatment, their mentors do not report any effect on externalizing problems. These variations in outcomes might be caused by a difference in perception between youths and their mentors. Other studies that were conducted in residential institutions have reported comparable results, with substantial disagreement between youths’ self-reports and mentor-reports, in particular for outcomes on externalizing problems (Bastiaansen et al. 2004; Grills and Ollendick 2003; Nijhof et al. 2011). It may be that group home workers are more critical in the assessment of youths’ behavioral progress compared with youths themselves, precluding them from noticing changes (Knorth et al. 2008).
Our results suggest that Dojo has a larger effect on anxiety than on externalizing problems. When we compare Dojo with conventional interventions that target anxiety and externalizing problems, there is a larger overlap with traditional components of anxiety treatment (i.e., skills training such as affect recognition, relaxation, and exposure; e.g., Kendall et al. 2003) than with aggression treatment (i.e. teaching prosocial behavior, anger control, and moral reasoning; Goldstein and Glick 1987). Although Dojo has one room that specifically targets anger, the main idea of the game is to teach youths to recognize their emotional and physical arousal, and to control this by practicing relaxation techniques.
Surprisingly, youths’ immediate decrease in self-reported anxiety and externalizing problems was not maintained at follow-up, only mentor-report anxiety showed an effect at both posttreatment and follow-up. This suggests that although Dojo could be an effective way to decrease anxiety and externalizing problems, from the youths’ perspective, the intervention may not help them to cope with future anxiety-provoking situations. In order to maintain immediate posttreatment effects, it may be necessary to provide youths with a ‘booster session,’ as done by The Growth Factory, a computerized mindset intervention developed for youths in residential care (Helmond et al. 2014).
Finally, we would like to discuss our results in the light of the findings of Scholten et al. (2016), who also tested Dojo, but as a method of prevention in a non-clinical sample of adolescents at risk for anxiety. They compared Dojo to the commercial videogame Rayman, and found equal reductions in anxiety for both conditions. This could mean that both videogames were equally effective in reducing anxiety, or that neither was effective. This question remains unanswered due to the lack of an inactive control group, but even when we assume that Dojo did not have an effect in their study, there are some explanations for the different results in our study. It could be due to the substantial differences between the two target populations. It may be that Dojo does work as an intervention for a clinical population with and without ID, but not as prevention for non-clinical adolescents. Another possibility is the difference in the ways Dojo was delivered. Scholten et al. (2016) allowed youths to play freely with minimal supervision, which made it possible for youths to skip the relaxation tutorials – which are hypothesized as Dojo’s working mechanisms – and play the mini games only.
Strengths and Limitations
To our knowledge the current study is the first to examine a videogame intervention in a residential treatment setting with youths with and without ID. Research in this clinical, high-risk context is critical to establish intervention effectiveness and requires minimal translation to implement its results in these settings. Attrition in high-risk samples is usually high, but we lost only two participants in each condition from baseline to posttreatment. Both participants in the Dojo condition decided to quit the sessions due to scheduling problems: the sessions had to be scheduled after school hours and interfered with their part time job and/or leisure activity appointments. One participant asked for the opportunity to quit the study but start again with the Dojo sessions after the summer, since he would have more spare time by then. This indicates that although he quit the study, it was not because he was not motivated to play Dojo. Youths’ positive game evaluations demonstrated that they liked the game and they reported high compliance during the relaxation tutorials. Moreover, not only was treatment fidelity high – all participants completed the eight scheduled sessions – all youths in the control condition still wanted to play Dojo after the follow-up measurement. They did not receive an incentive for this, which indicated that they were intrinsically motivated to play. This suggests that we met our goal as for engaging this hard-to-motivate population into treatment.
The biggest limitation of the present study is its small sample size. Its results should be interpreted with caution, since these were not robust. Outcomes changed depending on whether or not the missing data were imputed. This study was not powered to definitively test the effectiveness of Dojo. In the future, rigorously designed and adequately powered RCTs could establish the effectiveness of Dojo on anxiety and externalizing problems and examine potential mediating mechanisms.
Participants in the Dojo condition received the intervention as an addition to TAU, while the control condition only received TAU. This means that participants in the Dojo condition received extra individual attention compared to the controls. Although active control groups are more rigorous, these are only superior when participants in both conditions have the same type of attention and the same expectations of improvement (Boot et al. 2013). Thus, optimal control would be a videogame that is comparable, but does not include the working mechanisms. This was impossible to achieve for us, since the study was conducted within institutions which had restrictions for casual videogame play. Clinicians did not agree on implementing a condition in which youths were allowed to play a videogame that was not expected to lead to mental health benefits. Moreover, the primary purpose of this study was to test the effectiveness of Dojo as a beneficial addition to regular treatment, not to determine its superiority to another form of treatment, which makes TAU a valid control condition (Freedland et al. 2011).
Our study design included not only a posttreatment measurement to assess immediate intervention effects, but also a 4-months follow-up measurement to evaluate whether effects were maintained over time. While attrition rates in these high-risk populations are in particular high for follow-up measurements, all participants who were included at posttreatment, also completed the self-report measurement at follow-up. However, we lost a substantial amount of the mentor-reports at follow-up, for reasons explained above.
The gameplay sessions were supervised by the first author and two research assistants, while an ideal clinical study design does not include the researcher’s involvement in the intervention sessions. Given limited funds and personnel time, it was not feasible to hire an additional research assistant blind to the study goals. Gameplay supervision was done following a standardized protocol, to prevent possible supervisor effects. Moreover, since Dojo teaches the emotion-regulation techniques within the game, the only task for supervisors was to answer any questions and to ensure that the youths followed the instructions (e.g., completing the tutorials before starting with the game).
Although this study showed the potential of Dojo as a form of treatment, we do not propose Dojo as a stand-alone intervention that is able to replace interventions already in use. Dojo has advantages compared to conventional treatment, but also possible disadvantages. For example, an important element of traditional therapy that may have positive effects is therapeutic alliance (Shirk and Karver 2003), something that is missing for Dojo.
The findings of this study suggest that Dojo is worth further evaluation as an intervention for short-term reductions in anxiety and externalizing problems among high-risk youths in residential care. Playing Dojo resulted in reduced self-reported anxiety and externalizing problems, and mentor-reported anxiety at posttreatment. Of these three effects, only mentor-reported anxiety was maintained at follow-up. There was no effect on mentor-reported externalizing behavior. Although our results should be interpreted cautiously given the small sample size of this study, these findings contribute towards the evidence for a gaming approach in youth interventions. In the future, a RCT with a larger sample could be a useful next step. Not only would this allow mediator/moderator analyses, but Dojo could also be compared with active control groups, such as a closely matched control game or a standardized CBT-based intervention to assess whether Dojo is equally effective or perhaps even superior. To conclude, the results of this initial RCT suggest that Dojo may be an innovative, promising form of treatment for some of the most vulnerable and hard-to-treat populations in our society. A fully powered trial would be necessary to establish its effectiveness.
Albano, A. M., & Kendall, P. C. (2002). Cognitive behavioural therapy for children and youths with anxiety disorder: Clinical research advances. International Review of Psychiatry, 14, 129–134.
Association for the Treatment of Sexual Abusers. (2015). Assessment and treatment of adolescents with intellectual disabilities who exhibit sexual problems or offending behaviors. Beaverton: Author.
Bastiaansen, D., Koot, H. M., Ferdinand, R. F., & Verhulst, F. C. (2004). Quality of life in children with psychiatric disorders: Self-, parent, and clinician report. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 221–230.
Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013). The pervasive problem with placebos in psychology: Why active control groups are not sufficient to rule out placebo effects. Perspectives on Psychological Science, 8, 445–454.
Briggs, E. C., Greeson, J. K., Layne, C. M., Fairbank, J. A., Knoverek, A. M., & Pynoos, R. S. (2012). Trauma exposure, psychosocial functioning, and treatment needs of youth in residential care: Preliminary findings from the NCTSN core data set. Journal of Child and Adolescent Trauma, 5, 1–15.
Cohen, J. W. (1988). Statistical power analyses for the behavioral sciences (2nd ed.). Hillsdale: Lawrence Erlbaum Associate.
Connor, D. F., Doerfler, L. A., Toscano, P. F., Volungis, A. M., & Steingard, R. J. (2004). Characteristics of children and adolescents admitted to a residential treatment center. Journal of Child and Family Studies, 13, 497–510.
de Lange, M., Matthys, W., Foolen, N., Addink, A., Oudhof, M., & Vermeij, K. (2013). Richtlijn ernstige gedragsproblemen voor de jeugdzorg (Protocol severe behavior problems for youth care). Utrecht: Nederlands Jeugdinstituut.
de Swart, J. J. W., van den Broek, H., Stams, G. J. J. M., Asscher, J. J., van der Laan, P. H., Holsbrink-Engels, G. A., & van der Helm, G. H. P. (2012). The effectiveness of institutional youth care over the past three decades: A meta-analysis. Children and Youth Services Review, 34, 1818–1824.
Didden, R. (2006). Gedragsanalyse en cognitieve gedragstherapie bij mensen met een verstandelijke beperking: Een tussenbalans. In R. Didden (Ed.), Gedragsproblemen, psychiatrische stoornissen en lichte verstandelijke beperking (Behavioral problems, psychiatric disorders and intellectual disability). Houten: Bohn Stafleu van Lochum.
Ducharme, P., Wharff, E., Hutchinson, E., Kahn, J., Logan, G., & Gonzalez-Heydrich, J. (2012). Videogame assisted emotion regulation training: An ACT with RAGE-control case illustration. Clinical Social Work Journal, 40, 75–82.
Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G*power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
Freedland, K. E., Mohr, D. C., Davidson, K. W., & Schwartz, J. E. (2011). Usual and unusual care: Existing practice control groups in randomized controlled trials of behavioral interventions. Psychosomatic Medicine, 73, 323–335.
Frensch, K. M., & Cameron, G. (2002). Treatment of choice or a last resort? A review of residential mental health placements for children and youth. Child and Youth Care Forum, 31, 307–339.
Garrido, V., & Morales, L. A. (2007). Serious (violent and chronic) juvenile offenders: A systematic review of treatment effectiveness in secure corrections. In The Campbell collaboration reviews of intervention and policy evaluations (C2-RIPE). Philadelphia: Campbell Collaboration.
Glancy, G., & Saini, M. A. (2005). An evidence-based review of psychological treatments of anger and aggression. Brief Treatment and Crisis Intervention, 5, 229–248.
Goldstein, A. P., & Glick, B. (1987). Aggression replacement training. Champaign: Research Press.
Goodman, R. (1997). The Strenghts and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586.
Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576.
Granic, I. (2014). The role of anxiety in the development, maintenance and treatment of childhood aggression. Development and Psychopathology, 26, 1515–1530.
Granic, I., Lobel, A., & Engels, R. C. M. E. (2014). The benefits of playing video games. American Psychologist, 69, 66–78.
Grills, A. E., & Ollendick, T. H. (2003). Multiple informant agreement and the anxiety disorders interview schedule for parents and children. Journal of the American Academy of Child and Adolescent Psychiatry, 42, 30–40.
Harder, A. T., Knorth, E. J., & Zandberg, T. (2006). Residential youth care in the picture: A review study of its target group, methods, and outcomes. Amsterdam: SWP Publishers (in Dutch).
Harder, A. T., Knorth, E. J., & Kalverboer, M. E. (2011). Risky or needy? Dynamic risk factors and delinquent behavior of adolescents in secure residential youth care. International Journal of Offender Therapy and Comparative Criminology, 59, 1057–1065.
Harder, A. T., Knorth, E. J., & Kalverboer, M. E. (2012). Securing the downside up: Client and care factors associated with outcomes of secure residential youth care. Child and Youth Care Forum, 41, 259–276.
Helmond, P. E., Verberg, F., Vermaes, I. P. R., & Overbeek, G., 2014). De Groeifabriek. Het ontwikkelen van een groeimindset! (The Growth Factory. Developing a growth mindset!). Intervention rapport: Pluryn.
Kahn, J., Ducharme, P., Rotenberg, A., & Gonzalez-Heydrich, J. (2013). “RAGE-control”: A game to build emotional strength. Games for Health Journal, 2, 53–57.
Kaptein, S., Jansen, D. E. M. C., Vogels, A. G. C., & Reijneveld, S. A. (2008). Mental health problems in children with intellectual disability: Use of the strengths and difficulties questionnaire. Journal of Intellectual Disability Research, 52, 125–131.
Kazdin, A. E., & Blase, S. L. (2011). Rebooting psychotherapy research and practice to reduce the burden of mental illness. Perspectives on Psychological Science, 6, 21–37.
Kazdin, A. E., & Rabbitt, S. M. (2013). Novel models for delivering mental health services and reducing the burdens of mental illness. Clinical Psychological Science, 1, 170-191.
Kendall, P. C., Aschenbrand, S., & Hudson, J. L. (2003). Cognitive behavioral treatment of anxiety disordered youth. In A. E. Kazdin & J. R. Weisz (Eds.), Evidence-based psychotherapies for children and adolescents (pp. 81–100). New York: Guilford Press.
Knorth, E. J., Harder, A. T., Zandberg, T., & Kendrick, A. J. (2008). Under one roof: A review and selective meta-analysis on the outcomes of residential child and youth care. Children and Youth Services Review, 30, 123–140.
Kok, J. M., Menkehorst, G. A. B. M., Naayer, P. M. H., & Zandberg, T. (1991). Residentieel gedragstherapeutisch behandelingsprogramma. Ontwikkeling, invoering, effectmeting (residential behavioral-therapeutic treatment program. Development, implementation, effects). Assen: Dekker & van de Vegt.
Magiati, I., Tsakanikos, E., & Howlin, P. (2014). Psychological and social factors. In E. Tsakanikos & J. McCarthy (Eds.), Handbook of psychopathology in intellectual disability research, practice, and policy. New York: Springer.
McCurdy, B. L., & McIntyre, E. K. (2004). And what about residential…?’ Re-conceptualizing residential treatment as a stop-gap service for youths with emotional and behavioral disorders. Behavioral Interventions, 19, 137–158.
Muris, P., Merckelbach, H., Ollendick, T., King, N., & Bogie, N. (2002). Three traditional and three new childhood anxiety questionnaires: Their reliability and validity in a normal adolescent sample. Behavioral Assessment, 40, 753–772.
Muris, P., Meesters, C., & van der Berg, F. (2003). The strengths and difficulties questionnaire (SDQ). Further evidence for its reliability and validity in a community sample of Dutch children and adolescents. European Child & Adolescent Psychiatry, 12, 1–8.
Nijhof, K. S., Veerman, J. W., Engels, R. C. M. E., & Scholte, R. H. J. (2011). Compulsory residential care: An examination of treatment improvement of individual and family functioning. Children and Youth Services Review, 33, 1779–1785.
Pollard, J. A., & Hawkins, J. D. (1999). Risk and protection: Are both necessary to understand diverse behavioral outcomes in adolescents? Social Work Research, 23, 145–158.
Rapee, R. M., Wignall, A., Hudson, J. L., & Schniering, C. A. (2000). Treating anxious children and adolescents: An evidence-based approach. Oakland: Harbinger.
Rosas, R., Nussbaum, M., Cumsille, P., Marianov, V., Correa, M., Flores, P., et al. (2003). Beyond Nintendo: Design and assessment of educational video games for first and second grade students. Computers & Education, 40, 71–94.
Scholten, H., Malmberg, M., Lobel, A., Engels, R. C. M. E., & Granic, I. (2016). A randomized controlled trial to test the effectiveness of an immersive 3D video game for anxiety prevention among adolescents. PLoS One, 11, e0147763.
Schuurmans, A. A. T., Nijhof, K. S., Vermaes, I. P. R., Engels, R. C. M. E., & Granic, I. (2015). A pilot study evaluating “dojo,” a videogame intervention for youths with externalizing and anxiety problems. Games for Health Journal, 4, 401–408.
Shenk, C., & Brown, A. (2007). Cognitive-behavioral treatment of an adolescent sexual offender with an intellectual disability. A novel application of exposure and response prevention. Clinical Case Studies, 6, 307–324.
Shirk, S. R., & Karver, M. (2003). Prediction of treatment outcome from relationship variables in child and adolescent therapy: A meta-analytic review. Journal of Consulting and Clinical Psychology, 71, 452–464.
Spence, S. H. (1998). A measure of anxiety symptoms among children. Behaviour Research and Therapy, 36, 545–566.
Steinberg, L., & Avenevoli, S. (2000). The role of context in the development of psychopathology: A conceptual framework and some speculative propositions. Child Development, 71, 66–74.
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., et al. (2009). Multiple imputation for missing data in epidemiological and clinical research: Potential and pitfalls. BMJ, 338, b2393.
Sturney, P., & Hamelin, J. P. (2014). Psychological treatments. In E. Tsakanikos & J. McCarthy (Eds.), Handbook of psychopathology in intellectual disability research, practice, and policy. New York: Springer.
Sukhodolsky, D. G., & Scahill, L. (2012). Cognitive-behavioral therapy for anger and aggression in children and adolescents. New York: Guilford Press.
Taylor, J. L. (2002). A review of the assessment and treatment of anger and aggression in offenders with intellectual disability. Journal of Intellectual Disability Research, 46, 57–73.
Binsbergen, M. H. van (2003). Motivatie voor behandeling: Ontwikkeling van behandelmotivatie in een justitiële instelling (Motivation for treatment. Development of treatment motivation in juvenile justice institutions). University of Leiden: Doctoral dissertation.
van Nieuwenhuijzen, M. (2010). De (h)erkenning van jongeren met een licht verstandelijke Beperking (The recognition of youths with intellectual disability). Amsterdam: SWP.
van Widenfelt, B. M., Goedhart, A. W., Treffers, P. D. A., & Goodman, R. (2003). Dutch version of the Strenghts and difficulties questionnaires (SDQ). European Child and Adolescents Psychiatry, 12, 281–289.
Vitaro, F., Brendgen, M., & Barker, E. D. (2006). Subtypes of aggressive behaviors: A developmental perspective. International Journal of Behavioral Development, 30, 12–19.
Vygotsky, L. (1978). Mind in society: The development of higher psychological functions. Cambridge: Harvard Press.
Weisz, J. R., & Kazdin, A. E. (2010). Evidence-based psychotherapies for children and adolescents (2nd ed.). New York: Guilford Press.
Whittaker, J. K., del Valle, J. F., & Holmes, L. (2015). Therapeutic Residential Care For Children and Youth. Developing Evidence-Based International Practices. London and Philadelphia: Jessica Kingsley Publishers.
Yucha, C., & Mongomery, D. (2008). Evidence-based practice in biofeedback and neurofeedback. Colorado: Association for Applied Psychophysiology and Biofeedback.
The authors gratefully acknowledge GameDesk for the development of Dojo.
Conflicts of Interest
Angela A. T. Schuurmans, Karin S. Nijhof, Rutger C. M. E. Engels and Isabela Granic declare that they have no conflicts of interests.
All performed study procedures were in accordance with the ethical standards of Pluryn, the Faculty of Social Sciences, Radboud University Nijmegen, and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent was obtained from all individual participants included in the study.
About this article
Cite this article
Schuurmans, A.A.T., Nijhof, K.S., Engels, R.C.M.E. et al. Using a Videogame Intervention to Reduce Anxiety and Externalizing Problems among Youths in Residential Care: an Initial Randomized Controlled Trial. J Psychopathol Behav Assess 40, 344–354 (2018). https://doi.org/10.1007/s10862-017-9638-2
- Randomized controlled trial
- Residential care
- Externalizing problems
- Videogame intervention
- Intellectual disability