Learning struggles are a key characteristic of neurodevelopmental disorders—including specific learning disorder, attention deficit hyperactivity disorder (ADHD), and language disorder (American Psychiatric Association 2013)—but are also found among neurotypical children. Cognitive skill deficits are common among children who struggle to learn, especially deficits in working memory (Alloway et al. 2009), processing speed (Lewandowski et al. 2007), and executive functions (McQuade et al. 2011). Because cognitive skills are significant predictors of academic success (Freberg et al. 2008), finding interventions that remediate cognitive deficits must necessarily be a priority.

Cognitive training as a targeted intervention is receiving much attention. Research on cognitive training for older adults is growing rapidly, but research with children is limited. Further, the preponderance of existing research with children has been conducted using digital cognitive training programs. One of the advantages of digital programs is online delivery, which facilitates widespread accessibility at comparably lower costs. Digital programs, however, have historically demonstrated a significant limitation—they only target single constructs such as working memory to the exclusion of other neuropsychological constructs (Cortese et al. 2015). In addition, the results of studies on digital programs have been mixed with several reporting improvements in working memory (Holmes and Gathercole 2014; Klingberg et al. 2002; Klingberg et al. 2005), but others not consistently replicating positive results (Chacko et al. 2013; van der Donk et al. 2015). In similar research on action video games, researchers reported significant improvements in visual processing (Green and Bavelier 2007), visual selection attention (Green and Bavelier 2003), and visuospatial attention and visual-to-auditory attentional shifting in children with dyslexia (Franceschini et al. 2013; Franceschini et al. 2017). One trial of a digital cognitive training program reported a significant difference between treatment and control groups on inhibition and metacognition (Van der Oord et al. 2012), and another found significant changes in the treatment group on attention and ADHD symptoms (Beck et al. 2010); the outcome measures in both studies were based solely on parent and teacher reports. Further, much controversy surrounding cognitive training has specifically targeted computerized training (Melby-Lervag and Hulme 2016). To our knowledge, a similar debate about the general effects of clinician-delivered cognitive training interventions for children is absent in the peer-reviewed literature. Studies on clinician-delivered cognitive training programs for children have shown positive results that include transfer of training to tests of fluid intelligence and academic performance with effects that remain stable over time (Klauer and Phye 2008). Studies on Feuerstein’s Instrumental Enrichment (IE) training, for example, consistently revealed cognitive improvements among children with learning disabilities (Kozulin et al. 2010; Schnitzer et al. 2007), and children with ADHD realized improvements in executive functioning after direct attention training (Kerns et al. 1999; Tamm et al. 2010).

However, as the cognitive training field continues to grow, so do questions about the efficacy of each program. In particular, questions arise about differences between clinician-delivered cognitive training and computer-based training. Is one mode of delivery more efficacious than another? Is it possible to scale up the demonstrable benefits of clinician-delivered programs to online delivery for wider accessibility? Answering such questions could prove challenging due to the difficulty of disentangling effects in studies that compare, for example, an online program by one company to clinical program of a different company. One solution to that challenge is to compare two delivery methods of the same cognitive training program, which inherently controls for multiple variables and enables the researchers to focus on mode of delivery as the predictor of outcomes.

The current study examined similarities and differences in cognitive outcomes between two methods of delivering the same cognitive training program, ThinkRx (Gibson et al. 2003). To our knowledge, ThinkRx is the only clinician-delivered training program for children that targets multiple cognitive skills—including working memory, long-term memory, processing speed, visual processing, auditory processing, logic and reasoning, and attention. In prior studies examining ThinkRx with children, training effects were significant across multiple cognitive constructs as measured by standard neuropsychological tests, functional magnetic resonance imaging, and parent rating scales. In a randomized controlled trial examining the efficacy of ThinkRx for children ages 8–14 (n = 39), results showed statistically significant differences between the treatment and control groups on measures of working memory, long-term memory, processing speed, visual processing, auditory processing, fluid reasoning, and overall IQ score (Carpenter et al. 2016). In an earlier controlled study on ThinkRx with children ages 6–18 (n = 61), researchers found significant differences between treatment and controls on long-term memory, logic and reasoning, working memory, processing speed, auditory processing, and Word Attack skills (Gibson et al. 2015). Jedlicka (2017) examined changes in academic skills and behavior and found significantly reduced academic difficulties for school-age children with learning struggles in two treatment groups (60 h of ThinkRx training and 120 h of ThinkRx plus reading training) compared to a no contact control group. In a small study of children with attention problems (n = 13), the group trained with ThinkRx significantly outperformed the control group on measures of working memory, long-term memory, logic and reasoning, auditory processing, and IQ score and also noted improvements in self-confidence, self-discipline, and cooperative behaviors along with clinically significant change (Moore et al. 2018). In a randomized controlled trial with high school students (n = 30), resting state magnetic resonance imaging (fMRI) results showed significant differences between ThinkRx treatment and controls on overall global efficiency associated with visual and auditory processing, contextual associations, and the cerebellum (Ledbetter et al. 2016).

The proposed mechanism of change for the ThinkRx method is grounded in Feuerstein’s theory of structural cognitive modifiability (Feuerstein 1990; Feuerstein et al. 2010) and the phenomenon of neuroplasticity. Feuerstein suggests that mediated learning experiences delivered by an adult and the intentional focus of a child’s interaction with the world actually modify cognition. Indeed, research on his cognitive training programs has indicated such an effect. Further, the deliberate sustained exposure to targeted mental exercises capitalizes on the brain’s plasticity and changes the neural structure (Willis et al. 2006), which has been seen in ThinkRx research.

ThinkRx is routinely used in more than 70 cognitive training centers and 50 clinics in the USA and in another 40 countries around the world. In the USA, it is delivered one-on-one by a clinician or cognitive trainer. As such, clinic space is limited, and the cost per hour is comparable to counseling or therapy. The international centers utilize a more scalable model of delivering ThinkRx that combines clinician-delivered training with digital training to make the program more affordable and available to a larger number of children at one time. The obvious working assumption is equivalency between the two modes of delivery. However, that assumption has not been tested. In the current study, we examined the equivalency of the more scalable delivery model compared to the traditional solely one-on-one model of delivery. The current study was guided by the following research question: Are there statistically significant differences in cognitive growth between those who experience cognitive training entirely through a clinician-delivered model compared to those who experience cognitive training in a mixed-delivery model (online and clinician-delivered)? We answer this question through a randomized, controlled trial assessing the equivalence of two groups. Given that clients in international clinics using the mixed delivery model achieve similar results to clients in US clinics using the one-on-one delivery model and that adult participants in a ThinkRx study using a mixed delivery model achieved clinically significant results (Ledbetter et al. 2017), our hypothesis is that there is no significant difference in cognitive skill improvements between the two delivery methods of ThinkRx cognitive training.

Method

Participants

A sample of 39 participants between the ages of 8 and 14 were recruited through an email invitation to a large database of families who had contacted the Colorado Springs LearningRx training center for information about the program. LearningRx is a national network of clinics that offer a comprehensive clinician-delivered cognitive training curriculum for children and adults as well as intensive reading and math interventions for children and adolescents. The primary clientele of LearningRx centers is children with learning disabilities or academic struggles, including dyslexia, attention deficit hyperactivity disorder, and speech and language delays. Eligibility was limited to participants between the ages of 8 and 14 living within commuting range of Colorado Springs who scored between 70 and 130 on the general intellectual ability (GIA) composite of the Woodcock Johnson III – Tests of Cognitive Abilities during screening. A projected sample size of 40 was selected based on availability of research and development funding for the project. Using results from Carpenter et al. (2016), we also used G*Power to perform a priori power analysis to determine if this sample size would be sufficient. With an alpha of 0.05, power at 0.80, and effect sizes from Carpenter et al., the sample of 40 was more than adequate.

Of the 43 volunteers, 39 met the criteria for participation. In accordance with the Declaration of Helsinki, parents provided written informed consent and the minor participants assented to participating in the study. The study was approved by the Institutional Review Board (IRB) at the Gibson Institute of Cognitive Research and is registered at ClinicalTrials.gov as identifier NCT02927197. Recruitment began in June 2015 and final assessments concluded in March 2016.

Procedure

Using blocked sampling with siblings and individuals, participants were randomly assigned by the study coordinator either to the ThinkRx group (n = 20) to receive 60 h of one-on-one clinician-delivered cognitive training or to the mixed delivery group (n = 19) to receive 30 h of one-on-one clinician-delivered cognitive training plus 30 h of clinician-supervised, computer-based cognitive training. Blocking by sibling or individual status was chosen to reduce the risk of attrition and contamination if siblings were assigned to different groups. Randomization was conducted by a research team member not involved in the assessment or intervention who assigned the participants to the interventions. Allocation sequence was concealed using sequentially numbered opaque, sealed envelopes. Participants and their parents were blind to the condition in which they were assigned. They were simply told that we were examining differences between two versions of the same cognitive training program. There was no chance for contamination because the groups attended training sessions successively rather than simultaneously. That is, the one-on-one clinician-delivered ThinkRx group completed their training in the first phase of the study and the mixed delivery group began their training in the second phase of the study after the first group was finished. The mixed delivery group served as the waitlist control group for the first phase of the study (Carpenter et al. 2016).

Table 1 illustrates the pre-intervention demographics including diagnoses. A check of the random assignment indicated the groups were balanced, with no significant differences between groups based on personal characteristics (age: t = 0.217, p = 0.829; gender: χ2 = 1.80, p = 0.210; race/ethnicity: χ2 = 2.35, p = 0.218; no disability: χ2 = 0.54, p = 0.522). One participant in the one-on-one delivery group was on stimulant medication for ADHD, and the medication status remained stable throughout the study. Each participant underwent pre-testing and post-testing with subtests from the Woodcock Johnson III – Tests of Cognitive Abilities (Woodcock et al. 2001) administered by master’s-level clinicians blind to the condition of the participants. Both groups were tested by the same clinicians. Pre- and post-testing was supervised by a doctoral-level psychologist who was aware of the group assignments. In addition to conducting pre- and post-cognitive testing, the research team met with parents and cognitive trainers before the intervention, at the midpoint, and at the completion of the intervention to document qualitative changes observed in the participants.

Table 1 Pre-intervention demographics for each group
Full size table

Two methods of delivering ThinkRx cognitive training were administered. Participants in both groups attended three or four 90-min cognitive training sessions per week for a total of 40 sessions. Training was delivered or monitored by certified cognitive trainers at two sites: a cognitive training center and a cognitive science research facility with training rooms designed to mimic the rooms in the training center. Two master trainers monitored program fidelity based on a comprehensive plan that included fidelity in design to prevent contamination, training and observation of cognitive trainers using a training manual and observation checklist, and tracking of intervention delivery including number of visits, frequency of visits, length of visits, and consistency in skills targeted. All but one participant completed the entire 60-h training protocol. The remaining participant—a member of the mixed delivery group—was unable to complete the complex training exercises and was instead trained with a related cognitive training curriculum designed for younger children. The one-on-one delivery group received all 60 h of ThinkRx cognitive training delivered one-on-one by a cognitive trainer. The Mixed Delivery group received 30 h of cognitive training delivered one-on-one by a trainer and another 30 h of training via digital delivery on an iPad in a monitored computer lab at each intervention site. The same staff of cognitive trainers trained both groups.

Assessments

Subtests 1 through 7 of the Woodcock Johnson III Tests of Cognitive Abilities (Woodcock et al. 2001) were administered, so a general intellectual ability composite could subsequently be generated. Subtest 10 was also administered to obtain a measure of long-term memory. Although no measure of selective or divided attention is available on the Woodcock Johnson test battery, the Numbers Reversed subtest served as a measure of attentional capacity as well as working memory (Mather and Woodcock 2001). A description of the tests is listed in Table 2.

Table 2 Brief description of Woodcock Johnson III outcome measures
Full size table

Interventions

One-on-one Delivery

The traditional ThinkRx cognitive training program was administered one-on-one by a cognitive trainer during 90-min sessions 3 or 4 days per week for the duration of the study. The program is grounded in the Cattell-Horn-Carrol (CHC) theory of intelligence and the multiple-construct view of cognition (McGrew 2005). As such, the ThinkRx training procedures were developed to target multiple cognitive skills such as working memory, long-term memory, processing speed, visual processing, auditory processing, logic and reasoning, and attention. The 230-page curriculum includes 23 training tasks with more than 1000 variations and levels of difficulty. In the current study, the trainer sat across a table from the participant and utilized a variety of materials to deliver the intervention. Those materials included a metronome, timer, shape and number cards, Tangrams, speeded activity worksheets, visual logic cards, and even a footbag and mini-trampoline to encourage simultaneous motor skill and cognitive skill development. During each training session, the trainer loaded the training tasks with additional cognitive activities, such as counting aloud to the beat of a metronome, performing mathematical calculations, or answering questions about an unrelated task. To train divided and sustained attention skills and the ability to work in the presence of distractions, the trainer added deliberate distractions to the training tasks, such as clapping to a different metronome beat, asking silly questions, singing a song, or making funny sounds. Throughout the session, trainers provided immediate feedback and made adjustments as participants progressed through each level of task difficulty. The metronome is a key component of the ThinkRx program and used in nearly every training procedure to create intensity, develop attention skills, progressively improve processing speed, and ensure there are no mental breaks. Progress and mastery of each level of task were carefully tracked using a task flow sheet. Every participant followed the same scope and sequence—it was the speed of progression and the slope of the trajectory that varied from person to person based on abilities.

During the first training session, the cognitive trainer helped participants establish goals for improvements they would like to reach, such as better academic performance or improvements in sports or hobbies. Then, the first few minutes of each training session were spent revisiting goals and discussing real-life improvements. This important metacognitive aspect of the training program was designed to increase motivation for training, develop the relationship between the participant and the cognitive trainer, and help the participant apply any new skills outside of the training environment.

Mixed Delivery

Participants in the mixed delivery group attended 90-min training sessions 3 or 4 days per week exactly like the one-on-one delivery group. However, the mixed delivery group participants spent half of each training session working one-on-one with a cognitive trainer and the other half of each training session in a supervised computer lab where they completed digital training exercises on an iPad. A lab assistant monitored the participants to ensure they remained on-task and helped participants with any technical questions. Eight to ten members of the mixed delivery group were always scheduled at the same time to maximize the use of the computer lab as the scaling feature of the delivery model. Half began with the one-on-one portion of the training session while the remaining half began in the computer lab.

The digital training program, called Brainskills, is a computerized version of ten training exercises from the traditional ThinkRx program that progresses by levels of intensity and difficulty like the one-on-one version. That is, the ten computer-based tasks were designed to mimic ten of the trainer-delivered tasks and were the only tasks that could be digitally adapted with a high level of fidelity. Figure 1 shows an example of both versions of the same exercise: Reasoning Brain Cards. In this inductive logic exercise, participants use a set of rules to identify a three-card group from a set of 12 cards that each has four features: shape, color, size, and orientation. One task is to identify three cards that share all the same variable. Another task is to identify a card that is not shown but that would complete a set.

Fig. 1
figure 1

Reasoning Brain Cards. Both delivery versions of the same reasoning training procedure

Full size image

The software is web-based and accessible via the Internet on both tablets and computers. In the current study, Brainskills was accessed on iPad tablets on stands and used with headphones. A research assistant generated weekly Brainskills progress reports for the master trainers to review. This ensured that participants were progressing through the digital part of the training curriculum in the same way they progressed through it in their one-on-one portion of the training. If a participant was not progressing through the tasks as expected, the master trainer provided assistance during a subsequent digital training session.

Data Analysis

Data were analyzed using multivariate analysis of variance (MANOVA), with the dependent variables being the difference scores between the pre- and post-tests for each measure. In other words, the study used a difference-in-difference analysis for all measures. Given the number of pairwise comparisons (i.e., eight, one for each measure), a Bonferroni correction was applied to the multiple comparisons. Effect sizes were also calculated for all measures using Cohen’s d and described using Cohen’s (1988) general guidance of small (0.2), medium (0.5), or large (0.8) effects. To address the potential for Lord’s Paradox (Wainer 1991), we conducted an alternate series of individual analyses of covariance (ANCOVA) for each post-test score as a dependent variable with pre-test scores as covariates, including a Bonferroni correction for multiple comparisons. Because the results were conceptually similar, we chose to report the MANOVA findings, with one exception described below.

As mentioned above, the sample included sibling clusters in both groups. Because this violates the assumption of independence, we first used intra-class correlations (ICC) to determine if dependence among some of the cases was significant (Grawitch and Munz 2004). ICC results were statistically significant for three of the outcome measures—processing speed, working memory and attentional capacity, and GIA. Consequently, we analyzed the eight dependent variables (not just three significant variables, out of an abundance of caution) using OLS regression with clustered standard errors to determine if the clustering resulted in substantively different results compared to the MANOVA analysis. Regression results were substantively the same, so we report MANOVA results below.

In a final preliminary analysis, we examined if diagnosis was a significant predictor of the changes in each construct. Recall these participants reported some form of learning difficulty that may stem from a diagnosis of ADHD, dyslexia, or a number of other disabilities. Because such diagnoses could confound the results from the primary analysis, we performed a series of linear regression analyses with diagnosis as the predictor variable and gain scores as the dependent variables to determine the extent to which the diagnoses might need to be controlled. Diagnosis was not a significant predictor of change for any of the outcome variables: associative memory F (1,37) = 1.45, p = 0.24; visual processing F (1,37) = 0.781, p = 0.38; auditory processing F (1,37) = 1.29, p = 0.26; processing speed F (1,37) = 3.11, p = 0.09; working memory and attentional capacity F (1,37) = 1.38, p = 0.25; logic and reasoning F (1,37) = 0.758, p = 0.39; long-term memory F (1,37) = 1.83, p = 0.18; and IQ score F (1,37) = 0.38, p = 0.54.

Data Availability

The datasets generated during and/or analyzed during the current study are available in the Harvard Dataverse repository, https://dataverse.harvard.edu/dataverse/thinkrxvsbrainlab.

Results

Preliminary Results

Data screening indicated no missing data, and all variables were within tolerable ranges for skewness. Finally, comparisons of pre-test scores indicate groups were statistically equivalent on almost all measures (associative memory t = 1.21, p = 0.24; visual processing t = 1.68, p = 0.10; auditory processing t = − 1.87, p = 0.07; processing speed t = 0.47, p = 0.65; working memory and attentional capacity t = − 0.02, p = 0.98; logic and reasoning t = 0.50, p = 0.62; IQ t = − 0.28, p = 0.78), with the exception of long-term memory (t = 2.66, p = 0.01), where the one-on-one delivery group pre-test score (M = 98.65, SD = 19.55) was greater than the mixed delivery group pre-test score (M = 104.56, SD = 21.68). Due to the significant difference on this measure, we provide results below from the aforementioned ANCOVA, in addition to the MANOVA, as the former presents the post-test results after controlling for the pre-tests. One member of the mixed delivery group was excluded from the analysis because the participant was not able to participate due to the complexity of the training tasks. Instead, the participant was provided with a different cognitive training intervention designed for younger children.

Results of Statistical Significance Testing

As shown in Table 3, MANOVA results indicate an overall significant difference between groups (F = 3.36, p = 0.01), with pairwise comparisons indicating the significant difference between groups was on one of the eight measures—long-term memory. On that measure, the one-on-one delivery group showed greater growth than the mixed delivery group.

Table 3 Results of significance testing between groups
Full size table

Turning to effect sizes calculated as Cohen’s d indicating the magnitude of the significance, the greatest difference was measured on long-term memory, followed by visual processing (one-on-one delivery group was designated as the treatment in these calculations). The former saw a large effect size, with the latter approaching a large effect size. The smallest effect size was measured on working memory and attentional capacity, then IQ score and processing speed, all of which saw small effect sizes. As for the ANCOVA analysis for long-term memory, results indicate post-test scores were significantly greater for the one-on-one delivery group participants, after controlling for pre-test scores (F = 6.94, p = 0.01). The one-on-one delivery group participants conditionally scored approximately 13 points greater than mixed delivery group participants (Mone-on-one = 128.05, se = 3.36; Mmixed = 115.11, se = 3.55).

Discussion

This study compared the equivalency of two delivery methods of the same cognitive training program: ThinkRx delivered solely one-on-one by a cognitive trainer and ThinkRx delivered 50% one-on-one by a cognitive trainer and the other 50% via computerized training procedures in a supervised computer lab. Largely, the results were similar between the groups, suggesting that the mixed delivery model may be a feasible method of scaling the intervention to reach more children while achieving comparable results. As an alternative to a cognitive trainer working with one child every 90 min, the addition of a computer lab enables the trainer to work with two children in a 90-min session when they alternate between one-on-one training time and time in the computer lab.

However, there was a significant difference in outcomes between the groups on long-term memory, which needs to be addressed. The one-on-one delivery group gains from pre-test to post-test were more than twice those of the mixed delivery group. An evaluation of individual components of the delivery models should be considered to address the discrepancy in long-term memory outcomes. That is, what characteristics of the training models may have contributed to the difference? A check of the fidelity monitoring records indicated the one-on-one delivery group did not spend more training time working on long-term memory tasks with the cognitive trainer than the mixed delivery group. However, the computerized training procedures do not include multitasking components that target memory skills. In the absence of the trainer during the digital training time, there is no method for adding distractions and additional tasks to complete orally on top of the digital exercises. Thus, the one-on-one delivery group received more exposure to task loading by the cognitive trainer than the mixed delivery group. Although this would seemingly address working memory development, it may have contributed to the difference in long-term memory skills at post-test given the nature of the assessment task. That is, the task measures automaticity of retrieval (Schrank 2011), which the mixed delivery participants may not have developed as strongly as the one-on-one delivery group. After performing a replication study with a much larger sample, programming random distractions or adding a multitasking component to the digital Brainskills program may be a consideration for narrowing this gap on long-term memory outcomes if it indeed exists. There is, however, a small chance that the difference in outcomes is the result of the significant differences on long-term memory scores between the groups at pre-test. The mixed delivery group began the study with pre-test scores almost five points above the population mean while the one-on-one delivery group began the study with pre-test scores just below the population mean. Therefore, the 6.5-point difference at pre-test may have influenced the results. However, the difference between groups on the long-term memory measure remained significant even after controlling for these pre-test differences in the analysis.

It is important to note the training tasks are complex and qualitatively different from the assessment tasks, so the transfer to untrained tasks is not a result of “training to the test.” Further, the assessment tasks are designed to measure cognitive skills in isolation while each training procedure targets multiple skills simultaneously. To illustrate, consider the difference in how processing speed was assessed versus how it was trained. The assessment task for processing speed required participants to locate and circle pairs of identical images. One of the intervention tasks to train processing speed required participants to perform diverse actions on a set of related letters in a limited time period, such a circling every “p,” underlining every “d,” crossing out every “b,” and putting a box around every “q,” which trained not only processing speed, but also attention, working memory, visual discrimination, visual span, and sensory motor integration. The cognitive trainer might load the training task by asking the participant to recite the alphabet or count backwards from 100 on beat to the metronome at the same time. In the intervention, processing speed was also targeted in every training task by using the metronome or a stopwatch, which forced participants to make decisions more rapidly regardless of the construct the task was designed to improve.

There are a few limitations and implications for future research in the current study. First, the sample size is small. Although the sample is consistent with sample sizes in similar cognitive training studies, there is a risk that the results are due to the limited statistical power of the design. The field would benefit from a replication study with a larger sample size. Next, long-term outcomes in the current sample have not been assessed. An important gap in the literature is the long-term retention of cognitive training gains. Although beyond the scope of the current study, it will be important to conduct follow-up assessments of this study sample to determine if improvements in cognitive and functional skills persist as well as they have in prior follow-up assessments following the intervention used in the current study (Wainer and Moore 2016). A potential limitation may also be that working memory and attentional capacity were measured using the same assessment task. However, there is research to support the use of digit span tasks in measuring scope of attention and attentional control in children (Cowan et al. 2005). Indeed, the constructs are related and challenging to disentangle. It is important to note, though, that the inclusion of the associative memory task further delineates the memory constructs. That means that if we were to simply refer to the digit span task as the attentional control measure in the current study, we are still left with two additional measures of memory: associative memory and delayed recall. Thus, the potential confounding of constructs measured by the digit span task is mitigated. In future research, the addition of a continuous performance test would enable the assessment of divided and selective attention as well.

Future research should include formal measures of far transfer to behavioral and functional outcomes. In the current study, the researchers collected session notes from the cognitive trainers, which documented self-reported improvements noted by the participants. All participants in both groups reported improvements in academic skills, saying things such as, “I can comprehend faster and better in math and language arts,” “I’m remembering what I read,” and “In Algebra I can complete problems faster.” Almost all participants in both groups also noted changes in self-esteem, saying things like, “I’m less worried about making mistakes and questioning my answers,” “I’m less shy and I feel more confident,” and “I’m feeling good about myself.” Although participants also reported changes in relationships with others, self-discipline, skill in sports and hobbies, and overall cognitive improvement, we did not use an objective or quantitative measure of far transfer. This was a limitation of the current study and is an important area to address in future research.

Another limitation of the study is that it was not possible to compare the delivery methods of all the training tasks because only ten of the training tasks were suitable for adaptation to digital format. This is a limitation of digitization for the field overall, however, because many training procedures lend themselves only to delivery in person. Finally, we compared treatment delivery methods that overlapped rather than two methods that were completely distinct. The design creates some difficulty in teasing apart the different mechanisms at work in each method. However, there is substantial ecological validity to the design. The two delivery methods tested in the current study are the methods used in clinics around the world. Therefore, it was important to evaluate them “as is” in order to test for equivalency.

The question of placebo or expectancy effects often arises when considering the outcomes of intervention studies. In the current study, these effects are controlled by the design because both groups completed an intervention for the same number of hours in the presence of an adult, and neither group knew there was a difference in delivery methods. However, it is interesting to note that prior research using two control groups to test the presence of placebo effects in cognitive training studies has failed to find any, including two meta-analyses that revealed no statistically significant differences between active and passive controls (Au et al. 2015; Peng and Miller 2016). Thus, we suggest that expectancy or placebo effects were unlikely in the current study as well. Although the authors examined the difference between the one-on-one delivery method and a control group in the first phase of this study (Carpenter et al. 2016), future research could include a replication study with all three groups to confirm the lack of placebo effects with the current intervention.

The results of the current study provide support for the use of mixed delivery in scaling the ThinkRx program to reach more children, including those with neurodevelopmental and learning disorders. The results were consistent with prior studies on ThinkRx with children (Carpenter et al. 2016; Gibson et al. 2015; Jedlicka 2017; Moore et al. 2018) and with observational data on nearly 18,000 clients (Wainer and Moore 2016) in remediating multiple cognitive skills. This convergence of evidence is important for determining the benefit of adopting an intervention for use in clinical practice (Carey and Stiles 2016). The ability to scale a one-on-one intervention has important implications for its use in clinical practice where time is a limiting factor for scheduling clients. The mixed delivery model enables each clinician to see twice as many clients and provide the intervention to more children who need it.