Advertisement

Comparing Two Methods of Delivering ThinkRx Cognitive Training to Children Ages 8–14: a Randomized Controlled Trial of Equivalency

  • Amy Lawson Moore
  • Dick M. CarpenterII
  • Terissa M. Miller
  • Christina Ledbetter
Open Access
Original Article

Abstract

Cognitive training is growing in popularity as an intervention for children who struggle to learn. In the current study, we compared the equivalency of two delivery models of the same cognitive training program, ThinkRx, for children ages 8–14. In a randomized controlled trial assessing equivalence, we compared cognitive outcomes between a group who received 60 h of ThinkRx cognitive training delivered one-on-one by a cognitive trainer (n = 20) versus a group of children who received 30 h of ThinkRx delivered by a cognitive trainer and the other 30 h through digital training procedures (n = 18). Results showed no significant differences between groups on tests of working memory, logic and reasoning, auditory processing, visual processing, processing speed, or overall IQ score. Results were significantly different on the test of long-term memory. These results suggest that both delivery models are equivalent cognitive training interventions for children.

Keywords

Cognitive training Brain training Cognition Learning disabilities LearningRx 

Learning struggles are a key characteristic of neurodevelopmental disorders—including specific learning disorder, attention deficit hyperactivity disorder (ADHD), and language disorder (American Psychiatric Association 2013)—but are also found among neurotypical children. Cognitive skill deficits are common among children who struggle to learn, especially deficits in working memory (Alloway et al. 2009), processing speed (Lewandowski et al. 2007), and executive functions (McQuade et al. 2011). Because cognitive skills are significant predictors of academic success (Freberg et al. 2008), finding interventions that remediate cognitive deficits must necessarily be a priority.

Cognitive training as a targeted intervention is receiving much attention. Research on cognitive training for older adults is growing rapidly, but research with children is limited. Further, the preponderance of existing research with children has been conducted using digital cognitive training programs. One of the advantages of digital programs is online delivery, which facilitates widespread accessibility at comparably lower costs. Digital programs, however, have historically demonstrated a significant limitation—they only target single constructs such as working memory to the exclusion of other neuropsychological constructs (Cortese et al. 2015). In addition, the results of studies on digital programs have been mixed with several reporting improvements in working memory (Holmes and Gathercole 2014; Klingberg et al. 2002; Klingberg et al. 2005), but others not consistently replicating positive results (Chacko et al. 2013; van der Donk et al. 2015). In similar research on action video games, researchers reported significant improvements in visual processing (Green and Bavelier 2007), visual selection attention (Green and Bavelier 2003), and visuospatial attention and visual-to-auditory attentional shifting in children with dyslexia (Franceschini et al. 2013; Franceschini et al. 2017). One trial of a digital cognitive training program reported a significant difference between treatment and control groups on inhibition and metacognition (Van der Oord et al. 2012), and another found significant changes in the treatment group on attention and ADHD symptoms (Beck et al. 2010); the outcome measures in both studies were based solely on parent and teacher reports. Further, much controversy surrounding cognitive training has specifically targeted computerized training (Melby-Lervag and Hulme 2016). To our knowledge, a similar debate about the general effects of clinician-delivered cognitive training interventions for children is absent in the peer-reviewed literature. Studies on clinician-delivered cognitive training programs for children have shown positive results that include transfer of training to tests of fluid intelligence and academic performance with effects that remain stable over time (Klauer and Phye 2008). Studies on Feuerstein’s Instrumental Enrichment (IE) training, for example, consistently revealed cognitive improvements among children with learning disabilities (Kozulin et al. 2010; Schnitzer et al. 2007), and children with ADHD realized improvements in executive functioning after direct attention training (Kerns et al. 1999; Tamm et al. 2010).

However, as the cognitive training field continues to grow, so do questions about the efficacy of each program. In particular, questions arise about differences between clinician-delivered cognitive training and computer-based training. Is one mode of delivery more efficacious than another? Is it possible to scale up the demonstrable benefits of clinician-delivered programs to online delivery for wider accessibility? Answering such questions could prove challenging due to the difficulty of disentangling effects in studies that compare, for example, an online program by one company to clinical program of a different company. One solution to that challenge is to compare two delivery methods of the same cognitive training program, which inherently controls for multiple variables and enables the researchers to focus on mode of delivery as the predictor of outcomes.

The current study examined similarities and differences in cognitive outcomes between two methods of delivering the same cognitive training program, ThinkRx (Gibson et al. 2003). To our knowledge, ThinkRx is the only clinician-delivered training program for children that targets multiple cognitive skills—including working memory, long-term memory, processing speed, visual processing, auditory processing, logic and reasoning, and attention. In prior studies examining ThinkRx with children, training effects were significant across multiple cognitive constructs as measured by standard neuropsychological tests, functional magnetic resonance imaging, and parent rating scales. In a randomized controlled trial examining the efficacy of ThinkRx for children ages 8–14 (n = 39), results showed statistically significant differences between the treatment and control groups on measures of working memory, long-term memory, processing speed, visual processing, auditory processing, fluid reasoning, and overall IQ score (Carpenter et al. 2016). In an earlier controlled study on ThinkRx with children ages 6–18 (n = 61), researchers found significant differences between treatment and controls on long-term memory, logic and reasoning, working memory, processing speed, auditory processing, and Word Attack skills (Gibson et al. 2015). Jedlicka (2017) examined changes in academic skills and behavior and found significantly reduced academic difficulties for school-age children with learning struggles in two treatment groups (60 h of ThinkRx training and 120 h of ThinkRx plus reading training) compared to a no contact control group. In a small study of children with attention problems (n = 13), the group trained with ThinkRx significantly outperformed the control group on measures of working memory, long-term memory, logic and reasoning, auditory processing, and IQ score and also noted improvements in self-confidence, self-discipline, and cooperative behaviors along with clinically significant change (Moore et al. 2018). In a randomized controlled trial with high school students (n = 30), resting state magnetic resonance imaging (fMRI) results showed significant differences between ThinkRx treatment and controls on overall global efficiency associated with visual and auditory processing, contextual associations, and the cerebellum (Ledbetter et al. 2016).

The proposed mechanism of change for the ThinkRx method is grounded in Feuerstein’s theory of structural cognitive modifiability (Feuerstein 1990; Feuerstein et al. 2010) and the phenomenon of neuroplasticity. Feuerstein suggests that mediated learning experiences delivered by an adult and the intentional focus of a child’s interaction with the world actually modify cognition. Indeed, research on his cognitive training programs has indicated such an effect. Further, the deliberate sustained exposure to targeted mental exercises capitalizes on the brain’s plasticity and changes the neural structure (Willis et al. 2006), which has been seen in ThinkRx research.

ThinkRx is routinely used in more than 70 cognitive training centers and 50 clinics in the USA and in another 40 countries around the world. In the USA, it is delivered one-on-one by a clinician or cognitive trainer. As such, clinic space is limited, and the cost per hour is comparable to counseling or therapy. The international centers utilize a more scalable model of delivering ThinkRx that combines clinician-delivered training with digital training to make the program more affordable and available to a larger number of children at one time. The obvious working assumption is equivalency between the two modes of delivery. However, that assumption has not been tested. In the current study, we examined the equivalency of the more scalable delivery model compared to the traditional solely one-on-one model of delivery. The current study was guided by the following research question: Are there statistically significant differences in cognitive growth between those who experience cognitive training entirely through a clinician-delivered model compared to those who experience cognitive training in a mixed-delivery model (online and clinician-delivered)? We answer this question through a randomized, controlled trial assessing the equivalence of two groups. Given that clients in international clinics using the mixed delivery model achieve similar results to clients in US clinics using the one-on-one delivery model and that adult participants in a ThinkRx study using a mixed delivery model achieved clinically significant results (Ledbetter et al. 2017), our hypothesis is that there is no significant difference in cognitive skill improvements between the two delivery methods of ThinkRx cognitive training.

Method

Participants

A sample of 39 participants between the ages of 8 and 14 were recruited through an email invitation to a large database of families who had contacted the Colorado Springs LearningRx training center for information about the program. LearningRx is a national network of clinics that offer a comprehensive clinician-delivered cognitive training curriculum for children and adults as well as intensive reading and math interventions for children and adolescents. The primary clientele of LearningRx centers is children with learning disabilities or academic struggles, including dyslexia, attention deficit hyperactivity disorder, and speech and language delays. Eligibility was limited to participants between the ages of 8 and 14 living within commuting range of Colorado Springs who scored between 70 and 130 on the general intellectual ability (GIA) composite of the Woodcock Johnson III – Tests of Cognitive Abilities during screening. A projected sample size of 40 was selected based on availability of research and development funding for the project. Using results from Carpenter et al. (2016), we also used G*Power to perform a priori power analysis to determine if this sample size would be sufficient. With an alpha of 0.05, power at 0.80, and effect sizes from Carpenter et al., the sample of 40 was more than adequate.

Of the 43 volunteers, 39 met the criteria for participation. In accordance with the Declaration of Helsinki, parents provided written informed consent and the minor participants assented to participating in the study. The study was approved by the Institutional Review Board (IRB) at the Gibson Institute of Cognitive Research and is registered at ClinicalTrials.gov as identifier NCT02927197. Recruitment began in June 2015 and final assessments concluded in March 2016.

Procedure

Using blocked sampling with siblings and individuals, participants were randomly assigned by the study coordinator either to the ThinkRx group (n = 20) to receive 60 h of one-on-one clinician-delivered cognitive training or to the mixed delivery group (n = 19) to receive 30 h of one-on-one clinician-delivered cognitive training plus 30 h of clinician-supervised, computer-based cognitive training. Blocking by sibling or individual status was chosen to reduce the risk of attrition and contamination if siblings were assigned to different groups. Randomization was conducted by a research team member not involved in the assessment or intervention who assigned the participants to the interventions. Allocation sequence was concealed using sequentially numbered opaque, sealed envelopes. Participants and their parents were blind to the condition in which they were assigned. They were simply told that we were examining differences between two versions of the same cognitive training program. There was no chance for contamination because the groups attended training sessions successively rather than simultaneously. That is, the one-on-one clinician-delivered ThinkRx group completed their training in the first phase of the study and the mixed delivery group began their training in the second phase of the study after the first group was finished. The mixed delivery group served as the waitlist control group for the first phase of the study (Carpenter et al. 2016).

Table 1 illustrates the pre-intervention demographics including diagnoses. A check of the random assignment indicated the groups were balanced, with no significant differences between groups based on personal characteristics (age: t = 0.217, p = 0.829; gender: χ2 = 1.80, p = 0.210; race/ethnicity: χ2 = 2.35, p = 0.218; no disability: χ2 = 0.54, p = 0.522). One participant in the one-on-one delivery group was on stimulant medication for ADHD, and the medication status remained stable throughout the study. Each participant underwent pre-testing and post-testing with subtests from the Woodcock Johnson III – Tests of Cognitive Abilities (Woodcock et al. 2001) administered by master’s-level clinicians blind to the condition of the participants. Both groups were tested by the same clinicians. Pre- and post-testing was supervised by a doctoral-level psychologist who was aware of the group assignments. In addition to conducting pre- and post-cognitive testing, the research team met with parents and cognitive trainers before the intervention, at the midpoint, and at the completion of the intervention to document qualitative changes observed in the participants.
Table 1

Pre-intervention demographics for each group

 

One-on-one delivery

Mixed delivery

Mean age (SD)

11.3 (2.0)

11.2 (1.7)

Mean IQ at pre-test (SD)

103 (13)

102 (13)

Sex

 Male

9

12

 Female

11

7

Diagnosis

 ADHD

6

7

 Dyslexia

3

3

 LD

2

1

 Speech/language delay

2

1

 Traumatic brain injury

1

0

 Autism spectrum

1

0

Two methods of delivering ThinkRx cognitive training were administered. Participants in both groups attended three or four 90-min cognitive training sessions per week for a total of 40 sessions. Training was delivered or monitored by certified cognitive trainers at two sites: a cognitive training center and a cognitive science research facility with training rooms designed to mimic the rooms in the training center. Two master trainers monitored program fidelity based on a comprehensive plan that included fidelity in design to prevent contamination, training and observation of cognitive trainers using a training manual and observation checklist, and tracking of intervention delivery including number of visits, frequency of visits, length of visits, and consistency in skills targeted. All but one participant completed the entire 60-h training protocol. The remaining participant—a member of the mixed delivery group—was unable to complete the complex training exercises and was instead trained with a related cognitive training curriculum designed for younger children. The one-on-one delivery group received all 60 h of ThinkRx cognitive training delivered one-on-one by a cognitive trainer. The Mixed Delivery group received 30 h of cognitive training delivered one-on-one by a trainer and another 30 h of training via digital delivery on an iPad in a monitored computer lab at each intervention site. The same staff of cognitive trainers trained both groups.

Assessments

Subtests 1 through 7 of the Woodcock Johnson III Tests of Cognitive Abilities (Woodcock et al. 2001) were administered, so a general intellectual ability composite could subsequently be generated. Subtest 10 was also administered to obtain a measure of long-term memory. Although no measure of selective or divided attention is available on the Woodcock Johnson test battery, the Numbers Reversed subtest served as a measure of attentional capacity as well as working memory (Mather and Woodcock 2001). A description of the tests is listed in Table 2.
Table 2

Brief description of Woodcock Johnson III outcome measures

Variable

WJ III test

Description

Associative memory

Visual auditory learning

Participant learns a rebus and then recalls and recites the association between the pictures and the words.

Visual processing

Spatial relations

Participant visually matches individual puzzle pieces to a completed shape.

Auditory processing

Sound blending

Participant hears a series of phonemes and then blends them to form a word.

Logic and reasoning

Concept formation

Participant applies inductive rules to a set of shapes and indicates the rule that differentiates them.

Processing speed

Visual matching

In 3 min, participant identifies and circles pairs of matching numbers in each row.

Working memory and attentional capacity

Numbers reversed

Participant hears a list of numbers and repeats them in reverse order.

Long-term memory

Visual auditory learning—delayed

Participant recalls verbal-visual associations learned earlier by reading rebus passages.

General intellectual ability (GIA)

Composite score for g

GIA is a weighted composite of tests 1–7 on the WJ III.

Interventions

One-on-one Delivery

The traditional ThinkRx cognitive training program was administered one-on-one by a cognitive trainer during 90-min sessions 3 or 4 days per week for the duration of the study. The program is grounded in the Cattell-Horn-Carrol (CHC) theory of intelligence and the multiple-construct view of cognition (McGrew 2005). As such, the ThinkRx training procedures were developed to target multiple cognitive skills such as working memory, long-term memory, processing speed, visual processing, auditory processing, logic and reasoning, and attention. The 230-page curriculum includes 23 training tasks with more than 1000 variations and levels of difficulty. In the current study, the trainer sat across a table from the participant and utilized a variety of materials to deliver the intervention. Those materials included a metronome, timer, shape and number cards, Tangrams, speeded activity worksheets, visual logic cards, and even a footbag and mini-trampoline to encourage simultaneous motor skill and cognitive skill development. During each training session, the trainer loaded the training tasks with additional cognitive activities, such as counting aloud to the beat of a metronome, performing mathematical calculations, or answering questions about an unrelated task. To train divided and sustained attention skills and the ability to work in the presence of distractions, the trainer added deliberate distractions to the training tasks, such as clapping to a different metronome beat, asking silly questions, singing a song, or making funny sounds. Throughout the session, trainers provided immediate feedback and made adjustments as participants progressed through each level of task difficulty. The metronome is a key component of the ThinkRx program and used in nearly every training procedure to create intensity, develop attention skills, progressively improve processing speed, and ensure there are no mental breaks. Progress and mastery of each level of task were carefully tracked using a task flow sheet. Every participant followed the same scope and sequence—it was the speed of progression and the slope of the trajectory that varied from person to person based on abilities.

During the first training session, the cognitive trainer helped participants establish goals for improvements they would like to reach, such as better academic performance or improvements in sports or hobbies. Then, the first few minutes of each training session were spent revisiting goals and discussing real-life improvements. This important metacognitive aspect of the training program was designed to increase motivation for training, develop the relationship between the participant and the cognitive trainer, and help the participant apply any new skills outside of the training environment.

Mixed Delivery

Participants in the mixed delivery group attended 90-min training sessions 3 or 4 days per week exactly like the one-on-one delivery group. However, the mixed delivery group participants spent half of each training session working one-on-one with a cognitive trainer and the other half of each training session in a supervised computer lab where they completed digital training exercises on an iPad. A lab assistant monitored the participants to ensure they remained on-task and helped participants with any technical questions. Eight to ten members of the mixed delivery group were always scheduled at the same time to maximize the use of the computer lab as the scaling feature of the delivery model. Half began with the one-on-one portion of the training session while the remaining half began in the computer lab.

The digital training program, called Brainskills, is a computerized version of ten training exercises from the traditional ThinkRx program that progresses by levels of intensity and difficulty like the one-on-one version. That is, the ten computer-based tasks were designed to mimic ten of the trainer-delivered tasks and were the only tasks that could be digitally adapted with a high level of fidelity. Figure 1 shows an example of both versions of the same exercise: Reasoning Brain Cards. In this inductive logic exercise, participants use a set of rules to identify a three-card group from a set of 12 cards that each has four features: shape, color, size, and orientation. One task is to identify three cards that share all the same variable. Another task is to identify a card that is not shown but that would complete a set.
Fig. 1

Reasoning Brain Cards. Both delivery versions of the same reasoning training procedure

The software is web-based and accessible via the Internet on both tablets and computers. In the current study, Brainskills was accessed on iPad tablets on stands and used with headphones. A research assistant generated weekly Brainskills progress reports for the master trainers to review. This ensured that participants were progressing through the digital part of the training curriculum in the same way they progressed through it in their one-on-one portion of the training. If a participant was not progressing through the tasks as expected, the master trainer provided assistance during a subsequent digital training session.

Data Analysis

Data were analyzed using multivariate analysis of variance (MANOVA), with the dependent variables being the difference scores between the pre- and post-tests for each measure. In other words, the study used a difference-in-difference analysis for all measures. Given the number of pairwise comparisons (i.e., eight, one for each measure), a Bonferroni correction was applied to the multiple comparisons. Effect sizes were also calculated for all measures using Cohen’s d and described using Cohen’s (1988) general guidance of small (0.2), medium (0.5), or large (0.8) effects. To address the potential for Lord’s Paradox (Wainer 1991), we conducted an alternate series of individual analyses of covariance (ANCOVA) for each post-test score as a dependent variable with pre-test scores as covariates, including a Bonferroni correction for multiple comparisons. Because the results were conceptually similar, we chose to report the MANOVA findings, with one exception described below.

As mentioned above, the sample included sibling clusters in both groups. Because this violates the assumption of independence, we first used intra-class correlations (ICC) to determine if dependence among some of the cases was significant (Grawitch and Munz 2004). ICC results were statistically significant for three of the outcome measures—processing speed, working memory and attentional capacity, and GIA. Consequently, we analyzed the eight dependent variables (not just three significant variables, out of an abundance of caution) using OLS regression with clustered standard errors to determine if the clustering resulted in substantively different results compared to the MANOVA analysis. Regression results were substantively the same, so we report MANOVA results below.

In a final preliminary analysis, we examined if diagnosis was a significant predictor of the changes in each construct. Recall these participants reported some form of learning difficulty that may stem from a diagnosis of ADHD, dyslexia, or a number of other disabilities. Because such diagnoses could confound the results from the primary analysis, we performed a series of linear regression analyses with diagnosis as the predictor variable and gain scores as the dependent variables to determine the extent to which the diagnoses might need to be controlled. Diagnosis was not a significant predictor of change for any of the outcome variables: associative memory F (1,37) = 1.45, p = 0.24; visual processing F (1,37) = 0.781, p = 0.38; auditory processing F (1,37) = 1.29, p = 0.26; processing speed F (1,37) = 3.11, p = 0.09; working memory and attentional capacity F (1,37) = 1.38, p = 0.25; logic and reasoning F (1,37) = 0.758, p = 0.39; long-term memory F (1,37) = 1.83, p = 0.18; and IQ score F (1,37) = 0.38, p = 0.54.

Data Availability

The datasets generated during and/or analyzed during the current study are available in the Harvard Dataverse repository, https://dataverse.harvard.edu/dataverse/thinkrxvsbrainlab.

Results

Preliminary Results

Data screening indicated no missing data, and all variables were within tolerable ranges for skewness. Finally, comparisons of pre-test scores indicate groups were statistically equivalent on almost all measures (associative memory t = 1.21, p = 0.24; visual processing t = 1.68, p = 0.10; auditory processing t = − 1.87, p = 0.07; processing speed t = 0.47, p = 0.65; working memory and attentional capacity t = − 0.02, p = 0.98; logic and reasoning t = 0.50, p = 0.62; IQ t = − 0.28, p = 0.78), with the exception of long-term memory (t = 2.66, p = 0.01), where the one-on-one delivery group pre-test score (M = 98.65, SD = 19.55) was greater than the mixed delivery group pre-test score (M = 104.56, SD = 21.68). Due to the significant difference on this measure, we provide results below from the aforementioned ANCOVA, in addition to the MANOVA, as the former presents the post-test results after controlling for the pre-tests. One member of the mixed delivery group was excluded from the analysis because the participant was not able to participate due to the complexity of the training tasks. Instead, the participant was provided with a different cognitive training intervention designed for younger children.

Results of Statistical Significance Testing

As shown in Table 3, MANOVA results indicate an overall significant difference between groups (F = 3.36, p = 0.01), with pairwise comparisons indicating the significant difference between groups was on one of the eight measures—long-term memory. On that measure, the one-on-one delivery group showed greater growth than the mixed delivery group.
Table 3

Results of significance testing between groups

Variable

One-on-one delivery

Mixed delivery

Difference

Pre

Post

Change

Pre

Post

Change

M1–M2

F

p

d

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Mean (SD)

Associative memory

95.70 (15.19)

118.65 (19.13)

22.95 (13.61)

102.11 (15.71)

119.33 (12.78)

17.22 (15.66)

5.73

1.46

0.24

0.37

Visual processing

98.95 (8.79)

109.80 (10.59)

10.85 (9.75)

102.83 (8.98)

108.78 (10.57)

5.94 (8.03)

4.91

2.83

0.10

0.61

Auditory processing

122.35 (16.45)

135.65 (18.34)

13.30 (12.28)

120.22 (12.87)

141.06 (10.86)

20.83 (12.57)

− 7.53

3.49

0.07

− 0.60

Logic and reasoning

100.70 (17.28)

121.80 (11.03)

21.10 (18.50)

105.39 (10.01)

124.17 (9.35)

18.78 (6.85)

2.32

0.25

0.62

0.34

Processing speed

87.35 (16.83)

100.30 (14.26)

12.95 (9.53)

83.94 (13.69)

95.56 (14.98)

11.61 (8.07)

1.34

0.22

0.65

0.17

Working memory

95.85 (16.33)

108.90 (16.74)

13.05 (15.11)

92.22 (15.82)

105.39 (14.74)

13.17 (20.35)

− 0.12

0.00

0.98

− 0.01

Long-term memory

98.65 (19.55)

126.85 (18.71)

28.20 (22.38)

104.56 (21.68)

116.44 (15.30)

11.89 (13.92)

16.31*

7.09

0.01

1.17

IQ score

103.55 (13.30)

124.55 (14.16)

21.00 (13.49)

102.17 (12.56)

124.28 (9.40)

22.11 (10.29)

− 1.11

0.08

0.78

− 0.11

*Significant at p < 0.05

Turning to effect sizes calculated as Cohen’s d indicating the magnitude of the significance, the greatest difference was measured on long-term memory, followed by visual processing (one-on-one delivery group was designated as the treatment in these calculations). The former saw a large effect size, with the latter approaching a large effect size. The smallest effect size was measured on working memory and attentional capacity, then IQ score and processing speed, all of which saw small effect sizes. As for the ANCOVA analysis for long-term memory, results indicate post-test scores were significantly greater for the one-on-one delivery group participants, after controlling for pre-test scores (F = 6.94, p = 0.01). The one-on-one delivery group participants conditionally scored approximately 13 points greater than mixed delivery group participants (Mone-on-one = 128.05, se = 3.36; Mmixed = 115.11, se = 3.55).

Discussion

This study compared the equivalency of two delivery methods of the same cognitive training program: ThinkRx delivered solely one-on-one by a cognitive trainer and ThinkRx delivered 50% one-on-one by a cognitive trainer and the other 50% via computerized training procedures in a supervised computer lab. Largely, the results were similar between the groups, suggesting that the mixed delivery model may be a feasible method of scaling the intervention to reach more children while achieving comparable results. As an alternative to a cognitive trainer working with one child every 90 min, the addition of a computer lab enables the trainer to work with two children in a 90-min session when they alternate between one-on-one training time and time in the computer lab.

However, there was a significant difference in outcomes between the groups on long-term memory, which needs to be addressed. The one-on-one delivery group gains from pre-test to post-test were more than twice those of the mixed delivery group. An evaluation of individual components of the delivery models should be considered to address the discrepancy in long-term memory outcomes. That is, what characteristics of the training models may have contributed to the difference? A check of the fidelity monitoring records indicated the one-on-one delivery group did not spend more training time working on long-term memory tasks with the cognitive trainer than the mixed delivery group. However, the computerized training procedures do not include multitasking components that target memory skills. In the absence of the trainer during the digital training time, there is no method for adding distractions and additional tasks to complete orally on top of the digital exercises. Thus, the one-on-one delivery group received more exposure to task loading by the cognitive trainer than the mixed delivery group. Although this would seemingly address working memory development, it may have contributed to the difference in long-term memory skills at post-test given the nature of the assessment task. That is, the task measures automaticity of retrieval (Schrank 2011), which the mixed delivery participants may not have developed as strongly as the one-on-one delivery group. After performing a replication study with a much larger sample, programming random distractions or adding a multitasking component to the digital Brainskills program may be a consideration for narrowing this gap on long-term memory outcomes if it indeed exists. There is, however, a small chance that the difference in outcomes is the result of the significant differences on long-term memory scores between the groups at pre-test. The mixed delivery group began the study with pre-test scores almost five points above the population mean while the one-on-one delivery group began the study with pre-test scores just below the population mean. Therefore, the 6.5-point difference at pre-test may have influenced the results. However, the difference between groups on the long-term memory measure remained significant even after controlling for these pre-test differences in the analysis.

It is important to note the training tasks are complex and qualitatively different from the assessment tasks, so the transfer to untrained tasks is not a result of “training to the test.” Further, the assessment tasks are designed to measure cognitive skills in isolation while each training procedure targets multiple skills simultaneously. To illustrate, consider the difference in how processing speed was assessed versus how it was trained. The assessment task for processing speed required participants to locate and circle pairs of identical images. One of the intervention tasks to train processing speed required participants to perform diverse actions on a set of related letters in a limited time period, such a circling every “p,” underlining every “d,” crossing out every “b,” and putting a box around every “q,” which trained not only processing speed, but also attention, working memory, visual discrimination, visual span, and sensory motor integration. The cognitive trainer might load the training task by asking the participant to recite the alphabet or count backwards from 100 on beat to the metronome at the same time. In the intervention, processing speed was also targeted in every training task by using the metronome or a stopwatch, which forced participants to make decisions more rapidly regardless of the construct the task was designed to improve.

There are a few limitations and implications for future research in the current study. First, the sample size is small. Although the sample is consistent with sample sizes in similar cognitive training studies, there is a risk that the results are due to the limited statistical power of the design. The field would benefit from a replication study with a larger sample size. Next, long-term outcomes in the current sample have not been assessed. An important gap in the literature is the long-term retention of cognitive training gains. Although beyond the scope of the current study, it will be important to conduct follow-up assessments of this study sample to determine if improvements in cognitive and functional skills persist as well as they have in prior follow-up assessments following the intervention used in the current study (Wainer and Moore 2016). A potential limitation may also be that working memory and attentional capacity were measured using the same assessment task. However, there is research to support the use of digit span tasks in measuring scope of attention and attentional control in children (Cowan et al. 2005). Indeed, the constructs are related and challenging to disentangle. It is important to note, though, that the inclusion of the associative memory task further delineates the memory constructs. That means that if we were to simply refer to the digit span task as the attentional control measure in the current study, we are still left with two additional measures of memory: associative memory and delayed recall. Thus, the potential confounding of constructs measured by the digit span task is mitigated. In future research, the addition of a continuous performance test would enable the assessment of divided and selective attention as well.

Future research should include formal measures of far transfer to behavioral and functional outcomes. In the current study, the researchers collected session notes from the cognitive trainers, which documented self-reported improvements noted by the participants. All participants in both groups reported improvements in academic skills, saying things such as, “I can comprehend faster and better in math and language arts,” “I’m remembering what I read,” and “In Algebra I can complete problems faster.” Almost all participants in both groups also noted changes in self-esteem, saying things like, “I’m less worried about making mistakes and questioning my answers,” “I’m less shy and I feel more confident,” and “I’m feeling good about myself.” Although participants also reported changes in relationships with others, self-discipline, skill in sports and hobbies, and overall cognitive improvement, we did not use an objective or quantitative measure of far transfer. This was a limitation of the current study and is an important area to address in future research.

Another limitation of the study is that it was not possible to compare the delivery methods of all the training tasks because only ten of the training tasks were suitable for adaptation to digital format. This is a limitation of digitization for the field overall, however, because many training procedures lend themselves only to delivery in person. Finally, we compared treatment delivery methods that overlapped rather than two methods that were completely distinct. The design creates some difficulty in teasing apart the different mechanisms at work in each method. However, there is substantial ecological validity to the design. The two delivery methods tested in the current study are the methods used in clinics around the world. Therefore, it was important to evaluate them “as is” in order to test for equivalency.

The question of placebo or expectancy effects often arises when considering the outcomes of intervention studies. In the current study, these effects are controlled by the design because both groups completed an intervention for the same number of hours in the presence of an adult, and neither group knew there was a difference in delivery methods. However, it is interesting to note that prior research using two control groups to test the presence of placebo effects in cognitive training studies has failed to find any, including two meta-analyses that revealed no statistically significant differences between active and passive controls (Au et al. 2015; Peng and Miller 2016). Thus, we suggest that expectancy or placebo effects were unlikely in the current study as well. Although the authors examined the difference between the one-on-one delivery method and a control group in the first phase of this study (Carpenter et al. 2016), future research could include a replication study with all three groups to confirm the lack of placebo effects with the current intervention.

The results of the current study provide support for the use of mixed delivery in scaling the ThinkRx program to reach more children, including those with neurodevelopmental and learning disorders. The results were consistent with prior studies on ThinkRx with children (Carpenter et al. 2016; Gibson et al. 2015; Jedlicka 2017; Moore et al. 2018) and with observational data on nearly 18,000 clients (Wainer and Moore 2016) in remediating multiple cognitive skills. This convergence of evidence is important for determining the benefit of adopting an intervention for use in clinical practice (Carey and Stiles 2016). The ability to scale a one-on-one intervention has important implications for its use in clinical practice where time is a limiting factor for scheduling clients. The mixed delivery model enables each clinician to see twice as many clients and provide the intervention to more children who need it.

Notes

Funding

This study was funded through LearningRx research and development (R&D) funds.

Compliance with Ethical Standards

Conflict of Interest

The first and third authors are employed by the Gibson Institute of Cognitive Research, the nonprofit research arm of the interventions described in this paper. The other authors report no pecuniary or professional interest.

References

  1. Alloway, T. P., Rajendran, G., & Archibald, L. M. (2009). Working memory in children with developmental disorders. Journal of Learning Disabilities, 42(4), 372–382.  https://doi.org/10.1177/0022219409335214.CrossRefPubMedGoogle Scholar
  2. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.CrossRefGoogle Scholar
  3. Au, J., Sheehan, E., Tsai, N., Duncan, G. J., Buschkuehl, M., & Jaeggi, S. M. (2015). Improving fluid intelligence with training on working memory: a meta-analysis. Psychonomic Bulletin Review, 22(2), 366–377.  https://doi.org/10.3758/s13423-014-0699-x.CrossRefPubMedGoogle Scholar
  4. Beck, S. J., Hanson, C. A., & Puffenberger, S. S. (2010). A controlled trial of working memory training for children and adolescent with ADHD. Journal of Clinical Child and Adolescent Psychology, 39(6), 825–836.  https://doi.org/10.1080/15374416.2010.517162.CrossRefPubMedGoogle Scholar
  5. Carey, T. A., & Stiles, W. B. (2016). Some problems with randomized controlled trials and some viable alternatives. Clinical Psychology and Psychotherapy, 23(1), 87–95.  https://doi.org/10.1002/cpp.1942.CrossRefPubMedGoogle Scholar
  6. Carpenter, D., Ledbetter, C., & Moore, A. L. (2016). LearningRx cognitive training effects in children ages 8-14: a randomized controlled study. Applied Cognitive Psychology, 30(5).  https://doi.org/10.1002/acp.3257.CrossRefGoogle Scholar
  7. Chacko, A., Feirsen, N., Bedard, A. C., Marks, D., Uderman, J. Z., & Chimiklis, A. (2013). Cogmed Working Memory Training for youth with ADHD: a closer examination of efficacy utilizing evidence-based criteria. Journal of Clinical Child and Adolescent Psychology, 42(6), 769–783.  https://doi.org/10.1080/15374416.2013.787622.CrossRefPubMedGoogle Scholar
  8. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: Lawrence Erlbaum and Associates.Google Scholar
  9. Cortese, S., Ferrin, M., Brandies, D., Buitelaar, J., Daley, D., Dittmann, R. W., et al. (2015). Cognitive training for attention deficit hyperactivity disorder. A meta-analysis of clinical and neuropsychological outcomes from randomized controlled trials. Journal of the American Academy of Child and Adolescent Psychiatry, 54(3), 164–174.CrossRefGoogle Scholar
  10. Cowan, N., Elliott, E., Saults, J. S., Morey, C., Mattox, S., Hismjatullina, A., & Conway, A. (2005). On the capacity of attention: its estimation and its role in working memory and cognitive aptitudes. Cognitive Psychology, 51, 42–100.  https://doi.org/10.1016/j.cogpsych.2004.12.001.CrossRefPubMedPubMedCentralGoogle Scholar
  11. Feuerstein, R. (1990). The theory of structural modifiability. In learning and thinking styles: classroom interaction. Washington, DC: National Education Association.Google Scholar
  12. Feuerstein, R., Feuerstein, R. S., & Falik, L. H. (2010). Beyond smarter: mediated learning and the brain’s capacity for change. New York: Teacher’s College Press.Google Scholar
  13. Franceschini, S., Gori, S., Ruffino, M., Viola, S., Molteni, M., & Facoetti, A. (2013). Action video games make dyslexic children read better. Current Biology, 23(6), 462–466.  https://doi.org/10.1016/j.cub.2013.01.044.CrossRefPubMedGoogle Scholar
  14. Franceschini, S., Trevisan, P., Ronconi, L., Bertoni, S., Colmar, S., Double, K., et al. (2017). Action video games improve reading abilities and visual-to-auditory attentional shifting in English-speaking children with dyslexia. Scientific Reports, 7, 5863.  https://doi.org/10.1038/s41598-017-05826-8.CrossRefPubMedPubMedCentralGoogle Scholar
  15. Freberg, M. M., Vandiver, B. J., Watkins, M. W., & Canivez, G. L. (2008). Significant factor score variability and the validity of the WISC-III full scale IQ in predicting later academic achievement. Applied Neuropsychology, 15(2), 131–139.  https://doi.org/10.1080/09084280802084010.CrossRefPubMedGoogle Scholar
  16. Gibson, K., Mitchell, T., & Tenpas, D. (2003). ThinkRx. Colorado Springs: LearningRx.Google Scholar
  17. Gibson, K., Carpenter, D., Moore, A. L., & Mitchell, T. (2015). Training the brain to learn: Augmenting vision therapy. Vision Developmental and Rehabilitation, 1(2), 119–128 Retrieved from http://c.ymcdn.com/sites/www.covd.org/resource/resmgr/VDR/VDR_1_2/VDR1-2_article_Gibson_web_in.pdf.Google Scholar
  18. Grawitch, M. J., & Munz, D. C. (2004). Are your data nonindependent? A practical guide to evaluating nonindependence and within-group agreement. Understanding Statistics, 3(4), 231–257.CrossRefGoogle Scholar
  19. Green, C. S., & Bavelier, D. (2003). Action video game modifies visual selective attention. Nature, 423(6939), 534–537.  https://doi.org/10.1038/nature01647.CrossRefPubMedGoogle Scholar
  20. Green, C. S., & Bavelier, D. (2007). Action-video-game experience alters the spatial resolution of vision. Psychological Science, 18(1), 88–94.  https://doi.org/10.1111/j.1467-9280.2007.01853.x.CrossRefPubMedPubMedCentralGoogle Scholar
  21. Holmes, J., & Gathercole, E. (2014). Taking working memory training from the laboratory into schools. Educational Psychology, 34(4), 440–450.  https://doi.org/10.1080/01443410.2013.797338.CrossRefPubMedGoogle Scholar
  22. Jedlicka, E. (2017). LearningRx cognitive training for children and adolescents ages 5-18: effects on academic skills, behavior, and cognition. Frontiers in Education, 2(62).  https://doi.org/10.3389/feduc.2017.00062.
  23. Kerns, K. A., Eso, K., & Thomson, J. (1999). Investigation of a direct intervention for improving attention in young children with ADHD. Developmental Neuropsychology, 16(2), 273–295.CrossRefGoogle Scholar
  24. Klauer, K. J., & Phye, G. D. (2008). Inductive reasoning: a training approach. Review of Educational Research, 78(1), 85–123.  https://doi.org/10.3102/0034654307313402.CrossRefGoogle Scholar
  25. Klingberg, T., Forssberg, H., & Westerberg, H. (2002). Training of working memory in children with ADHD. Journal of Clinical and Experimental Neuropsychology, 24(6), 781–791.  https://doi.org/10.1076/jcen.24.6.781.8395.CrossRefPubMedGoogle Scholar
  26. Klingberg, T., Fernell, E., Oleson, P. J., Johnson, M., Gustafsson, P., & Dahlstrom, K. (2005). Computerized training of working memory in children with ADHD—a randomized, controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 44(2), 177–186.  https://doi.org/10.1097/00004583-200502000-00010.CrossRefPubMedGoogle Scholar
  27. Kozulin, A., Lebeer, J., Madella-Noja, A., Gonzalez, F., Jeffrey, I., Rosenthal, N., & Koslowsky, M. (2010). Cognitive modifiability of children with developmental disabilities: a multicenter study using Feuerstein’s Instrumental Enrichment-Basic program. Research in Developmental Disabilities, 31(2), 551–559.  https://doi.org/10.1016/j.ridd.2009.12.001.CrossRefPubMedGoogle Scholar
  28. Ledbetter, C., Faison, M. O., & Patterson, J. (2016). Correlation of cognitive training gains and resting state functional connectivity. Presented at Society for Neuroscience, November 2016, San Diego, CA.Google Scholar
  29. Ledbetter, C., Moore, A. L., & Mitchell, T. (2017). Cognitive effects of ThinkRx cognitive rehabilitation training for eleven soldiers with brain injury: a retrospective chart review. Frontiers in Psychology, 8(825).  https://doi.org/10.3389/fpsyg.2017.00825.
  30. Lewandowski, L. J., Lovett, B. J., Parolin, R. A., Gordon, M., & Codding, R. S. (2007). Extended time accommodations and the mathematics performance of students with and without ADHD. Journal of Psychoeducational Assessment, 25, 17–28.CrossRefGoogle Scholar
  31. Mather, N., & Woodcock, R. W. (2001). Examiner’s manual. Woodcock Johnson III: tests of cognitive abilities. Itasca: Riverside Publishing.Google Scholar
  32. McGrew, K. (2005). The Cattell-Horn-Carroll theory of cognitive abilities. In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary intellectual assessment: theories, tests, and issues (pp. 151–179). New York: Guilford.Google Scholar
  33. McQuade, J. D., Tomb, M., Hoza, B., Waschbusch, D. A., Hurt, E. A., & Vaughn, A. J. (2011). Cognitive deficits and positively biased self-perceptions in children with ADHD. Journal of Abnormal Child Psychology, 39(2), 307–319.  https://doi.org/10.1007/s10802-010-9453-7.CrossRefPubMedPubMedCentralGoogle Scholar
  34. Melby-Lervag, M., & Hulme, C. (2016). There is no convincing evidence that working memory training is effective: a reply to Au et al. (2014) and Karbach and Verhaeghen (2014). Psychonomic Bulletin Review, 23, 324–330.  https://doi.org/10.3758/s13423-015-0862-z.CrossRefPubMedGoogle Scholar
  35. Moore, A. L., Carpenter, D. M., Ledbetter, C., & Miller, T. M. (2018). Clinician-delivered cognitive training for children with attention problems: transfer effects on cognitive and behavior from the ThinkRx randomized controlled trial. Neuropsychiatric Disease and Treatment, 14, 1671–1683.  https://doi.org/10.2147/NDT.S165418.CrossRefPubMedPubMedCentralGoogle Scholar
  36. Peng, P., & Miller, A. C. (2016). Does attention training work? A selective meta-analysis to explore the effects of attention training and moderators. Learning and Individual Differences, 45, 77–87.  https://doi.org/10.1016/j.lindif.2015.11.012.CrossRefGoogle Scholar
  37. Schnitzer, G., Andries, C., & Lebeer, J. (2007). Usefulness of cognitive intervention programs for socio-emotional and behaviour problems in children with learning disabilities. Journal of Research in Special Educational Needs, 7(3), 161–171.CrossRefGoogle Scholar
  38. Schrank, F. A. (2011). Woodcock–Johnson III tests of cognitive abilities. In A. S. Davis (Ed.), Handbook of pediatric neuropsychology (pp. 415–434). New York: Springer.Google Scholar
  39. Tamm, L., Hughes, C., Ames, L., Pickering, J., Silver, C. H., Stavinoha, P., et al. (2010). Attention training for school-aged children with ADHD: results of an open trial. Journal of Attention Disorders, 14(1), 86–94.  https://doi.org/10.1177/1087054709347446.CrossRefPubMedGoogle Scholar
  40. van der Donk, M., Hiemstra-Beernink, A. C., Tjeenk-Kalff, A., van der Leij, A., & Lindauer, R. (2015). Cognitive training for children with ADHD: a randomized controlled trial of Cogmed working memory training and ‘paying attention in class’. Frontiers in Psychology, 6(1081).  https://doi.org/10.3389/fpsyg.2015.01081.
  41. Van der Oord, S., Ponsioen, A. J. G. B., Geurts, H. M., Ten Brink, E. L., & Prins, P. J. M. (2012). A pilot study of the efficacy of a computerized executive functioning remediation training with game elements for children with ADHD in an outpatient setting: outcome on parent-and-teacher-rated executive functioning and ADHD behavior. Journal of Attention Disorders, 18(8), 699–712.  https://doi.org/10.1177/1087054712453167.CrossRefPubMedGoogle Scholar
  42. Wainer, H. (1991). Adjusting for differential base-rates: Lord’s Paradox again. Psychological Bulletin, 109, 147–151.CrossRefGoogle Scholar
  43. Wainer, H., & Moore, A. L. (2016). LearningRx client outcomes and research results. Colorado Springs: Gibson Institute of Cognitive Research Retrieved from http://download.learningrx.com/results-report.pdf.Google Scholar
  44. Willis, S. L., Tennstedt, S. L., Marsiske, M., Ball, K., Elias, J., Koepke, K. M., Morris, J. N., Rebok, J. W., Unverzagt, F. W., Stoddard, A. M., & Wright, E. (2006). Long-term effects of cognitive training on everyday functional outcomes in older adults. American Medical Association, 296(23), 2805–2813.  https://doi.org/10.1001/jama.296.23.2805.CrossRefGoogle Scholar
  45. Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock Johnson - III. Itasca: Riverside Publishing.Google Scholar

Copyright information

© The Author(s) 2018

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Gibson Institute of Cognitive ResearchColorado SpringsUSA
  2. 2.University of Colorado Colorado SpringsCollege of EducationColorado SpringsUSA
  3. 3.Department of NeurologyLouisiana State University Health Sciences CenterShreveportUSA

Personalised recommendations