Advertisement

Journal of Abnormal Child Psychology

, Volume 45, Issue 8, pp 1477–1490 | Cite as

Is Poor Working Memory a Transdiagnostic Risk Factor for Psychopathology?

  • Cynthia Huang-Pollock
  • Zvi Shapiro
  • Hilary Galloway-Long
  • Alex Weigard
Article

Abstract

In contrast to historical conceptualizations that framed psychological disorders as distinct, categorical conditions, it is now widely understood that co- and multi-morbidities between disorders are extensive. As a result, there has been a call to better understand the dimensional liabilities that are common to and influence the development of multiple psychopathologies, as supported and exemplified by the National Institutes of Mental Health (NIMH) Research Domain Criteria (RDoC) framework. We use a latent variable SEM approach to examine the degree to which working memory deficits represent a cognitive liability associated with the development of common and discrete dimensions of psychopathology. In a sample of 415 community recruited children aged 8–12 (n = 170 girls), we fit a bi-factor model to parent reports of behavior from the DISC-4 and BASC-2, and included a latent working memory factor as a predictor of the internalizing, externalizing, and general “p-factor.” We found that both the general “p-factor” and externalizing (but not internalizing) latent factor were significantly associated with working memory. When a bi-factor model of externalizing symptomology was fit to further explore this relationship, working memory was only correlated with the general externalizing dimension; correlation with specific inattention, hyperactive/impulsive, and oppositional factors did not survive once the general externalizing dimension was taken into consideration. These findings held regardless of the sex of the child. Our results suggest that working memory deficits represent both a common cognitive liability for mental health disorders, and a specific liability for externalizing disorders.

Keywords

Executive function Working memory Bifactor P-factor Externalizing 

Comorbidity among mental health disorders is the rule (Angold et al. 1999; Costello et al. 2003; Ford et al. 2003; Kessler et al. 2005; Merikangas et al. 2010), and it is neither the result of methodological artifacts (e.g., referral bias or halo effects), nor of artifacts in our current diagnostic system (e.g., overlapping symptomology across disorders: Angold et al. 1999; Cramer et al. 2010). The prevalence of co- and multi-morbidities has long been a principal limitation of the current categorical nosology of psychiatric disorders, and is believed to be caused by the existence of latent liabilities that are shared by syndromes captured within two broad Externalizing and Internalizing dimensions (Achenbach and Edelbrock 1978; Krueger 1999; Lahey et al. 2008).

However, in large national as well as international datasets, strong (~0.50) correlations (Krueger 1999; Lahey et al. 2008; Wright et al. 2013), and frequent comorbidities (Angold et al. 1999; Lahey et al. 2008) are also observed across these domains, even among community samples where the influence of referral bias is reduced. Thus, a comprehensive taxonomy must account for both the common and discrete nature of mental health disorders. In response, recent work has found evidence that there also exists a General Psychopathology factor (or “p-factor”), reflecting latent liabilities shared by all mental health disorders. This bi-factor model has now been repeatedly validated in children (Caspi et al. 2014; Tackett et al. 2013), adolescents (Laceulle et al. 2015), and adults (Krueger 1999; Krueger et al. 1998; Lahey et al. 2012). The existence of the common p- (on which thought disorders load directly: Caspi et al. 2014; Laceulle et al. 2015), and more discrete Externalizing/ Internalizing domains may therefore explain why disorders tend not to be categorical structures, and why unique and distinguishing etiologic mechanisms between disorders by and large have not been found.

However, it remains to be seen whether these latent factors ultimately represent and can be used to identify a common set of transdiagnostic or interactive causal mechanisms predicted by the principle of multifinality (Cicchetti and Rogosch 1996), or whether their identity is limited to a statistical representation of psychopathology severity (Caspi et al. 2014; Laceulle et al. 2015). There is reason to be optimistic. Genome wide association studies have identified a limited set of shared genetic risk factors that are associated with multiple disorders (Malhotra and Sebat 2012; Smoller et al. 2013), and large twin studies have similarly found these broad latent factors represent shared genetic and familial influences (Kendler et al. 2003; Kendler et al. 1995; Young et al. 2009).

And what of the possible downstream psychological mechanisms that mediate the effects of these genetic risk factors on broader functioning? Indeed, it has been the promise of endophenotypes that they might close the causal gap between underlying biology and psychpathology (Gottesman and Gould 2003). Of the putative cognitive endophenotypes, executive function is arguably among the most plausible (Pennington and Ozonoff 1996; Snyder et al. 2015). Referring broadly to the cognitive control processes mediated by the prefrontal cortices that enable goal-directed behavior, evidence of executive dysfunction has been found across a wide range of mental health disorders including Attention Deficit Hyperactivity Disorder (Barkley 1997; Willcutt et al. 2005), learning disabilities (McLean and Hitch 1999; Willcutt et al. 2001), anxiety disorders (Bishop 2009; Eysenck and Derakshan 2011), depression (Paelecke-Habermann et al. 2005; Rogers et al. 2004), bipolar disorder (Quraishi and Frangou 2002), schizophrenia (Nieuwenstein et al. 2001), and autism (Hill 2004; Hughes et al. 1994).

However, extensive comorbidity makes it unclear whether a single or smaller set of disorders could be driving these group effects, or whether EF deficits are truly transdiagnostic. For example, evidence of executive and prefrontal dysfunction have been well documented in conduct disordered, delinquent, and criminal populations (Moffitt 1993; Raine et al. 1994; Raine et al. 2005; White et al. 1994) with an average effect size of 0.62 reported in a meta-analytic review of anti-social behavior (Morgan and Lilienfeld 2000). But many have since argued that the EF deficits observed in aggressive and conduct disordered youth are primarily due to comorbid ADHD (Barkley et al. 2001; Barnett et al. 2009; McAlonan et al. 2007; Oosterlaan et al. 1998; Schachar et al. 2000).

To address the contributions of executive dysfunction to the development of both common p and discrete internalizing/externalizing domains, Caspi et al. (2014) found that worse performance on two of three EF tasks (CANTAB Rapid Visual Information Processing: A’-Prime and Trails B, measures of sustained attention and set shifting, broadly speaking) were each associated with greater severity on the p-factor, but not with severity on the externalizing or internalizing dimensions. Only Mental Control from the WMS-III, a measure of verbal fluency, was also associated with the externalizing dimension. There is therefore at least some evidence that executive dysfunction may be a common risk factor for the development of psychopathology in general, alongside evidence that verbal dysfluency confers specific liability for externalizing disorders. However, these analyses are limited by the study’s use of traditional neuropsychological tasks, which, in the interest of external validity, are known to tap multiple executive as well as non-executive processes, and leads to concerns of task impurity. The issue of task impurity is compounded by the use of a single index of performance. The formation of a latent variable, determined by multiple indices of the construct, would provide a more pure and reliable measurement of the putative endophenotype of interest.

To address these issues in the current study, we utilize an SEM approach to evaluate the degree to which a well-specified cognitive process, working memory, is a critical mechanism in the development of both broad and discrete forms of psychopathology. Working memory is a prototypical executive function, and refers to the ability to actively maintain information in temporary storage while simultaneously manipulating that information. Central to the construct is an assumption of a limited-capacity domain-general executive, similar to a controlled attention or supervisory attentional construct (Norman and Shallice 1986; Shiffrin and Schneider 1977). If executive dysfunction is a transdiagnostic mechanism for general childhood psychopathology, then we would expect that a latent WM factor would not be associated with either of the externalizing or internalizing domains after variance associated with the general p-factor was parsed.

Methods

Participants

Between 2008 and 2015, N = 415 children (n = 170 girls) between the ages of 8 and 12 were recruited from Centre, York, and Dauphin counties of Pennsylvania to participate in a study on attention and learning conducted at The Pennsylvania State University. Reflecting demographics of the region, the sample ethnicity was as follows: 75.7 % Caucasian/non-Hispanic, 7.0 % African American/non-Hispanic, 4.1 % Caucasian/Hispanic, 1.2 % African American/Hispanic, 1.4 % Asian, 7.2 % Mixed, and 3.1 % other or unknown. Children were excluded if they (a) were currently prescribed and taking a non-stimulant medication, or (b) had a parent-reported pervasive developmental disorder, intellectual or sensorimotor disability, psychosis, or neurological disorder.

To be included in the sample, children were required to meet one of two criteria. Either: (a) both parent and teacher report of behavior on the Attention, Hyperactivity, or ADHD subscales of the Behavioral Assessment Scale for Children (BASC-2: Reynolds and Kamphaus 2004) or the Conners’ Rating Scales (Conners 2008) exceeded the 85th percentile (T-score > 60). Or, (b) both parent and teacher report on the same listed indices were below the 80th percentile T-score ≤ 58).

Procedures

All participants completed the following measures as part of a larger test battery completed during two 3-h test sessions. Any children prescribed a psychostimulant medication (N = 95, 23 %) were required to complete a medication-free 1–2 day “wash-out” period (mean = 75 h, median = 57, range = 22–544) before testing. All data were collected in compliance with human subjects’ approval from the Pennsylvania State University Institutional Review Board (IRB#32,126). Informed written consent from parents and verbal assent from children were obtained prior to participation. Children received a small prize for participation. Parents received monetary compensation and informal clinical feedback.

Measures of Psychopathology

Parent report (88 % mothers) of behavior and socioemotional functioning on the BASC-2, as well as past-year symptom counts for Generalized Anxiety Disorder (GAD), Major Depressive Disorder (MDD), Dysthymia (DD), Oppositional Defiant Disorder (ODD), and ADHD on the Diagnostic Interview Schedule for Children-IV (DISC-IV: Shaffer et al. 2000) were obtained as indices of psychopathology.

Working Memory Tasks

A mix of verbal and non-verbal complex and backwards span tasks was used to form the latent WM factor. For all tasks, one point was awarded per correct recall of the entire trial. Reading span. This computer administrated program written in Eprime was obtained from Randall Engle and colleagues, and modified for use in school aged children. Children read aloud simple sentences based on Towse et al. (1998) and made true/false decisions with a right or left mouse click. Immediately following their response, a letter of the alphabet appeared, and children were told to remember the letter. The number of sentence/letter pairs increased in size from two to seven, and after all pairs of an element were presented, children were asked to recall the letters/targets in the order they were presented. Three items were presented per set size, and the task was discontinued if children failed all items of a set size. Digits backwards. Children completed the Digits Backwards subtest of the WISC-IV (Wechsler 2003). Children listen to a trained research assistant read a series of digits at a rate of one per second. They were then asked to recall the digits out loud in the correct backwards sequence. Two sets of digits are recited per digit span length, and the task is discontinued when the child could not correctly recall either set of digits within the same span length. Finger windows backwards. This task was adapted from Finger Windows Forwards subtest of the WRAML-2 (Sheslow and Adams 2003). Children watched a trained research assistant place the tip of a pen through holes or “windows” on an opaque plastic board one at a time, at the rate of one per second. Children were asked to place their finger in the holes in the correct backwards sequence. Two sets of window sequences were performed per span length, and the task was discontinued when the child could not correctly recall either set of windows with the same span length.

Data Analyses

Modeling was carried out using Mplus 7 (Muthén and Muthén 1998–2012). A maximum likelihood estimator with robust standard errors (MLR) was used to account for the non-normal distribution of the continuous BASC variables and DISC symptom counts (models 1, 1b and 2). In models where manifest variables were composed of binary ADHD and ODD symptoms, a weighted least squares means and variance adjusted (WLSMV) estimator (Brown 2015; Enders 2010) was used to account for non-normal distributions of these variables (models 3 and 4). MLR and WLSMV estimators are recommended for use with these variables types and provide adequate model estimates when missing values are relatively few (Brown 2015; Enders 2010), as they were herein (See Table 1).
Table 1

Descriptives

 

Min

Max

Mean

N meeting dx

Skewness

Kurtosis

%

#Sxs

#Sxs

#Sxs (SD)

criteria

#Sxs

#Sxs

Missing

Inattention

0

9

4.85 (3.48)

269 (any ADHD subtype)

-0.27

-1.55

0 %

Hyperactive/Impulsive

0

9

3.17 (3.00)

 

0.51

-1.11

0 %

ODD

0

8

2.11 (2.40)

110

0.90

-0.49

0 %

MDD/DD

0

9

0.54 (1.51)

13/5

3.04

8.82

0 %

GAD

0

7

0.8 (1.68)

33

2.31

4.48

0 %

 

Min

Max

 

N (%)

   
 

T-score

T-score

 

T-score > 60

   

BASC-2 Ext T-score

34

104

55.4313 (13.05)

126 (30.4 %)

0.89

0.50

0 %

BASC-2 Int T-score

30

120

53.0867 (14.01)

96 (23.1 %)

1.11

1.76

0 %

 

Min

Max

 

N (%) SS < 8

   

DSB raw score

0

12

6.61 (1.67)

129 (17.1 %)

0.44

0.93

0 %

FWB raw score

1

20

8.99 (3.45)

N/A

0.21

-0.11

6.02 %

Reading span raw score

0

48

8.24 (8.03)

N/A

1.56

2.86

2.89 %

Sxs symptoms, Dx diagnostic, ODD oppositional defiant disorder, MDD/DD major depressive/dysthymic disorder, GAD generalized anxiety disorder, Ext externalizing composite, Int internalizing composite, SS scaled score, DSB digit span backwards, FWB finger windows backwards

ADHD diagnoses (see Huang-Pollock et al. 2016, for full details) were made via standardized ratings of behaviors provided by parents and teachers, as well as structured diagnostic interview of the primary care provider (DISC-IV) to confirm age of onset, duration, cross-situational severity, impairment, and symptom count (using the “or” algorithm to integrate DISC and teacher report, following DSM-IV field trials: Lahey et al. 1994). Other DSM-IV diagnoses were identified using DISC-IV algorithms that include duration, impairment, and symptom count (age of onset restrictions and cross situational severity are not required for diagnoses of ODD, MDD/DD, or GAD)

Because chi-square is sensitive to large sample size, model fit was also evaluated using the following indices of practical fit: TLI (Bentler and Bonett 1980; Hu and Bentler 1999; Tucker and Lewis 1973), CFI (Bentler 1990), and RMSEA (Browne and Cudeck 1992; Steiger and Lind 1980).

Results

A full account of descriptive values including skew, kurtosis, value ranges, and % missing data can be found in Table 1.

Model 1: Bifactor Model of Psychopathology

Using a confirmatory factor analysis (CFA), we fit a bifactor model in which (a) parent reported symptom counts on the DISC for GAD and MDD/DD, as well as the Internalizing composite score of the BASC-2 loaded onto an Internalizing factor; (b) parent reported DISC symptom counts for ODD, Inattention, Hyperactivity/Impulsivity, as well as the Externalizing composite score for the BASC-2 loaded on the Externalizing factor; (c) and a General Psychopathology factor (p-factor) on which all indices loaded. The solution for this initial model was inadmissible due to negative residual variance. We then tested an alternative model where we assumed the loadings of Inattention and Hyperactivity/Impulsivity composite scores on the Externalizing and p- factors were equal. We did this under the assumption that each contributes equal information regarding the presence of ADHD symptomology (Marsh et al. 1992). The model converged, but model fit was poor: χ 2(9, N = 415) = 52.21, CFI = 0.959, TLI = 0.904, RMSEA = 0.108, 90 % CI [0.08–0.137]. Examination of the modification indices indicated that correlations between inattention symptoms and the BASC internalizing score, and between ODD and MDD/DD symptoms, remained unaccounted for by the model. Due to conceptual and symptom overlap between inattention and internalizing symptomology, and between ODD and MDD/DD symptomology (e.g., inattention, irritability), these residuals were allowed to correlate in Model 1b. Results for Model 1b are shown in Table 2, and the model is depicted in Fig. 1. This model fit the data well: χ 2(7, N = 415) = 8.729, CFI = 0.998, TLI = 0.995, RMSEA = 0.024, 90 % CI [0.000–0.068].
Table 2

Model fit statistics

Model

X2

df

CFI

TLI

RMSEA

1

52.21*

9

0.959

0.904

0.108

1b

8.729

7

0.998

0.995

0.024

2

44.995*

25

0.987

0.976

0.042

3

353.036*

273

0.995

0.994

0.027

4

463.449*

347

0.993

0.992

0.028

For chi-squares, N = 415. RMSEA root-mean-square error of approximation, CFI comparative fit index, TLI tucker-lewis index. *p < .01

Fig. 1

Model 1b, bifactor model of psychopathology. Non-significant paths shown as dotted lines. Int Prob = BASC-2 Internalizing problems composite; GAD = Generalized Anxiety Disorder; MDD/DD = Major Depressive/Dysthymic disorder; Ext Prob = BASC-2 Externalizing problems composite; ODD = Oppositional Defiant Disorder; IN = Inattention; HI = Hyperactivity/Impulsivity

Model 2: Does WM Represent a General Cognitive Risk Factor for Psychopathology?

We next tested the degree to which working memory capacity could represent the cognitive liability associated with general psychopathology. Working memory capacity was represented by a latent variable composed of Reading Span, Digit Span Backwards, and Finger Windows Backwards. Loadings onto the Working Memory factor were all positive and highly significant (all ps < 0.001). Standardized coefficient estimates for these loadings averaged to 0.572. Results are shown in Table 2, and the model is depicted in Fig. 2. This model fit the data well: χ2(25, N = 415) = 42.995, CFI = 0.987, TLI = 0.976, RMSEA = 0.042, 90 % CI [0.019–0.062].
Fig. 2

Model 2, working memory (WM) as latent liability for general psychopathology (p) and externalizing (Ext) but not internalizing symptomology (Int). Non-significant paths shown as dotted lines. Int Prob = BASC-2 internalizing problems composite; GAD = Generalized Anxiety Disorder; MDD/DD = Major Depressive/Dysthymic Disorders; Ext Prob = BASC-2 Externalizing problems composite; ODD = Oppositional Defiant Disorder; IN = Inattention; HI = Hyperactivity/Impulsivity

Working Memory significantly predicted the externalizing factor (p < 0.001), with a standardized estimate value of −0.407, as well as the general p-factor (p < 0.001), with a standardized estimate value of −0.253. Working memory was not significantly associated with the internalizing factor (p = 0.711). Therefore, working memory continued to be independently associated with externalizing factors even after variance associated with the p-factor was accounted for, but the same was not true for internalizing disorders.

Are there more nuanced symptom profiles that are driving this apparent association between WM and the externalizing dimension? Bifactor models of ADHD and the disruptive behavior disorders have also been fit (Arias et al. 2016; Martel et al. 2010a, b, 2011, 2012; Toplak et al. 2009, 2012) and significant bivariate correlations have been reported between (a) performance on the stop signal reaction time task (a measure of inhibitory control) and Trails A/B (a broad measure of set shifting) and (b) latent factor scores for hyperactivity/impulsivity and a general ADHD (but not a specific inattention) factor (Martel et al. 2011). That being said, it’s not clear whether the associations between the specific hyperactivity factor and performance would have remained significant if the relationship to general ADHD had been simultaneously parceled, or if more robust/latent indices of executive control had been used.

In the next set of analyses, we attempt to replicate and extend previous findings. We fit a bifactor model to ADHD and ODD symptoms, and determine the degree to which the relationship between working memory and externalizing disorders in Model 2 reflects (a) its importance to the development of disruptive behavior disorders, generally, or, (b) whether the association of working memory with the externalizing dimension is driven by specific inattentive, hyperactive/impulsive, or oppositional behavior.

Model 3: Bifactor Model of Externalizing Disorders

Using individual symptom counts from the DISC-IV, we next fit a model in which the nine inattention items loaded onto an Inattention (IA) factor; the 9 hyperactive/impulsive items loaded onto a hyperactive/impulsive (HI) factor; the 8 oppositional defiant items loaded onto an oppositional (ODD) factor; and a general externalizing factor, for which all indices loaded. Results are shown in Table 3, and the model is depicted in Fig. 3. The model fit the data well: χ2(273, N = 415) = 353.036, CFI (0.995), TLI (0.994) and RMSEA = 0.027, 90 % CI [0.018–0.034].
Table 3

Correlations of manifest variables in model 2

 

IN

HI

ODD

MDD/DD

GAD

EXT

INT

DSB

FWB

HI

0.674**

        

ODD

0.535**

0.565**

       

MDD/DD

0.219**

0.220**

0.435**

      

GAD

0.278**

0.320**

0.323**

0.468**

     

EXT

0.632**

0.705**

0.662**

0.354**

0.382**

    

INT

0.425**

0.343**

0.374**

0.441**

0.623**

0.504**

   

DSB

-0.233**

-0.222**

-0.154*

-0.018

-0.006

-0.136**

-0.072

  

FWB

-0.320**

-0.272**

-0.212**

-0.091

-0.107*

-0.263**

-0.140**

0.302**

 

RSPAN

-0.171**

-0.091

-0.164**

-0.092

-0.041

-0.135**

-0.109*

0.334**

0.354**

IN inattentive, HI hyperactive/impulsive, ODD oppositional defiant disorder, MDD/DD major depressive/dysthymic disorder, GAD generalized anxiety disorder, EXT BASC-2 externalizing problems composite, INT BASC-2 internalizing problems composite, DSB digit span backward, FWB finger windows backward, RSPAN reading span

* p < 0.05 **p < 0.01

Fig. 3

Model 3, bifactor model of externalizing disorders (Ext), comprised of inattention (IN), hyperactivity/impulsivity (HI) and Oppositional Defiant Disorder (ODD) symptoms. Nonsignificant paths shown as dotted lines

Model 4: Does WM Represent a Cognitive Risk Factor for Externalizing Disorders broadly?

In the last series of analyses, we tested the degree to which WM was associated with the broad vs. discreet externalizing dimensions. Results are shown in Table 3, and the model is depicted in Fig. 4. The model fit the data well: χ2(347, N = 415) = 463.449, CFI (0.993) TLI (0.992), and RMSEA = 0.028, 90 % CI [0.021–0.035].
Fig. 4

Model 4, working memory (WM) as a latent liability for general externalizing psychopathology (Ext) but not the specific inattentive (IN) or hyperactive/impulsive (HI) factors. Nonsignificant paths shown as dotted lines

Working Memory was negatively associated with the broad Externalizing factor (p ≤ 0.001), with standardized estimate values of −0.576; none of the specific factors were significantly predicted by working memory (all β ≤ 0.294, all p > 0.09).

Inclusion of Conduct Disorder Symptoms

We excluded CD from analyses because in this age range, the base rate for the majority of symptoms (e.g., rapes, fire setting, running away overnight, etc.) are generally too low to allow their inclusion. However, results and interpretations did not change when the CD symptoms that could be included (i.e., lying, stealing, bullies, cruelty to animals, and destruction of property) were included. For Model 2, WM predicted both the general p-factor, β = −0.288, p < 0.001, and externalizing, β = −0.383, p < 0.001, but not internalizing factor, β = 0.026, p = 0.77. Similarly, for Model 4, WM was associated with the broad externalizing factor, β = −0.558, p < 0.001, but not the specific inattention, hyperactive, or ODD/CD factors (all β < 0.248, all p > 0.14).

Evaluation of Possible Sex Effects

When the factor scores for externalizing, internalizing, and general psychopathology were output and saved, boys had greater externalizing, r(413) = .-0.171, p < 0.01, and general psychopathology, r(413) = .-0.143, p < 0.01, but there were no gender differences in general internalizing psychopathology, r(413) = −0.083, p > 0.05. To determine whether the relationship between WM and psychopathology was equivalent across girls and boys, we examined model 2 based on Joreskog’s hierarchy (Jöreskog 1971). Fit statistics for each step of the model can be found in Table 4. We first fit the model separately for boys (Model 2.0 M) and girls (Model 2.0F). Fit was also good in a two-group model (Model 2.1) where all parameters were estimated separately in the two gender groups. Because model 2.1 fit well, we then tested a model (Model 2.2) in which factor loadings were constrained to be equal across both groups. Again, the fit statistics suggested this model fit the data well. Comparison of models 2.1 and 2.2 using the Satorra-Bentler Scaled Chi Square difference (Satorra and Bentler 2001, 2010) was not statistically significant, χ2(df = 9) = 11.27, ns. This indicates that the factor loadings in the two groups are statistically invariant, and that there are no meaningful difference in the factor structure between boys and girls. Finally, regression weights from the WM factor to the internalizing factor, externalizing factor, and general p-factor were constrained to be equal across groups (model 3a). Again, the model fit well and was statistically invariant from Model 2.2, χ2(df = 3) = 3.46, ns.
Table 4

Fit statistics for models assessing factor loading and path invariance across boys and girls

Model

χ2

df

RMSEA

CFI

TLI

2.0 M

44.030

25

0.056

0.975

0.955

2.0F

27.683

25

0.025

0.995

0.992

2.1

72.539

52

0.044

0.985

0.974

2.2

83.376

61

0.042

0.983

0.976

Model 2.2 vs Model 2.1

11.27 (ns)

9

0.006

0.01

0.01

3a

86.890

64

0.043

0.983

0.975

Model 3a vs Model 2.2

3.46 (ns)

3

0.001

0.000

0.001

Discussion

Supported by a substantial body of literature, contemporary understanding of psychiatric taxonomy includes both broad and discrete dimensional liabilities. But, the external validation of these liabilities and demonstration of their ultimate usefulness for identifying underlying mechanism is ongoing, and is less commonly addressed. Existing work reporting significant bivariate correlations between dimensional factor scores and individual measures of neuropsychological performance have found that sustained attention and set shifting are associated with the general psychopathology factor, and that verbal fluency is associated with both the general psychopathology and the specific externalizing dimension (Caspi et al. 2014). Within an ADHD bifactor model, performance on inhibitory control and set shifting tasks are associated with a general ADHD factor as well as a specific hyperactivity/impulsivity (but not inattention) factor (Martel et al. 2011).

However, the analytic approach adopted by this prior work does not answer whether the associations between the specific factors and neuropsychological performance would survive after the more general factors are taken into consideration, or if more robust/latent indices of executive control and cognitive performance had been used. Thus, a clear strength of the current study was its use of an SEM approach capable of simultaneously evaluating the unique relationships of a well-specified latent cognitive process (WM), to both specific and general liabilities for psychopathology.

We found that externalizing disorders were independently and disproportionately associated with WM impairments after accounting for the relationship of WM with general psychopathology, upholding the general pattern of relationships Caspi et al. (2014) reported. When a bifactor model of externalizing symptomology was fit to further explore this relationship, WM capacity was only correlated with the general externalizing dimension; correlation with the specific inattention, hyperactive/impulsive, and oppositional factors did not survive once the general dimension was taken into consideration. Though theory-based explanations might be advanced by way of explaining discrepancies with Martel et al. (2011) utilized a wider 6–18 year age range), it is more likely that the association of cognitive performance to the specific hyperactivity/impulsivity dimension would not have survived after the more general factor were taken into consideration, as it did not in our analyses. To better characterize developmental timing effects it would be important for future studies to combine an SEM approach with a wider age range than allowed by the current study. Overall, these results indicate that although individual differences in WM capacity predict general psychiatric severity, WM deficits are particularly and uniquely associated with the severity of externalizing disorders.

In line with major conceptualizations of WM (e.g., Baddeley 1986; Daneman and Carpenter 1980; Engle et al. 1999), we included both verbal and visuospatial working memory tasks that allowed us to model the domain-general central executive which is at the core of the WM construct (Barrouillet et al. 2004; Kane et al. 2007; Unsworth and Engle 2006, 2007). As an index of variance shared among three well-validated measures of WM, our latent factor was less vulnerable than single indices of performance to concerns of task impurity, unreliability, and measurement error, which provided a degree of confidence and ease of interpretation that was missing from previous studies. This approach may also be used in the future to clarify the specific contributions of other potential endophenotypes including latent indices of “set shifting” and “common” EF (Snyder et al. 2015).

Interestingly, in a sample of 5–11 year old girls followed longitudinally for 5 years, Lahey et al. (2015) found that over and above the association with general psychopathology, the externalizing dimension was independently associated with concurrent and prospective academic difficulty (i.e., grade retention and the use of special education services), as well as with prospective teacher reported academic achievement in reading, spelling, and mathematics. Because WM is crucial to the development of skilled cognition and behavior (Anderson 1982; Logan 1992) and demonstrates strong longitudinal associations with academic achievement (Bull et al. 2008; Geary 2011; Raghubar et al. 2010), together, the pattern of these results suggest that working memory deficits may be a common mechanism that places children at specific risk for both externalizing disorders and poor academic outcomes.

Though our formation of a latent WM construct remains a strength of the study, recall accuracy was the manifest outcome variable for the complex and backwards span tasks used herein. This represents a standard approach, even though global processing speed (alongside the central executive) is known to drive both individual (Karalunas and Huang-Pollock 2013; Weigard and Huang-Pollock 2016) and developmental (Case et al. 1982; Fry and Hale 1996, 2000; Kail 1992, 2007; Kail and Salthouse 1994) differences in performance. Arguably one of the best ways to incorporate accuracy and speed of performance into a single set of indices is through a computational approach known as diffusion modelling (Ratcliff and McKoon 2008). This approach, which has long been used in the cognitive sciences and cognitive neurosciences, has recently begun to be adopted in the developmental (Cohen-Gilbert et al. 2014; Ratcliff et al. 2012), aging (Ratcliff et al. 2004; Ratcliff et al. 2011; Starns and Ratcliff 2010), and clinical (Huang-Pollock et al. 2016; Huang-Pollock et al. 2012; Karalunas et al. 2012; Moustafa et al. 2015; Weigard et al. 2016; Weigard and Huang-Pollock 2014; Wiecki et al. 2015) literatures.

Unlike performance indices that are restricted to mean reaction time or mean accuracy, this approach relies on the shape of the reaction time distributions for both error and correct responses to output a comprehensive set of performance parameters. It thereby provides a more complete picture of performance than variables that rely on accuracy or RT alone. However, the diffusion model is only applicable for forced choice RT tasks, so that methodology could not be used in the current study. But, future work utilizing well-validated EF tasks that are amenable to that type of analysis and data collection, would be important. It may be that these more sensitive performance indices might alter the patterns of associations and interpretations that were found here.

In addition to considering how alternative indices of cognitive performance might influence results, it also bears mentioning that the identity of the reporter (parent, teacher, or child) and the strategy used to combine those reports (Youngstrom et al. 2000) can alter rates of comorbidity (Achenbach et al. 1987; Collishaw et al. 2009; De Los Reyes and Kazdin 2005; Youngstrom et al. 2000). Because teachers may be less sensitive to internalizing symptoms (Abikoff et al. 1993), and children similarly demonstrate poor insight into their own externalizing behaviors (Youngstrom et al. 2000), we chose to utilize parent report of behavior in the absence of clear guidelines on how to incorporate multiple informant reports (De Los Reyes and Kazdin 2005). Reassuringly, previous research has found that child indices of cognitive functioning are equally associated with parent and teacher ratings of psychopathology (Collishaw et al. 2009), but future studies investigating this further would of course be important. Similarly, future studies examining how these relationships may or may not change when self-report, father, or other primary caregiver report is utilized, as well as at different stages of development (e.g., adolescence), would also be important.

In contrast to findings for the externalizing domain, WM capacity was not significantly associated with the internalizing dimension once variance attributed to general and externalizing psychopathology were taken into consideration. These results may not be entirely surprising. For example, although models of anxiety have suggested that an important consequence of chronic rumination and worry should be manifest as worse working memory (Eysenck and Derakshan 2011; Pessoa 2009), as well as loss of inhibitory control over time due to ego depletion (Granic 2014), empirically, broad evidence of such impairments have been difficult to consistently document (Berggren and Derakshan 2013). Ongoing work in the area suggests that chronic rumination and worry may simultaneously increase motivation to perform well, thus cancelling out any performance deficits that might otherwise have been observed (Braver et al. 2014; Edwards et al. 2015; Pessoa 2009). Similarly, substantial heterogeneity in neurocognitive performance is also found in depression (McClintock et al. 2010), with evidence that executive dysfunction is not observed among depressed patients who demonstrate valid effort during testing (Benitez et al. 2011; Rohling et al. 2002). However, even though motivation-cognition interactions on performance are relevant to a wide range of processes outside of WM (Botvinick and Braver 2015; Braver et al. 2014) and are also observed among externalizing disorders (Luman et al. 2005), the association between externalizing behavior and executive dyscontrol survives even when task engagement is controlled (Huang-Pollock et al. 2016; Huang-Pollock et al. 2007; Shanahan et al. 2008; Shiels et al. 2008).

Among the school aged children in our study, externalizing and general psychopathology was greater among boys; there were no gender differences in internalizing disorders. Such results are consistent with other developmental work in this age range demonstrating greater preponderance of externalizing disorders in boys. It is also consistent with work finding the female preponderance for depression and anxiety is most clearly evident in the teenage years (Crick and Zahn-Waxler 2003; Essex et al. 2006; Kessler et al. 1994; Zahn-Waxler et al. 2008). However, there were no meaningful gender differences in factor structure, and the regression weights between WM and psychopathology latent factors were equivalent between groups. Thus, regardless of how gendered the expression of psychopathology may be, we find that the cognitive liability WM deficits confer to the severity of psychopathology in general, and to the specific externalizing direction, are the same regardless of the gender of the child.

Our sample represented a range of severity from typically developing children to those with psychiatric disorders, but was primarily driven to recruit children with ADHD and their non-ADHD peers. We believe our results to be broadly applicable to understanding the cognitive mechanisms involved in the development of psychopathology generally, particularly because ADHD represents one of the most common childhood psychiatric disorders, in which 25–50 % of children meet criteria for a concurrent anxiety disorder (Angold et al. 1999; Biederman et al. 1991; Jensen et al. 1997; Tannock 2009), 20–30 % meet criteria for a concurrent depressive disorder (Angold et al. 1999; Meinzer et al. 2014), and 30–50 % meet criteria for concurrent ODD/CD (Angold et al. 1999; Biederman et al. 1991). Thus, in many ways, ADHD represents the ideal childhood mental health disorder in which to conduct such an inquiry. Indeed, our results are strikingly consistent with data reported in the large longitudinal and epidemiological Dunedin sample which found neuropsychological performance to be associated with both the general psychopathology and specific externalizing dimensions (Caspi et al. 2014). However, even conservatively interpreted within an ADHD framework, our findings still suggest that individual differences in working memory predicts overall psychiatric severity among children with ADHD, but that such capacity is particularly and uniquely associated with externalizing severity in that population.

Conclusions

Overall, we found evidence that working memory deficits are uniquely and disproportionately associated with externalizing disorders, over and above that of general psychopathology, and regardless of the gender of the child. If such findings were to hold in longitudinal and epidemiological samples, it would suggest that poor working memory raises the risk for the development of psychopathology, generally, while simultaneously raising the risk for an externalizing disorder, specifically. The same could not be said for internalizing disorders, despite the fact that executive function impairments (and working memory specifically) have been invoked in many well regarded theories of those disorders. These findings are consistent with the ongoing discussion and search for dimensional liabilities that influence the development of mental health problems.

Notes

Compliance with Ethical Standards

Funding

This work was supported in part by National Institute of Mental Health Grant R01 MH084947 to Cynthia Huang-Pollock. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health or the National Institutes of Health.

Conflicts of Interest

The authors declare that there are no conflicts of interest.

Ethical Approval

All procedures were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent (assent for children) was obtained from all individual participants included in the study.

References

  1. Abikoff, H., Courtney, M., Pelham, W. E., & Koplewicz, H. S. (1993). Teachers ratings of disruptive behaviors: the influence of halo effects. Journal of Abnormal Child Psychology, 21, 519–533. doi: 10.1007/bf00916317.PubMedCrossRefGoogle Scholar
  2. Achenbach, T. M., & Edelbrock, C. S. (1978). Classification of child psychopathology: review and analysis of empirical efforts. Psychological Bulletin, 85, 1275–1301. doi: 10.1037//0033-2909.85.6.1275.PubMedCrossRefGoogle Scholar
  3. Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232. doi: 10.1037/0033-2909.101.2.213.PubMedCrossRefGoogle Scholar
  4. Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review, 89, 369–406.CrossRefGoogle Scholar
  5. Angold, A., Costello, E. J., & Erkanli, A. (1999). Comorbidity. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 40, 57–87. doi: 10.1017/s0021963098003448.PubMedCrossRefGoogle Scholar
  6. Arias, V. B., Ponce, F. P., Martínez-Molina, A., Arias, B., & Núñez, D. (2016). General and specific attention-deficit/hyperactivity disorder factors of children 4 to 6 years of age: an exploratory structural equation modeling approach to assessing symptom multidimensionality. Journal of Abnormal Psychology, 125, 125–137. doi: 10.1037/abn0000115.PubMedCrossRefGoogle Scholar
  7. Baddeley, A. D. (1986). Working memory. Oxford: Oxford University Press.Google Scholar
  8. Barkley, R. A. (1997). Behavioral inhibition, sustained attention, and executive functions: constructing a unifying theory of ADHD. Psychological Bulletin, 121, 65–94.PubMedCrossRefGoogle Scholar
  9. Barkley, R. A., Edwards, G., Laneri, M., Fletcher, K., & Metevia, L. (2001). Executive functioning, temporal discounting, and sense of time in adolescents with attention deficit hyperactivity disorder (ADHD) and oppositional defiant disorder (ODD). Journal of Abnormal Child Psychology, 29, 541–556.PubMedCrossRefGoogle Scholar
  10. Barnett, R., Maruff, P., & Vance, A. (2009). Neurocognitive function in attention-deficit-hyperactivity disorder with and without comorbid disruptive behaviour disorders. Australian and New Zealand Journal of Psychiatry, 43, 722–730. doi: 10.1080/00048670903001927.PubMedCrossRefGoogle Scholar
  11. Barrouillet, P., Bernardin, S., & Camos, V. (2004). Time constraints and resource sharing in adults' working memory spans. Journal of Experimental Psychology. General, 133, 83–100. doi: 10.1037/0096-3445.133.1.83.PubMedCrossRefGoogle Scholar
  12. Benitez, A., Horner, M. D., & Bachman, D. (2011). Intact cognition in depressed elderly veterans providing adequate effort. Archives of Clinical Neuropsychology, 26, 184–193. doi: 10.1093/arclin/acr001.PubMedCrossRefGoogle Scholar
  13. Bentler, P. M. (1990). Comparative fit indices in structural models. Psychological Bulletin, 107, 238–246. doi: 10.1037/0033-2909.107.2.238.PubMedCrossRefGoogle Scholar
  14. Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606. doi: 10.1037/0033-2909.88.3.588.CrossRefGoogle Scholar
  15. Berggren, N., & Derakshan, N. (2013). Attentional control deficits in trait anxiety: why you see them and why you don't. Biological Psychology, 92, 440–446. doi: 10.1016/j.biopsycho.2012.03.007.PubMedCrossRefGoogle Scholar
  16. Biederman, J., Newcorn, J., & Sprich, S. (1991). Comorbidity of attention deficit hyperactivity disorder with conduct, depressive, anxiety, and other disorders. American Journal of Psychiatry, 148, 564–577.PubMedCrossRefGoogle Scholar
  17. Bishop, S. J. (2009). Trait anxiety and impoverished prefrontal control of attention. Nature Neuroscience, 12, 92–98. doi: 10.1038/nn.2242.PubMedCrossRefGoogle Scholar
  18. Botvinick, M., & Braver, T. (2015). Motivation and cognitive control: from behavior to neural mechanism. Annual Review of Psychology, 66, 83–113. doi: 10.1146/annurev-psych-010814-015044.PubMedCrossRefGoogle Scholar
  19. Braver, T. S., Krug, M. K., Chiew, K. S., Kool, W., Westbrook, J. A., Clement, N. J., & Somerville, L. H. (2014). Mechanisms of motivation-cognition interaction: challenges and opportunities. Cognitive, Affective, & Behavioral Neuroscience, 14, 443–472. doi: 10.3758/s13415-014-0300-0.CrossRefGoogle Scholar
  20. Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). New York: Guilford Publications.Google Scholar
  21. Browne, M. W., & Cudeck, R. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230–258. doi: 10.1177/0049124192021002005.CrossRefGoogle Scholar
  22. Bull, R., Espy, K. A., & Wiebe, S. A. (2008). Short-term memory, working memory, and executive functioning in preschoolers: longitudinal predictors of mathematical achievement at age 7 years. Developmental Neuropsychology, 33, 205–228. doi: 10.1080/87565640801982312.PubMedPubMedCentralCrossRefGoogle Scholar
  23. Case, R., Kurland, D. M., & Goldberg, J. (1982). Operational efficiency and the growth of short-term-memory span. Journal of Experimental Child Psychology, 33, 386–404.CrossRefGoogle Scholar
  24. Caspi, A., Houts, R. M., Belsky, D. W., Goldman-Mellor, S. J., Harrington, H., Israel, S., & Moffitt, T. E. (2014). The p factor: one general psychopathology factor in the structure of psychiatric disorders? Clinical Psychological Science, 2, 119–137. doi: 10.1177/2167702613497473.PubMedPubMedCentralCrossRefGoogle Scholar
  25. Cicchetti, D., & Rogosch, F. A. (1996). Equifinality and multifinality in developmental psychopathology. Development and Psychopathology, 8, 597–600.CrossRefGoogle Scholar
  26. Cohen-Gilbert, J. E., Killgore, W. D. S., White, C. N., Schwab, Z. J., Crowley, D. J., Covell, M. J., et al. (2014). Differential influence of safe versus threatening facial expressions on decision-making during an inhibitory control task in adolescence and adulthood. Developmental Science, 17, 212–223. doi: 10.1111/desc.12123.PubMedPubMedCentralCrossRefGoogle Scholar
  27. Collishaw, S., Goodman, R., Ford, T., Rabe-Hesketh, S., & Pickles, A. (2009). How far are associations between child, family and community factors and child psychopathology informant-specific and informant-general? Journal of Child Psychology and Psychiatry, 50, 571–580. doi: 10.1111/j.1469-7610.2008.02026.x.PubMedCrossRefGoogle Scholar
  28. Conners, C. K. (2008). Conners’ rating scales—3 technical manual. New York: Multi-Health Systems Inc..Google Scholar
  29. Costello, E. J., Mustillo, S., Erkanli, A., Keeler, G., & Angold, A. (2003). Prevalence and development of psychiatric disorders in childhood and adolescence. Archives of General Psychiatry, 60, 837–844. doi: 10.1001/archpsyc.60.8.837.PubMedCrossRefGoogle Scholar
  30. Cramer, A. O. J., Waldorp, L. J., van der Maas, H. L. J., & Borsboom, D. (2010). Comorbidity: a network perspective. Behavioral and Brain Sciences, 33, 137–150. doi: 10.1017/s0140525x09991567.PubMedCrossRefGoogle Scholar
  31. Crick, N. R., & Zahn-Waxler, C. (2003). The development of psychopathology in females and males: current progress and future challenges. Development and Psychopathology, 15, 719–742. doi: 10.1017/s095457940300035x.PubMedCrossRefGoogle Scholar
  32. Daneman, M., & Carpenter, P. A. (1980). Individual-differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior, 19, 450–466.CrossRefGoogle Scholar
  33. De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: a critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131, 483–509. doi: 10.1037/0033-2909.131.4.483.PubMedCrossRefGoogle Scholar
  34. Edwards, E. J., Edwards, M. S., & Lyvers, M. (2015). Cognitive trait anxiety, situational stress, and mental effort predict shifting efficiency: implications for attentional control theory. Emotion, 15, 350–359. doi: 10.1037/emo0000051.PubMedCrossRefGoogle Scholar
  35. Enders, C. K. (2010). Applied missing data analysis. New York: Guilford Press.Google Scholar
  36. Engle, R., Kane, M., & Tuholski, S. (1999). Individual differences in working memory capacity and what they tell us about controlled attention, general fluid intelligence, and functions of the prefrontal cortex. In A. Miyake & P. Shah (Eds.), Models of working memory: mechanisms of active maintenance and executive control (pp. 102–134). New York: Cambridge University Press.CrossRefGoogle Scholar
  37. Essex, M. J., Kraemer, H. C., Armstrong, J. M., Boyce, T., Goldsmith, H. H., Klein, M. H., & Kupfer, D. J. (2006). Exploring risk factors for the emergence of children's mental health problems. Archives of General Psychiatry, 63, 1246–1256. doi: 10.1001/archpsyc.63.11.1246.PubMedCrossRefGoogle Scholar
  38. Eysenck, M. W., & Derakshan, N. (2011). New perspectives in attentional control theory. Personality and Individual Differences, 50, 955–960. doi: 10.1016/j.paid.2010.08.019.CrossRefGoogle Scholar
  39. Ford, T., Goodman, R., & Meltzer, H. (2003). The British child and adolescent mental health survey 1999: the prevalence of DSM-IV disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 42, 1203–1211. doi: 10.1097/01.chi.0000081820.25107.ae.PubMedCrossRefGoogle Scholar
  40. Fry, A. F., & Hale, S. (1996). Processing speed, working memory, and fluid intelligence: evidence for a developmental cascade. Psychological Science, 7, 237–241.CrossRefGoogle Scholar
  41. Fry, A. F., & Hale, S. (2000). Relationships among processing speed, working memory, and fluid intelligence in children. Biological Psychology, 54, 1–34.PubMedCrossRefGoogle Scholar
  42. Geary, D. C. (2011). Cognitive predictors of achievement growth in mathematics: a 5-year longitudinal study. Developmental Psychology, 47, 1539–1552. doi: 10.1037/a0025510.PubMedPubMedCentralCrossRefGoogle Scholar
  43. Gottesman, I. I., & Gould, T. D. (2003). The endophenotype concept in psychiatry: etymology and strategic intentions. American Journal of Psychiatry, 160, 636–645.PubMedCrossRefGoogle Scholar
  44. Granic, I. (2014). The role of anxiety in the development, maintenance, and treatment of childhood aggression. Development and Psychopathology, 26, 1515–1530. doi: 10.1017/s0954579414001175.PubMedCrossRefGoogle Scholar
  45. Hill, E. L. (2004). Executive dysfunction in autism. Trends in Cognitive Sciences, 8, 26–32. doi: 10.1016/j.tics.2003.11.003.PubMedCrossRefGoogle Scholar
  46. Hu, L.-T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. doi: 10.1080/10705519909540118.CrossRefGoogle Scholar
  47. Huang-Pollock, C. L., Mikami, A. Y., Pfiffner, L., & McBurnett, K. (2007). ADHD subtype differences in motivational responsivity but not inhibitory control: evidence from a reward-based variation of the stop signal paradigm. Journal of Clinical Child and Adolescent Psychology, 36, 127–136.PubMedCrossRefGoogle Scholar
  48. Huang-Pollock, C. L., Karalunas, S. L., Tam, H., & Moore, A. N. (2012). Evaluating vigilance deficits in ADHD: a meta-analysis of CPT performance. Journal of Abnormal Psychology, 121, 360–371. doi: 10.1037/a0027205.PubMedPubMedCentralCrossRefGoogle Scholar
  49. Huang-Pollock, C., Ratcliff, R., McKoon, G., Shapiro, Z., Weigard, A., & Galloway-Long, H. (2016). Using the diffusion model to explain cognitive deficits in attention deficit hyperactivity disorder. Journal of Abnormal Child Psychology, 1–12. doi: 10.1007/s10802-016-0151-y.
  50. Hughes, C., Russell, J., & Robbins, T. W. (1994). Evidence for executive dysfunction in autism. Neuropsychologia, 32, 477–492. doi: 10.1016/0028-3932(94)90092-2.PubMedCrossRefGoogle Scholar
  51. Jensen, P. S., Martin, D., & Cantwell, D. P. (1997). Comorbidity in ADHD: implications for research, practice, and DSM-V. Journal of the American Academy of Child and Adolescent Psychiatry, 36, 1065–1079.PubMedCrossRefGoogle Scholar
  52. Jöreskog, K. G. (1971). Simultaneous factor analysis in several populations. Psychometrika, 36, 409–426. doi: 10.1007/BF02291366.CrossRefGoogle Scholar
  53. Kail, R. (1992). Processing speed, speech rate, and memory. Developmental Psychology, 28, 899–904.CrossRefGoogle Scholar
  54. Kail, R. (2007). Longitudinal evidence that increases in processing speed and working memory enhance children's reasoning. Psychological Science, 18, 312–313.PubMedCrossRefGoogle Scholar
  55. Kail, R., & Salthouse, T. A. (1994). Processing speed as a mental-capacity. Acta Psychologica, 86, 199–225.PubMedCrossRefGoogle Scholar
  56. Kane, M. J., Conway, A., Hambrick, D. Z., & Engle, R. W. (2007). Variation in working memory capacity as variation in executive attention and control. In A. R. A. Conway, C. Jarrold, M. J. Kane, A. Miyake, & J. N. Towse (Eds.), Variation in working memory. New York: Oxford University Press.Google Scholar
  57. Karalunas, S. L., & Huang-Pollock, C. L. (2013). Integrating impairments in reaction time and executive function using a diffusion model framework. Journal of Abnormal Child Psychology, 41, 837–850. doi: 10.1007/s10802-013-9715-2.PubMedPubMedCentralCrossRefGoogle Scholar
  58. Karalunas, S. L., Huang-Pollock, C. L., & Nigg, J. T. (2012). Decomposing attention-deficit/hyperactivity disorder (ADHD)-related effects in response speed and variability. Neuropsychology, 26, 684–694. doi: 10.1037/a0029936.PubMedPubMedCentralCrossRefGoogle Scholar
  59. Kendler, K. S., Walters, E. E., Neale, M. C., Kessler, R. C., Heath, A. C., & Eaves, L. J. (1995). The structure of the genetic and environmental risk factors for 6 major psychiatric disorders in women: phobia, generalized anxiety DIsorder, panic disorder, bulimia, major depression, and alcoholism. Archives of General Psychiatry, 52, 374–383.PubMedCrossRefGoogle Scholar
  60. Kendler, K. S., Prescott, C. A., Myers, J., & Neale, M. C. (2003). The structure of genetic and environmental risk factors for common psychiatric and substance use disorders in men and women. Archives of General Psychiatry, 60, 929–937. doi: 10.1001/archpsyc.60.9.929.PubMedCrossRefGoogle Scholar
  61. Kessler, R. C., McGonagle, K. A., Zhao, S. Y., Nelson, C. B., Hughes, M., Eshleman, S., & Kendler, K. S. (1994). Lifetime and 12-month prevalence of DSM-III psychiatric disorders in the United States: results from the National Comorbidity Survey. Archives of General Psychiatry, 51, 8–19.PubMedCrossRefGoogle Scholar
  62. Kessler, R. C., Chiu, W. T., Demler, O., & Walters, E. E. (2005). Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry, 62, 617–627. doi: 10.1001/archpsyc.62.6.617.PubMedPubMedCentralCrossRefGoogle Scholar
  63. Krueger, R. F. (1999). The structure of common mental disorders. Archives of General Psychiatry, 56, 921–926. doi: 10.1001/archpsyc.56.10.921.PubMedCrossRefGoogle Scholar
  64. Krueger, R. F., Caspi, A., Moffitt, T. E., & Silva, P. A. (1998). The structure and stability of common mental disorders (DSM-III-R): a longitudinal-epidemiological study. Journal of Abnormal Psychology, 107, 216–227. doi: 10.1037/0021-843x.107.2.216.PubMedCrossRefGoogle Scholar
  65. Laceulle, O. M., Vollebergh, W. A. M., & Ormel, J. (2015). The structure of psychopathology in adolescence: replication of a general psychopathology factor in the TRAILS study. Clinical Psychological Science, 3, 850–860. doi: 10.1177/2167702614560750.CrossRefGoogle Scholar
  66. Lahey, B. B., Applegate, B., McBurnett, K., Biederman, J., Greenhill, L., Hynd, G. W., & Shaffer, D. (1994). DSM-IV field trials for attention-deficit hyperactivity disorder in children and adolescents. American Journal of Psychiatry, 151, 1673–1685.PubMedCrossRefGoogle Scholar
  67. Lahey, B. B., Rathouz, P. J., Van Hulle, C., Urbano, R. C., Krueger, R. F., Applegate, B., et al. (2008). Testing structural models of DSM-IV symptoms of common forms of child and adolescent psychopathology. Journal of Abnormal Child Psychology, 36, 187–206. doi: 10.1007/s10802-007-9169-5.PubMedCrossRefGoogle Scholar
  68. Lahey, B. B., Applegate, B., Hakes, J. K., Zald, D. H., Hariri, A. R., & Rathouz, P. J. (2012). Is there a general factor of prevalent psychopathology during adulthood? Journal of Abnormal Psychology, 121, 971–977. doi: 10.1037/a0028355.PubMedPubMedCentralCrossRefGoogle Scholar
  69. Lahey, B. B., Rathouz, P. J., Keenan, K., Stepp, S. D., Loeber, R., & Hipwell, A. E. (2015). Criterion validity of the general factor of psychopathology in a prospective study of girls. Journal of Child Psychology and Psychiatry, 56, 415–422. doi: 10.1111/jcpp.12300.PubMedCrossRefGoogle Scholar
  70. Logan, G. (1992). Attention and preattention in theories of automaticity. American Journal of Psychology, 105, 317–339.PubMedCrossRefGoogle Scholar
  71. Luman, M., Oosterlaan, J., & Sergeant, J. A. (2005). The impact of reinforcement contingencies on AD/HD: a review and theoretical appraisal. Clinical Psychology Review, 25, 183–213. doi: 10.1016/j.cpr.2004.11.001.PubMedCrossRefGoogle Scholar
  72. Malhotra, D., & Sebat, J. (2012). CNVs: harbingers of a rare variant revolution in psychiatric genetics. Cell, 148, 1223–1241. doi: 10.1016/j.cell.2012.02.039.PubMedPubMedCentralCrossRefGoogle Scholar
  73. Marsh, H. W., Byrne, B. M., & Craven, R. (1992). Overcoming problems in confirmatory factor analysis of MTMM data: the correlated uniqueness model and factorial invariance. Multivariate Behavioral Research, 27, 489–507. doi: 10.1207/s15327906mbr2704_1.PubMedCrossRefGoogle Scholar
  74. Martel, M. M., Gremillion, M., Roberts, B., von Eye, A., & Nigg, J. T. (2010a). The structure of childhood disruptive behaviors. Psychological Assessment, 22, 816–826. doi: 10.1037/a0020975.PubMedPubMedCentralCrossRefGoogle Scholar
  75. Martel, M. M., von Eye, A., & Nigg, J. T. (2010b). Revisiting the latent structure of ADHD: is there a 'g' factor? Journal of Child Psychology and Psychiatry, 51, 905–914. doi: 10.1111/j.1469-7610.2010.02232.x.PubMedPubMedCentralCrossRefGoogle Scholar
  76. Martel, M. M., Roberts, B., Gremillion, M., von Eye, A., & Nigg, J. T. (2011). External validation of bifactor model of ADHD: explaining heterogeneity in psychiatric comorbidity, cognitive control, and personality trait profiles within DSM-IV ADHD. Journal of Abnormal Child Psychology, 39, 1111–1123. doi: 10.1007/s10802-011-9538-y.PubMedPubMedCentralCrossRefGoogle Scholar
  77. Martel, M. M., von Eye, A., & Nigg, J. (2012). Developmental differences in structure of attention-deficit/hyperactivity disorder (ADHD) between childhood and adulthood. International Journal of Behavioral Development, 36, 279–292. doi: 10.1177/0165025412444077.PubMedPubMedCentralCrossRefGoogle Scholar
  78. McAlonan, G. M., Cheung, V., Cheung, C., Chua, S. E., Murphy, D. G. M., Suckling, J., et al. (2007). Mapping brain structure in attention deficit-hyperactivity disorder: a voxel-based MRI study of regional grey and white matter volume. Psychiatry Research: Neuroimaging, 154, 171–180. doi: 10.1016/j.pscychresns.2006.09.006.PubMedCrossRefGoogle Scholar
  79. McClintock, S. A., Husain, M. M., Greer, T. L., & Cullum, C. M. (2010). Association between depression severity and neurocognitive function in major depressive disorder: a review and synthesis. Neuropsychology, 24, 9–34. doi: 10.1037/a0017336.PubMedCrossRefGoogle Scholar
  80. McLean, J. F., & Hitch, G. J. (1999). Working memory impairments in children with specific arithmetic learning difficulties. Journal of Experimental Child Psychology, 74, 240–260. doi: 10.1006/jecp.1999.2516.PubMedCrossRefGoogle Scholar
  81. Meinzer, M. C., Pettit, J. W., & Viswesvaran, C. (2014). The co-occurrence of attention-deficit/hyperactivity disorder and unipolar depression in children and adolescents: a meta-analytic review. Clinical Psychology Review, 34, 595–607. doi: 10.1016/j.cpr.2014.10.002.PubMedCrossRefGoogle Scholar
  82. Merikangas, K. R., He, J. P., Burstein, M., Swanson, S. A., Avenevoli, S., Cui, L. H., et al. (2010). Lifetime prevalence of mental disorders in U.S. adolescents: results from the National Comorbidity Survey Replication-Adolescent Supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 49, 980–989. doi: 10.1016/j.jaac.2010.05.017.PubMedPubMedCentralCrossRefGoogle Scholar
  83. Moffitt, T. E. (1993). The neuropsychology of conduct disorder. Development and Psychopathology, 5, 135–151.CrossRefGoogle Scholar
  84. Morgan, A. B., & Lilienfeld, S. O. (2000). A meta-analytic review of the relation between antisocial behavior and neuropsychological measures of executive function. Clinical Psychology Review, 20, 113–136.PubMedCrossRefGoogle Scholar
  85. Moustafa, A. A., Keri, S., Somlai, Z., Balsdon, T., Frydecka, D., Misiak, B., & White, C. (2015). Drift diffusion model of reward and punishment learning in schizophrenia: modeling and experimental data. Behavioural Brain Research, 291, 147–154. doi: 10.1016/j.bbr.2015.05.024.PubMedCrossRefGoogle Scholar
  86. Muthén, L.K., & Muthén, B.O. (1998-2012). Mplus user’s guide, 7th Ed. Los Angeles, CA: Muthén & MuthénGoogle Scholar
  87. Nieuwenstein, M. R., Aleman, A., & de Haan, E. H. F. (2001). Relationship between symptom dimensions and neurocognitive functioning in schizophrenia: a meta-analysis of WCST and CPT studies. Journal of Psychiatric Research, 35, 119–125.PubMedCrossRefGoogle Scholar
  88. Norman, D. A., & Shallice, T. (1986). Attention to action: willed and automatic control of behavior. In R. J. Davidson, G. E. Schwartz, & D. Shapiro (Eds.), Consciousness and self-regulation: advances in research and theory (Vol. 4, pp. 1–18). New York: Plenum Press.Google Scholar
  89. Oosterlaan, J., Logan, G., & Sergeant, J. A. (1998). Response inhibition in AD/HD, CD, comorbid AD/HD + CD, anxious, and control children: a meta-analysis of studies with the stop task. Journal of Child Psychology and Psychiatry, 39, 411–425.PubMedCrossRefGoogle Scholar
  90. Paelecke-Habermann, Y., Pohl, J., & Leplow, B. (2005). Attention and executive functions in remitted major depression patients. Journal of Affective Disorders, 89, 125–135. doi: 10.1016/j.jad.2005.09.006.PubMedCrossRefGoogle Scholar
  91. Pennington, B. F., & Ozonoff, S. (1996). Executive functions and developmental psychopathology. Journal of Child Psychology and Psychiatry, 37, 51–87.PubMedCrossRefGoogle Scholar
  92. Pessoa, L. (2009). How do emotion and motivation direct executive control? Trends in Cognitive Sciences, 13, 160–166. doi: 10.1016/j.tics.2009.01.006.PubMedPubMedCentralCrossRefGoogle Scholar
  93. Quraishi, S., & Frangou, S. (2002). Neuropsychology of bipolar disorder: a review. Journal of Affective Disorders, 72, 209–226. doi: 10.1016/s0165-0327(02)00091-5.PubMedCrossRefGoogle Scholar
  94. Raghubar, K. P., Barnes, M. A., & Hecht, S. A. (2010). Working memory and mathematics: a review of developmental, individual difference, and cognitive approaches. Learning and Individual Differences, 20, 110–122. doi: 10.1016/j.lindif.2009.10.005.CrossRefGoogle Scholar
  95. Raine, A., Buchsbaum, M. S., Stanley, J., Lottenberg, S., Abel, L., & Stoddard, J. (1994). Selective reductions in prefrontal glucose metabolism in murderers. Biological Psychiatry, 36, 365–373. doi: 10.1016/0006-3223(94)91211-4.PubMedCrossRefGoogle Scholar
  96. Raine, A., Moffitt, T. E., Caspi, A., Loeber, R., Stouthamer-Loeber, M., & Lynam, D. (2005). Neurocognitive impairments in boys on the life-course persistent antisocial path. Journal of Abnormal Psychology, 114, 38–49. doi: 10.1037/0021-843x.114.1.38.PubMedCrossRefGoogle Scholar
  97. Ratcliff, R., & McKoon, G. (2008). The diffusion decision model: theory and data for two-choice decision tasks. Neural Computation, 20, 873–922. doi: 10.1162/neco.2008.12-06-420.PubMedPubMedCentralCrossRefGoogle Scholar
  98. Ratcliff, R., Thapar, A., & McKoon, G. (2004). A diffusion model analysis of the effects of aging on recognition memory. Journal of Memory and Language, 50, 408–424. doi: 10.1016/j.jml.2003.11.002.CrossRefGoogle Scholar
  99. Ratcliff, R., Thapar, A., & McKoon, G. (2011). Effects of aging and IQ on item and associative memory. Journal of Experimental Psychology. General, 140, 464–487. doi: 10.1037/a0023810.PubMedPubMedCentralCrossRefGoogle Scholar
  100. Ratcliff, R., Love, J., Thompson, C. A., & Opfer, J. E. (2012). Children are not like older adults: a diffusion model analysis of developmental changes in speeded responses. Child Development, 83, 367–381. doi: 10.1111/j.1467-8624.2011.01683.x.PubMedCrossRefGoogle Scholar
  101. Reynolds, C., & Kamphaus, R. (2004). Behavioral assessment system for children, 2nd Ed. Manual. MN: AGS Publishing.Google Scholar
  102. Rogers, M. A., Kasai, K., Koji, M., Fukuda, R., Iwanami, A., Nakagome, K., et al. (2004). Executive and prefrontal dysfunction in unipolar depression: a review of neuropsychological and imaging evidence. Neuroscience Research, 50, 1–11. doi: 10.1016/j.neures.2004.05.003.PubMedCrossRefGoogle Scholar
  103. Rohling, M. L., Green, P., Allen, L. M., & Iverson, G. L. (2002). Depressive symptoms and neurocognitive test scores in patients passing symptom validity tests. Archives of Clinical Neuropsychology, 17, 205–222. doi: 10.1016/s0887-6177(01)00109-3.PubMedCrossRefGoogle Scholar
  104. Satorra, A., & Bentler, P. M. (2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66, 507–514. doi: 10.1007/bf02296192.CrossRefGoogle Scholar
  105. Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference chi-square test statistic. Psychometrika, 75, 243–248. doi: 10.1007/s11336-009-9135-y.PubMedPubMedCentralCrossRefGoogle Scholar
  106. Schachar, R., Mota, V. L., Logan, G. D., Tannock, R., & Klim, P. (2000). Confirmation of an inhibitory control deficit in attention-deficit/hyperactivity disorder. Journal of Abnormal Child Psychology, 28, 227–235. doi: 10.1023/a:1005140103162.PubMedCrossRefGoogle Scholar
  107. Shaffer, D., Fisher, P., Lucas, C. P., Dulcan, M. K., & Schwab-Stone, M. E. (2000). NIMH diagnostic interview schedule for children version IV (NIMH DISC-IV): description, differences from previous versions, and reliability of some common diagnoses. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 28–38.PubMedCrossRefGoogle Scholar
  108. Shanahan, M. A., Pennington, B. F., & Willcutt, E. W. (2008). Do motivational incentives reduce the inhibition deficit in ADHD? Developmental Neuropsychology, 33, 137–159. doi: 10.1080/87565640701884238.PubMedCrossRefGoogle Scholar
  109. Sheslow, D., & Adams, W. (2003). Wide range assessment of memory and learning, 2nd Ed (WRAML-2): administration and technical manual. DE: Wide Range.Google Scholar
  110. Shiels, K., Hawk Jr., L. W., Lysczek, C. L., Tannock, R., Pelham Jr., W. E., Spencer, S. V., et al. (2008). The effects of incentives on visual-spatial working memory in children with attention-deficit/hyperactivity disorder. Journal of Abnormal Child Psychology, 36, 903–913. doi: 10.1007/s10802-008-9221-0.PubMedPubMedCentralCrossRefGoogle Scholar
  111. Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information processing. II. Perceptual learning, automatic attending, and a general theory. Psychological Review, 84, 127–190.CrossRefGoogle Scholar
  112. Smoller, J. W., Craddock, N., Kendler, K., Lee, P. H., Neale, B. M., Nurnberger, J. I., et al. (2013). Identification of risk loci with shared effects on five major psychiatric disorders: a genome-wide analysis. Lancet, 381, 1371–1379. doi: 10.1016/s0140-6736(12)62129-1.CrossRefGoogle Scholar
  113. Snyder, H. R., Miyake, A., & Hankin, B. L. (2015). Advancing understanding of executive function impairments and psychopathology: bridging the gap between clinical and cognitive approaches. Frontiers in Psychology, 6, 24. doi: 10.3389/fpsyg.2015.00728.CrossRefGoogle Scholar
  114. Starns, J. J., & Ratcliff, R. (2010). The effects of aging on the speed-accuracy compromise: boundary optimality in the diffusion model. Psychology and Aging, 25, 377–390. doi: 10.1037/a0018022.PubMedPubMedCentralCrossRefGoogle Scholar
  115. Steiger, J., & Lind, J. (1980). Statistically based tests for the number of common factors. Paper presented at the psychometric society annual meeting, Iowa City.Google Scholar
  116. Tackett, J. L., Lahey, B. B., van Hulle, C., Waldman, I., Krueger, R. F., & Rathouz, P. J. (2013). Common genetic influences on negative emotionality and a general psychopathology factor in childhood and adolescence. Journal of Abnormal Psychology, 122, 1142–1153. doi: 10.1037/a0034151.PubMedPubMedCentralCrossRefGoogle Scholar
  117. Tannock, R. (2009). ADHD with anxiety disorders. In T. E. Brown (Ed.), ADHD comorbidities: handbook for ADHD complications in children and adults (pp. 131–155). Washington, D.C.: American Psychiatric Publishing, Inc..Google Scholar
  118. Toplak, M. E., Pitch, A., Flora, D. B., Iwenofu, L., Ghelani, K., Jain, U., & Tannock, R. (2009). The unity and diversity of inattention and hyperactivity/impulsivity in ADHD: evidence for a general factor with separable dimensions. Journal of Abnormal Child Psychology, 37, 1137–1150. doi: 10.1007/s10802-009-9336-y.PubMedCrossRefGoogle Scholar
  119. Toplak, M. E., Sorge, G. B., Flora, D. B., Chen, W., Banaschewski, T., Buitelaar, J., et al. (2012). The hierarchical factor model of ADHD: invariant across age and national groupings? Journal of Child Psychology and Psychiatry, 53, 292–303. doi: 10.1111/j.1469-7610.2011.02500.x.PubMedCrossRefGoogle Scholar
  120. Towse, J., Hutton, U., & Hitch, G. (1998). Grass is coloured…red? Further sentence completion norms for children during a working memory reading task (Tech Rep No CDRG3). Retrieved from http://www.pc.rhbnc.ac.uk/papers/tr.html.
  121. Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1–10. doi: 10.1007/BF02291170.CrossRefGoogle Scholar
  122. Unsworth, N., & Engle, R. W. (2006). Simple and complex memory spans and their relation to fluid abilities: evidence from list-length effects. Journal of Memory and Language, 54, 68–80. doi: 10.1016/j.jml.2005.06.003.CrossRefGoogle Scholar
  123. Unsworth, N., & Engle, R. W. (2007). The nature of individual differences in working memory capacity: active maintenance in primary memory and controlled search from secondary memory. Psychological Review, 114, 104–132. doi: 10.1037/0033-295x.114.1.104.PubMedCrossRefGoogle Scholar
  124. Wechsler, D. (2003). Wechsler intelligence scale for children- fourth edition: technical and interpretive manual. San Antonio: Harcourt Brace.Google Scholar
  125. Weigard, A., & Huang-Pollock, C. L. (2014). A diffusion modeling approach to understanding contextual cueing effects in children with ADHD. Journal of Child Psychology and Psychiatry, 55, 1336–1344. doi: 10.1111/jcpp.12250.PubMedPubMedCentralCrossRefGoogle Scholar
  126. Weigard, A., & Huang-Pollock, C. L. (2016). The role of speed in ADHD-related working memory deficits: a time-based resource-sharing and diffusion model account. Clinical Psychological Science, (in press).Google Scholar
  127. Weigard, A., Huang-Pollock, C., & Brown, S. (2016). Evaluating the consequences of impaired monitoring of learned behavior in attention-deficit/hyperactivity disorder using a Bayesian hierarchical model of choice response time. Neuropsychology, 30, 502–515. doi: 10.1037/neu0000257.PubMedPubMedCentralCrossRefGoogle Scholar
  128. White, J. L., Moffitt, T. E., Caspi, A., Bartusch, D. J., Needles, D. J., & Stouthamerloeber, M. (1994). Measuring impulsivity and examining its relationship to delinquency. Journal of Abnormal Psychology, 103, 192–205. doi: 10.1037/0021-843x.103.2.192.PubMedCrossRefGoogle Scholar
  129. Wiecki, T. V., Poland, J., & Frank, M. J. (2015). Model-based cognitive neuroscience approaches to computational psychiatry: clustering and classification. Clinical Psychological Science, 3, 378–399. doi: 10.1177/2167702614565359.CrossRefGoogle Scholar
  130. Willcutt, E. G., Pennington, B. F., Boada, R., Ogline, J. S., Tunick, R. A., Chhabildas, N. A., & Olson, R. K. (2001). A comparison of the cognitive deficits in reading disability and attention-deficit/hyperactivity disorder. Journal of Abnormal Psychology, 110, 157–172. doi: 10.1037//0021-843x.1001.1.157.PubMedCrossRefGoogle Scholar
  131. Willcutt, E. G., Doyle, A., Nigg, J. T., Faraone, S. V., & Pennington, B. F. (2005). Validity of the executive function theory of attention-deficit/hyperactivity disorder: a meta-analytic review. Biological Psychiatry, 57, 1336–1346. doi: 10.1016/j.biopsych.2005.02.006.PubMedCrossRefGoogle Scholar
  132. Wright, A. G. C., Krueger, R. F., Hobbs, M. J., Markon, K. E., Eaton, N. R., & Slade, T. (2013). The structure of psychopathology: toward an expanded quantitative empirical model. Journal of Abnormal Psychology, 122, 281–294. doi: 10.1037/a0030133.PubMedCrossRefGoogle Scholar
  133. Young, S. E., Friedman, N. P., Miyake, A., Willcutt, E. G., Corley, R. P., Haberstick, B. C., & Hewitt, J. K. (2009). Behavioral disinhibition: liability for externalizing spectrum disorders and its genetic and environmental relation to response inhibition across adolescence. Journal of Abnormal Psychology, 118, 117–130. doi: 10.1037/a0014657.PubMedPubMedCentralCrossRefGoogle Scholar
  134. Youngstrom, E., Loeber, R., & Stouthamer-Loeber, M. (2000). Patterns and correlates of agreement between parent, teacher, and male adolescent ratings of externalizing and internalizing problems. Journal of Consulting and Clinical Psychology, 68, 1038–1050. doi: 10.1037/0022-006x.68.6.1038.PubMedCrossRefGoogle Scholar
  135. Zahn-Waxler, C., Shirtcliff, E. A., & Marceau, K. (2008). Disorders of childhood and adolescence: Gender and psychopathology. Annual Review of Clinical Psychology, 4, 275–303.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of Psychology, 130 Moore BuildingThe Pennsylvania State UniversityUniversity ParkUSA

Personalised recommendations