Skip to main content
SpringerLink
Log in

The Generalizability of Executive Function Ratings Across Dimensions and Parent and Self-reports in a School-Based Adolescent Sample

  • Published:
Contemporary School Psychology Aims and scope Submit manuscript

Abstract

Behavior rating scales measuring executive function have grown popular among school psychologists and other professions. To examine the generalizability of executive function rating scales, 42 parent-adolescent dyads, recruited from a school-based sample, completed the Behavior Rating Inventory of Executive Function, Second Edition and the Comprehensive Executive Function Inventory. Caregivers were mostly mothers (79%), and adolescents were mostly girls (81%). Both groups included approximately two-thirds participants who were White and not-Hispanic, and more than a quarter of participants were Black and not-Hispanic. A total of 1260 scores representing general and specific executive functions were submitted to generalizability theory analyses to evaluate rater effects, instrument effects, dimension effects, and all interactions on the variation across the total scores and subscale scores from the rating scales. The generalizability of scores was markedly low (.09–.19) and much lower than expected given prior research and analysis evaluating these effects in isolation. Partial models examining variance components for each rater in isolation indicated higher yet inadequate generalizability (.59–.69). These findings indicate that practitioners should be cautious when generalizing ratings of executive function across different raters, similar instruments, and specific executive functions—especially in school-based populations like those from which we sampled.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Results were the same using the CEFI Organizing subscale and can be obtained by contacting the authors.

References 

  • Achenbach, T. M. (1991). Manual for the Child Behavior Checklist/4–18 and 1991 Profile. University of Vermont Department of Psychiatry.

    Google Scholar 

  • Achenbach, T. M. (2011). Commentary: Definitely more than measurement error: But how should we understand and deal with informant discrepancies? Journal of Clinical Child & Adolescent Psychology, 40, 80–86. https://doi.org/10.1080/15374416.2011.533416

    Article  Google Scholar 

  • Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232. https://doi.org/10.1037/0033-2909.101.2.213

    Article  PubMed  Google Scholar 

  • Ahmed, S. F., Tang, S., Waters, N. E., & Davis-Kean, P. (2019). Executive function and academic achievement: Longitudinal relations from early childhood to adolescence. Journal of Educational Psychology, 111, 446–458. https://doi.org/10.1037/edu0000296

    Article  Google Scholar 

  • Alderson, R. M., Rapport, M. D., & Kofler, M. J. (2007). Attention-deficit/hyperactivity disorder and behavioral inhibition: A meta-analytic review of the stop-signal paradigm. Journal of Abnormal Child Psychology, 35(5), 745–758. https://doi.org/10.1007/s10802-007-9131-6

    Article  PubMed  Google Scholar 

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

  • Atilgan, H. (2013). Sample size for estimation of G and Phi coefficients in generalizability theory. Eurasian Journal of Educational Research, 51, 215–227.

    Google Scholar 

  • Baddeley, A. (2007). Working memory, thought, and action. Oxford University Press.

    Book  Google Scholar 

  • Benson, N. F., Floyd, R. G., Kranzler, J. H., Eckert, T. L., Fefer, S. A., & Morgan, G. B. (2019). Test use and assessment practices of school psychologists in the United States: Findings from the 2017 National Survey. Journal of School Psychology, 72, 29–48. https://doi.org/10.1016/j.jsp.2018.12.004

    Article  PubMed  Google Scholar 

  • Bergeron, R., Floyd, R. G., McCormack, A. C., & Farmer, W. L. (2008). The generalizability of externalizing behavior composites and subscale scores across time, rater, and instrument. School Psychology Review, 37, 91–108. https://doi.org/10.1080/02796015.2008.12087911

    Article  Google Scholar 

  • Bernes, G. A., Villodas, M., Coles, C. D., Kable, J. A., May, P. A., Kalberg, W. O., … & Mattson, S.N. (2021). Validity and reliability of executive function measures in children with heavy prenatal alcohol exposure: Correspondence between multiple raters and laboratory measures. Alcoholism Clinical and Experimental Research, 45, 596–607. https://doi.org/10.1111/acer.14547

  • Best, J. R., & Miller, P. H. (2010). A developmental perspective on executive function. Child Development, 81, 1641–1660. https://doi.org/10.1111/j.1467-8624.2010.01499.x

    Article  PubMed  PubMed Central  Google Scholar 

  • Blanco-Villaseñor, A., & Escolano-Pérez, E. (2017). Observational data analysis using generalizability theory and general and mixed linear models: An empirical study of infant learning and development. Anales De Psicología, 33, 450–460.

    Article  Google Scholar 

  • Brennan, R. L. (2011). Generalizability theory and classical test theory. Applied Measurement in Education, 24, 1–21. https://doi.org/10.1080/08957347.2011.532417

    Article  Google Scholar 

  • Briesch, A. M., Swaminathan, H., Welsh, M., & Chafouleas, S. M. (2014). Generalizability theory: A practical guide to study design, implementation, and interpretation. Journal of School Psychology, 52, 13–35. https://doi.org/10.1016/j.jsp.2013.11.008

    Article  PubMed  Google Scholar 

  • Conners, C. K. (2008). Conners rating scales–Third edition. Multi-Health Systems.

    Google Scholar 

  • Cortese, S., Kelly, C., Chabernaud, C., Proal, E., Di Martino, A., Milham, M. P., & Castellanos, F. X. (2012). Toward systems neuroscience of ADHD: A meta-analysis of 55 fMRI studies. American Journal of Psychiatry, 169, 1038–1055. https://doi.org/10.1176/appi.ajp.2012.11101521

    Article  PubMed  Google Scholar 

  • Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137–163. https://doi.org/10.1111/j.2044-8317.1963.tb00206.x

    Article  Google Scholar 

  • De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131, 483–509. https://doi.org/10.1037/0033-2909.131.4.483

    Article  Google Scholar 

  • De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Drabick, D., Burgers, D. E., & Rabinowitz, J. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141, 858–900. https://doi.org/10.1037/a0038498

    Article  PubMed Central  Google Scholar 

  • Demaray, M. K., & Jenkins, L. N. (2011). Relations among academic enablers and academic achievement in children with and without high levels of parent-rated symptoms of inattention, impulsivity, and hyperactivity. Psychology in the Schools, 48, 573–586. https://doi.org/10.1002/pits.20578

    Article  Google Scholar 

  • Duhig, A., Renk, K., Epstein, M., & Phares, V. (2000). Interparental agreement on internalizing, externalizing, and total behavior problems: A meta-analysis. Clinical Psychology: Science and Practice, 7, 435–453. https://doi.org/10.1093/clipsy.7.4.435

    Article  Google Scholar 

  • DuPaul, G. J., Anastopoulos, A. D., Power, T. J., Murphy, K., & Barkley, R. A. (1998). ADHD Rating Scale-IV: Checklists, norms, and clinical interpretation. Guilford Press. https://doi.org/10.1037/t00680-000

    Article  Google Scholar 

  • Egan, K. N., Cohen, L. A., & Limbers, C. (2019). Parent–child agreement on the Behavior Rating Inventory of Executive Functioning (BRIEF) in a community sample of adolescents. Applied Neuropsychology: Child, 8, 61–68. https://doi.org/10.1080/21622965.2018.1438896

    Article  Google Scholar 

  • Eslinger, E. J. (1996). Conceptualizing, describing, and measuring components of executive function. In G. R. Lyon & N. A. Krasnegor (Eds.), Attention, memory, and executive function (pp. 367–395). Brooks.

    Google Scholar 

  • Gioia, G. A., Isquith, P. K., Guy, S. C., & Kenworthy, L. (2000). Behavior Rating Inventory of Executive Function. Psychological Assessment Resources. https://doi.org/10.1076/chin.6.3.235.3152

    Article  Google Scholar 

  • Gioia, G. A., Isquith, P. K., Guy, S. C., & Kenworthy, L. (2015a). The Behavior Rating Inventory of Executive Function-Second Edition. Psychological Assessment Resources.

  • Gioia, G. A., Isquith, P. K., Guy, S. C., & Kenworthy, L. (2015b). The Behavior Rating Inventory of Executive Function-Second Edition professional manual. Psychological Assessment Resources.

  • Greenbaum, P. E., Dedrick, R. F., Prange, M. E., & Friedman, R. M. (1994). Parent, teacher, and child ratings of problem behaviors of youngsters with serious emotional disturbances. Psychological Assessment, 6, 141–148. https://doi.org/10.1037/1040-3590.6.2.141

    Article  Google Scholar 

  • Groves, N. B., Wells, E. L., Soto, E. F. et al. (2022). Executive Functioning and Emotion Regulation in Children with and without ADHD. Research on Child Adolescent Psychopathology. https://doi.org/10.1007/s10802-021-00883-0

  • IBM Corp. (2016). IBM SPSS Statistics for Windows, Version 24.0. IBM Corp.

    Google Scholar 

  • Isquith, P. K., Roth, R. M., & Gioia, G. (2013). Contribution of rating scales to the assessment of executive functions. Applied Neuropsychology: Child, 2, 125–132. https://doi.org/10.1080/21622965.2013.748389

    Article  PubMed  Google Scholar 

  • Karr, J. E., Areshenkoff, C. N., Rast, P., Hofer, S. M., Iverson, G. L., & Garcia-Barrera, M. A. (2018). The unity and diversity of executive functions: A systematic review and re-analysis of latent variable studies. Psychological Bulletin, 144, 1147–1185. https://doi.org/10.1037/bul0000160

    Article  PubMed  PubMed Central  Google Scholar 

  • Keppel, G. & Wickens, T. D. (2004). Design and analysis: A researcher's handbook, Prentice Hall.

  • Kofler, M. J., Rapport, M. D., Bolden, J., Sarver, D. E., Raiker, J. S., & Alderson, R. M. (2011). Working memory deficits and social problems in children with ADHD. Journal of Abnormal Child Psychology, 39, 805–817. https://doi.org/10.1007/s10802-011-9492-8

    Article  PubMed  Google Scholar 

  • Kofler, M. J., Sarver, D. E., Harmon, S. L., Moltisanti, A., Aduen, P. A., Soto, E. F., & Ferretti, N. (2018a). Working memory and organizational skills problems in ADHD. Journal of Child Psychology and Psychiatry and Allied Disciplines, 59, 57–67. https://doi.org/10.1111/jcpp.12773

  • Kofler, M. J., Harmon, S. L., Aduen, P. A., Day, T. N., Austin, K., Spiegel, J. A., Irwin, L., & Sarver, D. E. (2018b). Neurocognitive and behavioral predictors of social problems in ADHD: A bayesian framework. Neuropsychology, 32, 344–355. https://doi.org/10.1037/neu0000416

  • Kofler, M. J., Sarver, D. E., & Wells, E. L. (2020). Working memory and increased activity level (hyperactivity) in ADHD: Experimental evidence for a functional relation. Journal of Attention Disorders, 24, 1330–1344. https://doi.org/10.1177/1087054715608439

    Article  PubMed  Google Scholar 

  • Krieger, V., Amador-Campos, J. A., & Peró-Cebollero, M. (2019). Interrater agreement on behavioral executive function measures in adolescents with attention deficit hyperactivity disorder. International Journal of Clinical and Health Psychology, 19, 141–149. https://doi.org/10.1016/j.ijchp.2019.02.007

    Article  PubMed  PubMed Central  Google Scholar 

  • McAuley, T., Chen, S., Goos, L., Schachar, R., & Crosbie, J. (2010). Is the behavior rating inventory of executive function more strongly associated with measures of impairment or executive function? Journal of the International Neuropsychological Society, 16, 495–505.

    Article  PubMed  Google Scholar 

  • Merydith, S. P., Prout, H. T., & Blaha, J. (2003). Social desirability and behavior rating scales: An exploratory study with the Child Behavior Checklist/4-18. Psychology in the Schools, 40, 225–235. https://doi.org/10.1002/pits.10077

    Article  Google Scholar 

  • Miyake, A., & Friedman, N. P. (2012). The nature and organization of individual differences in executive functions: Four general conclusions. Current Directions in Psychological Science, 21, 8–14. https://doi.org/10.1177/0963721411429458

    Article  PubMed  PubMed Central  Google Scholar 

  • Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., Howerter, A., & Wager, T. D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41, 49–100.

    Article  PubMed  Google Scholar 

  • Naglieri, J. A., & Goldstein, S. (2013a). Comprehensive Executive Function Inventory. Multi-Health Systems.

  • Naglieri, J. A., & Goldstein, S. (2013b). Comprehensive Executive Function Inventory: Technical manual. Multi-Health Systems.National Association of School Psychologists. (2020). Model for comprehensive and integrated school psychological services. National Association of School Psychologists.

  • National Association of School Psychologists. (2020). The Professional Standards of the National Association of School Psychologists. National Association of School Psychologists.

  • Norfolk, P. A., & Floyd, R. G. (2016). Detecting parental deception using a behavior rating scale during assessment of attention-deficit/ hyperactivity disorder: An experimental study. Psychology in the Schools, 53, 158–172. https://doi.org/10.1002/pits.21894

    Article  Google Scholar 

  • Ohan, J. L., Visser, T. A. W., Strain, M. C., & Allen, L. (2010). Teachers’ and education students’ perceptions of and reactions to children with and without the diagnostic label “ADHD.” Journal of School Psychology, 49, 81–105. https://doi.org/10.1016/j.jsp.2010.10.001

    Article  PubMed  Google Scholar 

  • Pa, J., Possin, K. L., Wilson, S. M., Quitania, L. C., Kramer, J. H., Boxer, A. L., et al. (2010). Gray matter correlates of set-shifting among neurodegenerative disease, mild cognitive impairment, and healthy older adults. Journal of the International Neuropsychological Society, 16, 640–650. https://doi.org/10.1017/S1355617710000408

    Article  PubMed  PubMed Central  Google Scholar 

  • Reynolds, C., & Kamphaus, R. (2004). Behavioral Assessment System for Children (2nd ed.). Pearson Assessments.

    Google Scholar 

  • Salvia, J., Ysselydke, J. E., & Bolt, S. (2010). Assessment in special and inclusive education (11th ed.). Houghton Mifflin.

    Google Scholar 

  • Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Sage.

    Google Scholar 

  • Shavelson, R. J., Webb, N. M., & Rowley, G. L. (1989). Generalizability theory. American Psychologist, 44, 922–932. https://doi.org/10.1037/0003-066X.44.6.922

    Article  Google Scholar 

  • Sibley, M. H., Pelham, W. E., Jr., Molina, B. S. G., Gnagy, E. M., Waschbusch, D. A., Garefino, A. C., . . . Karch, K. M. (2012). Diagnosing ADHD in adolescence. Journal of Consulting and Clinical Psychology, 80, 139-150. https://doi.org/10.1037/a0026577

  • Singh, L. J., Gaye, F., Cole, A. M., Chan, E. S. M., & Kofler, Michael J. (2022). Central executive training for ADHD: Effects on academic achievement, productivity, and success in the classroom. Neuropsychology.

  • Snyder, H. R., Miyake, A., & Hankin, B. L. (2015). Advancing understanding of executive function impairments and psychopathology: Bridging the gap between clinical and cognitive approaches. Frontiers in Psychology, 6, 328. https://doi.org/10.3389/fpsyg.2015.00328

    Article  PubMed  PubMed Central  Google Scholar 

  • Soto, E. F., Kofler, M. J., Singh, L. J., Wells, E. L., Irwin, L. N., Groves, N. B., & Miller, C. E. (2020). Executive functioning rating scales: Ecologically valid or construct invalid? Neuropsychology, 34(6), 605–619. https://doi.org/10.1037/neu0000681

    Article  PubMed  PubMed Central  Google Scholar 

  • Soto, E. F., Irwin, L. N., Chan, E. S. M., Spiegel, J.  A., & Kofler, M.  J. (2021). Executive functions and writing skills in children with and without ADHD. Neuropsychology, 35(8), 792–808.

  • Spiegel, J. A., Lonigan, C. J., & Phillips, B. M. (2017). Factor structure and utility of the Behavior Rating Inventory of Executive Function-Preschool Version. Psychological Assessment, 29, 172–185. https://doi.org/10.1037/pas0000324

    Article  PubMed  Google Scholar 

  • Spiegel, J., Goodrich, J., Morris, B., Osborne, C., & Lonigan, C. (2021). Relations between executive functions and academics in elementary school children: A meta-analysis. Psychological Bulletin. https://doi.org/10.1037/bul0000322

    Book  Google Scholar 

  • Spooner, D. M., & Pachana, N. A. (2006). Ecological validity in neuropsychological assessment: A case for greater consideration in research with neurologically intact populations. Archives of Clinical Neuropsychology, 21, 327–337. https://doi.org/10.1016/j.acn.2006.04.004

    Article  PubMed  Google Scholar 

  • St Clair-Thompson, H. L., & Gathercole, S. E. (2006). Executive functions and achievements in school: Shifting, updating, inhibition, and working memory. Quarterly Journal of Experimental Psychology, 59(4), 745–759. https://doi.org/10.1080/17470210500162854

    Article  Google Scholar 

  • Steward, K. A., Tan, A., Delgaty, L., Gonzales, M. M., & Bunner, M. (2017). Self-awareness in executive functioning deficits in adolescents with ADHD. Journal of Attention Disorders, 27, 316–322. https://doi.org/10.1177/1087054714530782

    Article  Google Scholar 

  • Sulik, M. J., Huerta, S., Zerr, A. A., Eisenberg, N., Spinrad, T. L., & Valiente, C.,…& Taylor, H. B. (2010). The factor structure of effortful control and measurement invariance across ethnicity and sex in a high-risk sample. Journal of Psychopathology and Behavior Assessment, 32, 8–22. https://doi.org/10.1007/s10862-009-9164-y

    Article  Google Scholar 

  • Tabachnick, B. G., & Fidell, L. S. (Eds.). (2018). Using multivariate statistics (7th ed.). Pearson.

    Google Scholar 

  • Toplak, M. E., West, R. F., & Stanovich, S. E. (2013). Practitioner review: Do performance-based measures and ratings of executive function assess the same construct? Journal of Child Psychology and Psychiatry, 54, 131–143. https://doi.org/10.1111/jcpp.12001

    Article  PubMed  Google Scholar 

  • van der Ven, S. H., Kroesbergen, E. H., Boom, J., & Leseman, P. P. (2013). The structure of executive functions in children: A closer examination of inhibition, shifting, and updating. British Journal of Developmental Psychology, 31, 70–87. https://doi.org/10.1111/j.2044-835X.2012.02079.x

    Article  PubMed  Google Scholar 

  • Wager, T. D., & Smith, E. E. (2003). Neuroimaging studies of working memory. Cognitive, Affective, & Behavioral Neuroscience, 3(4), 255–274. https://doi.org/10.3758/CABN.3.4.255

    Article  Google Scholar 

  • Walker, J. M., & D’Amato, R. C. (2006). Test review. [Review of the test Behavior Rating Inventory of Executive Function- Self-Report version]. Journal of Psychoeducational Assessment, 24, 394–398. https://doi.org/10.1177/0734282906288390

    Article  Google Scholar 

  • Whitcomb, S. A. (2017). Behavioral, social, and emotional assessment of children and adolescents (5th ed.). Routledge. https://doi.org/10.4324/978131574

    Book  Google Scholar 

  • Willard, V. W., Conklin, H. M., Huang, L., Zhang, H., & Kahalley, L. S. (2016). Concordance of parent-, teacher- and self- report ratings on the Conners 3 in adolescent survivors of cancer. Psychological Assessment, 26, 1110–1118. https://doi.org/10.1037/pas0000265

    Article  Google Scholar 

  • Wilson, K. R., Donders, J., & Nguyen, L. (2011). Self and parent ratings of executive functioning after adolescent traumatic brain injury. Rehabilitation Psychology, 56, 100–106. https://doi.org/10.1037/a0023446

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This manuscript is based on a dissertation submitted by the first author under supervision of the second author in partial fulfillment of the requirements for a doctoral degree in school psychology. We thank dissertation committee members Dr. Elizabeth Meisinger, Dr. Lilya Jiang, and Dr. Kristoffer Berlin for their helpful comments. We appreciate the middle and high schools and their administrators, teachers, and parents for supporting this research.

Author information

Authors and Affiliations

  1. Department of Psychology, The University of Memphis, Memphis, USA

    Leah J. Singh & Randy G. Floyd

  2. Department of Psychology, Florida State University, 1107 W. Call Street, Tallahassee, FL, 32306-4301, USA

    Leah J. Singh

Authors
  1. Leah J. Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Randy G. Floyd
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Both authors contributed collaboratively and interactively in the production of this manuscript, and both authors read and approved the final manuscript.

Corresponding author

Correspondence to Leah J. Singh.

Ethics declarations

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee.

Consent to Participate and Publish

Informed consent was obtained from all individual participants included in the study.

Competing Interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Singh, L.J., Floyd, R.G. The Generalizability of Executive Function Ratings Across Dimensions and Parent and Self-reports in a School-Based Adolescent Sample. Contemp School Psychol 27, 606–622 (2023). https://doi.org/10.1007/s40688-022-00436-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40688-022-00436-2

Keywords

Search

Navigation