Skip to main content
SpringerLink
Log in

Computerized adaptive testing to screen pre-school children for emotional and behavioral problems

  • RESEARCH
  • Published:
European Journal of Pediatrics Aims and scope Submit manuscript

Abstract

Questionnaires to detect emotional and behavioral (EB) problems in preventive child healthcare (PCH) should be short; this potentially affects their validity and reliability. Computerized adaptive testing (CAT) could overcome this weakness. The aim of this study was to (1) develop a CAT to measure EB problems among pre-school children and (2) assess the efficiency and validity of this CAT. We used a Dutch national dataset obtained from parents of pre-school children undergoing a well-child care assessment by PCH (n = 2192, response 70%). Data regarded 197 items on EB problems, based on four questionnaires, the Strengths and Difficulties Questionnaire (SDQ), the Child Behavior Checklist (CBCL), the Ages and Stages Questionnaire: Social Emotional (ASQ:SE), and the Brief Infant–Toddler Social and Emotional Assessment (BITSEA). Using 80% of the sample, we calculated item parameters necessary for a CAT and defined a cutoff for EB problems. With the remaining part of the sample, we used simulation techniques to determine the validity and efficiency of this CAT, using as criterion a total clinical score on the CBCL. Item criteria were met by 193 items. This CAT needed, on average, 16 items to identify children with EB problems. Sensitivity and specificity compared to a clinical score on the CBCL were 0.89 and 0.91, respectively, for total problems; 0.80 and 0.93 for emotional problems; and 0.94 and 0.91 for behavioral problems.

    Conclusion: A CAT is very promising for the identification of EB problems in pre-school children, as it seems to yield an efficient, yet high-quality identification. This conclusion should be confirmed by real-life administration of this CAT.

What is Known:

• Studies indicate the validity of using computerized adaptive test (CAT) applications to identify emotional and behavioral problems in school-aged children.

• Evidence is as yet limited on whether CAT applications can also be used with pre-school children.

What is New:

• The results of this study show that a computerized adaptive test is very promising for the identification of emotional and behavior problems in pre-school children, as it appears to yield an efficient and high-quality identification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

ASQ:SE:

Ages and Stages Questionnaire: Social Emotional

AUC:

Area under the curve

BITSEA:

Brief Infant–Toddler Social and Emotional Assessment

CAT:

Computerized adaptive test

CBCL:

Child Behavior Checklist

CI:

Confidence interval

EB:

Emotional and behavioral

IRT:

Item response theory

PCH:

Preventive child healthcare

PF:

Parent form

SDQ:

Strengths and Difficulties Questionnaire

TPS:

Total problem score

References

  1. Brugman E, Reijneveld SA, Verhulst FC, Verloove-Vanhorick SP (2001) Identification and management of psychosocial problems by preventive child health care. Arch Pediatr Adolesc Med 155(4):462–469

    Article  CAS  PubMed  Google Scholar 

  2. Patel V, Flisher AJ, Hetrick S, McGorry P (2007) Mental health of young people: a global public-health challenge. Lancet 369(9569):1302–1313

    Article  PubMed  Google Scholar 

  3. Kieling C, Baker-Henningham H, Belfer M et al (2011) Child and adolescent mental health worldwide: evidence for action. Lancet 378(9801):1515–1525

    Article  PubMed  Google Scholar 

  4. Tremblay L, Limbos M (2009) Body image disturbance and psychopathology in children: research evidence and implications for prevention and treatment. Current Psychiatry Reviews 5(1):62–72

    Article  Google Scholar 

  5. Ormel J, Oldehinkel AJ, Sijtsema J et al (2012) The TRacking adolescents’ individual lives survey (TRAILS): design, current status, and selected findings. J Am Acad Child Adolesc Psychiatry 51(10):1020–1036

    Article  PubMed  Google Scholar 

  6. Jaspers M, de Winter AF, Huisman M et al (2012) Trajectories of psychosocial problems in adolescents predicted by findings from early well-child assessments. J Adolesc Health 51(5):475–483

    Article  PubMed  Google Scholar 

  7. Verboom CE, Sijtsema JJ, Verhulst FC, Penninx, Brenda WJH, Ormel J (2014) Longitudinal associations between depressive problems, academic performance, and social functioning in adolescent boys and girls. Dev Psychol 50(1):247–257.  https://doi.org/10.1037/a0032547

  8. Lavigne JV, Meyers KM, Feldman M (2016) Systematic review: classification accuracy of behavioral screening measures for use in integrated primary care settings. J Pediatr Psychol 41(10):1091–1109

    Article  PubMed  Google Scholar 

  9. Geeraert L, van den Noortgate W, Grietens H, Onghena P (2004) The effects of early prevention programs for families with young children at risk for physical child abuse and neglect: a meta-analysis. Child Maltreat 9(3):277–291

    Article  PubMed  Google Scholar 

  10. Cuijpers P, Van Straten A, Smit F (2005) Preventing the incidence of new cases of mental disorders: a meta-analytic review. J Nerv Ment Dis 193(2):119–125

    Article  PubMed  Google Scholar 

  11. Goodman R (1997) The strengths and difficulties questionnaire: a research note. J Child Psychol Psychiatry 38:581–586

    Article  CAS  PubMed  Google Scholar 

  12. Theunissen MH, Vogels AG, Reijneveld SA (2012) Early detection of psychosocial problems in children aged 5 to 6 years by preventive child healthcare: has it improved? J Pediatr 160(3):500–504

    Article  PubMed  Google Scholar 

  13. Glascoe FP (2005) Screening for developmental and behavioral problems. Ment Retard Dev Disabil Res Rev 11(3):173–179

    Article  PubMed  Google Scholar 

  14. Sheldrick RC, Merchant S, Perrin EC (2011) Identification of developmental-behavioral problems in primary care: a systematic review. Pediatrics 128(2):356–363

    Article  PubMed  Google Scholar 

  15. Vogels AG, Crone MR, Hoekstra F, Reijneveld SA (2009) Comparing three short questionnaires to detect psychosocial dysfunction among primary school children: a randomized method. BMC Public Health 9(1):489

    Article  PubMed  PubMed Central  Google Scholar 

  16. Theunissen MH, Vogels AG, de Wolff MS, Reijneveld SA (2013) Characteristics of the strengths and difficulties questionnaire in preschool children. Pediatrics 131(2):446

    Article  Google Scholar 

  17. Theunissen MHC, de Wolff MS, Eekhout I, Mieloo CL, Stone LL, Reijneveld SA (2022) The strengths and difficulties questionnaire parent form: Dutch norms and validity. BMC Pediatr 22(1):202–206. https://doi.org/10.1186/s12887-022-03274-6

    Article  PubMed  PubMed Central  Google Scholar 

  18. Vugteveen J, de Bildt A, Theunissen M, Reijneveld M, Timmerman M (2021) Validity aspects of the strengths and difficulties questionnaire (SDQ) adolescent self-report and parent-report versions among dutch adolescents. Assessment 28(2):601–616

    Article  PubMed  Google Scholar 

  19. Stone LL, Otten R, Engels RC, Vermulst AA, Janssens JM (2010) Psychometric properties of the parent and teacher versions of the strengths and difficulties questionnaire for 4- to 12-year-olds: a review. Clin Child Fam Psychol Rev 13(3):254–274

    Article  PubMed  PubMed Central  Google Scholar 

  20. Vogels AG, Jacobusse GW, Reijneveld SA (2011) An accurate and efficient identification of children with psychosocial problems by means of computerized adaptive testing. BMC Med Res Methodol 11:111–111

    Article  PubMed  PubMed Central  Google Scholar 

  21. Theunissen MHC, de Wolff MS, Deurloo JA, Vogels AGC, Reijneveld SA (2020) Computerized adaptive testing to screen children for emotional and behavioral problems by preventive child healthcare. BMC Pediatr 20(1):119–121

    Article  PubMed  PubMed Central  Google Scholar 

  22. Barthel D, Otto C, Nolte S et al (2017) The validation of a computer-adaptive test (CAT) for assessing health-related quality of life in children and adolescents in a clinical sample: study design, methods and first results of the kids-CAT study. Qual Life Res 26(5):1105–1117

    Article  CAS  PubMed  Google Scholar 

  23. Devine J, Otto C, Rose M et al (2015) A new computerized adaptive test advancing the measurement of health-related quality of life (HRQoL) in children: the kids-CAT. Qual Life Res 24(4):871–884

    Article  CAS  PubMed  Google Scholar 

  24. Gardner W, Kelleher KJ, Pajer KA (2002) Multidimensional adaptive testing for mental health problems in primary care. Med Care 40(9):812–823

    Article  PubMed  Google Scholar 

  25. Hill CD, Edwards MC, Thissen D et al (2007) Practical issues in the application of item response theory: a demonstration using items from the pediatric quality of life inventory (PedsQL) 4.0 generic core scales. Med Care 45(5 Suppl 1):39

    Article  Google Scholar 

  26. Sherlock P, Blackwell CK, Kallen MA et al (2022) Measuring PROMIS® emotional distress in early childhood. J Pediatr Psychol 47(5):547–558. https://doi.org/10.1093/jpepsy/jsac029

    Article  PubMed  PubMed Central  Google Scholar 

  27. Theunissen MH, Vogels AG, de Wolff MS, Crone MR, Reijneveld SA (2015) Comparing three short questionnaires to detect psychosocial problems among 3 to 4-year olds. BMC Pediatr 15:84-y

    Article  Google Scholar 

  28. de Wolff MS, Theunissen MH, Vogels AG, Reijneveld SA (2013) Three questionnaires to detect psychosocial problems in toddlers: a comparison of the BITSEA, ASQ:SE, and KIPPPI. Acad Pediatr 13(6):587–592

    Article  PubMed  Google Scholar 

  29. Achenbach T, Rescorla L (2000) Manual for the ASEBA preschool forms and profiles. University of Vermont, Burlington

    Google Scholar 

  30. Ferdinand RF, Visser JH, Hoogerheide KN et al (2004) Improving estimation of the prognosis of childhood psychopathology; combination of DSM-III-R/DISC diagnoses and CBCL scores. J Child Psychol Psychiatry 45(3):599–608

    Article  PubMed  Google Scholar 

  31. Koot HM, Van Den Oord EJ, Verhulst FC, Boomsma DI (1997) Behavioral and emotional problems in young preschoolers: cross-cultural testing of the validity of the child behavior checklist/2-3. J Abnorm Child Psychol 25(3):183–196

    Article  CAS  PubMed  Google Scholar 

  32. Goodman R, Ford T, Simmons H, Gatward R, Meltzer H (2000) Using the strengths and difficulties questionnaire (SDQ) to screen for child psychiatric disorders in a community sample. Br J Psychiatry 177:534–539

    Article  CAS  PubMed  Google Scholar 

  33. Bricker D, Squires J (1999) Ages & stages questionnaires: A parent-completed, child-monitoring system. Vol 2. Baltimore: Brooks

  34. Squires J, Bricker D, Twombly E (2003) Ages & stages questionnaires: Social-emotional. Brookes Publishing, A parent-completed child-monitoring system for social-emotional behaviors. Baltimore

    Google Scholar 

  35. Guillemin F (1995) Cross-cultural adaptation and validation of health status measures. Scand J Rheumatol 24(2):61–63

    Article  CAS  PubMed  Google Scholar 

  36. Briggs-Gowan MJ, Carter AS, Irwin JR, Wachtel K, Cicchetti DV (2004) The brief infant-toddler social and emotional assessment: screening for social-emotional problems and delays in competence. J Pediatr Psychol 29(2):143–155

    Article  PubMed  Google Scholar 

  37. Visser JC, Smeekens S, Riksen-Walraven M et al (2001) Nederlandse vertaling van de BITSEA [dutch translation of the brief infant toddler social, emotional Assessment–Revised]. Radboud Universiteit Nijmegen, Nijmegen, The Netherlands

    Google Scholar 

  38. Masters GN (1982) A rasch model for partial credit scoring. Psychometrika 47:149–174. https://doi.org/10.1007/BF02296272

    Article  Google Scholar 

  39. Baker FB (1992) Item response theory: Parameter estimation techniques. Marcel Dekker, New York

    Google Scholar 

  40. Bock RD, Mislevy RJ (1982) Adaptive EAP estimation of ability in a microcomputer environment. Appl Psychol Meas 6:431–444. https://doi.org/10.1177/014662168200600405

    Article  Google Scholar 

  41. Crone MR, Vogels AG, Hoekstra F, Treffers PD, Reijneveld SA (2008) A comparison of four scoring methods based on the parent-rated strengths and difficulties questionnaire as used in the dutch preventive child health care system. BMC Public Health 8:106

    Article  PubMed  PubMed Central  Google Scholar 

  42. Zero To Three (2005) Diagnostic classification of mental health and development disorders in infancy and early childhood, (rev.ed.). Washington, DC: Zero To Three Press

  43. Oyetunji A, Chandra P (2020) Postpartum stress and infant outcome: a review of current literature. Psychiatry Res 284:112769. https://www.sciencedirect.com/science/article/pii/S0165178119318803 https://doi.org/10.1016/j.psychres.2020.112769

  44. Weber AM, Rubio-Codina M, Walker SP et al (2019) The D-score: a metric for interpreting the early development of infants and toddlers across global settings. BMJ Glob Health 4(6):e001724–001724. eCollection. https://doi.org/10.1136/bmjgh-2019-001724

Download references

Funding

This research received financial support from the Netherlands Organization for Health Research and Development (ZonMw). No honorarium, grant, or other form of payment was given to anyone to produce the manuscript.

Author information

Authors and Affiliations

  1. Child Health, TNO (Netherlands Organisation of Applied Scientific Research), Leiden, The Netherlands

    Meinou H. C. Theunissen, Iris Eekhout & Sijmen A. Reijneveld

  2. Department of Health Sciences, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands

    Sijmen A. Reijneveld

Authors
  1. Meinou H. C. Theunissen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Iris Eekhout
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sijmen A. Reijneveld
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.T. and S.R. conceptualized and designed the study, drafted the initial manuscript, and critically reviewed and revised the manuscript. I.E. conceptualized and designed the study, carried out the analyses and critically reviewed and revised the manuscript. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

Corresponding author

Correspondence to Meinou H. C. Theunissen.

Ethics declarations

Ethics approval

Ethical approval for this study was granted by the Medical Research Ethics Committee (METC) of Leiden University Medical Center. The study was conducted according to the Declaration of Helsinki code of ethics.

Consent to participate

Informed consent was not deemed necessary by the METC concerned.

Consent for publication

N/A.

Competing interests

The authors declare no competing interests.

Additional information

Communicated by Peter de Winter

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 411 KB)

Appendix. A visualization of the algorithm and the stop rules involved

Appendix. A visualization of the algorithm and the stop rules involved

figure a

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Theunissen, M.H.C., Eekhout, I. & Reijneveld, S.A. Computerized adaptive testing to screen pre-school children for emotional and behavioral problems. Eur J Pediatr 183, 1777–1787 (2024). https://doi.org/10.1007/s00431-023-05414-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00431-023-05414-1

Keywords

Search

Navigation