Abstract
The emotional competence screening and the social competence screening for parents and teachers were developed in Romania as brief, multiinformant, strength-focused assessment tools to identify children at risk of underdeveloped social–emotional competencies. The objective of the current study was to gather further reliability and validity evidence concerning the teacher and parent versions of the screeners. Findings provided evidence of high internal consistency, test–retest reliability (3-month interval), and moderate levels of cross-informant reliability. Principal axis factoring was employed for testing latent dimensions associated with the theoretical models underlying the scales’ development. Construct validity was also established using measures of children’s social–emotional competencies and internalizing/externalizing problems and yielded moderate-to-strong correlations across all forms of the screeners. Diagnostic accuracy was assessed relative to the clinical range scores from the internalizing/externalizing problem scales from Achenbach System of Empirically Based Assessment and yielded acceptable specificity and sensitivity indices. Receiver operating characteristic analyses and area under the curve values indicate that these measures have moderate-to-strong utility (AUCs ≥ .78) in detecting children at risk of internalizing and externalizing problems based on ratings of their social–emotional skills. These data support the notion that these instruments might prove potential candidates for school-wide screening practices.
Similar content being viewed by others
Notes
ECS and SCS versions translated in English are available on request from the first author.
References
Achenbach, T. M., Dumenci, L., & Rescorla, L. A. (2002). Ten-year comparisons of problems and competencies for national samples of youth: Self, parent, and teacher reports. Journal of Emotional and Behavioral Disorders, 10, 194–203.
Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross informant correlations for situational specificity. Psychological Bulletin, 101, 213–232. doi:10.1037/0033-2909.101.2.213.
Achenbach, T. M., & Rescorla, L. A. (2000). Manual for the ASEBA preschool forms and profiles. Burlington, VT: University of Vermont.
Blandon, A. Y., Calkins, S. D., Grimm, K. J., Keane, S. P., & O’Brien, M. (2010). Testing a developmental cascade model of emotional and social competence and early peer acceptance. Development and Psychopathology, 22, 737–748. doi:10.1017/S0954579410000428.
Bornstein, M. H., Hahn, C.-S., & Haynes, O. M. (2010). Social competence, externalizing, and internalizing behavioral adjustment from early childhood through early adolescence: Developmental cascades. Development and Psychopathology, 22, 717–735. doi:10.1017/S0954579410000416.
Briggs-Gowan, M. J., & Carter, A. S. (2008). Social emotional screening status in early childhood predicts elementary school outcomes. Pediatrics, 121, 957–962. doi:10.1542/peds.2007-1948.
Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.
Campbell, S. B., Spieker, S., Burchinal, M., & Poe, M. D. (2006). Trajectories of aggression from toddlerhood to age 9 predict academic and social functioning through age 12. Journal of Child Psychology and Psychiatry, 47, 791–800. doi:10.1111/j.1469-7610.2006.01636.x.
Carter, A. S., Briggs-Gowan, M. J., & Davis, N. O. (2004). Assessment of young children’s social-emotional development and psychopathology: Recent advances and recommendations for practice. Journal of Child Psychology and Psychiatry, 45, 109–134. doi:10.1046/j.0021-9630.2003.00316.x.
Cicchetti, D. V., Volkmar, F., Klin, A., & Showalter, D. (1995). Diagnosing autism using ICD-10 criteria: A comparison of neural networks and standard multivariate procedures. Child Neuropsychology, 1, 26–37.
Costello, E. J., Mustillo, S., Erkanli, A., Keeler, G., & Angold, A. (2003). Prevalence and development of psychiatric disorders in childhood and adolescence. Archives of General Psychiatry, 60, 837–844. doi:10.1001/archpsyc.60.8.837.
Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research and Evaluation, 10, 1–9.
Council on Children with Disabilities. (2001). Developmental surveillance and screening of infants and young children. Pediatrics, 108, 192–196. doi:10.1542/peds.108.1.192.
De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131, 483–509. doi:10.1037/0033-2909.131.4.483.
Denham, S. A. (2006). Social-emotional competence as support for school readiness: What is it and how do we assess it? Early Education and Development, 17, 57–89. doi:10.1207/s15566935eed1701_4.
Denham, S. A., Bassett, H. H., Way, E., Mincic, M., Zinsser, K., & Graling, K. (2012). Preschoolers’ emotion knowledge: Self-regulatory foundations, and predictions of early school success. Cognition and Emotion, 26, 667–679. doi:10.1080/02699931.2011.602049.
Denham, S. A., Blair, K. A., DeMulder, E., Levitas, J., Sawyer, K., Auerbach-Major, S., et al. (2003). Preschool emotional competence: Pathway to social competence? Child Development, 74, 238–256. doi:10.1111/1467-8624.00533.
Denham, S. A., Blair, K. A., Schmidt, M. S., Blair, K., DeMulder, E., & Caal, S. (2002). Preschool understanding of emotions: Contributions to classroom anger and aggression. Journal of Child Psychology and Psychiatry, 43, 901–916. doi:10.1111/1469-7610.00139.
Denham, S., Mason, T., Caverly, S., Schmidt, M., Hackney, R., Caswell, C., et al. (2001). Preschoolers at play: Co-socializers of emotional and social competence. International Journal of Behavioral Development, 25, 90–101. doi:10.1080/016502501143000067.
Department of Psychology Task Force. (2012). National study report on Romanian children’s mental health and well-being. Unpublished report.
Dobrean, A. (2009). The Romanian version of the Achenbach system of empirically based assessment. Cluj-Napoca: RTS Publishing.
Dowdy, E., Chin, J. K., & Quirk, M. P. (2013). Preschool screening an examination of the behavioral and emotional screening system preschool teacher form (BESS preschool). Journal of Psychoeducational Assessment, 31, 578–584. doi:10.1177/0734282913475779.
Dowdy, E., Ritchey, K., & Kamphaus, R. (2010). School based screening: A population-based approach to inform and monitor children’s mental health needs. School Mental Health, 2, 166–176. doi:10.1007/s12310-010-9036-3.
Duchesne, S., Vitaro, F., Larose, S., & Tremblay, R. E. (2008). Trajectories of anxiety during elementary-school years and the prediction of high school noncompletion. Journal of Youth and Adolescence, 39, 1134–1146. doi:10.1007/s10964-007-9224-0.
Egger, H. L., & Angold, A. (2006). Common emotional and behavioral disorders in preschool children: Presentation, nosology, and epidemiology. Journal of Child Psychology and Psychiatry, 47, 313–337. doi:10.1111/j.1469-7610.2006.01618.x.
Eisenberg, N., Cumberland, A., Spinrad, T. L., Fabes, R. A., Shepard, S. A., & Reiser, M. (2001). The relations of regulation and emotionality to children’s externalizing and internalizing problem behavior. Child Development, 72, 1112–1134. doi:10.1111/1467-8624.00337.
Eklund, K., & Dowdy, E. (2014). Screening for behavioral and emotional risk versus traditional school identification methods. School Mental Health, 6, 40–49. doi:10.1007/s12310-013-9109-1.
Elliot, S. N., & Gresham, F. M. (2007). Social Skills Improvement System: Performance Screening Guide. Minneapolis, MN: NCS Pearson.
Essex, M. J., Kraemer, H. C., Slattery, M. J., Burk, L. R., Boyce, W. T., Woodward, H. R., et al. (2009). Screening for childhood mental health problems: Outcomes and early identification. Journal of Child Psychology and Psychiatry, 50, 562–570. doi:10.1111/j.1469-7610.2008.02015.x.
Evans, J. D. (1996). Straightforward statistics for the behavioral sciences. Pacific Grove, California: Brooks/Cole Publishing.
Feeney-Kettler, K. A., Kratochwill, T. R., Kaiser, A. P., Hemmeter, M. L., & Kettler, R. J. (2010). Screening young children’s risk for mental health problems: A review of four measures. Assessment for Effective Intervention, 35, 218–230. doi:10.1177/1534508410380557.
Feeney-Kettler, K. A., Kratochwill, T. R., & Kettler, R. J. (2009). Preschool behavior screening system. Unpublished instrument, Vanderbilt University.
Feeney-Kettler, K. A., Kratochwill, T. R., & Kettler, R. J. (2011). Identification of preschool children at risk for emotional and behavioral disorders: Development and validation of a universal screening system. Journal of School Psychology, 49, 197–216. doi:10.1016/j.jsp.2011.02.001.
Flanagan, D. P., Alfonso, V. C., Primavera, L. H., Povall, L., & Higgins, D. (1996). Convergent validity of the BASC and SSRS: Implications for social skills assessment. Psychology in the Schools, 33, 13–23. doi:10.1002/(SICI)1520-6807(199601)33:1<13:AID-PITS2>3.0.CO;2-X.
George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference, 11.0 update (4th ed.). Boston: Allyn and Bacon.
Gimpel, G. A., & Merrell, K. W. (1998). On the nature and value of social skills. In G. A. Gimpel & K. W. Merrell (Eds.), Social skills of children and adolescents: Conceptualization, assessment, treatment (pp. 1–28). Mahwah, NJ: Lawrence Erlbaum Associates.
Glorfeld, L. W. (1995). An improvement on Horn’s parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55, 377–393. doi:10.1177/0013164495055003002.
Glover, T. A., & Albers, C. A. (2007). Considerations for evaluating universal screening instruments. Journal of School Psychology, 45, 117–135. doi:10.1016/j.jsp.2006.05.005.
Goodman, R. (1997). The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586. doi:10.1097/00004583-200111000-00015.
Greer, F. W., DiStefano, C. A., Liu, J., & Cain, L. K. (2015). Preliminary psychometric evidence of the Behavioral and Emotional Screening System Teacher Rating Scale-Preschool. Assessment for Effective Intervention, 40, 240–246. doi:10.1177/1534508415571594.
Gresham, F. M., & Elliott, S. N. (1990). Social skills rating system. Circle Pines, MN: American Guidance Service.
Gridley, B. E., Mucha, L., & Hatfield, B. B. (1995). Best practices in preschool screening. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology III (pp. 213–225). Washington, DC: National Association of School Psychologists.
Halberstadt, A. G., Denham, S. A., & Dunsmore, J. C. (2001). Affective social competence. Social Development, 10, 10–20. doi:10.1111/1467-9507.00150.
Hayes, A. F. (2006). A primer on multilevel modeling. Human Communication Research, 32, 385–410.
Henson, R. K., & Roberts, J. K. (2006). Use of exploratory factor analysis in published research. Educational and Psychological Measurement, 66, 393–416. doi:10.1177/0013-164405282485.
Herndon, K. J., Bailey, C. S., Shewark, E., Denham, S. A., & Bassett, H. H. (2013). Preschoolers’ emotion expression and regulation: Relations with school adjustment. The Journal of Genetic Psychology, 174, 642–663. doi:10.1080/2F00221325.2012.759525.
Hill, A. L., Degnan, K. A., Calkins, S. A., & Keane, S. P. (2006). Profiles of externalizing behavior problems for boys and girls across preschool: The roles of emotion regulation and inattention. Developmental Psychology, 42, 913–928. doi:10.1037/0012-1649.42.5.913.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118.
Ivanova, M. Y., Achenbach, T. M., Dumenci, L., Rescorla, L. A., Almqvist, F., Weintraub, S., et al. (2007a). Testing the 8-syndrome structure of the child behavior checklist in 30 societies. Journal of Clinical Child and Adolescent Psychology, 36, 405–417. doi:10.1037/0022-006X.75.5.729.
Ivanova, M. Y., Achenbach, T. M., Rescorla, L. A., Dumenci, L., Almqvist, F., Bathiche, M., et al. (2007b). Testing the Teacher’s Report Form syndromes in 20 societies. School Psychology Review, 36, 468–483.
Izard, C., Fine, S., Schultz, D., Mostow, A., Ackerman, B., & Youngstrom, E. (2001). Emotion knowledge as a predictor of social behavior and academic competence in children at risk. Psychological Science, 12, 18–23. doi:10.1111/1467-9280.00304.
Kamphaus, R. W., & Reynolds, C. R. (2007). BASC-2 behavioral and emotional screening system. Minneapolis, MN: Pearson.
Kataoka, S. H., Zhang, L., & Wells, K. B. (2002). Unmet need for mental health care among U.S. children: Variation by ethnicity and insurance status. The American Journal of Psychiatry, 159, 1548–1555. doi:10.1176/appi.ajp.159.9.1548.
Kettler, R. J., & Feeney-Kettler, K. A. (2011). Screening systems and decision making at the preschool level: Application of a comprehensive validity framework. Psychology in the Schools, 48, 430–441. doi:10.1002/pits.20565.
Kratochwill, T. R., Albers, C. A., & Shernoff, E. (2004). School-based interventions. Child and Adolescent Psychiatric Clinics of North America, 13, 885–903. doi:10.1016/j.chc.2004.05.003.
Kreft, I. G. G., & de Leeuw, J. (1998). Introducing multilevel modeling. Newbury Park, CA: Sage Publications.
LaFreniere, P. J., & Dumas, J. E. (1995). Social competence and behavior evaluation (Preschool ed.). Los Angeles, CA: Western Psychological Services.
Lane, K. L., Oakes, W. P., Lusk, M. E., Cantwell, E. D., & Schatschneider, C. (2015). Screening for intensive intervention needs in secondary schools: Directions for the future. Journal of Emotional and Behavioral Disorders,. doi:10.1177/1063426615618624.
Lane, K. L., & Walker, H. M. (2015). The connection between assessment and intervention: How does screening lead to better interventions? In B. Bateman, M. Tankersley, & J. Lloyd (Eds.), Enduring issues in special education: Personal perspectives (pp. 283–301). New York, NY: Routledge.
Lee, D. L., Belfiore, P. J., & Gormley, S. (2008). Riding the wave: Creating a momentum of school success. Teaching Exceptional Children, 40, 65–70.
Levitt, J. M., Saka, N., Hunter Romanelli, L., & Hoagwood, K. (2007). Early identification of mental health problems in schools: The status of instrumentation. Journal of School Psychology, 45, 163–191. doi:10.1016/j.jsp.2006.11.005.
Masten, A. S., Burt, K. B., & Coatsworth, J. D. (2006). Competence and psychopathology in development. In D. Cicchetti & D. J. Cohen (Eds.), Developmental psychopathology (2nd ed., Vol. 3, pp. 696–738). New York, NY: Wiley.
McCabe, P. C., & Altamura, M. (2011). Empirically valid strategies to improve social and emotional competence of preschool children. Psychology in the Schools, 48, 513–540. doi:10.1002/pits.20570.
Meisels, S., & Fenichel, E. (1996). New visions for the developmental assessment of infants and young children. Washington, DC: Zero to Three, National Center for Infants, Toddlers, and Families.
Metcalfe, L. A., Harvey, E. A., & Laws, H. B. (2013). The longitudinal relation between academic/cognitive skills and externalizing behavior problems in preschool children. Journal of Educational Psychology, 105, 881–894. doi:10.1037/2Fa0032624.
Milgrom, J., Mendelsohn, J., & Gemmill, A. W. (2011). Does postnatal depression screening work? Throwing out the bathwater, keeping the baby. Journal of Affective Disorders, 132, 301–310.
Muthen, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. doi:10.2307/271070.
Naglieri, J. A., & LeBuffe, P. A. (2005). Measuring resilience in children. In S. Goldstein & R. Brooks (Eds.), Handbook of resilience in children (pp. 107–124). New York: Kluwer/Academic Publishers.
Naglieri, J. A., LeBuffe, P. A., & Shapiro, V. (2010). Devereux student strengths assessment-mini. Lewisville, NC: Kaplan Press.
Naglieri, J. A., LeBuffe, P. A., & Shapiro, V. B. (2011). Universal screening for social-emotional competencies: A study of the reliability and validity of the Dessa-mini. Psychology in the Schools, 48, 660–671. doi:10.1002/pits.20586.
National Association for the Education of Young Children. (2005). Code of ethical conduct & statement of commitment. Brochure (Revised ed.). Washington, DC: National Association for the Education of Young Children. www.naeyc.org/positionstatements/ethical_conduct.
Owens, J. S., Holdaway, A. S., Serrano, V. J., Himawan, L. K., Watanabe, Y., et al. (2016). Screening for social, emotional, and behavioral problems at kindergarten entry: The utility of two teacher rating scales. School Mental Health, published on-line.,. doi:10.1007/s12310-016-9176-1.
Owens, J. S., Storer, J., Holdaway, A. S., Serrano, V. J., Watabe, Y., Himawan, L. K., et al. (2015). Screening for social, emotional, and behavioral problems at kindergarten entry: Utility and incremental validity of parent report. School Psychology Review, 44, 21–40. doi:10.17105/SPR44-1.21-40.
Pintea, S., & Moldovan, R. (2009). The receiver-operating characteristic (ROC) analysis: Fundamentals and applications in clinical psychology. Journal of Cognitive and Behavioral Psychotherapies, 9, 49–66.
Rubin, K. H., Bukowski, W. M., & Parker, J. G. (2006). Peer interactions, relationships, and groups. In W. Damon, R. M. Lerner, & N. Eisenberg (Eds.), Handbook of child psychology (6th ed., Vol. 3, pp. 571–645). New York, NY: Wiley.
Sandall, S., McLean, M. E., & Smith, B. J. (2000). DEC recommended practices in early intervention/early childhood special education. Longmont, CO: Sopris West.
Sattler, J. M. (2008). Assessment of children: Cognitive applications (4th ed.). La Mesa, CA: Author.
Seligman, M. P. (2008). Positive health. Applied Psychology, 57, 3–18. doi:10.1111/j.1464-0597.2008.00351.x.
Severson, H. H., & Walker, H. M. (2002). Pro-active approaches for identifying children at risk for socio-behavioral problems. In K. L. Lane, F. M. Gresham, & T. E. O’Shaughnessy (Eds.), Interventions for children with or at-risk for emotional and behavioral disorders (pp. 33–54). Boston: Allyn & Bacon.
Severson, H. H., Walker, H. M., Hope-Doolittle, J., Kratochwill, T. R., & Gresham, F. M. (2007). Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. Journal of School Psychology, 45, 193–223. doi:10.1016/j.jsp.2006.11.003.
Shin, N., Vaughn, B. E., Akers, V., Kim, M., Stevens, S., et al. (2011). Are happy children socially successful? Testing a central premise of positive psychology in a sample of preschool children. The Journal of Positive Psychology, 6, 355–367. doi:10.1080/17439760.2011.584549.
Snijders, T., & Bosker, R. (1999). Multilevel analysis. London: Sage Publication.
Squires, J., Bricker, D., & Twombly, E. (2003). Ages and stages questionnaires: Social-emotional. Baltimore, MD: Brookes.
Ştefan, C. A., Bălaj, A., Porumb, M., Albu, M., & Miclea, M. (2009). Preschool screening for social and emotional competencies–development and psychometric properties. Cognition, Brain, Behavior. An Interdisciplinary Journal, 13, 121–146.
Ştefan, C. A., & Miclea, M. (2012). Classroom effects of a hybrid universal and indicated prevention program for preschool children: A comparative analysis based on social and emotional competence screening. Early Education and Development, 23, 393–426. doi:10.1080/10409289.2011.554756.
Streiner, D. L., & Cairney, J. (2007). What’s under the ROC? An introduction to receiver operating characteristic curves. The Canadian Journal of Psychiatry, 52, 121–128.
Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston, MA: Allyn and Bacon.
Trentacosta, C. J., & Fine, S. E. (2010). Emotion knowledge, Social competence, and behavior problems in childhood and adolescence: A meta-analytic review. Social Development (Oxford, England), 19, 1–29. doi:10.1111/j.1467-9507.2009.00543.x.
Walker, H. M., Severson, H. H., & Feil, E. G. (1995). Early screening project: User manual. Longmont, CO: Sopris West.
Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432–442. doi:10.1037/0033-2909.99.3.432.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Catrinel A. Ştefan and Mircea Miclea declare that they have no conflict of interest.
Human and Animal Rights
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Informed Consent
Informed consent was obtained from all individual participants included in the study.
Ethical Approval
This article does not contain any studies with animals performed by any of the authors.
Rights and permissions
About this article
Cite this article
Ştefan, C.A., Miclea, M. Reliability and Validity of Two Brief Screening Measures of Preschoolers’ Social–Emotional Competencies. School Mental Health 9, 44–65 (2017). https://doi.org/10.1007/s12310-016-9200-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12310-016-9200-5