Skip to main content
SpringerLink
Log in

Psychometric Properties of Standardized Instruments that are Used to Measure Pragmatic Intervention Effects in Children with Developmental Language Disorder: A Systematic Review

  • Original Paper
  • Published:
Journal of Autism and Developmental Disorders Aims and scope Submit manuscript

Abstract

This review aimed to analyze the psychometric properties of standardized instruments that are used to measure pragmatic intervention effects in children with developmental language disorder. Following PRISMA structure, a comprehensive literature research was conducted in five electronic indexing databases: Central, PubMed, Medline, Web of Science and Scopus. Six studies between 2005 and 2019 were included and seven standardized instruments have been analyzed. All instruments present some evidence of validity and reliability, but none reported responsiveness. The instruments reviewed were not used for their original purpose, which may explain the results. The small number of studies prevent drawing clear conclusions. Future studies should focus on appropriate, valid, reliable, and responsive measures. Research focusing on psychometric properties is crucial, mostly about responsiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Abdoola, F., Flack, P. S., & Karrim, S. B. (2017). Facilitating pragmatic skills through role-play in learners with language learning disability. The South African Journal of Communication Disorders, 64(1), e1–e12. https://doi.org/10.4102/sajcd.v64i1.187

    Article  PubMed  Google Scholar 

  • Adams, C., & Lloyd, J. (2007). The effects of speech and language therapy intervention on children with pragmatic language impairments in mainstream school. British Journal of Special Education, 34(4), 226–233. https://doi.org/10.1111/j.1467-8578.2007.00483.x

    Article  Google Scholar 

  • Adams, C., Cooke, R., Crutchley, A., Hesketh, A., & Reeves, D. (2001). Assessment of comprehension and expression 6–11 (ACE 6–11). NFER-Nelson.

    Google Scholar 

  • Adams, C., Baxendale, J., Lloyd, J., & Aldred, C. (2005). Pragmatic language impairment: Case studies of social and pragmatic language therapy. Child Language Teaching and Therapy, 21(3), 227–250. https://doi.org/10.1191/0265659005ct290oa

    Article  Google Scholar 

  • Adams, C., Lloyd, J., Aldred, C., & Baxendale, J. (2006). Exploring the effects of communication intervention for developmental pragmatic language impairments: A signal-generation study. International Journal of Language & Communication Disorders, 41(1), 41–65. https://doi.org/10.1080/13693780500179793

    Article  Google Scholar 

  • Adams, C., Lockton, E., Freed, J., Gaile, J., Earl, G., McBean, K., Nash, M., Green, J., Vail, A., & Law, J. (2012). The social communication intervention project: A randomized controlled trial of the effectiveness of speech and language therapy for school-age children who have pragmatic and social communication problems with or without autism spectrum disorder. International Journal of Language & Communication Disorders, 47(3), 233–244. https://doi.org/10.1111/j.1460-6984.2011.00146.x

    Article  Google Scholar 

  • American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders—fifth edition. American Psychiatric Association Publishing.

  • Andresen, S. (2000). Criteria for assessing the tools of disability outcomes research. Archives of Physical Medicine and Rehabilitation, 81(12), S15–S20. https://doi.org/10.1053/apmr.2000.20619

    Article  PubMed  Google Scholar 

  • Andersson, L. (2005). Determining the adequacy of tests of children’s language. Communication Disorders Quartely, 26(4), 207–225. https://doi.org/10.1177/15257401050260040301

    Article  Google Scholar 

  • Betz, S. K., Eickhoff, J. R., & Sullivan, S. F. (2013). Factors influencing the selection of standardized tests for the diagnosis of specific language impairment. Language, Speech and Hearing Services in Schools, 44(2), 133–146. https://doi.org/10.1044/0161-1461(2012/12-0093)

    Article  PubMed  Google Scholar 

  • Bishop, D. V. M. (1998). Development of the children’s communication checklist (CCC): A method for assessing qualitative aspects of communicative impairment in children. Journal of Child Psychology and Psychiatry and Allied Disciplines, 39(6), 879–891. https://doi.org/10.1111/1469-7610.00388

    Article  PubMed  Google Scholar 

  • Bishop, D. V. M. (2003). The children’s communication checklist—2. Psychological Corporation.

    Google Scholar 

  • Bishop, D. V. M. (2004). Expression, reception and recall of narrative instrument: ERRNI. Harcourt Assessment.

    Google Scholar 

  • Bishop, D. V. M. (2006). Children’s communication checklist—2. Harcourt Assessment.

    Google Scholar 

  • Bishop, D. V. M. (2017). Why is it so hard to reach agreement on terminology? The case of developmental language disorder (DLD). International Journal of Language & Communication Disorders, 52(6), 671–680. https://doi.org/10.1111/1460-6984.12335

    Article  Google Scholar 

  • Bishop, D. V. M., & Rosenbloom, L. (1987). Classification of childhood language disorders. Language Development and Disorders, 22(1), 61–81.

    Google Scholar 

  • Bishop, D. V. M., Snowling, M. J., Thompson, P. A., Greenhalgh, T., CATALISE-2 Consortium. (2016). CATALISE: A multinational and multidisciplinary delphi consensus study. Identifying language impairments in children. PLoS ONE. https://doi.org/10.1371/journal.pone.0158753

    Article  PubMed  PubMed Central  Google Scholar 

  • Bishop, D. V. M., Snowling, M. J., Thompson, P. A., Greenhalgh, T., CATALISE Consortium. (2017). Phase 2 of CATALISE: A multinational and multidisciplinary delphi consensus study of problems with language development: Terminology. Journal of Child Psychology and Psychiatry and Allied Disciplines, 58(10), 1068–1080. https://doi.org/10.1111/jcpp.12721

    Article  PubMed  Google Scholar 

  • Bowers, L., Huisingh, R., & LoGuidice, C. (2018). Test of problem solving—third edition elementary: Normative update. LinguiSystems.

    Google Scholar 

  • Brownell, M. D. (2002). Musically adapted social stories to modify behaviors in students with autism: Four case studies. Journal of Music Therapy, 39, 117–144. https://doi.org/10.1093/jmt/39.2.117

    Article  PubMed  Google Scholar 

  • Conti-Ramsden, G., & Botting, N. (2004). Social difficulties and victimization in children with SLI at 11 years of age. Journal of Speech, Language and Hearing Research, 47(1), 145–161. https://doi.org/10.1044/1092-4388(2004/013)

    Article  Google Scholar 

  • Corkum, P., Corbin, N., & Pike, M. (2010). Evaluation of a school-based social skills program for children with attention-deficit/hyperactivity disorder. Child and Family Behavior Therapy, 32(2), 139–151. https://doi.org/10.1080/07317101003776472

    Article  Google Scholar 

  • Cummings, L. (2017). Research in clinical pragmatics. In Perspectives in pragmatics, philosophy & psychology. (Vol. 11). Springer.

    Google Scholar 

  • Denman, D., Speyer, R., Munro, N., Pearce, W. M., Chen, Y. W., & Cordier, R. (2017). Psychometric properties of language assessments for children aged 4–12 years: A systematic review. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2017.01515

    Article  PubMed  PubMed Central  Google Scholar 

  • Docking, K., Munro, N., Cordier, R., & Ellis, P. (2013). Examining the language skills of children with ADHD following a play-based intervention. Child Language Teaching and Therapy, 29(3), 291–304. https://doi.org/10.1177/0265659012469042

    Article  Google Scholar 

  • Dockrell, J. E., & Marshall, C. R. (2015). Measurement issues: Assessing language skills in young children. Child and Adolescent Mental Health, 20(2), 116–125. https://doi.org/10.1111/camh.12072

    Article  PubMed  Google Scholar 

  • Dunn, L. M., Dunn, L. M., & Whetton, C. (1997). The British Picture Vocabulary Scale (2nd ed.). NFER-Nelson.

    Google Scholar 

  • Feinberg, M. J. (2002). Using social stories to teach specific social skills to individuals diagnosed with autism [Unpublished Doctoral Dissertation]. California School of Professional Psychology.

  • Fleiss, J. L. (1986). Reliability of measurements. Wiley.

    Google Scholar 

  • Friberg, J. C. (2010). Considerations for test selection: How do validity and reliability impact of diagnostic decisions? Child Language Teaching and Therapy, 26, 77–92. https://doi.org/10.1177/0265659009349972

    Article  Google Scholar 

  • Friedman, N., & Novogrodsky, R. (2008). Subtypes of SLI: SySLI, PhoSLI. Cambridge Scholars Press.

    Google Scholar 

  • Gerber, S., Brice, A., Capone, N., Fujiki, M., & Timler, G. (2012). Language use in social interactions of school-age children with language impairments: An evidence-based systematic review of treatment. Language Speech and Hearing Services in Schools. https://doi.org/10.1044/0161-1461(2011/10-0047)

    Article  PubMed  Google Scholar 

  • Geurts, H., & Embrechts, M. (2008). Language profiles in ASD, SLI, and ADHD. Journal of Autism and Developmental Disorders, 38(10), 1931–1943. https://doi.org/10.1007/s10803-008-0587-1

    Article  PubMed  Google Scholar 

  • Goodyer, I. M. (2000). Language difficulties and psychopathology. In D. V. M. Bishop & L. B. Leonard (Eds.), Speech and language impairments in children: Causes, characteristics, intervention and outcome (pp. 227–244). Psychology Press.

    Google Scholar 

  • Gray, C. A. (1994). Comic strip conversations: Illustrated interactions that teach conversation skills to students with autism and related disorders. Future Horizons.

    Google Scholar 

  • Gray, C. A., & Garand, J. D. (1993). Social stories: Improving responses of students with autism with accurate social information. Focus on Autistic Behaviour, 8(1), 1–10. https://doi.org/10.1177/108835769300800101

    Article  Google Scholar 

  • Gresham, F. M., & Elliott, S. N. (1990). The social skills rating system. American Guidance Service.

    Google Scholar 

  • Gresham, F. M., & Elliott, S. N. (2008). Social skills improvement system: Rating scales. Pearson Assessments.

    Google Scholar 

  • Hoffman, L. M., Loeb, D. F., Brandel, J., & Gillam, R. B. (2011). Concurrent and construct validity of oral language measures with school-age children with specific language impairment. Journal of Speech, Language and Hearing Research, 54(6), 1597–1608. https://doi.org/10.1044/1092-4388(2011/10-0213)

    Article  Google Scholar 

  • Huisingh, R., Bowers, L., & LoGiudice, C. (2005). The test of problem solving, third edition—elementary. LinguiSystems.

    Google Scholar 

  • Hyman, S. L., Levy, S. E., Myers, S. M., App Council on Children with Disabilities Section on Developmental and Behavioral Pediatrics. (2020). Identification, evaluation, and management of children with autism spectrum disorder. Pediatrics, 145(1), e20193447. https://doi.org/10.1542/peds.2019-3447

    Article  PubMed  Google Scholar 

  • Katelaars, M. P., Cuperus, J., Jansonius, K., & Verhoeven, L. (2010). Pragmatic language impairment and associated behavioral problems. International Journal of Language & Communication Disorders, 45(2), 204–214. https://doi.org/10.3109/13682820902863090

    Article  Google Scholar 

  • Kohnert, K., Windsor, J., & Ebert, K. D. (2009). Primary or “specific” language impairment and children learning a second language. Brain and Language, 102(2–3), 101–111. https://doi.org/10.1016/j.bandl.2008.01.009

    Article  Google Scholar 

  • Law, J., Wendy, L., Roulstone, S., Wren, Y., Zeng, B., & Lindsay, G. (2012). "What works": Interventions for children and young people with speech, language and communication needs [Research Report DFE-RR247-BCRP10]. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/556912/DFERR247-BCRP10.pdf

  • Lorimer, P. A., Simpson, R. L., Myles, B. S., & Ganz, B. J. (2002). The use of social stories as a preventative behavioral intervention in a home setting with a child with autism. Journal of Positive Behavior Interventions, 4(1), 53–60. https://doi.org/10.1177/109830070200400109

    Article  Google Scholar 

  • Moher, D., Liberati, A., Tetzlaff, J., Altamn, D., & Group, P. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 6, 264–269. https://doi.org/10.7326/0003-4819-151-4-200908180-00135

    Article  Google Scholar 

  • Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., Bouter, L. M., & de Vet, H. C. (2010). The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemology, 63(7), 737–745. https://doi.org/10.1016/j.jclinepi.2010.02.006

    Article  Google Scholar 

  • Nelson, K. (1978). How young children represent knowledge of their world in and out of language. In R. S. Siegler (Ed.), Children’s thinking: What develops? (pp. 225–273). Erlbaum.

    Google Scholar 

  • Norbury, C. F., Gooch, D., Wray, C., Baird, G., Charman, T., Simonoff, E., Vamvakas, G., & Pickles, A. (2016). The Impact of nonverbal ability on prevalence and clinical presentation of language disorder: Evidence from a population study. Journal of Child Psychology and Psychiatry, 57(11), 1247–1257. https://doi.org/10.1111/jcpp.12573

    Article  PubMed  Google Scholar 

  • Parsons, L., Cordier, R., Munro, N., Joosten, A., & Speyer, R. (2017). A systematic review of pragmatic language interventions for children with autism spectrum disorder. PLoS ONE, 12(4), e0172242. https://doi.org/10.1371/journal.pone.0172242

    Article  PubMed  PubMed Central  Google Scholar 

  • Pereira, T., Lousada, M., Ramalho, M., & Machado, B. (2019). Programa de intervenção em competências pragmáticas—Versão Pré-Escolar [Manual de Aplicação]. Edição de Autor.

  • Pereira, T., Lousada, M., Ramalho, M., & Machado, B. (2021). Programa de Intervenção em Competências Pragmáticas: Desenvolvimento, validação de conteúdo e estudo piloto de aceitabilidade. Revista Portuguesa de Terapia da Fala. https://doi.org/11.2181/rptf.2021.12.02

  • Polit, D. (2015). Assessing measurement in health: Beyond reliability and validity. International Journal of Nursing Studies, 52(11), 1746–1753. https://doi.org/10.1016/j.ijnurstu.2015.07.002

    Article  PubMed  Google Scholar 

  • Polit, D., & Yang, M. (2016). Measurement and the measurement of change. Wolters Kluwer.

    Google Scholar 

  • Price, P., Jhangiani, R., Chiang, I. C., Leighton, D., & Cuttler, C. (2017). Research methods in psychology (3rd American ed.). University of Minnesota Libraries Publishing. https://opentext.wsu.edu/carriecuttler/.

  • Prutting, C. A., & Kirchner, D. M. (1987). A clinical appraisal of the pragmatic aspects of language. Journal of Speech and Hearing Disorders, 52(2), 105–119. https://doi.org/10.1044/jshd.5202.105

    Article  PubMed  Google Scholar 

  • Rapin, I., & Allen, D. A. (1983). Developmental language disorders: Nosologic considerations. Academic Press.

    Google Scholar 

  • Reynhout, G., & Carter, M. (2006). Social Stories for children with disabilities. Journal of Autism and Developmental Disorders, 34(4), 445–469. https://doi.org/10.1007/s10803-006-0086-1

    Article  Google Scholar 

  • Rinaldi, W. (1995). The social use of language programme (primary and pre-school teaching pack). NFER.

    Google Scholar 

  • Russel, R. L., & Grizzle, K. L. (2008). Assessing child and adolescent pragmatic language competencies: Toward evidence-based assessments. Clinical Child and Family Psychology Review, 11(1–2), 59–73. https://doi.org/10.1007/s10567-008-0032-1

    Article  Google Scholar 

  • Salter, K., Jutai, J., Teasell, R., Foley, N., & Bitensky, J. (2005). Issues for selection of outcome measures in stroke rehabilitation: ICF body functions. Disability and Rehabilitation, 27(4), 191–207. https://doi.org/10.1080/09638280400008537

    Article  PubMed  Google Scholar 

  • Semel, E., Wiig, E. H., & Secord, W. A. (2000). Clinical evaluation of language fundamentals (3rd ed.). Psychological Corporation.

    Google Scholar 

  • Semel, E., Wiig, E. H., & Secord, W. A. (2006). Clinical evaluation of language fundamentals (4th ed.). Pearson Assessments.

    Google Scholar 

  • Shipley, K. G., & McAfee, J. G. (2021). Assessment in speech-language pathology: A resource manual (6th ed.). Plural Publishing.

    Google Scholar 

  • Spaulding, T. J., Swartwout Szulga, M., & Figueroa, C. (2012). Using norm-referenced tests to determine severity of language impairment in children: Disconnect between U.S. policy makers and test developers. Language, Speech and Hearing Services in Schools, 43(2), 176–190. https://doi.org/10.1044/0161-1461(2011/10-0103)

    Article  PubMed  Google Scholar 

  • Statley, M. J. (2002). An investigation of social-story effectiveness using reversal and multiple-baseline designs [Unpublished Doctoral Dissertation].

  • Tavakol, M., & Dennick, R. (2011). Making sense of Cronback’s alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd

    Article  PubMed  PubMed Central  Google Scholar 

  • Terwee, C. B., Mokkink, L. B., Knol, D. L., Ostelo, R. W., Bouter, L. M., & de Vet, H. C. W. (2012). Rating the methodological quality in systematic reviews of studies on measurement properties: A scoring system for the COSMIN checklist. Quality of Life Research, 21(4), 651–657. https://doi.org/10.1007/s11136-011-9960-1

    Article  PubMed  Google Scholar 

  • Vermon, D. S., Schumaker, J. B., & Deshler, D. D. (1996). Score skills: Social skills for cooperative groups. Edge Enterprises Inc.

    Google Scholar 

  • Volden, J., Dodd, E., Engel, K., Smith, I. M., Szatmari, P., Fombonne, E., Zwaigenbaum, L., Mirenda, P., Bryson, S., Roberts, W., Vaillancourt, T., Waddell, C., Elsabbagh, M., Teresa, B., Bennett, T., Georgiades, S., Duku, E., study team, P. A. S. D. (2017). Beyond sentences: Using the expression, reception, and recall of narratives instrument to assess communication in school-aged children with autism spectrum disorder. Journal of Speech, Language and Hearing Research, 60(8), 2228–2240. https://doi.org/10.1044/2017_JSLHR-L-16-0168

    Article  Google Scholar 

  • Weismer, S. E. (2013). Developmental language disorders: Challenges and implications of cross-group comparisons. Folia Phoniatrica Et Logopaedica, 65(2), 68–77. https://doi.org/10.1159/000353896

    Article  Google Scholar 

  • Whitehouse, A. J. O., Watt, H. J., Line, E. A., & Bishop, D. V. M. (2009). Adult psychosocial outcomes of children with specific language impairment, pragmatic language impairment and autism. International Journal of Language & Communication Disorders, 44(4), 511–528. https://doi.org/10.1080/13682820802708098

    Article  Google Scholar 

  • Wiig, E. H., Semel, E., & Secord, W. A. (2013). Clinical evaluation of language fundamentals (5th ed.). Pearson Assessments.

    Google Scholar 

  • Winner, M. G., & Crooke, P. J. (2009). Social thinking: A developmental treatment approach for students with social learning/social pragmatic challenges. Perspectives on Language Learning and Education, 16(2), 62–69. https://doi.org/10.1044/lle16.2.62

    Article  Google Scholar 

  • Yusoff, M. (2019). ABC of content validation and content validity index calculation. Education in Medicine Journal, 11(2), 49–54. https://doi.org/10.21315/eimj2019.11.2.6

    Article  Google Scholar 

  • Ziegenfusz, S., Coughlan, A., Paynter, J., Simpson, K., & Westerveld, M. F. (2018). Measuring language progress in students with developmental language disorder while attending a specialist school: A retrospective analysis. Journal of Clinical Practice in Speech-Language Pathology, 20(1), 1–7.

    Google Scholar 

  • Zingg, W., Castro-Sanchez, E., Secci, F. V., Edwards, R., Drumright, L. N., Sevdalis, N., & Holmes, A. H. (2016). Innovative tools for quality assessment integrated quality criteria for review of multiple study designs (ICROMS). Public Health, 133, 19–37. https://doi.org/10.1016/j.puhe.2015.10.012

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

This work was supported by the European Regional Development Fund (ERDF) under Grant POCI-01-0145-FEDER-007746 funded by the Operational Programme Competitiveness and Internationalization–COMPETE2020 and by the national funds through FCT–Fundação para a Ciência e a Tecnologia, I.P., within CINTESIS, R&D Unit (reference UIDB/4255/2020 and UIDP/4255/2020), CLUL (reference UIDB/00214/2020) and a PhD Grant (reference 2020.08569.BD).

Funding

Funding was provided by FCT-Fundação para a Ciência e a Tecnologia, I.P., (Grant Nos. 2020.08569.BD, UIDB/4255/2020, UIDP/4255/2020, and UIDB/00214/2020), European Regional Development Fund (Grant No. POCI-01-0145-FEDER-007746).

Author information

Authors and Affiliations

  1. Center for Health Technology and Services Research (CINTESIS.UA), Aveiro, Portugal

    Tatiana Pereira & Marisa Lousada

  2. Center of Linguistics of the University of Lisbon (CLUL), Lisbon, Portugal

    Tatiana Pereira & Marisa Lousada

  3. University of Aveiro, Aveiro, Portugal

    Tatiana Pereira

  4. School of Health Sciences (ESSUA), University of Aveiro, Aveiro, Portugal

    Marisa Lousada

  5. University of Aveiro, Campus Universitário de Santiago, 3810-193, Aveiro, Portugal

    Tatiana Pereira

Authors
  1. Tatiana Pereira
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marisa Lousada
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed equally to this review and approved the final version.

Corresponding author

Correspondence to Tatiana Pereira.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pereira, T., Lousada, M. Psychometric Properties of Standardized Instruments that are Used to Measure Pragmatic Intervention Effects in Children with Developmental Language Disorder: A Systematic Review. J Autism Dev Disord 53, 1764–1780 (2023). https://doi.org/10.1007/s10803-022-05481-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10803-022-05481-7

Keywords

Search

Navigation