Skip to main content

Four methods of identifying change in the context of a multiple component reading intervention for struggling middle school readers

Reading and Writing volume 26, pages 539–563 (2013)Cite this article

Abstract

The results from controlled intervention research have indicated that effective reading interventions exist for children with reading difficulties. Effect sizes for older struggling readers, however, typically have not matched the large effects demonstrated with younger children. Standardized effect sizes for intervention/control comparisons obscure important individual differences within intervention and control groups—differences potentially relevant to the who and why of intervention success. The present study reports the outcomes of PHAST Reading, a research-based multiple component reading intervention. Participants were 270 Grade 6, 7, and 8 students reading significantly below age-level expectations, who participated in a year-long intensive small-group intervention. Four methods were applied to characterize individual change: (a) normalization relative to age-appropriate standards; (b) statistically-reliable pre–post change using the Jacobson–Truax index; (c) individually-estimated growth rates using hierarchical linear modeling; and (d) change to a fixed criterion across multiple measures. Each method was evaluated for its ability to identify intervention outcomes, replicate traditional group-based effect size metrics, and characterize individual differences across participants depending on whether change was demonstrated. Each method replicated traditional group-based effect sizes, with advantages in consistency and predictive power for the reliable change index and growth curve approaches.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Notes

  1. The standard error of the difference was calculated via \( \sqrt {2S_{E}^{2} } \), where S E represents the standard error of measurement of the test score, and was estimated via \( \sqrt {S_{x}^{2} r_{{xx^{\prime } }} } \), where S x is the standard deviation of pretest scores and \( r_{{xx^{\prime } }} \) is the reliability of the test in the standardization sample.

References

  • Al Otaiba, S., & Fuchs, D. (2002). Characteristics of children who are unresponsive to early literacy intervention: A review of the literature. Remedial and Special Education, 23, 300–316.

    Article  Google Scholar 

  • Aunola, K., Leskinen, E., Onatsu-Arvilommi, T., & Nurmi, J.-E. (2002). Three methods for studying developmental change: A case of reading skills and self-concept. British Journal of Educational Psychology, 72(3), 343–364.

    Article  Google Scholar 

  • Barth, A. E., Stuebing, K. K., Anthony, J. L., Denton, C. A., Mathes, P. G., Fletcher, J. M., et al. (2008). Agreement among response to intervention criteria for identifying responder status. Learning and Individual Differences, 18(3), 296–307.

    Google Scholar 

  • Bauer, S. B., Lambert, M. J., & Nielsen, S. L. (2004). Clinical significance methods: A comparison of statistical techniques. Journal of Personality Assessment, 82(1), 60–70.

    Article  Google Scholar 

  • Beeson, P. I. M., & Robey, R. R. (2006). Evaluating single-subject treatment research: Lessons learned from the aphasia literature. Neuropsychology Review, 16(4), 161–169. doi:10.1007/s11065-006-9013-7.

    Article  Google Scholar 

  • Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, B, 57, 289–300.

    Google Scholar 

  • Bryk, A. S., & Raudenbush, S. W. (1987). Application of hierarchical linear models to assessing change. Psychological Bulletin, 101(1), 147–158.

    Article  Google Scholar 

  • Chinn, S. (2000). A simple method for converting an odds ratio to effect size for use in meta-analysis. Statistics in Medicine, 19, 3127–3131.

    Article  Google Scholar 

  • Christensen, L., & Mendoza, J. L. (1986). A method of assessing change in a single subject: An alteration of the RC index. Behavior Therapy, 17, 305–308.

    Article  Google Scholar 

  • Cirino, P. T., Rashid, F. L., Sevcik, R. A., Lovett, M. W., Frijters, J. C., Wolf, M., et al. (2002). Psychometric stability of nationally normed and experimental decoding and related measures in children with reading disability. Journal of Learning Disabilities, 35(6), 526–539.

    Article  Google Scholar 

  • Compton, D. L. (2000). Modeling the response of normally achieving and at-risk first grade children to word reading instruction. Annals of Dyslexia, 50, 53–84.

    Article  Google Scholar 

  • Compton, D. L. (2003). Modeling the relationship between growth in rapid naming speed and growth in decoding skill in first-grade children. Journal of Educational Psychology, 95(2), 225–239.

    Article  Google Scholar 

  • Edmonds, M. S., Vaughn, S., Wexler, J., Reutebuch, C., Cable, A., Klinger-Tackett, K., et al. (2009). A Synthesis of reading interventions and effects on reading comprehension outcomes for older struggling readers. Review of Educational Research, 79(1), 262–300.

    Article  Google Scholar 

  • Evans, C. (1988). Reliable change criterion calculator. Retrieved from http://www.psyctc.org/stats/rcsc1.htm.

  • Fletcher, J. M., Stuebing, K. K., & Barth, A. E. (2011). Cognitive correlates of inadequate response to reading intervention. School Psychology Review, 40(1), 3–22.

    Google Scholar 

  • Foorman, B. R., Francis, D. J., Fletcher, J. M., Schatschneider, C., & Mehta, P. (1998). The role of instruction in learning to read: Preventing reading failure in at-risk children. Journal of Educational Psychology, 90(1), 37–55.

    Article  Google Scholar 

  • Francis, D. J., Shaywitz, S. E., Stuebing, K. K., Shaywitz, B. A., & Fletcher, J. M. (1996). Developmental lag versus deficit models of reading disability: A longitudinal, individual growth curves analysis. Journal of Educational Psychology, 88(1), 3–17.

    Article  Google Scholar 

  • Frijters, J. C., Lovett, M. W., Steinbach, K. A., Wolf, M., Sevcik, R. A., & Morris, R. (2011). Neurocognitive predictors of reading outcomes for children with reading disabilities. Journal of Learning Disabilities, 44(2), 150–166.

    Article  Google Scholar 

  • Fuchs, L. S. (2003). Assessing intervention responsiveness: Conceptual and technical issues. Learning Disabilities Research and Practice, 18(3), 172–186.

    Article  Google Scholar 

  • Gathercole, S. E., Alloway, T. P., Willis, C., & Adams, A.-M. (2005). Working memory in children with reading disabilities. Journal of Experimental Child Psychology, 93, 265–281.

    Article  Google Scholar 

  • Greenberg, D., Wise, J. C., Morris, R., Fredrick, L. D., Rodrigo, V., Nanda, A. O., et al. (2011). A randomized control study of instructional approaches for struggling adult readers. Journal of Research on Educational Effectiveness, 4(2), 101–117.

    Google Scholar 

  • IBM. (2010). IBM SPSS statistics (version 19.0). Chicago, IL.

  • Jacobson, N. S., Follette, W. C., & Revenstorf, D. (1984). Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance. Behavior Therapy, 15, 336–352.

    Article  Google Scholar 

  • Kromrey, J. D., & Foster-Johnson, L. (1996). Determining the efficacy of intervention: The use of effect sizes for data analysis in single-subject research. Journal of Experimental Education, 65(1), 73–93.

    Article  Google Scholar 

  • Lovett, M. W., De Palma, M., Frijters, J. C., Steinbach, K. A., Temple, M., Benson, N. J., et al. (2008). Interventions for reading difficulties: A comparison of response to intervention by ELL and EFL struggling readers. Journal of Learning Disabilities, 41(4), 333–352.

    Article  Google Scholar 

  • Lovett, M. W., Lacerenza, L., & Borden, S. L. (2000). Putting struggling readers on the PHAST track: A program to integrate phonological and strategy‐based remedial reading instruction and maximize outcomes. Journal of Learning Disabilities, 33(5), 458–476.

    Google Scholar 

  • Lovett, M. W., Lacerenza, L., Borden, S. L., Frijters, J. C., Steinbach, K. A., & De Palma, M. (2000). Components of effective remediation for developmental reading disabilities: Combining phonological and strategy-based instruction to improve outcomes. Journal of Educational Psychology, 92(2), 263–283.

    Article  Google Scholar 

  • Lovett, M. W., Lacerenza, L., De Palma, M., & Frijters, J. C. (2012). Evaluating the efficacy of remediation for struggling readers in high school. Journal of Learning Disabilities, 45(2), 151–169.

    Article  Google Scholar 

  • Maasen, G. H. (2004). The standard error in the Jacobson and Truax reliable change index: The classical approach to the assessment of reliable change. Journal of the International Neuropsychological Society, 10, 888–893.

    Google Scholar 

  • Magnusson, D., & Allen, V. L. (1983). An interactional perspective for human development. In D. Magnusson & V. L. Allen (Eds.), Human development: An interactional perspective (pp. 369–387). New York, NY: Academic Press.

    Google Scholar 

  • Magnusson, D., & Bergman, L. R. (1988). Individual and variable-based approaches to longitudinal research on early risk factors. In M. Rutter (Ed.), Studies of psychosocial risk: The power of longitudinal data (pp. 45–61). Cambridge: Cambridge University Press.

    Google Scholar 

  • McArdle, J. J. (2001). A latent difference score approach to longitudinal dynamic structural analysis. In R. Cudeck, S. du Toit, & D. Sorbom (Eds.), Structural equation modeling: Present and future (pp. 342–380). Lincolnwood, IL: Scientific Software International.

    Google Scholar 

  • McGlinchey, J. B., Atkins, D. C., & Jacobson, N. S. (2002). Clinical significance methods: Which one to use and how useful are they? Behavior Therapy, 33, 529–550.

    Google Scholar 

  • Morris, R. D., Lovett, M. W., Wolf, M., Sevcik, R. A., Steinbach, K. A., Frijters, J. C., et al. (2012). Multiple-component remediation for developmental reading disabilities: IQ, SES, and race as factors on remedial outcome. Journal of Learning Disabilities, 45(2), 99–127.

    Article  Google Scholar 

  • Muthén, L. K., & Muthén, B. O. (1998–2010). Mplus user’s guide (6th ed.). Los Angeles, CA: Muthén & Muthén.

  • Muthén, B., & Muthén, L. K. (2000). Integrating person-centered and variable-centered analyses: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical and Experimental Research, 24(6), 882–891.

    Google Scholar 

  • Nelson, J. R., Benner, G. J., & Gonzalez, J. (2003). Learner characteristics that influence the treatment effectiveness of early literacy interventions: A meta-analytic review. Learning Disabilities Research and Practice, 18, 255–267.

    Article  Google Scholar 

  • Olson, R. K., Wise, B., Ring, J., & Johnson, M. (1997). Computer-based remedial training in phoneme awareness and phonological decoding: Effects on the posttraining development of word recognition. Scientific Studies of Reading, 1(3), 235–254.

    Article  Google Scholar 

  • Overall, J. E. (1994). Issues in the design and analysis of controlled clinical trials. Journal of Clinical Psychology, 51(1), 95–102.

    Article  Google Scholar 

  • Raudenbush, S. W., Bryk, A. S., & Congdon, R. T. (2007). HLM 6: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International.

    Google Scholar 

  • Reed, D., & Vaughn, S. (2010). Reading interventions for older students. In T. A. Glover & S. Vaughn (Eds.), The promise of response to intervention: Evaluating current science and practice (pp. 143–186). New York, NY: The Guilford Press.

    Google Scholar 

  • Rogosa, D. R. (1995). Myths and methods: “Myths About Longitudinal Research” plus supplemental questions. In J. Gottman (Ed.), The analysis of change (pp. 3–66). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Rogosa, D., Brandt, D., & Zimowski, M. (1982). A growth curve approach to the measurement of change. Psychological Bulletin, 92(3), 726–748.

    Article  Google Scholar 

  • Rogosa, D. R., & Willet, J. B. (1983). Demonstrating the reliability of the difference score in the measurement of change. Journal of Educational Measurement, 20(4), 335–343.

    Article  Google Scholar 

  • Sabatini, J. P., Shore, J., Holtzman, S., & Scarborough, H. S. (2011). Relative effectiveness of reading intervention programs for adults with low literacy. Journal of Research on Educational Effectiveness, 4(2), 118–133.

    Google Scholar 

  • SAS Institute, I. (2008). SAS/STAT system (Version 9.2). Cary, NC.

  • Scarborough, H. (2001). Connecting early language and literacy to later reading (dis)abilities: Evidence, theory, and practice. In S. Neuman & D. Dickinson (Eds.), Handbook for research in early literacy (pp. 97–110). New York: Guilford Press.

    Google Scholar 

  • Singer, J. D., & Willet, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York: Oxford University Press.

    Book  Google Scholar 

  • Smart, D., Sanson, A., & Prior, M. (1996). Connections between reading disability and behavior problems: Testing temporal and causal hypotheses. Journal of Abnormal Child Psychology, 24(3), 363–383.

    Article  Google Scholar 

  • Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage Publications.

    Google Scholar 

  • Swanson, H. L. (1999). Reading research for students with learning disabilities: A meta-analysis of intervention outcomes. Journal of Learning Disabilities, 32(6), 504–532.

    Article  Google Scholar 

  • Swanson, H., Trainin, G., Necoechea, D. M., & Hammill, D. D. (2003). Rapid naming, phonological awareness, and reading: A meta-analysis of the correlation evidence. Review of Educational Research, 73(4), 407–440.

    Article  Google Scholar 

  • Temkin, N. R. (2004). Standard error in the Jacobson and Truax Reliable Change Index: The “classical approach” leads to poor estimates. Journal of the International Neuropsychological Society, 2004(10), 899–901.

    Google Scholar 

  • Tiu, R. D., Thompson, L. A., & Lewis, B. A. (2003). The role of IQ in a component model of reading. Journal of Learning Disabilities, 36(5), 424–436.

    Article  Google Scholar 

  • Torgesen, J. K., Alexander, A. W., Wagner, R. K., Rashotte, C. A., Voeller, K. K. S., & Conway, T. (2001). Intensive remedial instruction for children with severe reading disabilities: Immediate and long-term outcomes from two instructional approaches. Journal of Learning Disabilities, 34(1), 33–58.

    Article  Google Scholar 

  • Torgesen, J. K., Wagner, R. K., Rashotte, C. A., Alexander, A. W., & Conway, T. (1997). Preventive and remedial interventions for children with severe reading disabilities. Learning Disabilities: A Multidisciplinary Journal, 8, 51–62.

    Google Scholar 

  • Vellutino, F. R., Scanlon, D. M., & Lyon, G. R. (2000). Differentiating between difficult-to-remediate and readily remediated poor readers. Journal of Learning Disabilities, 33(3), 223–238.

    Article  Google Scholar 

  • Vellutino, F. R., Scanlon, D. M., Sipay, E. R., Small, S. G., Pratt, A., Chen, R., et al. (1996). Cognitive profiles of difficult-to-remediate and readily remediated poor readers: Early intervention as a vehicle for distinguishing between cognitive and experiential deficits as basic causes of specific reading disability. Journal of Educational Psychology, 88(4), 601–638.

    Article  Google Scholar 

  • Wagner, R. K., & Torgesen, J. K. (1987). The nature of phonological processing and its causal role in the acquisition of reading skills. Psychological Bulletin, 101(2), 192–212.

    Article  Google Scholar 

  • Wagner, R., Torgesen, J. K., & Rashotte, C. A. (1999). Comprehensive tests of phonological processing (CTOPP). Austin, TX: Pro-Ed.

    Google Scholar 

  • Wechsler, D. (1999). Wechsler abbreviated scale of intelligence (WASI). New York: The Psychological Corporation.

    Google Scholar 

  • Willett, J. B., & Sayer, A. G. (1996). Cross-domain analysis of change over time: Combining growth modeling and covariance structure analysis. In G. A. Marcoulides & R. E. Schumacker (Eds.), Advanced structural equation modeling. Issues and techniques. Mahweh, NJ: Lawrence Erlbaum.

  • Wise, B. W., Ring, J., Sessions, L., & Olson, R. K. (1997). Phonological awareness with and without articulation: A preliminary study. Learning Disability Quarterly, 20(3), 211–225.

    Article  Google Scholar 

  • Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock-Johnson III tests of achievement (WJ-III). Itasca, IL: Riverside Publishing.

    Google Scholar 

  • Yoder, P., & Compton, D. (2004). Identifying predictors of treatment response. Mental Retardation and Developmental Disabilities Research Reviews, 10, 162–168.

    Article  Google Scholar 

Download references

Acknowledgments

Supported by a grant from the Institute for Educational Sciences, Georgia State University (#R324G06005).

Author information

Authors and Affiliations

  1. Departments of Child and Youth Studies and Psychology, Brock University, 500 Glenridge Avenue, St. Catharines, ON, L2S 3A1, Canada

    Jan C. Frijters

  2. Department of Paediatrics, The Hospital for Sick Children, University of Toronto, Toronto, ON, Canada

    Maureen W. Lovett

  3. Department of Psychology, Georgia State University, Atlanta, GA, USA

    Rose A. Sevcik & Robin D. Morris

Authors
  1. Jan C. Frijters
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maureen W. Lovett
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rose A. Sevcik
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Robin D. Morris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jan C. Frijters.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Frijters, J.C., Lovett, M.W., Sevcik, R.A. et al. Four methods of identifying change in the context of a multiple component reading intervention for struggling middle school readers. Read Writ 26, 539–563 (2013). https://doi.org/10.1007/s11145-012-9418-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11145-012-9418-z

Keywords

  • Reading intervention
  • Outcomes
  • Treatment evaluation
  • Reading disability
  • Middleschool children
  • Adolescents
  • Dyslexia