Abstract
In light of their training in psychometrics, evaluation, and statistical analysis, school psychologists may have the most knowledge, of any professional in the school setting, of evaluating the impact of educational and psychotherapeutic interventions. School settings provide a natural opportunity for important, real-world, and meaningful data that will enhance the delivery of interventions and decision-making as to who benefits from what intervention under what conditions. In this chapter, we provide detailed guidelines for strategies for the working school psychologist to measure system-level change, classroom-wide change, and individual change. We first present various methodologies for collecting data to evaluate change at all levels of intervention implementation. Next, we describe in detail straightforward approaches to calculate the effectiveness of an intervention. Throughout the chapter, we also review the practical barriers that may exist in the Australian educational system that challenge the collection of outcome data, and we present potential strategies to help manage these barriers.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Albers, C. A., & Kettler, R. J. (2014). Best practices in universal screening. In P. L. Harrison & A. Thomas (Eds.), Best practices in school psychology: Data-based and collaborative decision making (pp. 121–131). Washington, DC: National Association of School Psychologists.
Alberto, P., & Troutman, A. C. (2013). Applied behaviour analysis for teachers (9th ed.). Upper Saddle River,NJ: Pearson Education, Inc.
American Psychological Association. (2002). Criteria for evaluating treatment guidelines. American Psychologist, 57, 1052–1059. doi:10.1037/0003-066X.57.12.1052.
Australian Psychological Society. (2013). Framework for the effective delivery of school psychological services. Flinders Lane, VIC: The Australian Psychological Society Limited.
Batsche, G. M., Elliot, J., Graden, J. L., Grimes, J., Kovaleski, J. F., Prasse, D., et al. (2005). Response to intervention: Policy considerations and implementation. Alexandria, VA: National Association of State Directors of Special Education.
Berryhill, J. C., & Prinz, R. J. (2003). Environmental interventions to enhance student adjustment: Implications for prevention. Prevention Science, 4, 65–87.
Bradshaw, C., Mitchell, M., & Leaf, P. (2010). Examining the effects of school-wide positive behavioural interventions and supports on student outcomes. Journal of Positive Behavior Interventions, 12, 133–148.
Brown, R., Steege, M. W., & Bickford, R. (2014). Best practices in evaluating the effectiveness of interventions using single-case methods. In P. L. Harrison & A. Thomas (Eds.), Best practices in school psychology: Foundations (pp. 371–380). Washington, DC: National Association of School Psychologists.
Campbell, J. M., & Hammond, R. K. (2014). Best practices in rating scale assessment of children’s behaviour. In P. L. Harrison & A. Thomas (Eds.), Best practices in school psychology: Data-based and collaborative decision making (pp. 287–304). Washington, DC: National Association of School Psychologists.
Cappella, E., Massetti, G. M., & Yampolsky, S. (2009). Rigorous, responsive, and responsible: Experimental designs in school intervention research. In L. M. Dinella (Ed.), Conducting science based psychology research in schools (pp. 51–78). Washington, DC: American Psychological Association. doi:10.1037/11881-003.
Castillo, J. M. (2014). Best practices in program evaluation in a model of response to intervention/multitiered system of supports. In P. L. Harrison & A. Thomas (Eds.), Best practices in school psychology: Foundations (pp. 329–342). Washington, DC: National Association of School Psychologists.
Chafouleas, S. M., Riley-Tillman, T. C., & Christ, T. J. (2009). Direct Behavior Rating (DBR): An emerging method for assessing social behaviour within a tiered intervention system. Assessment for Effective Intervention, 34, 195–200. doi:10.1177/1534508409340391.
Commonwealth of Australia. Australian Education Act 2013, No. 67, 2013. Retrieved from https://www.comlaw.gov.au/Details/C2013A00067/Html/Text#_Toc361054816
Cone, J. D. (2001). Evaluating outcomes: Empirical tools for effective practice. Washington, DC: American Psychological Association. doi:10.1037/10384-000.
Cooke, N., Heward, W. L., Test, D. W., Spooner, F., & Courson, F. H. (1991). Student performance data in the classroom: Measurement and evaluation of student progress. Teacher Education and Special Education, 14, 155–161. doi:10.1177/088840649101400301.
Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behaviour analysis (2nd ed.). Upper Saddle River, NJ: Pearson Prentice Hall.
Cordray, D. S., & Pion, G. M. (2006). Treatment strength and integrity: Models and methods. In R. R. Bootzin & P. E. McKnight (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 103–124). Washington, DC: American Psychological Association. doi:10.1037/11384-006.
Cummings, J. A., & Martinez, R. S. (2012). Visual representation of progress monitoring and academic achievement data. In R. Brown-Chidsey & K. J. Andren (Eds.), Assessment for intervention: A problem-solving approach (2nd ed., pp. 321–343). New York, NY: Guilford Press.
Deno, S. L. (1986). Formative evaluation of individual student programs. A new role for school psychologists. School Psychology Review, 15, 358–374.
Dix, K. L., Slee, P. T., Lawson, M. J., & Keeves, J. P. (2011). Implementation quality of whole-school mental health promotion and students’ academic performance. Child and Adolescent Mental Health, 17(1), 45–51.
Durlak, J. A., & Dupre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting the implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0.
Erchul, W. P., & Martens, B. K. (2010). School consultation: Conceptual and empirical bases of practice (3rd ed.). New York, NY: Springer Science + Business Media.
Fagan, T. K., & Wise, P. S. (2000). Chapter 4: Roles and functions of school psychologists (School psychology: Past, present, and future). Bethesda, MD: NASP Publications.
Farlow, L., & Snell, M. (1994). Making the most of student performance data. Washington, DC: American Association on Mental Retardation.
Fuchs, L. S., & Deno, S. L. (1991). Paradigmatic distinctions between instructionally relevant measurement models. Exceptional Children, 57(6), 488–500.
Fuchs, L., & Fuchs, D. (1986). Effects of systematic formative evaluation: A meta-analysis. Exceptional Children, 53, 199–208.
Gibbons, K., & Brown, S. (2014). Best practices in evaluating psychoeducational services based on student outcome data. In P. L. Harrison & A. Thomas (Eds.), Best practices in school psychology: Foundations (pp. 355–370). Washington, DC: National Association of School Psychologists.
Glover, T. A., & DiPerna, J. C. (2007). Service delivery for response to intervention: Core components and directions for future research. School Psychology Review, 36(4), 526–540.
Goffreda, C. T., & DiPerna, J. C. (2010). An empirical review of psychometric evidence for the Dynamic Indicators of Basic Early Literacy Skills. School Psychology Review, 39(3), 463–483.
Gresham, F. M., & Elliott, S. N. (2008). Social skills improvement system: Rating scales. Bloomington, MN: Pearson Assessments.
Hamre, B. K., & Pianta, R. C. (2007). Learning opportunities in preschool and early elementary classrooms. In R. Pianta, M. Cox, & K. Snow (Eds.), School readiness & the transition to school (pp. 49–84). Baltimore: Brookes.
Hamre, B. K., Pianta, R. C., & Chomat-Mooney, L. (2009). Conducting classroom observations in school-based research. In L. M. Dinella (Ed.), Conducting science based psychology research in schools (pp. 79–105). Washington, DC: American Psychological Association. doi:10.1037/11881-004.
Horner, R. H., & Spaulding, S. A. (2010). Single-case research designs. In N. J. Salkind (Ed.), Encyclopedia of research design (pp. 1386–1394). Thousand Oaks, CA: SAGE.
Howell, K. W., & Hosp, J. L. (2014). Best practices in curriculum-based evaluation. In P. L. Harrison & A. Thomas (Eds.), Best practices in school psychology: Data-based and collaborative decision making (pp. 159–171). Washington, DC: National Association of School Psychologists.
Howell, M. K., Hosp, J. L., & Howell, K. W. (2007). The ABCs of CBM: A practical guide to curriculum-based measurement. New York, NY: Guilford Press.
Howell, K. W., Hosp, J. L., & Kurns, S. (2008). Best practices in curriculum-based evaluation. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology (Vth ed., pp. 349–362). Bethesda, MD: National Association of School Psychologists.
Individuals With Disabilities Education Act, 20 U.S.C. § 1400 (2004).
Kazdin, A. E. (1981). Drawing valid inferences from case studies. Journal of Consulting and Clinical Psychology, 49, 183–192. doi:10.1037/0022-006X.49.2.183.
Kazdin, A. E. (1998). Research design in clinical psychology (3rd ed.). Boston: Allyn & Bacon.
Kazdin, A. E. (2011). Single-case research designs: Methods for clinical and applied settings (2nd ed.). New York, NY: Oxford University Press.
Kennedy, C. (2005). Single-case designs for educational research. Boston: Allyn & Bacon.
Kettler, R. J., Glover, T. A., Albers, C. A., & Feeney-Kettler, K. A. (2014). An introduction to universal screening in educational settings. In R. J. Kettler, T. A. Glover, C. A. Albers, & K. A. Feeney-Kettler (Eds.), Universal screening in educational settings: Evidence-based decision making for schools (pp. 3–16). Washington, DC: American Psychological Association. doi:10.1037/14316-001.
Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M. et al. (2010). Single-case designs technical documentation. In What Works Clearinghouse: Procedures and standards handbook (version 2.0). Retrieved from What Works Clearinghouse website: http://ies.ed.gov/ncee/wwc/pdf/wwc_procedures_v2_standards_handbook.pdf.
Kratochwill, T. R., Hitchock, J. H., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., et al. (2013). Single-case intervention research design standards. Remedial and Special Education, 34, 26–38. doi:10.1177/0741932512452794.
Kratochwill, T. R., & Piersel, W. C. (1983). Time-series research: Contributions to empirical clinical practice. Behavioral Assessment, 5, 165–176.
Mager, R. (1997). Preparing instructional objectives (3rd ed.). Atlanta, GA: The Center for Effective Performance, Inc.
Martens, B. K., & McIntyre, L. L. (2009). The importance of treatment integrity in school-based behavioural interventions. In A. Akin-Little, S. Little, M. Bray, & T. Kehle (Eds.), Behavioral interventions in schools: Evidence-based positive strategies (pp. 59–71). Washington, DC: American Psychological Association. doi:10.1037/11886-004.
McConaughy, S. H., & Ritter, D. R. (2008). Best practices in multimethod assessment of emotional and behavioural disorders. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology (Vth ed., pp. 697–715). Bethesda, MD: National Association of School Psychologists.
McDougal, J. L., Bardos, A. N., & Meier, S. T. (2010). Behavior intervention monitoring assessment system (BIMAS). Toronto: Multi-Health Systems.
McKevitt, B. C., & Braaksma Fynaardt, A. (2014). Best practices in developing a positive behaviour support system at the school level. In P. L. Harrison & A. Thomas (Eds.), Best practices in school psychology: Systems-level services (pp. 165–180). Washington, DC: National Association of School Psychologists.
Merrell, K. W. (2008). Behavioral, social, and emotional assessment of children and adolescents (3rd ed.). New York, NY: Lawrence Erlbaum Associates.
Miltenberger, R. G. (2012). Behavior modifications: Principles and procedures (5th ed.). Belmont, CA: Thomson/Wadsworth.
National Assessment Program – Literacy and Numeracy (NAPLAN). (2013). Retrieved from http://www.nap.edu.au/.
National Association of School Psychologists. (2010). Model for comprehensive and integrated school psychological services. Bethesda, MD: National Association of School Psychologists. Retrieved from: http://www.nasponline.org/standards/2010standards/2_PracticeModel.pdf.
Neely, L., Davis, H., Davis, J., & Rispoli, M. (2015). Review of reliability and treatment integrity trends in autism-focused research. Research in Autism Spectrum Disorders, 9(1), 1–12. doi:10.1016/j.rasd.2014.09.011.
No Child Left Behind (NCLB) Act of 2001, Pub. L. No. 107-110, § 115, Stat. 1425 (2002).
Reynolds, C. R., & Kamphaus, R. W. (2009). BASC-2 progress monitor manual. Bloomington, MN: Pearson.
Riley-Tillman, C., & Burns, M. K. (2009). Evaluating educational interventions: Single-case designs for measuring response to intervention. New York, NY: Guilford Press.
Sandall, S. R., Schwartz, I. S., & Lacroix, B. (2004). Interventionists’ perspectives about data collection in integrated early childhood classrooms. Journal of Early Intervention, 26, 161–174. doi:10.1177/105381510402600301.
Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction To the special series. School Psychology Review, 38, 445–459. Retrieved from: http://www.nasponline.org/.
Sanetti, L. M., & Kratochwill, T. R. (Eds.). (2014). Treatment integrity: A foundation for evidence-based practice in applied psychology. Washington, DC: American Psychological Association.
Shapiro, E. S. (2003). Behavioral Observation of Students in Schools (BOSS). Computer Software. San Antonio, TX: Psychological Corporation.
Shapiro, E. S., & Heick, P. F. (2004). School psychologist assessment practices in the evaluation of students referred for social/behavioural/emotional problems. Psychology in the Schools, 41, 551–561. doi:10.1002/pits.10176.
Sheridan, S. M., Swanger-Gagné, M., Welch, G. W., Kwon, K., & Garbacz, S. A. (2009). Fidelity measurement in consultation: Psychometric issues and preliminary examination. School Psychology Review, 38(4), 476–495.
Shinn, M. R. (Ed.). (1998). Advanced applications of curriculum-based measurement. New York: Guilford.
Shinn, M. R. (2007). Identifying students at risk, monitoring performance, and determining eligibility within response to intervention: Research on educational need and benefit from academic intervention. School Psychology Review, 36, 601–617.
Shinn, M. R. (2010). Building a scientifically based data system for progress monitoring and universal screening across three tiers, including RTI using curriculum-based measurement. In M. R. Shinn & H. M. Walker (Eds.), Interventions for achievement and behaviour problems in a three-tier model including RTI (pp. 259–292). Bethesda, MD: National Association of School Psychologists.
Slee, P. T., Lawson, M. J., Russell, A., Askell-Williams, H., Dix, K. L., Owens, L., et al. (2009). KidsMatter Primary Evaluation Final Report. Centre for Analysis of Educational Futures, Flinders University of South Australia.
Stecker, P. M., Fuchs, L. S., & Fuchs, D. (2005). Using curriculum-based measurement to improve student achievement: Review of research. Psychology in the Schools, 42(8), 795–819. doi:10.1002/pits.20113.
Stoiber, K. C., & Kratochwill, T. R. (2002). Outcomes: Planning, monitoring, evaluating. San Antonio, TX: The Psychological Corporation.
Sugai, G., & Horner, R. H. (2006). A promising approach for expanding and sustaining school-wide positive behaviour support. School Psychology Review, 35, 245–259.
Sugai, G., Horner, R. H., & Lewis-Palmer, T. (2001). Team implementation checklist. Eugene, OR: Educational and Community Supports, University of Oregon.
Sugai, G., Lewis-Palmer, T. L., Todd, A. W., & Horner, R. H. (2001). School-wide evaluation tool (SET). Eugene, OR: Center for Positive Behavioral Supports, University of Oregon.
Technical Assistance Center on Positive Behavioral Interventions and Supports. (2010). Implementation blueprint and self-assessment: Positive behavioural interventions and supports. Washington, DC: Technical Assistance Center on Positive Behavioral Interventions and Supports. Retrieved from http://www.pbis.org/common/pbisresources/publications/SWPBS_ImplementationBlueprint_vSep_23_2010.pdf.
University of Oregon Center on Teaching and Learning. (2009). Dynamic indicators of basic early literacy skills. Retrieved January 10, 2009, from http://dibels.uoregon.edu/.
Upah, K. R. F. (2008). Best practices in designing, implementing, and evaluating quality interventions. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology (Vth ed., pp. 209–223). Bethesda, MD: National Association of School Psychologist.
VanDerHeyden, A. M., & Burns, M. K. (2005). Using curriculum-based assessment and curriculum-based measurement to guide elementary mathematics instruction: Effect on individual and group accountability scores. Assessment for Effective Intervention, 30(3), 15–31. doi:10.1177/073724770503000302.
VanDerHeyden, A., & Harvey, M. (2013). Using data to advance learning outcomes in schools. Journal of Positive Behavior Interventions, 15(4), 205–213. doi:10.1177/1098300712442387.
Vollmer, T. R., Roane, H. S., Ringdahl, J. E., & Marcus, B. A. (1999). Evaluating treatment challenges with differential reinforcement of alternative behavior. Journal of Applied Behavior Analysis, 32(1), 9–23. doi:10.1901/jaba.1999.32-9.
Volpe, R. J., DiPerna, J. C., Hintze, J. M., and Shapiro, E. S. (2005). Observing students in classroom settings: A review of seven coding schemes. School Psychology Review, 34, 454–474.
Waldron, N., & Prus, J. (2006). A guide for performance-based assessment, accountability, and program development in school psychology training programs (2nd ed.). Bethesda, MD: National Association of School Psychologists.
Walker, H. M., & Shinn, M. R. (2010). Systemic, evidence-based approaches for promoting positive student outcomes within a multitier framework: Moving from efficacy to effectiveness. In M. R. Shinn & H. M. Walker (Eds.), Interventions for achievement and behaviour problems in a three-tier model including RTI (pp. 1–26). Bethesda, MD: National Association of School Psychologists.
Watson, T. S., & Watson, T. S. (2009). Behavioral assessment in the schools. In A. Akin-Little, S. G. Little, M. A. Bray, & T. J. Kehle (Eds.), Behavioral interventions in schools: Evidence-based positive strategies (pp. 27–41). Washington, DC: American Psychological Association. doi:10.1037/11886-002.
Wilder, D. A., Atwell, J., & Wine, B. (2006). The effects of varying levels of treatment integrity on child compliance during treatment with a three-step prompting procedure. Journal of Applied Behavior Analysis, 39, 369–373. doi:10.1901/jaba.2006.144-05.
Wolery, M., Bailey, D. B., & Sugai, G. M. (1988). Effective teaching: Principles and procedures of applied behaviour analysis with exceptional children. Boston: Allyn & Bacon.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Chezan, L.C., Kratochwill, T.R., Terjesen, M.D., Nguyen, K.V.H. (2017). Measuring Outcomes in Schools. In: Thielking, M., Terjesen, M. (eds) Handbook of Australian School Psychology. Springer, Cham. https://doi.org/10.1007/978-3-319-45166-4_35
Download citation
DOI: https://doi.org/10.1007/978-3-319-45166-4_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-45164-0
Online ISBN: 978-3-319-45166-4
eBook Packages: Behavioral Science and PsychologyBehavioral Science and Psychology (R0)