Skip to main content
SpringerLink
Log in

Increasing the Rigor of Procedural Fidelity Assessment: An Empirical Comparison of Direct Observation and Permanent Product Review Methods

  • Original Paper
  • Published:
Journal of Behavioral Education Aims and scope Submit manuscript

Abstract

Although it is widely accepted that procedural fidelity data are important for making valid decisions about intervention effectiveness, there is little empirical guidance for researchers and practitioners regarding how to assess procedural fidelity. A first step in moving procedural fidelity assessment research forward is to develop a comprehensive understanding of the relation between currently recommended assessment methods, systematic direct observation (SDO) and permanent product review (PPR). To this end, we first discuss methodological issues related to procedural fidelity assessment via SDO and PPR, and how they are evidenced in the behavioral intervention outcome literature. Second, we present a single-case design study in which teachers’ implementation of behavior support plans was assessed via SDO and PPR. Third, we analyze the SDO and PPR data with regard to (a) relation between the methods; (b) methodological issues including definitions of procedural fidelity and intervention steps, representativeness of intervention steps, and collection of an adequate sample; and (c) association with student outcomes. Finally, we discuss implications for research and practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Ball, C. R., & Christ, T. J. (2012). Supporting valid decision making: Uses and misuses of assessment data within the context of RTI. Psychology in the Schools, 49, 231–244.

    Article  Google Scholar 

  • Cochrane, W. S., & Laux, J. M. (2008). A survey investigating school psychologists’ measurement of treatment integrity in school-based interventions and their beliefs about its importance. Psychology in the Schools, 45, 499–507.

    Article  Google Scholar 

  • Codding, R. S., Feinberg, A. B., Dunn, E. K., & Pace, G. M. (2005). Effects of immediate performance feedback on implementation of behavior support plans. Journal of Applied Behavior Analysis, 38, 205–219.

    Article  PubMed  Google Scholar 

  • Codding, R. S., Livanis, A., Pace, G. M., & Vaca, L. (2008). Using performance feedback to improve treatment integrity of classwide behavior plans: An investigation of observer reactivity. Journal of Applied Behavior Analysis, 41, 417–422.

    Article  PubMed  Google Scholar 

  • Collier-Meek, M. A., Fallon, L. M., Sanetti, L. M. H., & Maggin, D. M. (2013). Focus on implementation: Strategies for problem-solving teams to assess and promote treatment fidelity. Teaching Exceptional Children, 45, 52–59.

    Google Scholar 

  • Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45.

    Article  PubMed  Google Scholar 

  • DiGennaro, F. D., Martens, B. K., & Kleinmann, A. E. (2007). A comparison of performance feedback procedures on teachers’ treatment implementation integrity and students’ inappropriate behavior in special education classrooms. Journal of Applied Behavior Analysis, 40, 447–461.

    Article  PubMed  Google Scholar 

  • DiGennaro, F. D., Martens, B. K., & McIntyre, L. L. (2005). Increasing treatment integrity through negative reinforcement: Effects on teacher and student behavior. School Psychology Review, 34, 220–231.

    Google Scholar 

  • Fiske, K. (2008). Treatment integrity of school-based behavior analytic interventions: A review of the research. Behavior Analysis in Practice, 1, 19–25.

    PubMed Central  PubMed  Google Scholar 

  • Gansle, K. A. (2005). The effectiveness of school-based anger interventions and programs: A meta-analysis. Journal of School Psychology, 43, 321–341.

    Article  Google Scholar 

  • Gansle, K. A., & McMahon, C. M. (1997). Component integrity of teacher intervention management behavior using a student self-monitoring treatment: An experimental analysis. Journal of Behavioral Education, 7, 405–419.

    Article  Google Scholar 

  • Gresham, F. M. (1989). Assessment of treatment integrity in school consultation and prereferral intervention. School Psychology Review, 18, 37–50.

    Google Scholar 

  • Gresham, F. M., MacMillan, D. L., Beebe-Frankenberger, M. E., & Bocian, K. M. (2000). Treatment integrity in learning disabilities intervention research: Do we really know how treatments are implemented? Learning Disabilities Research and Practice, 15, 198–205.

    Article  Google Scholar 

  • Hawkins, S. M., & Heflin, L. J. (2011). Increasing secondary teachers’ behavior-specific praise using a video self-modeling and visual performance feedback intervention. Journal of Positive Behavior Interventions, 13, 97–108.

    Article  Google Scholar 

  • Hemmeter, M. L., Snyder, P., Kinder, K., & Artman, K. (2011). Impact of performance feedback delivered via electronic mail on preschool teachers’ use of descriptive praise. Early Childhood Research Quarterly, 26, 96–109.

    Article  Google Scholar 

  • Henggeler, S. W., Melton, G. B., Brondino, M. J., Scherer, D. G., & Hanley, J. H. (1997). Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65, 821–833.

    Article  PubMed  Google Scholar 

  • Hintze, J. M. (2005). Psychometrics of direct observation. School Psychology Review, 34, 507–519.

    Google Scholar 

  • Hirschstein, M. K., Edstrom, L. V., Frey, K. S., Snell, J. L., & MacKenzie, E. P. (2007). Walking the talk in bully prevention: Teacher implementation variable related to initial impact of the Steps to Respect program. School Psychology Review, 36, 3–21.

    Google Scholar 

  • Holcombe, A., Wolery, M., & Snyder, E. (1994). Effects of two levels of procedural fidelity with constant time delay on children’s learning. Journal of Behavioral Education, 4, 49–73.

    Article  Google Scholar 

  • Institute of Education Sciences. (2012). Request for applications: Education research grants. Retrieved from http://ies.ed.gov/funding/ncser_progs.asp.

  • Jones, H. A., Clarke, A. T., & Power, T. J. (2008). Expanding the concept of intervention integrity: A multidimensional model of participant engagement. Balance, 23, 4–5.

    Google Scholar 

  • Jones, K. M., Friman, K. F., & Fryman, P. C. (1997). The effects of observational feedback on treatment integrity in school-based behavioral consultation. School Psychology Quarterly, 12, 316–326.

    Article  Google Scholar 

  • Lane, K. L., Bocian, K. M., MacMillan, D. L., & Gresham, F. M. (2004). Treatment integrity: An essential-but often forgotten-component of school-based interventions. Preventing School Failure, 48, 36–43.

    Article  Google Scholar 

  • McGivern, J. E., & Walter, M. J. (in press). Legal and ethical issues related to treatment integrity in psychology and education. In L. M. H. Sanetti, & T. R. Kratochwill (Eds.), Treatment integrity: Conceptual, methodological, and applied considerations for practitioners and researchers. Washington, DC: American Psychological Association.

  • McIntyre, L. L., Gresham, F. M., DiGennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in Journal of Applied Behavior Analysis studies from 1991 to 2005. Journal of Applied Behavior Analysis, 40, 659–672.

    PubMed  Google Scholar 

  • Noell, G. H. (2008). Research examining the relationships among consultation process, treatment integrity, and outcomes. In W. P. Erchul & S. M. Sheridan (Eds.), Handbook of research in school consultation: Empirical foundations for the field (pp. 315–334). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Noell, G. H., Duhon, G. J., Gatti, S. L., & Connell, J. E. (2002). Consultation, follow-up, and implementation of behavior management interventions in general education. School Psychology Review, 31, 217–234.

    Google Scholar 

  • Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., Williams, K. L., et al. (2005). Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 34, 87–106.

    Google Scholar 

  • O’Neill, R. E., Horner, R. H., Albin, R. W., Sprague, J. R., Storey, K., & Newton, J. S. (1997). Functional assessment and program development for problem behavior. Pacific Grove, CA: Brooks/Cole Publishing.

    Google Scholar 

  • Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75, 829–841.

    Article  PubMed  Google Scholar 

  • Power, T. J., Blom-Hoffman, J., Clarke, A. T., Riley-Tillman, T. C., Kellerher, C., & Manz, P. (2005). Reconceptualizing intervention integrity: A partnership-based framework for linking research with practice. Psychology in the Schools, 42, 495–507.

    Article  Google Scholar 

  • Reinke, W. M., Lewis-Palmer, T., & Martin, E. (2007). The effect of visual performance feedback on teacher behavior-specific praise. Behavior Modification, 31(3), 247–263.

    Article  PubMed  Google Scholar 

  • Sanetti, L. M. H., Chafouleas, S. M., Christ, T. J., & Gritter, K. L. (2009). Extending use of Direct Behavior Rating beyond student assessment: Applications to treatment integrity assessment within a multi-tier model of school-based intervention delivery. Assessment for Effective Intervention, 34, 251–258.

    Article  Google Scholar 

  • Sanetti, L. M. H., Collier-Meek, M. A., Long, A. C. J., Kim, J., & Kratochwill, T. R. (2013a). Using implementation planning to increase teachers’ adherence and quality to behavior support plans (submitted).

  • Sanetti, L. M. H., & DiGennaro Reed, F. M. (2012). Barriers to implementing treatment integrity procedures in school psychology research: Survey of treatment outcome researchers. Assessment for Effective Intervention, 37, 195–202.

    Article  Google Scholar 

  • Sanetti, L. M. H., & Fallon, L. M. (2011). Treatment integrity assessment: How estimates of adherence, quality, and exposure influence interpretation of implementation. Journal of Educational and Psychological Consultation, 21, 209–232.

    Article  Google Scholar 

  • Sanetti, L. M. H., Fallon, L. M., & Collier-Meek, M. A. (2011a). Treatment integrity assessment and intervention by school-based personnel: Practical applications based on a preliminary study. School Psychology Forum, 5, 87–102.

    Google Scholar 

  • Sanetti, L. M. H., Fallon, L. M., & Collier-Meek, M. A. (2013b). Increasing teacher treatment integrity through performance feedback provided by school personnel. Psychology in the Schools, 50, 134–150.

    Article  Google Scholar 

  • Sanetti, L. M. H., Gritter, K. L., & Dobey, L. (2011b). Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. School Psychology Review, 40, 72–84.

    Google Scholar 

  • Sanetti, L. M. H., & Kratochwill, T. R. (2009a). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445–459.

    Google Scholar 

  • Sanetti, L. M. H., & Kratochwill, T. R. (2009b). Treatment integrity assessment in the schools: An evaluation of the Treatment Integrity Planning Protocol (TIPP). School Psychology Quarterly, 24, 24–35.

    Article  Google Scholar 

  • Sanetti, L. M. H., & Kratochwill, T. R. (2011). An evaluation of the treatment integrity planning protocol and two schedules of treatment integrity self-report: Impact on implementation and report accuracy. Journal of Educational and Psychological Consultation, 21, 284–308.

    Article  Google Scholar 

  • Sanetti, L. M. H., & Kratochwill, T. R. (2013). Treatment integrity assessment within the problem-solving model. In R. Brown-Chidsey (Ed.), Assessment for intervention: A problem-solving approach (2nd ed., pp. 297–320). New York: Guilford Publications, Inc.

    Google Scholar 

  • Sanetti, L. M. H., Kratochwill, T. R., & Long, A. C. J. (2013c). Applying adult behavior change theory to support mediator-based intervention implementation. School Psychology Quarterly, 28, 47–62.

    Article  PubMed  Google Scholar 

  • Sanetti, L. M. H., Luiselli, J. K., & Handler, M. W. (2007). Effects of verbal and graphic performance feedback on behavior support plan implementation in public elementary school. Behavior Modification, 31, 454–465.

    Article  Google Scholar 

  • Schulte, A. C., Easton, J., & Parker, J. (2009). Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 38, 460–475.

    Google Scholar 

  • Sheridan, S. M., Bovaird, J. A., Glover, T. A., Garbacz, A., Witte, A., & Kwon, K. (2012). A randomized trail examining the effects of conjoint behavioral consultation and the mediating role of the parent-teacher relationship. School Psychology Review, 41, 23–46.

    Google Scholar 

  • Swanger-Gagne, M. S., Garbacz, A. S., & Sheridan, S. M. (2009). Intervention integrity within conjoint behavioral consultation: Strategies for working with families. School Mental Health, 1, 131–142.

    Article  Google Scholar 

  • Vollmer, T. R., Roane, H. S., Ringdahl, J. E., & Marcus, B. A. (1999). Evaluating treatment challenges with differential reinforcement of alternative behavior. Journal of Applied Behavior Analysis, 32, 9–23.

    Article  Google Scholar 

  • Vollmer, T. R., Sloman, K. N., & Pipkin, C. S. (2008). Practical implications of data reliability and treatment integrity monitoring. Behavior Analysis in Practice, 1, 4–11.

    PubMed Central  PubMed  Google Scholar 

  • Wickstrom, K. F., Jones, K. M., LaFleur, L. H., & Witt, J. C. (1998). An analysis of treatment integrity in school-based behavioral consultation. School Psychology Quarterly, 13, 141–154.

    Article  Google Scholar 

  • Wilder, D. A., Atwell, J., & Wine, B. (2006). The effects of varying levels of treatment integrity on child compliance during treatment with a three-step prompting procedure. Journal of Applied Behavior Analysis, 39, 369–373.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

Research reported here was supposed by the Institute of Education Sciences, U.S. Department of Education, through Grant R324A10005 to the University of Connecticut. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Author information

Authors and Affiliations

  1. Department of Educational Psychology, U-3064, University of Connecticut, Storrs, CT, 06269-3064, USA

    Lisa M. Hagermoser Sanetti & Melissa A. Collier-Meek

Authors
  1. Lisa M. Hagermoser Sanetti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Melissa A. Collier-Meek
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lisa M. Hagermoser Sanetti.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sanetti, L.M.H., Collier-Meek, M.A. Increasing the Rigor of Procedural Fidelity Assessment: An Empirical Comparison of Direct Observation and Permanent Product Review Methods. J Behav Educ 23, 60–88 (2014). https://doi.org/10.1007/s10864-013-9179-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10864-013-9179-z

Keywords

Search

Navigation