Advertisement

Reframing conversations about teacher quality: school and district administrators’ perceptions of the validity, reliability, and justifiability of a new teacher evaluation system

  • Noelle A. PauflerEmail author
  • Chris Clark
Article
First Online:

Abstract

In changing accountability contexts, policymakers are engaging in international dialogue and collaborative efforts with new opportunities to reframe conversations about how to measure teacher quality and to (re) design and implement evaluation systems accordingly to ensure that they are fair, useable, feasible, and accurate. This study examined the lived experiences of school and district administrators in a large, fast-growth, suburban district in the USA regarding their districts’ new teacher evaluation system to better understand their perceptions of the system’s validity and reliability such that justifiable conclusions may be drawn about teachers’ effectiveness. Given concerns regarding validity and reliability, administrators generally discouraged external, high-stakes uses of evaluation results but valued the evaluation process and the data it provides for supporting teacher growth. As part of a larger study including teachers, findings can inform policymakers seeking to reform teacher evaluation frameworks to emphasize professional growth over high-stakes consequences.

Keywords

Teacher evaluation Accountability Educational policy Framework for personnel evaluation Administrator leadership Professional growth 
This is a preview of subscription content, log in to check access.

Notes

References

  1. Adnot, M., Dee, T., Katz, V., & Wyckoff, J. (2017). Teacher turnover, teacher quality, and student achievement in DCPS. Educational Evaluation and Policy Analysis, 39(1), 54–76.  https://doi.org/10.3102/0162373716663646.Google Scholar
  2. Amrein-Beardsley, A. (2008). Methodological concerns about the education value-added assessment system. Educational Researcher, 37(2), 65–75.  https://doi.org/10.3102/0013189X08316420.Google Scholar
  3. Amrein-Beardsley, A. (2014). Rethinking value-added models in education: critical perspectives on tests and assessment-based accountability. New York, NY: Routledge.Google Scholar
  4. Amrein-Beardsley, A., & Collins, C. (2012). The SAS education value-added assessment system (SAS® EVAAS®) in the Houston Independent School District (HISD): intended and unintended consequences. Education Policy Analysis Archives, 20.  https://doi.org/10.14507/epaa.v20n12.2012.
  5. Baker, E. L., Barton, P. E., Darling-Hammond, L., Haertel, E., Ladd, H. F., Linn, R. L., et al. (2010). Problems with the use of student test scores to evaluate teachers. Washington, DC: Economic Policy Institute Retrieved from http://www.epi.org/publication/bp278/. Accessed 5 May 2013.Google Scholar
  6. Ballou, D., & Springer, M. G. (2015). Using student test scores to measure teacher performance: some problems in the design and implementation of evaluation systems. Educational Researcher, 44(2), 77–86.  https://doi.org/10.3102/0013189X15574904.Google Scholar
  7. Berliner, D. C. (2018). Between Scylla and Charybdis: reflections on and problems associated with the evaluation of teachers in an era of metrification. Education Policy Analysis Archives, 26(54).  https://doi.org/10.14507/epaa.26.3820.
  8. Borg, S. (2018). Teacher evaluation: global perspectives and their implications for English language teaching. A literature review. London, UK: British Council Retrieved from https://www.teachingenglish.org.uk/sites/teacheng/files/pub_Teacher_evaluation_Global_perspectives_implications_ELT.pdf. Accessed 21 Dec 2018.Google Scholar
  9. Braun, H. I. (2005). Using student progress to evaluate teachers: a primer on value-added models. Princeton, NJ: Educational Testing Service Retrieved from http://www.ets.org/Media/Research/pdf/PICVAM.pdf. Accessed 5 May 2013.Google Scholar
  10. Briggs, D., & Domingue, B. (2011). Due diligence and the evaluation of teachers: a review of the value-added analysis underlying the effectiveness rankings of Los Angeles Unified School District teachers by the Los Angeles Times. Boulder, CO: National Education Policy Center Retrieved from http://nepc.colorado.edu/publication/due-diligence. Accessed 4 Feb 2019.Google Scholar
  11. Cannata, M., Rubin, M., Goldring, E., Grissom, J. A., Neumerski, C. M., Drake, T. A., & Schuermann, P. (2017). Using teacher effectiveness data for information-rich hiring. Educational Administration Quarterly, 53(2), 180–222.  https://doi.org/10.1177/0013161X16681629.Google Scholar
  12. Centre for Development and Enterprise (CDE). (2015a). Teacher evaluation: lessons from other countries [number 8]. Johannesburg, South Africa: Author. Retrieved from https://www.cde.org.za/teacher-evaluation-lessons-from-other-countries/. Accessed 22 Dec 2018.
  13. Centre for Development and Enterprise (CDE). (2015b). Teacher evaluation in South African schools [number 9]. Johannesburg, South Africa: Author Retrieved from https://www.cde.org.za/teacher-evaluation-in-south-african-schools/. Accessed 22 Dec 2018.
  14. Collins, C., & Amrein-Beardsley, A. (2014). Putting growth and value-added models on the map: a national overview. Teachers College Record, 116(1), 1-34. Retrieved from http://www.tcrecord.org/Content.asp?ContentId=17291. Accessed 04 Feb 2019.
  15. Corcoran, S. P. (2010). Can teachers be evaluated by their students’ test scores? Should they be? The use of value-added measures of teacher effectiveness in policy and practice. Providence, RI: Annenberg Institute for School Reform Retrieved from http://www.annenberginstitute.org/publications/can-teachers-be-evaluated-their-students%E2%80%99-test-scores-should-they-be-use-value-added-me. Accessed 5 May 2013.Google Scholar
  16. Cuevas, R., Ntoumanis, N., Fernandez-Bustos, J. G., & Bartholomew, K. (2018). Does teacher evaluation based on student performance predict motivation, well-being, and ill-being? Journal of School Psychology, 68, 154–162.  https://doi.org/10.1016/j.jsp.2018.03.005.Google Scholar
  17. Danielson, C. (2013). The framework for teaching evaluation instrument. Princeton, NJ: The Danielson Group Retrieved from http://danielsongroup.org/download/?download=448. Accessed 4 Feb 2019.Google Scholar
  18. Darling-Hammond, L. (2013). Getting teacher evaluation right: what really matters for effectiveness and improvement. New York, NY: Teachers College.Google Scholar
  19. Darling-Hammond, L. (2015). Can value added add value to teacher evaluation? Educational Researcher, 44(2), 132–137.  https://doi.org/10.3102/0013189X15575346.Google Scholar
  20. Darling-Hammond, L., Amrein-Beardsley, A., Haertel, E., & Rothstein, J. (2012). Evaluating teacher evaluation. Phi Delta Kappan, 93(6), 8–15.  https://doi.org/10.1177/003172171209300603.Google Scholar
  21. Derrington, M. L. (2016). Implementing teacher evaluation: lattice of leadership. Journal of Research on Leadership Education, 11(2), 181–199.  https://doi.org/10.1177/1942775116658689.Google Scholar
  22. Derrington, M. L., & Campbell, J. W. (2017). Teacher evaluation policy tools: principals’ selective use in instructional leadership. Leadership and Policy in Schools., 17, 568–590.  https://doi.org/10.1080/15700763.2017.1326143.Google Scholar
  23. Derrington, M. L., & Campbell, J. W. (2018). High-stakes teacher evaluation policy: US principals’ perspectives and variations in practice. Teachers and Teaching: Theory and Practice, 24(3), 246–263.  https://doi.org/10.1080/13540602.2017.1421164.Google Scholar
  24. Dodson, R. L. (2017). An analysis of principals’ perceptions of the primary teaching evaluation system used in eight U.S. states. International Journal of Education Policy and Leadership, 12(5), 1–22.  https://doi.org/10.22230/ijelp.2017v12n5a773.Google Scholar
  25. Donaldson, M. L. (2011). Principals’ approaches to developing teacher quality: constraints and opportunities in hiring, assigning, evaluating, and developing teachers. Washington, DC: Center for American Progress Retrieved from http://cdn.americanprogress.org/wp-content/uploads/issues/2011/02/pdf/principal_report.pdf. Accessed 8 Aug 2013.Google Scholar
  26. Erickson, F. (1986). Qualitative methods in research on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching (pp. 119–161). New York, NY: Macmillan.Google Scholar
  27. Every Student Succeeds Act (ESSA) of 2015, Pub. L. No. 114-95, § 129 Stat. 1802. (2015).Google Scholar
  28. Fink, A. (1995). Evaluation for education and psychology. Thousand Oaks, CA: Sage.Google Scholar
  29. Finster, M., & Milanowski, A. (2018). Teacher perceptions of a new performance evaluation system and their influence on practice: a within- and between-school level analysis. Education Policy Analysis Archives, 26(41).  https://doi.org/10.14507/epaa.26.3500.
  30. Flores, M. A., & Derrington, M. L. (2017). School principals’ views of teacher evaluation policy: lessons learned from two empirical studies. International Journal of Leadership in Education, 20(4), 416–431.  https://doi.org/10.1080/13603124.2015.1094144.Google Scholar
  31. Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: strategies for qualitative research. Chicago, IL: Aldine.Google Scholar
  32. Goldring, E., Grissom, J. A., Rubin, M., Neumerski, C. M., Cannata, M., Drake, T., & Schuermann, P. (2015). Make room value added: principals’ human capital decisions and the emergence of teacher observation data. Educational Researcher, 44(2), 96–104.  https://doi.org/10.3102/0013189X15575031.Google Scholar
  33. Harris, D. N. (2011). Value-added measures in education: what every educator needs to know. Cambridge, MA: Harvard Education Press.Google Scholar
  34. Harris, D. N., Ingle, W. K., & Rutledge, S. A. (2014). How teacher evaluation methods matter for accountability: a comparative analysis of teacher effectiveness ratings by principals and teacher value-added measures. American Educational Journal, 51(1), 73–112.  https://doi.org/10.3102/0002831213517130.Google Scholar
  35. Hazi, H. M. (2017). VAM under scrutiny: teacher evaluation litigation in the states. The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 90(5–6), 184–190.  https://doi.org/10.1080/00098655.2017.1366803.Google Scholar
  36. Herlihy, C., Karger, E., Pollard, C., Hill, H. C., Kraft, M. A., Williams, M., & Howard, S. (2014). State and local efforts to investigate the validity and reliability of scores from teacher evaluation systems. Teachers College Record, 116, 1–28.Google Scholar
  37. Hopkins, P. (2016). Teacher voice: how teachers perceive evaluations and how leaders can use this knowledge to help teachers grow professionally. NASSP Bulletin, 100(1), 5–25.  https://doi.org/10.1177/0192636516670771.Google Scholar
  38. Hopkins, E., Hendry, H., Garrod, F., McClare, S., Pettit, D., Smith, L., Burrell, H., & Temple, J. (2016). Teachers’ views of the impact of school evaluation and external inspection processes. Improving Schools, 19(1), 52–61.  https://doi.org/10.1177/1365480215627894.Google Scholar
  39. Jiang, J. Y., Sporte, S. E., & Luppescu, S. (2015). Teacher perspectives on evaluation reform: Chicago’s REACH students. Educational Researcher, 44(2), 105–116.  https://doi.org/10.3102/0013189X15575517.Google Scholar
  40. Johnson, S. M. (2005). The prospects for teaching as a profession. In L. V. Hedges & B. Schneider (Eds.), The social organization of schooling (pp. 72–90). New York, NY: Russell Sage Foundation.Google Scholar
  41. Joint Committee on Standards for Educational Evaluation. (2008). The personnel evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.Google Scholar
  42. Kane, M. T. (2008). Terminology, emphasis, and utility in validation. Educational Researcher, 37(2), 76–82.  https://doi.org/10.3102/0013189X08315390.Google Scholar
  43. Kraft, M. A., & Gilmour, A. F. (2016). Can principals promote teacher development as evaluators? A case study of principals’ views and experiences. Educational Administration Quarterly, 52(5), 711–753.  https://doi.org/10.1177/001316IXI16653445.Google Scholar
  44. Kraft, M. A., & Gilmour, A. F. (2017). Revisiting the widget effect: teacher evaluation reforms and the distribution of teacher effectiveness. Educational Researcher, 46(5), 234–249.  https://doi.org/10.3102/0013189X17718797.Google Scholar
  45. Lavigne, A. L. (2014). Exploring the intended and unintended consequences of high-stakes teacher evaluation on schools, teachers, and students. Teachers College Record, 116, 1–29 Retrieved from http://www.tcrecord.org/library/abstract.asp?contentid=17294. Accessed 24 Jan 2015.Google Scholar
  46. Lavigne, A. L., & Chamberlain, R. W. (2017). Teacher evaluation in Illinois: school leaders’ perceptions and practices. Educational Assessment, Evaluation and Accountability, 29, 179–209.  https://doi.org/10.1007/s11092-016-9250-0.Google Scholar
  47. Lavigne, A. L., & Good, T. L. (2014). Teacher and student evaluation: moving beyond the failure of school reform. New York, NY: Routledge.Google Scholar
  48. Liu, S., & Zhao, D. (2013). Teacher evaluation in China: latest trends and future directions. Educational Assessment, Evaluation and Accountability, 25(3), 231–250.  https://doi.org/10.1007/s11092-013-9168-8.Google Scholar
  49. Loewus, L. (2017). Are states changing course on teacher evaluation? Test-score growth plays lesser role in six states. Education Week, 37(13), 1–17 Retrieved from https://www.edweek.org/ew/articles/2017/11/15/are-states-changing-course-on-teacher. Accessed 28 Feb 2018.Google Scholar
  50. Martinez, F., Taut, S., & Schaaf, K. (2016). Classroom observation for evaluating and improving teaching: an international perspective. Studies in Educational Evaluation, 49, 15–29.  https://doi.org/10.1016/j.stueduc.2016.03.002.Google Scholar
  51. Messick, S. (1975). The standard problem: Meaning and values in measurement and evaluation. American Psychologist, 30, 955–966.  https://doi.org/10.1037/0003-066X.30.10.955.Google Scholar
  52. Messick, S. (1980). Test validity and the ethics of assessment. American Psychologist, 35(11), 1012–1027.  https://doi.org/10.1037/0003-066X.35.11.1012.Google Scholar
  53. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage Publications.Google Scholar
  54. No Child Left Behind (NCLB) Act of 2001, Public Law 107–110, § 115 Stat. 1425. (2002).Google Scholar
  55. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw Hill.Google Scholar
  56. O’Pry, S. C., & Schumacher, G. (2012). New teachers’ perceptions of a standards-based performance appraisal system. Educational Assessment, Evaluation and Accountability, 24(4), 325–350.  https://doi.org/10.1007/s11092-012-9148-4.Google Scholar
  57. Organisation for Economic Co-operation and Development (OECD). (2013). Teachers for the 21st century: Using evaluation to improve teaching. Paris: OECD Publishing.Google Scholar
  58. Papay, J. P. (2010). Different tests, different answers: the stability of teacher value-added estimates across outcome measures. American Educational Research Journal, 48(1), 163–193.  https://doi.org/10.3102/00002831210362589.Google Scholar
  59. Paufler, N. A. (2018a). Declining morale, diminishing autonomy, and decreasing value: principal reflections on a high-stakes teacher evaluation system. International Journal of Education Policy & Leadership, 13(8).  https://doi.org/10.22230/ijepl.2018v13n8a831.
  60. Paufler, N. A. (2018b). The value a teacher evaluation system adds in practice: school administrator and teacher perceptions of system effectiveness. Manuscript submitted for publication.Google Scholar
  61. Popham, W. J. (1988). Educational evaluation (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.Google Scholar
  62. Reddy, L. A., Dudek, C. M., Peters, S., Alperin, A., Kettler, R. J., & Kurz, A. (2018). Teachers’ and school administrators’ attitudes and beliefs of teacher evaluation: a preliminary investigation of high poverty school districts. Educational Assessment, Evaluation and Accountability, 30, 47–40.  https://doi.org/10.1007/s11092-017-9263-3.Google Scholar
  63. Reid, D. B. (2018). School principals acting as middle leaders implementing new teacher evaluation systems. School Leadership & Management., 1–17.  https://doi.org/10.1080/13632434.2018.1508013.
  64. Saldaña, J. (2013). The coding manual for qualitative researchers (2nd ed.). London: Sage.Google Scholar
  65. Smith, M. L. (1997). Mixing and matching: methods and models. New Directions for Evaluation, 74, 73–85.  https://doi.org/10.1002/ev.1073.Google Scholar
  66. Stake, R. E. (1978). The case study method in social inquiry. Educational Researcher, 7(2), 5–8.  https://doi.org/10.3102/0013189X007002005.Google Scholar
  67. Stake, R. E., & Trumbull, D. (1982). Naturalistic generalizations. Review Journal of Philosophy and Social Science, 7, 1–12.Google Scholar
  68. Stewart, V. (2013). Teacher quality: the 2013 International Summit on the Teaching Profession. Asia Society: Partnership for Global Learning Retrieved from https://asiasociety.org/files/teachingsummit2013.pdf. Accessed 21 Dec 2018.Google Scholar
  69. Stewart, V. (2015). Implementing highly effective teacher policy and practice. Asia Society: Partnership for Global Learning Retrieved from https://asiasociety.org/files/2015-istp-report.pdf. Accessed 22 Dec 2018.Google Scholar
  70. Stewart, V. (2016). Teachers’ professional learning and growth: creating the conditions to achieve quality teaching for excellent learning outcomes. Asia Society: Partnership for Global Learning Retrieved from https://asiasociety.org/files/2016-istp-report-small.pdf. Accessed 22 Dec 2018.Google Scholar
  71. Stewart, V. (2018). New challenges and opportunities facing the teaching profession in public education. Asia Society: Partnership for Global Learning Retrieved from https://asiasociety.org/sites/default/files/inline-files/2018-international-summit-on-the-teaching-profession-edu-istp.pdf.Google Scholar
  72. Strauss, A. L., & Corbin, J. (1995). Basics of qualitative research: grounded theory procedures and techniques. Newbury Park, CA: Sage Publications.Google Scholar
  73. Taut, S., & Sun, Y. (2014). The development and implementation of a national, standards-based, multi-method teacher performance assessment system in Chile. Education Policy Analysis Archives, 22(71).  https://doi.org/10.14507/epaa.v22n71.2014.
  74. Taylor, E. S., & Tyler, J. H. (2012). Can teacher evaluation improve teaching? Education Next, 12(4) Retrieved from https://www.educationnext.org/can-teacher-evaluation-improve-teaching/. Accessed 4 Feb 2019.
  75. Tucker, P. D., & Stronge, J. H. (2005). Linking teacher evaluation and student learning. Alexandria, VA: Association for Supervision and Curriculum Development.Google Scholar
  76. United Nations Educational, Scientific and Cultural Organization (UNESCO). (2016). Education 2030. Incheon declaration and framework for action. Retrieved from https://unesdoc.unesco.org/ark:/48223/pf0000245656. Accessed 22 Dec 2018.Google Scholar
  77. United States Department of Education. (2009). Race to the Top program: executive summary. Retrieved from http://www2.ed.gov/programs/racetothetop/executive-summary.pdf. Accessed 6 July 2013.
  78. United States Department of Education. (2010). Teacher Incentive Fund. Retrieved from http://www2.ed.gov/programs/teacherincentive/index.html. Accessed 6 July 2013.
  79. Weisberg, D., Sexton, S., Mulhern, J., & Keeling, D. (2009). The widget effect: our national failure to acknowledge and act of differences in teacher effectiveness (2nd ed.). Brooklyn, NY: The New Teacher Project (TNTP). Retrieved from http://tntp.org/ideas-and-innovations/view/the-widget-effect. Accessed 31 Jan 2018.
  80. Will, M. (2016). Assessing quality of teaching staff still complex despite ESSA’s leeway. Education Week, 36(16), 31–32 Retrieved from http://www.edweek.org/ew/articles/2017/01/04/assessing-quality-of-teaching-staff-still-complex.html?intc=EW-QC17-TOC&_ga=1.138540723.1051944855.1481128421. Accessed 9 Apr 2017.Google Scholar
  81. World Bank. (2013). What matters most in teacher policies? A framework for building a more effective teaching profession. Retrieved from https://openknowledge.worldbank.org/bitstream/handle/10986/20143/901820NWP0no4000Box385307B00PUBLIC0.pdf?sequence=1&isAllowed=y. Accessed 4 Feb 2019.
  82. World Bank. (2018). World development report: learning to realize education’s promise. Washington, DC: World Bank Retrieved from www.worldbank.org/en/publication/wdr2018. Accessed 4 Feb 2019.Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.College of EducationUniversity of North TexasDentonUSA

Personalised recommendations

Cite article

Buy options