Skip to main content

A Generalizability Study of Teach, a Classroom Observation Tool

  • Conference paper
  • First Online:
Quantitative Psychology (IMPS 2020)

Abstract

The use of classroom observation tools has grown in developing countries due to the consistent positive relationship between teaching quality and students’ learning outcomes. In this study we present results using the Generalizability Theory model for Teach, a classroom observation tool that has been used worldwide to measure the quality of interactions between teachers and students in the classroom. Data from four countries across different world regions was used to analyze to what extent Teach scores are impacted by raters, teachers, and items. Results across countries consistently showed a pattern in which most of the score variance is explained by the items and teachers, and to a lower extent by the raters. Results here obtained are similar to those reported for other classroom observation tools. Directions and lines of future psychometric research for Teach and other classroom observation tools are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Araujo, M. C., Carneiro, P., Cruz-Aguayo, Y., & Schady, N. (2016). Teacher quality and learning outcomes in kindergarten. The Quarterly Journal of Economics, 131(3), 1415–1453.

    Article  Google Scholar 

  • Azam, M., & Kingdon, G. G. (2015). Assessing teacher quality in India. Journal of Development Economics, 117, 74–83.

    Article  Google Scholar 

  • Bau, N., & Das, J. (2017). The misallocation of pay and productivity in the public sector: Evidence from the labor market for teachers. The World Bank.

    Google Scholar 

  • Brennan, R. L. (1992). Generalizability theory. Educational Measurement: Issues and Practice, 11(4), 27–34.

    Article  Google Scholar 

  • Brennan, R. L. (2000). Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement, 24(4), 339–353.

    Article  Google Scholar 

  • Buhl-Wiggers, J., Kerwin, J., Smith, J., & Thornton, R. (2017). The impact of teacher effectiveness on student learning in Africa. Paper presented at the Centre for the Study of African Economies Conference 2017.

    Google Scholar 

  • Burchinal, M., Howes, C., Pianta, R., Bryant, D., Early, D., Clifford, R., & Barbarin, O. (2008). Predicting child outcomes at the end of kindergarten from the quality of pre-kindergarten teacher-child interactions and instruction. Applied Developmental Science, 12(3), 140–153.

    Article  Google Scholar 

  • Cadima, J., Leal, T., & Burchinal, M. (2010). The quality of teacher-student interactions: Associations with first graders' academic and behavioral outcomes. Journal of School Psychology, 48(6), 457–482.

    Article  Google Scholar 

  • Core Team, R. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

    Google Scholar 

  • Cronbach, L. J., Nageswari, R., & Gleser, G. C. (1963). Theory of generalizability: A liberation of reliability theory. The British Journal of Statistical Psychology, 16, 137–163.

    Article  Google Scholar 

  • Curby, T. W., Brock, L. L., & Hamre, B. K. (2013). Teachers’ emotional support consistency predicts children’s achievement gains and social skills. Early Education and Development, 24(3), 292–309.

    Article  Google Scholar 

  • Hanushek, E. A., & Rivkin, S. G. (2010). Generalizations about using value-added measures of teacher quality. American Economic Review, 100(2), 267–271.

    Article  Google Scholar 

  • Hatfield, B. E., Hestenes, L. L., Kintner-Duffy, V. L., & O’Brien, M. (2012). Classroom emotional support predicts differences in preschool children’s cortisol and alpha-amylase levels. Early Childhood Research Quarterly, 28(2), 347–356.

    Article  Google Scholar 

  • Kane, T. J., & Staiger, D. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Bill and Melinda Gates Foundation.

    Google Scholar 

  • Kane, T. J., Taylor, E. S., Tyler, J. H., & Wooten, A. (2011). Identifying effective classroom practices using student achievement data. Journal of Human Resources, 46(3), 587–613.

    Article  Google Scholar 

  • Ladics, J., Molina, E., Wilichowski, T, and Yarrow, N. (2018). The measurement crisis: An assessment of how countries measure classroom practices. Working paper. https://riseprogramme.org/sites/default/files/inline-files/Molina.pdf.

    Google Scholar 

  • Linacre, J. (1989). Many-facet Rasch measurement. MESA Press.

    Google Scholar 

  • Mashburn, A. J., Pianta, R. C., Hamre, B. K., Downer, J. T., Barbarin, O. A., Bryant, D., Burchinal, M., Early, D. M., & Howes, C. (2008). Measures of classroom quality in prekindergarten and children’s development of academic, language, and social skills. Child Development, 79(3), 732–749.

    Article  Google Scholar 

  • Mashburn, A. J., Downer, J. T., Rivers, S. E., Brackett, M. A., & Martinez, A. (2014). Improving the power of an efficacy study of a social and emotional learning program: Application of generalizability theory to the measurement of classroom-level outcomes. Prevention Science, 15, 146–155.

    Article  Google Scholar 

  • Molina, E., Fatima, S. F., Ho, A. D., Melo, C., Wilichowksi, T. M., & Pushparatnam, A. (2020). Measuring the quality of teaching practices in primary schools: Assessing the validity of the Teach classroom observation tool in Punjab, Pakistan. Teaching and Teacher Education, 96, 103–121. https://doi.org/10.1016/j.tate.2020.103171

    Article  Google Scholar 

  • Moore, C. T. (2016). gtheory: Apply generalizability theory with R. R package version 0.1.2. URL: https://CRAN.R-project.org/package=gtheory

  • Morris, P., Lloyd, C. M., Millenky, M., Leacock, N., Raver, C., & Bangser, M. (2012). Using classroom management to improve preschoolers’ social and emotional skills: Final impact and implementation findings from the Foundations of Learning Demonstration in Newark and Chicago (Report). MDRC.

    Google Scholar 

  • Muijs, D., Kyriakides, L., van der Werf, G., Creemers, B., Timerley, H., & Earl, L. (2014). State of the art – Teacher effectiveness and professional learning. School Effectiveness and School Improvement, 25(2), 231–256.

    Article  Google Scholar 

  • Rimm-Kaufman, S. E., Curby, T., Grimm, K., Nathanson, L., & Brock, L. (2009). The contribution of children’s self-regulation and classroom quality to children’s adaptive behaviors in the kindergarten classroom. Developmental Psychology, 45(4), 958–972.

    Article  Google Scholar 

  • Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Sage.

    MATH  Google Scholar 

  • Snilstveit, B., Stevenson, J., Menon, R., Phillips, D., Gallagher, E., Geleen, M., Jobse, H., Schmidt, T., & Jimenez, E. (2016). The impact of education programmes on learning and school participation in low- and middle-income countries: A systematic review summary report. Systematic review summary. International Initiative for Impact Evaluation.

    Book  Google Scholar 

Download references

Acknowledgments

Finally, the authors would like to thank the World Bank teams, government counterparts, and partner organizations for their generosity in sharing their data from past measurement efforts. We want to thank Alice Danon for her assistance in the data management for the analyses here presented.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Diego Luna-Bazaldua .

Editor information

Editors and Affiliations

Appendix: Code to Estimate G Models in R Using Teach Data

Appendix: Code to Estimate G Models in R Using Teach Data

The code in R presented here exemplifies the estimation of G model in R. The G model is estimated using the gstudy() function on the set of Teach items arranged in a single data column. Each Teach item is identified by its name. Facets explored include item name (task, t), rater ID (rater, r) and Teacher ID (person, p).

# Call libraries that will be needed for this exercise. library(gtheory) # For generalizability models. ### 1. After calling the data to R, models are estimated. # G Study model with main effects ### Function gstudy() comes from the ‘gtheory’ package. ### Command “data” identifies the data set in R environment. ### Command “formula” defines in an R formula the scores and each of the facets to analyze. Quotation marks have to be used to define the formula when using the gstudy() function. Gmodel1 <- gstudy(data = Data_TR_long, formula = "as.numeric(score) ~ (1 | teachers) + (1 | rater) + (1 | item)" )

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Luna-Bazaldua, D., Molina, E., Pushparatnam, A. (2021). A Generalizability Study of Teach, a Classroom Observation Tool. In: Wiberg, M., Molenaar, D., González, J., Böckenholt, U., Kim, JS. (eds) Quantitative Psychology. IMPS 2020. Springer Proceedings in Mathematics & Statistics, vol 353. Springer, Cham. https://doi.org/10.1007/978-3-030-74772-5_42

Download citation

Publish with us

Policies and ethics