Learning Environments Research

, Volume 12, Issue 2, pp 85–99 | Cite as

Statistical tests conducted with school environment data: The effect of teachers being clustered in schools

Original Paper

Abstract

This article discusses the effect of clustering on statistical tests conducted with school environment data. Because most school environment studies involve the collection of data from teachers nested within schools, the hierarchical nature to these data cannot be ignored. In particular, this article considers the influence of intraschool correlations on tests of statistical significance conducted with the individual teacher as the unit of analysis. Theory that adjusts t test scores for nested data in two-group comparisons is presented and applied to school environment data. This article demonstrates that Type I error rates inflate greatly as the intraschool correlation increases. Because data analysis techniques that recognise the clustering of teachers in schools are essential, it is recommended that either multilevel analysis or adjustments to statistical parameters be undertaken in school environment studies involving nested data.

Keywords

Effect of clustering School environment Statistical testing 

References

  1. Aitkin, M., & Longford, N. (1986). Statistical modelling issues in school effectiveness studies (with discussion). Journal of the Royal Statistical Society. Series A (General), 149, 1–43. doi: 10.2307/2981882.CrossRefGoogle Scholar
  2. Alker, H. R. (1969). A typology of ecological fallacies. In H. Dogan & S. Rokkan (Eds.), Quantitative ecological analysis in the social sciences (pp. 69–86). London: MIT press.Google Scholar
  3. Anderson, C. S. (1982). The search for school climate: A review of the research. Review of Educational Research, 52, 368–420.Google Scholar
  4. Anstine-Templeton, R., Johnson, C. E., Lee, H., & Wan, G. (2006). Becoming more efficient: One college’s use of the SLEQ. In D. L. Fisher & M. S. Khine (Eds.), Contemporary approaches to research on learning environments (pp. 313–336). Singapore: World Scientific.Google Scholar
  5. Bailey, M. (1979). The art of positive principalship. Momentum, 10(2), 46–47.Google Scholar
  6. Brookover, W. B., Schweitzer, J. H., Schneider, J. M., Beady, C. H., Flood, P. K., & Wisenbaker, J. M. (1978). Elementary school social climate and school achievement. American Educational Research Journal, 15, 301–318.Google Scholar
  7. Burstein, L. (1980). The analysis of multi-level data in educational research and evaluation. In D. C. Berliner (Ed.), Review of research in education, Vol. 8 (pp. 158–233). Washington, DC: American Educational Research Association.Google Scholar
  8. Cresswell, J., & Fisher, D. L. (1999, April). A school level environment study in Australia. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.Google Scholar
  9. Cronbach, L. J. (1976). Research on classrooms and schools: Formulation of questions, design and analysis (Occasional paper of the Stanford Evaluation Consortium). Palo Alto, CA: Stanford University.Google Scholar
  10. den Brok, P., Brekelmans, M., & Wubbels, T. (2006). Multilevel issues in research using students’ perceptions of learning environments: The case of the Questionnaire on Teacher Interaction. Learning Environments Research, 9, 199–213. doi: 10.1007/s10984-006-9013-9.CrossRefGoogle Scholar
  11. Docker, J. G., Fraser, B. J., & Fisher, D. L. (1989). Differences in the psychosocial work environment of different types of schools. Journal of Research in Childhood Education, 4, 5–17.CrossRefGoogle Scholar
  12. Dorman, J. P. (1994). A study of school and classroom environments in Queensland Catholic secondary schools. Unpublished PhD thesis, Curtin University of Technology, Perth, Australia.Google Scholar
  13. Dorman, J. P. (1996a). School environment questionnaire: An instrument developed for Australian Catholic secondary schools. Journal of Christian Education, 39(1), 31–44.Google Scholar
  14. Dorman, J. P. (1996b). Use of teacher perceptions in school environment research. School Organisation, 16, 187–202. doi: 10.1080/02601369650037985.CrossRefGoogle Scholar
  15. Dorman, J. P. (2002). Classroom environment research: Progress and possibilities. Queensland Journal of Educational Research, 18, 112–140.Google Scholar
  16. Dorman, J. P., Fraser, B. J., & McRobbie, C. J. (1997). Relationship between school-level and classroom-level environment in secondary schools. Journal of Educational Administration, 35, 74–91. doi: 10.1108/09578239710156999.CrossRefGoogle Scholar
  17. Ely, M. J. (1971). The management of schools in New South Wales 1848–1880: Local initiative suppressed? Journal of Educational Administration, 9, 79–97. doi: 10.1108/eb009659.CrossRefGoogle Scholar
  18. Fisher, D. L., & Cresswell, J. (1998). Actual and ideal principal interpersonal behaviour. Learning Environments Research, 1, 231–247. doi: 10.1023/A:1009914102379.CrossRefGoogle Scholar
  19. Fisher, D. L., & Fraser, B. J. (1983). Use of WES to assess science teachers’ perceptions of school environment. European Journal of Science Education, 5, 231–233.Google Scholar
  20. Fisher, D. L., Fraser, B. J., & Wubbels, T. (1993). Interpersonal teacher behavior and school environment. Do you know what you look like? In T. Wubbels & J. Levy (Eds.), Interpersonal relationships in education (pp. 103–112). London: Falmer Press.Google Scholar
  21. Fraser, B. J. (1994). Research on classroom and school climate. In D. Gabel (Ed.), Handbook of research on science teaching and learning (pp. 493–541). New York: Macmillan.Google Scholar
  22. Fraser, B. J. (1998). Science learning environments: Assessments, effects and determinants. In B. J. Fraser & K. G. Tobin (Eds.), International handbook of science education (pp. 527–564). Dordrecht, The Netherlands: Kluwer.Google Scholar
  23. Fraser, B. J. (2007). Classroom learning environments. In S. K. Abell & N. G. Lederman (Eds.), Handbook of research on science education (pp. 103–124). Mahwah, NJ: Erlbaum.Google Scholar
  24. Fraser, B. J., & Rentoul, A. J. (1982). Relationships between school-level and classroom-level environment. The Alberta Journal of Educational Research, 28, 212–225.Google Scholar
  25. Freedman, D. A. (1999). Ecological inference and the ecological fallacy (Report prepared for the International Encyclopedia of the Social and Behavioral Sciences, Technical Report no. 549). Retrieved August 30, 2007 from www.stanford.edu/class/ed260/freedman549.pdf.
  26. Goldstein, H. (2004). Some observations on the definition and estimation of effect sizes. In I. Schagen & K. Elliot (Eds.), But what does it mean? The use of effect sizes in educational research (pp. 67–71). Slough, UK: NFER.Google Scholar
  27. Goldstein, H., Browne, W., & Rasbash, J. (2002). Partitioning variation in multilevel models. Understanding Statistics, 1, 223–231. doi: 10.1207/S15328031US0104_02.CrossRefGoogle Scholar
  28. Gregoire, M., & Algina, J.(2000, April). Reconceptualizing the debate on school climate and students’ academic motivation and achievement: A multilevel analysis. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.Google Scholar
  29. Halawah, I. (2005). The relationship between effective communication of high school principal and school climate. Education, 126, 334–345.Google Scholar
  30. Halpin, A. W., & Croft, D. B. (1963). Organisational climate of schools. Chicago: Midwest Administration Center, University of Chicago.Google Scholar
  31. Hedges, L. V. (2007). Correcting a significance test for clustering. Journal of Educational and Behavioral Statistics, 32, 151–179. doi: 10.3102/1076998606298040.CrossRefGoogle Scholar
  32. Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlations for planning group-randomised trials in education. Educational Evaluation and Policy Analysis, 29, 60–87. doi: 10.3102/0162373707299706.CrossRefGoogle Scholar
  33. Hox, J. J. (2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Erlbaum.Google Scholar
  34. Hudley, C., Daoud, A., Polanco, T., Wright-Castro, R., & Hershberg, R.(2003, April). Student engagement, school climate, and future expectations in high school. Paper presented at the biennial meeting of the Society for Research in Child Development, Tampa, FL.Google Scholar
  35. Johnson, B., & Stevens, J. J. (2001). Exploratory and confirmatory factor analysis of the School Level Environment Questionnaire (SLEQ). Learning Environments Research, 4, 325–344. doi: 10.1023/A:1014486821714.CrossRefGoogle Scholar
  36. Johnson, B., & Stevens, J. (2006). Student achievement and elementary teachers’ perceptions of school climate. Learning Environments Research, 9, 111–122. doi: 10.1007/s10984-006-9007-7.CrossRefGoogle Scholar
  37. Kelley, R. C., Thornton, B., & Daugherty, R. (2005). Relationships between measures of leadership and school climate. Education, 126, 17–25.Google Scholar
  38. Kilian, J. M., Fish, M. C., & Maniago, E. B. (2007). Making schools safe: A system-wide school intervention to increase student prosocial behaviors and enhance school climate. Journal of Applied School Psychology, 23, 1–30. doi: 10.1300/J370v23n01_01.CrossRefGoogle Scholar
  39. Kitsantas, A., Ware, H. W., & Martinez-Arias, R. (2004). Students’ perceptions of school safety: Effects by community, school environment, and substance use variables. The Journal of Early Adolescence, 24, 412–430. doi: 10.1177/0272431604268712.CrossRefGoogle Scholar
  40. Kottkamp, R. B., Mulhern, J. A., & Hoy, W. K. (1987). Secondary school climate: A revision of the OCDQ. Educational Administration Quarterly, 23(3), 31–48. doi: 10.1177/0013161X87023003003.CrossRefGoogle Scholar
  41. Lee, V. E. (2000). Using hierarchical linear modeling to study social contexts: The case of school effects. Educational Psychologist, 35, 125–141. doi: 10.1207/S15326985EP3502_6.CrossRefGoogle Scholar
  42. Loukas, A., & Murphy, J. L. (2007). Middle school student perceptions of school climate: Examining protective functions on subsequent adjustment problems. Journal of School Psychology, 45, 293–309. doi: 10.1016/j.jsp.2006.10.001.CrossRefGoogle Scholar
  43. Loukas, A., Suzuki, R., & Horton, K. D. (2006). Examining school connectedness as a mediator of school climate effects. Journal of Research on Adolescence, 16, 491–502. doi: 10.1111/j.1532-7795.2006.00504.x.CrossRefGoogle Scholar
  44. McEvoy, A., & Welker, R. (2000). Antisocial behavior, academic failure and school climate: A critical review. Journal of Emotional and Behavioral Disorders, 8(3), 130–140. doi: 10.1177/106342660000800301.CrossRefGoogle Scholar
  45. Moos, R. H. (1986). Work environment scale manual (2nd ed.). Palo Alto, CA: Consulting Psychologists Press.Google Scholar
  46. Noonan, J. (2004). School climate and the safe school: Seven contributing factors. Educational Horizons, 83(1), 61–65.Google Scholar
  47. Pepper, K., & Hamilton Thomas, L. (2002). Making a change: The effects of the leadership role on school climate. Learning Environments Research, 5, 155–166. doi: 10.1023/A:1020326829745.CrossRefGoogle Scholar
  48. Peterson, R. L., & Skiba, R. (2001). Creating school climates that prevent violence. Social Studies, 92, 167–175.Google Scholar
  49. Rasbash, J., Steele, F., Browne, W., & Prosser, B. (2005). A user’s guide to MLwiN version 2.0. Bristol, UK: Centre for Multilevel Modelling, University of Bristol.Google Scholar
  50. Raudenbush, S., & Bryk, A. (2002). Hierarchical linear models. Thousand Oaks, CA: Sage.Google Scholar
  51. Raudenbush, S. W., & Willms, J. D. (Eds.). (1991). Schools, classrooms and pupils: International studies of schooling from multilevel perspective. San Diego, CA: Academic.Google Scholar
  52. Rentoul, A. J., & Fraser, B. J. (1983). Development of a school-level environment questionnaire. Journal of Educational Administration, 21(1), 21–39. doi: 10.1108/eb009866.CrossRefGoogle Scholar
  53. Rowe, K. J. (2007). Practical multilevel analysis with MLwiN & LISREL: An integrated course (7th ed.). Melbourne, Australia: Australian Council for Educational Research.Google Scholar
  54. Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.Google Scholar
  55. Tabachnick, B. G., & Fidell, L. S. (2007). Understanding multivariate statistics (5th ed.). Boston: Pearson.Google Scholar
  56. Vail, K. (2005). Create great school climate. Education digest, 71(4), 11.Google Scholar
  57. van den Noortgate, W., Opdenakker, M.-C., & Onghena, P. (2005). The effects of ignoring a level in multilevel analysis. School Effectiveness and School Improvement, 16, 281–303. doi: 10.1080/09243450500114850.CrossRefGoogle Scholar
  58. Vyskocil, J. R., & Goens, G. A. (1979). Collective bargaining and supervision: A matter of climate. Educational Leadership, 37, 175–177.Google Scholar
  59. Walker, S. O., Petrill, S. A., & Plomin, R. (2005). A genetically sensitive investigation of the effects of the school environment and socio-economic status on academic achievement in seven-year-olds. Educational Psychology, 25, 55–73. doi: 10.1080/0144341042000294895.CrossRefGoogle Scholar
  60. Wilson, D. (2004). The interface of school climate and school connectedness and relationships with aggression and victimization. The Journal of School Health, 74, 293–299.CrossRefGoogle Scholar
  61. Wood, N. B., Lawrenz, F. P., Huffman, D., & Schultz, M. (2006). Viewing the school environment through multiple lenses: In search of school-level variables tied to student achievement. Journal of Research in Science Teaching, 43, 237–254. doi: 10.1002/tea.20108.CrossRefGoogle Scholar
  62. Wubbels, T., & Levy, J. (Eds.). (1993). Do you know what you look like? Interpersonal relationships in education. London: Falmer Press.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2009

Authors and Affiliations

  1. 1.School of EducationAustralian Catholic UniversityVirginiaAustralia

Personalised recommendations