# Who is the expert? Construct and criteria validity of student and teacher ratings of instruction

- 1.5k Downloads
- 111 Citations

## Abstract

In this article, we examine the construct and criteria validity of student and teacher ratings as indicators of instructional features. As parsimonious measures, student and teacher reports are frequently used to assess aspects of the learning environment. From a methodological perspective, however, both approaches have been questioned. Whereas student ratings are occasionally criticised as being undifferentiated and easily influenced by personal preferences, teacher reports are sometimes considered to be biased by self-serving strategies or teaching ideals. Instead of pitting one method against the other, our study aimed at establishing the *specific* value of each approach. The analyses draw on data from a German extension to the 2003 PISA assessment, in which 288 mathematics teachers and their students completed questionnaires using identical items to tap various aspects of instruction. Mean student ratings were computed for each class. Exploratory and confirmatory factor analyses revealed specific conceptual structures for student and teacher ratings, with teachers elaborating on the use of tasks and methods, and students focusing on their teacher’s support in personal and learning matters. Three shared dimensions were also established: occurrence of classroom management problems, degree of cognitive autonomy during lessons, and tempo of interaction. Teacher/student agreement on these dimensions varied across constructs, with considerable agreement on classroom management, low agreement on cognitive autonomy, and no significant agreement on whether or not the interaction tempo was appropriate. Accordingly, correlations with external criteria (student motivation, teacher motivation, mathematical achievement scores, and characteristics of the tasks set in class) also varied across constructs. We conclude that student and teacher ratings are best suited to tapping different aspects of the learning environment, and recommend that care be taken in choosing a data source appropriate for the construct to be measured.

## Keywords

Agreement Classroom instruction Instructional quality Questionnaires Student ratings Teacher ratings Validity## Notes

### Acknowledgements

The research reported is based on data from the COACTIV study (COACTIV: Professional Competence of Teachers, Cognitively Activating Instruction, and the Development of Students’ Mathematical Literacy) directed by Jürgen Baumert (Max Planck Institute for Human Development, Berlin), Werner Blum (Kassel University) and Michael Neubrand (Carl von Ossietzky University, Oldenburg). The project is funded by the German Research Foundation (DFG; BA 1461/2-2) as part of its priority program on School Quality (BIQUA). We thank our co-researchers Martin Brunner, Uta Klusmann, Stefan Krauss, Katrin Löwen and Yi-Miau Tsai for their support in the COACTIV project. We also thank Oliver Lüdtke for his advice and Susannah Goss for her language editing on this article.

## References

- Aleamoni, L. M. (1999). Student rating myths versus research facts from 1924 to 1998.
*Journal of Personnel Evaluation in Education*,*13*(2), 153–166.CrossRefGoogle Scholar - Baumert, J., Kunter, M., Brunner, M., Krauss, S., Blum, W., & Neubrand, M. (2004). Mathematikunterricht aus Sicht der PISA-Schülerinnen und -Schüler und ihrer Lehrkräfte [Mathematics teaching from the perspective of the PISA students and their teachers]. In The German PISA Consortium (Ed.),
*PISA 2003: Der Bildungsstand der Jugendlichen in Deutschland—Ergebnisse des zweiten internationalen Vergleichs*[PISA 2003: The educational level of adolescents in Germany—the second international comparison] (pp. 314–354). Münster, Germany: Waxmann.Google Scholar - Ben-Chaim, D., & Zoller, U. (2001). Self-perception versus students’ perception of teachers’ personal style in college science and mathematics courses.
*Research in Science Education, 31*(3), 437–454.CrossRefGoogle Scholar - Brophy, J. (1999).
*Teaching*. Geneva, Switzerland: International Academy of Education/International Bureau of Education.Google Scholar - Clausen, M. (2002).
*Unterrichtsqualität: Eine Frage der Perspektive?*[Instructional quality: A question of perspectives?] Münster, Germany: Waxmann.Google Scholar - de Corte, E., Greer, B., & Verschaffel, L. (1996). Mathematics teaching and learning. In D. C. Berliner & R. C. Calfee (Eds.),
*Handbook of educational psychology*(pp. 491–549). New York: Simon & Schuster Macmillan.Google Scholar - De Jong, R., & Westerhof, K. J. (2001). The quality of student ratings of teacher behaviour.
*Learning Environments Research, 4*(1), 51–85.CrossRefGoogle Scholar - den Brok, P., Bergen, T., & Brekelmans, M. (2003, April).
*Comparing student’ and teachers’ perceptions of instructional behavior in Dutch secondary education*. Paper presented at the annual meeting of the American Educational Research Association, Chicago.Google Scholar - den Brok, P., Bergen, T., & Brekelmans, M. (2006). Convergence and divergence between students’ and teachers’ perceptions of instructional behaviour in Dutch secondary education. In D. L. Fisher & M. S. Khine (Eds.),
*Contemporary approaches to research on learning environments: World views*(pp. 125–160). Singapore: World Scientific.Google Scholar - Floden, R. E. (2002). The measurement of opportunity to learn. In A. C. Porter, A. Gamoran, & National Research Council/Board on International Comparative Studies in Education (Eds.),
*Methodological advances in cross-national surveys of educational achievement*(pp. 231–266). Washington, DC: National Academy Press.Google Scholar - Fraser, B. (1991). Two decades of classroom environment research. In H. J. Walberg (Eds.),
*Educational environments: Evaluation, antecedents and consequences*(pp. 3–27). Elmsford, NY: Pergamon.Google Scholar - Gentry, M., Gable, R. K., & Rizza, M. G. (2002). Students’ perceptions of classroom activities: Are there grade-level and gender differences?
*Journal of Educational Psychology, 94*(3), 539–544.CrossRefGoogle Scholar - Greenwald, A. G. (1997). Validity concerns and usefulness of student ratings of instruction.
*American Psychologist, 52*(11), 1182–1186.CrossRefGoogle Scholar - Jordan, A., Ross, N., Krauss, S., Neubrand, M., Baumert, J., Brunner, M. et al. (Eds.). (2006). Klassifikationsschema für Mathematikaufgaben: Dokumentation der Aufgabenkategorisierung im COACTIV-Projekt [Classification Scheme for Mathematical Problems: A documentation of the coding system for mathematics tasks in the COACTIV project]. Berlin, Germany: Max Planck Institute for Human Development.Google Scholar
- Klieme, E., Schümer, G., & Knoll, S. (2001). Mathematikunterricht in der Sekundarstufe I: “Aufgabenkultur” und Unterrichtsgestaltung [Mathematical instruction in secondary school: “Task culture” and instructional settings]. In Federal Ministry of Education and Research (Ed.),
*TIMSS—Impulse für Schule und Unterricht*[TIMSS—Impulses for school and classroom instruction] (pp. 43–57). Bonn, Germany: Federal Ministry of Education and Research.Google Scholar - Krauss, S., Kunter, M., Brunner, M., Baumert, J., Blum, W., Neubrand, M. et al. (2004). COACTIV: Professionswissen von Lehrkräften, kognitiv aktivierender Mathematikunterricht und die Entwicklung von mathematischer Kompetenz [Professional knowledge of teachers, cognitively activating mathematics instruction and the development of mathematical literacy]. In J. Doll & M. Prenzel (Eds.),
*Bildungsqualität von Schule: Lehrerprofessionalisierung, Unterrichtsentwicklung und Schülerforderung als Strategien der Qualitätsverbesserung*[School quality: Improving quality through teacher education, instruction, and student support] (pp. 31–53). Münster, Germany: Waxmann.Google Scholar - Kunter, M., & Baumert, J. (2006). Linking TIMSS to research on learning and instruction: A re-analysis of the German TIMSS and TIMSS video data. In S. J. Howie & T. Plomp (Eds.),
*Learning mathematics and science: Lessons learned from TIMSS*(pp. 335–351). London: Routledge.Google Scholar - Kunter, M., Brunner, M., Baumert, J., Klusmann, U., Krauss, S., Blum, W., Jordan, A., & Neubrand, M. (2005). Der Mathematikunterricht der PISA-Schülerinnen und-Schüler: Schulformunterschiede in der Unterrichtsqualität [Quality of mathematics instruction across school types: Findings from PISA 2003].
*Zeitschrift für Erziehungswissenschaft, 8*(4), 502–520.CrossRefGoogle Scholar - Lanahan, L., McGrath, D. J., McLaughlin, M., Burian-Fitzgerald, M., & Salganik, L. (2005). Fundamental problems in the measurement of instructional processes: Estimating reasonable effect sizes and conceptualizing what is important to measure. Washington, DC: American Institutes for Research.Google Scholar
- Marsh, H. W., Trautwein, U., Lüdtke, O., Köller, O., & Baumert, J. (2005). Academic self-concept, interest, grades and standardized test scores: Reciprocal effects models of causal ordering.
*Child Development, 76*(2), 397–416.CrossRefGoogle Scholar - Mayer, D. P. (1999). Measuring instructional practice: Can policy makers trust survey data?
*Educational Evaluation and Policy Analysis, 21*(1), 29–45.CrossRefGoogle Scholar - Mayer, R. E. (2004). Teaching of subject matter.
*Annual Review of Psychology, 55*, 715–744.CrossRefGoogle Scholar - Muthén, L. K., & Muthén, B. O. (1998–2004).
*Mplus user’s guide*(3rd ed.). Los Angeles: Muthén & Muthén.Google Scholar - Porter, A. C. (2002). Measuring the content of instruction: Uses in research and practice.
*Educational Researcher, 31*(7), 3–14.Google Scholar - The German PISA Consortium (Ed.). (2004).
*PISA 2003. Der Bildungsstand der Jugendlichen in Deutschland—Ergebnisse des zweiten internationalen Vergleich*s [PISA 2003: Literacy of Adolescents in Germany]. Münster, Germany: Waxmann.Google Scholar - Urdan, T. (2003, August).
*Using multiple methods to assess students’ perceptions of classroom goal structures*. Paper presented at the 10th Conference of the European Association for Research in Learning and Instruction (EARLI), Padua, Italy.Google Scholar - Weinert, F. E., Schrader, F.-W., & Helmke, A. (1989). Quality of instruction and achievement outcomes.
*International Journal of Educational Research, 13*(8), 895–914.CrossRefGoogle Scholar - Wubbels, T., Brekelmans, M., & Hooymayers, H. P. (1992). Do teacher ideals distort the self-reports of their interpersonal behavior?
*Teaching and Teacher Education, 8*(1), 47–58.CrossRefGoogle Scholar