Quality & Quantity

, Volume 45, Issue 6, pp 1415–1427 | Cite as

A multivariate method for analyzing and improving the use of student evaluation of teaching questionnaires: a case study

  • Mónica Martínez-Gómez
  • Jose Miguel Carot Sierra
  • José Jabaloyes
  • Manuel Zarzo
Article
First Online:

Abstract

Student evaluation of teaching (SET) questionnaires are the most common methods of evaluation used by European universities to assess the quality of teaching delivered by their lecturers. A series of multivariate statistical methods were applied to analyze the underlying structure of the SET questionnaire used by the Universidad Politecnica de Valencia (UPV) in order to develop an appropriate methodology for extracting, analyzing, and interpreting the information contained in the questionnaire. In a first step, a confirmatory factorial analysis (CFA) was developed in order to evaluate the reliability, validity and dimensionality of it, by means of two relatively new parameters commonly used in structural equation modelling: the compound reliability and extracted variance for each latent construct. In a second step, cluster analysis (CA) was used to test the ability of the questionnaire for the identification of different categories of lecturers. In the last step, a tree classification method, the chi-squared automatic interaction detector (CHAID), was used in order to characterize the different lecturer’s categories obtained with CA according to all available information regarding the teaching staff and subjects.

Keywords

Quality indicator Confirmatory factor analysis Student questionnaire for teaching assessment Cluster analysis CHAID 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Batista, J.M., Coenders, G. (eds): Modelos de Ecuaciones Estructurales. Cuadernos de estadística, 6. La Muralla, Madrid (2000)Google Scholar
  2. Batista J.M., Coenders G., Alonso J.: Análisis Factorial Confirmatorio. Su utilidad en validación de cuestionarios relacionados con la salud. Med. Clin. (Barc.) 122(1), 21–27 (2004)CrossRefGoogle Scholar
  3. Bentler P.M.: Comparative fit indexes in structural models. Psychol. Bull. 107(2), 238–246 (1990)CrossRefGoogle Scholar
  4. Bentler P.M., Bonett D.G.: Significance tests and goodness of fit in the analysis of covariance structures. Psychol. Bull. 88, 588–606 (1980)CrossRefGoogle Scholar
  5. Berk R.A.: Survey of 12 strategies to measure teaching effectiveness. Int. J. Teach. Learn. High. Educ. 17, 48–62 (2005)Google Scholar
  6. Bollen, K.A. (eds): Structural Equations with Latent Variables. Wiley, New York (1989)Google Scholar
  7. Bollen, K.A., Long, J.S. (eds): Testing Structural Equation Models. Sage, Newbury Park (1993)Google Scholar
  8. Brown M.W., Cudeck R.: Alternative ways of assessing model fit. In: Bollen, K.A., Long, J.S. (eds) Testing Structural Equation Models, pp. 136–162. Sage, Newbury Park (1993)Google Scholar
  9. Cajide J.: Análisis factorial confirmatorio de las características de calidad docente universitaria (solución LISREL). Bordón 46(4), 389–405 (1994)Google Scholar
  10. Campbell D.T., Fiske D.W.: Convergent and discriminant validation by the multitrait–multimethod matrix. Psychol. Bull. 56(2), 81–105 (1959)CrossRefGoogle Scholar
  11. Cashin W.E.: Concerns about using student ratings in community colleges. N. Dir. Community Coll. 11(1), 57–65 (1983)Google Scholar
  12. Chiu, T., Fang, D., Chen, J., Wang, Y., Jeris, C.: A robust and scalable clustering algorithm for mixed type attributes in large database environment. In: Proceedings of Seventh ACM SIGKDD, International Conference on Knowledge Discovery and Data Mining, San Francisco, 2001Google Scholar
  13. Feldman K.A.: The significance of circumstances for college student’s ratings of their teachers and courses. Res. High. Educ. 10, 149–172 (1979)CrossRefGoogle Scholar
  14. Göb R., McCollin C., Fernanda Ramalhoto M.: Ordinal methodology in the analysis of likert scales. Qual. Quant. 41, 601–626 (2007)CrossRefGoogle Scholar
  15. González Such, J.: Estudio de un Instrumento para la Evaluación del Profesorado Universitario. Thesis, Universitat de València, Valencia (1997)Google Scholar
  16. Gursoy D., Umbreit W.T.: Exploring student’s evaluations of teaching effectiveness: what factors are important? J. Hosp. Tour. Res. 29(1), 91–109 (2005)CrossRefGoogle Scholar
  17. Hair, J.F., Anderson, R.E., Tatham, R.L., Black, W.C. (eds): Multivariate Data Analysis. Prentice Hall International, New York (1995)Google Scholar
  18. Hu L., Bentler P.M.: Cut off criteria for fit indexes in covariance. Structure analysis: conventional criteria versus new alternatives. Struct. Equ. Model. 6(1), 1–55 (1999)CrossRefGoogle Scholar
  19. Jackson D., Teal C.R., Raines J.S., Nansel T.R., Force R.C., Burdsal C.A.: The dimensions of students’ perceptions of teaching effectiveness. Educ. Psychol. Meas. 59(4), 580–596 (1999)CrossRefGoogle Scholar
  20. Jöreskog K.G.: Testing structural equation models. In: Bollen, K.A., Long, J.S. (eds) Testing Structural Equation Models, pp. 294–316. Sage, Newbury Park (1993)Google Scholar
  21. Jöreskog, K.G., Sörbom, D. (eds): LISREl V. Analysis of Linear Structural Relations by the Method of Maximum Likelihood. SPSS Publications, Chicago (1981)Google Scholar
  22. Jöreskog, K.G., Sörbom, D. (eds): LISREL VI. Analysis of Linear Structural Relationships by Maximum Likelihood, Instrumental Variable, and Least Squares Methods. Scientific Software, Moresville (1986)Google Scholar
  23. Kass G.V.: An exploratory technique for investigating large quantities of categorical data. Appl. Stat. 29(2), 119–127 (1980)CrossRefGoogle Scholar
  24. Lalla M., Facchinetti G., Maestroleo G.: Ordinal scales and fuzzy set systems to measure agreement: an application to the evaluation of teaching activity. Qual. Quant. 38, 577–601 (2004)CrossRefGoogle Scholar
  25. Marsh H.W.: Validity of students’ evaluations of college teaching: a multitrait multimethod analysis. J. Educ. Psychol. 74(2), 219–237 (1982)CrossRefGoogle Scholar
  26. Marsh H.W.: Students’ evaluations of university teaching: dimensionality, reliability, validity, potential biases, and utility. J. Educ. Psychol. 76(5), 707–754 (1984)CrossRefGoogle Scholar
  27. Marsh H.W.: Students’ evaluations of university teaching: research, findings, methodological issues and directions for future research. J. Educ. Res. 11(3), 253–388 (1987)Google Scholar
  28. Marsh H.W.: Weighting for the right criteria in the Instructional Development and Effectiveness Assessment (IDEA) system: global and specific ratings of teaching effectiveness and their relation to course objectives. J. Educ. Psychol. 86, 631–648 (1994)CrossRefGoogle Scholar
  29. Marsh H.W., Hattie J.: The relation between research productivity and teaching effectiveness. J. High. Educ. 73, 603–641 (2002)CrossRefGoogle Scholar
  30. Mulaik S.A., James L.R., Van Alstine J., Bennett N., Lind S., Stilwell C.D.: Evaluation of goodness-of-fit indices for structural equation models. Psychol. Bull. 105(3), 430–445 (1989)CrossRefGoogle Scholar
  31. Rodríguez Sabiote, C., Gutiérrez Pérez, J.: Debilidades de la evaluación de la calidad en la universidad española. Causas, consecuencias y propuestas de mejora. Revista Electron. de Investig. Educ. 5(1) (2003). http://redie.ens.uabc.mx/vol5no1/contenido-sabiote.html. Retrieved 20 April 2005
  32. Ting K.F.: A multilevel perspective on student ratings of instruction: lessons from the Chinese experience. Res. High. Educ. 41, 637–661 (2000)CrossRefGoogle Scholar
  33. Toland M.D., De Ayala R.J.: A multilevel factor analysis of student’s evaluations of teaching. Educ. Psychol. Meas. 65(2), 272–296 (2005)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2010

Authors and Affiliations

  • Mónica Martínez-Gómez
    • 1
  • Jose Miguel Carot Sierra
    • 1
  • José Jabaloyes
    • 1
  • Manuel Zarzo
    • 1
  1. 1.Department of Applied Statistics, Operations Research and QualityUniversidad Politécnica de ValenciaValenciaSpain

Personalised recommendations