Abstract
Student evaluation of teaching (SET) questionnaires are the most common methods of evaluation used by European universities to assess the quality of teaching delivered by their lecturers. A series of multivariate statistical methods were applied to analyze the underlying structure of the SET questionnaire used by the Universidad Politecnica de Valencia (UPV) in order to develop an appropriate methodology for extracting, analyzing, and interpreting the information contained in the questionnaire. In a first step, a confirmatory factorial analysis (CFA) was developed in order to evaluate the reliability, validity and dimensionality of it, by means of two relatively new parameters commonly used in structural equation modelling: the compound reliability and extracted variance for each latent construct. In a second step, cluster analysis (CA) was used to test the ability of the questionnaire for the identification of different categories of lecturers. In the last step, a tree classification method, the chi-squared automatic interaction detector (CHAID), was used in order to characterize the different lecturer’s categories obtained with CA according to all available information regarding the teaching staff and subjects.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Batista, J.M., Coenders, G. (eds): Modelos de Ecuaciones Estructurales. Cuadernos de estadística, 6. La Muralla, Madrid (2000)
Batista J.M., Coenders G., Alonso J.: Análisis Factorial Confirmatorio. Su utilidad en validación de cuestionarios relacionados con la salud. Med. Clin. (Barc.) 122(1), 21–27 (2004)
Bentler P.M.: Comparative fit indexes in structural models. Psychol. Bull. 107(2), 238–246 (1990)
Bentler P.M., Bonett D.G.: Significance tests and goodness of fit in the analysis of covariance structures. Psychol. Bull. 88, 588–606 (1980)
Berk R.A.: Survey of 12 strategies to measure teaching effectiveness. Int. J. Teach. Learn. High. Educ. 17, 48–62 (2005)
Bollen, K.A. (eds): Structural Equations with Latent Variables. Wiley, New York (1989)
Bollen, K.A., Long, J.S. (eds): Testing Structural Equation Models. Sage, Newbury Park (1993)
Brown M.W., Cudeck R.: Alternative ways of assessing model fit. In: Bollen, K.A., Long, J.S. (eds) Testing Structural Equation Models, pp. 136–162. Sage, Newbury Park (1993)
Cajide J.: Análisis factorial confirmatorio de las características de calidad docente universitaria (solución LISREL). Bordón 46(4), 389–405 (1994)
Campbell D.T., Fiske D.W.: Convergent and discriminant validation by the multitrait–multimethod matrix. Psychol. Bull. 56(2), 81–105 (1959)
Cashin W.E.: Concerns about using student ratings in community colleges. N. Dir. Community Coll. 11(1), 57–65 (1983)
Chiu, T., Fang, D., Chen, J., Wang, Y., Jeris, C.: A robust and scalable clustering algorithm for mixed type attributes in large database environment. In: Proceedings of Seventh ACM SIGKDD, International Conference on Knowledge Discovery and Data Mining, San Francisco, 2001
Feldman K.A.: The significance of circumstances for college student’s ratings of their teachers and courses. Res. High. Educ. 10, 149–172 (1979)
Göb R., McCollin C., Fernanda Ramalhoto M.: Ordinal methodology in the analysis of likert scales. Qual. Quant. 41, 601–626 (2007)
González Such, J.: Estudio de un Instrumento para la Evaluación del Profesorado Universitario. Thesis, Universitat de València, Valencia (1997)
Gursoy D., Umbreit W.T.: Exploring student’s evaluations of teaching effectiveness: what factors are important? J. Hosp. Tour. Res. 29(1), 91–109 (2005)
Hair, J.F., Anderson, R.E., Tatham, R.L., Black, W.C. (eds): Multivariate Data Analysis. Prentice Hall International, New York (1995)
Hu L., Bentler P.M.: Cut off criteria for fit indexes in covariance. Structure analysis: conventional criteria versus new alternatives. Struct. Equ. Model. 6(1), 1–55 (1999)
Jackson D., Teal C.R., Raines J.S., Nansel T.R., Force R.C., Burdsal C.A.: The dimensions of students’ perceptions of teaching effectiveness. Educ. Psychol. Meas. 59(4), 580–596 (1999)
Jöreskog K.G.: Testing structural equation models. In: Bollen, K.A., Long, J.S. (eds) Testing Structural Equation Models, pp. 294–316. Sage, Newbury Park (1993)
Jöreskog, K.G., Sörbom, D. (eds): LISREl V. Analysis of Linear Structural Relations by the Method of Maximum Likelihood. SPSS Publications, Chicago (1981)
Jöreskog, K.G., Sörbom, D. (eds): LISREL VI. Analysis of Linear Structural Relationships by Maximum Likelihood, Instrumental Variable, and Least Squares Methods. Scientific Software, Moresville (1986)
Kass G.V.: An exploratory technique for investigating large quantities of categorical data. Appl. Stat. 29(2), 119–127 (1980)
Lalla M., Facchinetti G., Maestroleo G.: Ordinal scales and fuzzy set systems to measure agreement: an application to the evaluation of teaching activity. Qual. Quant. 38, 577–601 (2004)
Marsh H.W.: Validity of students’ evaluations of college teaching: a multitrait multimethod analysis. J. Educ. Psychol. 74(2), 219–237 (1982)
Marsh H.W.: Students’ evaluations of university teaching: dimensionality, reliability, validity, potential biases, and utility. J. Educ. Psychol. 76(5), 707–754 (1984)
Marsh H.W.: Students’ evaluations of university teaching: research, findings, methodological issues and directions for future research. J. Educ. Res. 11(3), 253–388 (1987)
Marsh H.W.: Weighting for the right criteria in the Instructional Development and Effectiveness Assessment (IDEA) system: global and specific ratings of teaching effectiveness and their relation to course objectives. J. Educ. Psychol. 86, 631–648 (1994)
Marsh H.W., Hattie J.: The relation between research productivity and teaching effectiveness. J. High. Educ. 73, 603–641 (2002)
Mulaik S.A., James L.R., Van Alstine J., Bennett N., Lind S., Stilwell C.D.: Evaluation of goodness-of-fit indices for structural equation models. Psychol. Bull. 105(3), 430–445 (1989)
Rodríguez Sabiote, C., Gutiérrez Pérez, J.: Debilidades de la evaluación de la calidad en la universidad española. Causas, consecuencias y propuestas de mejora. Revista Electron. de Investig. Educ. 5(1) (2003). http://redie.ens.uabc.mx/vol5no1/contenido-sabiote.html. Retrieved 20 April 2005
Ting K.F.: A multilevel perspective on student ratings of instruction: lessons from the Chinese experience. Res. High. Educ. 41, 637–661 (2000)
Toland M.D., De Ayala R.J.: A multilevel factor analysis of student’s evaluations of teaching. Educ. Psychol. Meas. 65(2), 272–296 (2005)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Martínez-Gómez, M., Sierra, J.M.C., Jabaloyes, J. et al. A multivariate method for analyzing and improving the use of student evaluation of teaching questionnaires: a case study. Qual Quant 45, 1415–1427 (2011). https://doi.org/10.1007/s11135-010-9345-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-010-9345-5