Skip to main content

A new approach to evaluation of university teaching considering heterogeneity of students’ preferences

Higher Education volume 66, pages 153–171 (2013)Cite this article

Abstract

Students’ evaluations of teaching are increasingly used by universities to evaluate teaching performance. However, these evaluations are controversial mainly due to fact that students value various aspects of excellent teaching differently. Therefore, in this paper we propose a new approach to students’ evaluations of university teaching based on data from conjoint analysis. Conjoint analysis is a multivariate technique used to analyze the structure of individuals’ preference. In particular, our approach accounts for different importance students attach to various aspects of teaching. Moreover, it accounts explicitly for heterogeneity arising from students’ preferences, and incorporates it to form comprehensive teaching evaluation score. We have conducted survey and confirmed applicability and efficiency of the proposed approach.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3

References

  • Abrami, P. C., & d’Apollonia, S. (1991). Multidimensional students’ evaluations of teaching effectiveness; The generalizability of “N = 1” research: comment on Marsh (1991). Journal of Educational Psychology, 83, 411–415.

    Article  Google Scholar 

  • Arias, J. T. G. (1996). Conjoint-based preferential segmentation in the design of a new financial service. International Journal of Bank Marketing, 14(3), 30–32.

    Article  Google Scholar 

  • Berk, R. A. (2005). Survey of 12 strategies to measure teaching effectiveness. International Journal of Teaching and Learning in Higher Education, 17(1), 48–62.

    Google Scholar 

  • Biesma, R. G., Pavlova, M., van Merode, G. G., & Groot, W. (2007). Using conjoint analysis to estimate employers’ preferences for key competencies of master level Dutch graduates entering the public health field. Economics of Education Review, 26(3), 375–386.

    Article  Google Scholar 

  • Cashin, W. E., & Downey, R. G. (1992). Using global student rating items for summative evaluation. Journal of Educational Psychology, 84, 563–572.

    Article  Google Scholar 

  • Choi, Y. R., & Shepherd, D. A. (2004). Entrepreneurs’ decision to exploit opportunities. Journal of Management, 30(3), 377–395.

    Article  Google Scholar 

  • Davies, M., Hirschberg, J. G., Lye, J. N., Johnston, C., & McDonald, I. M. (2007). Systematic influences on teaching evaluations: The case for caution. Australian Economic Papers, 46(1), 18–38.

    Article  Google Scholar 

  • Ellis, L., Burke, D. M., Lomire, P., & McCormack, D. R. (2003). Student grades and average ratings of instructional quality: The need for adjustment. The Journal of Educational Research, 97(1), 35–40.

    Article  Google Scholar 

  • Feldman, K. A. (1997). Identifying exemplary teachers and teaching: Evidence from student ratings. In R. P. Perry & J. C. Smart (Eds.), Effective teaching in higher education: Research and practice (pp. 368–395). New York: Agathon Press.

    Google Scholar 

  • Gray, M., & Bergmann, B. R. (2003). Student teaching evaluations: Inaccurate, demeaning, misused. Academe, 89(5), 44–46.

    Article  Google Scholar 

  • Green, P. E., & Rao, V. R. (1971). Conjoint measurement for quantifying judgmental data. Journal of Marketing Research, 8, 355–363.

    Article  Google Scholar 

  • Gursoy, D., & Umbreit, W. T. (2005). Exploring student’s evaluations of teaching effectiveness: What factors are important? Journal of Hospitality and Tourism Research, 29(1), 91–109.

    Article  Google Scholar 

  • Haddad, Y., Haddad, J., Olabi, A., Shuayto, N., Haddad, T., & Toufeili, I. (2007). Mapping determinants of purchase intent of concentrated yogurt (Labneh) by conjoint analysis. Food Quality and Preference, 18, 795–802.

    Article  Google Scholar 

  • Hair, J. F., Anderson, R. E., Tatham, R. L. & Black W.C. (1998). Multivariate data analysis, 5th Ed. Upper Saddle River, New Jersey. Prentice-Hall International.

  • Hensher, D. (2001). The valuation of commuter travel time savings for car drivers: Evaluating alternative model specifications. Transportation, 28, 101–118.

    Article  Google Scholar 

  • Hur, J. S., & Pak, R. J. (2007). Conjoint analysis for the preferred subjects of elementary school computer education. Journal of the Korean Data and Information Science Society, 18(2), 357–364.

    Google Scholar 

  • Kim, C., Choe, S., Choi, C., & Park, Y. (2008). A systematic approach to new mobile service creation. Expert Systems with Applications, 35, 762–771.

    Article  Google Scholar 

  • Kim, A., Son, Y. D., & Sohn, S. Y. (2009). Conjoint analysis of enhanced English Medium instruction for college students. Expert Systems with Applications, 36, 10197–10203.

    Article  Google Scholar 

  • Kulik, J. A. (2001). Student ratings: validity, utility, and controversy. New Directions for Institutional Research, 109, 9–25.

    Article  Google Scholar 

  • Kuzmanovic, M., & Martic, M. (2012a). An approach to competitive product line design using conjoint data. Expert Systems with Application, 39(8), 7262–7269. doi:10.1016/j.eswa.2012.01.097.

    Article  Google Scholar 

  • Kuzmanovic, M., & Martic, M. (2012b). Using conjoint analysis to create superior value to customers. Metalurgia International, 17(2), 93–99.

    Google Scholar 

  • Kuzmanovic, M., Panic, B., & Martic, M. (2011). Identification of key positioning factors in the retail sector: A conjoint analysis approach. African Journal of Business Management, 5(26), 10376–10386. doi:10.5897/AJBM11.535.

    Google Scholar 

  • Kuzmanovic, M., Vujosevic, M., & Martic, M. (2012). Using conjoint analysis to elicit patients’ preferences for public primary care service in serbia. HealthMED, 6(2), 496–504.

    Google Scholar 

  • Levy, D. S. (1995). Modern marketing research techniques and the property professional. Property Management, 13, 33–40.

    Article  Google Scholar 

  • Liaw, S.-H., & Goh, K.-L. (2003). Evidence and control of biases in student evaluations of teaching. The International Journal of Educational Management, 17(1), 37–43.

    Google Scholar 

  • Lin, Y., McKeachie, W. J., & Tucker, D. G. (1984). The use of student ratings in promotion decisions. Journal of Higher Education, 55, 583–589.

    Article  Google Scholar 

  • Luce, R. D., & Tukey, J. W. (1964). Simultaneous conjoint measurement: A new type of fundamental measurement. Journal of Mathematical Psychology, 1, 1–27.

    Article  Google Scholar 

  • Makila, M. (2004). Retaining students in retail banking through price bundling: Evidence from the Swedish Market. European Journal of Operational Research, 155, 299–316.

    Article  Google Scholar 

  • Marsh, H. W. (1987). Students’ evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11, 253–388.

    Article  Google Scholar 

  • Marsh, H. W. (2007). Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness. In R. P. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 319–384). New York: Springer.

    Chapter  Google Scholar 

  • Marsh, H. W., Ginns, P., Morin, A. J. S., Nagengast, B., & Martin, A. J. (2011). Use of student ratings to benchmark universities: Multilevel modeling of responses to the australian course experience questionnaire (CEQ). Journal of Educational Psychology, 103(3), 733–748.

    Article  Google Scholar 

  • Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective. American Psychologist, 52, 1187–1197. doi:10.1037/0003-066X.52.11.1187.

    Article  Google Scholar 

  • Marsh, H. W., & Roche, L. A. (2000). Effects of grading leniency and low workloads on students’ evaluations of teaching: Popular myth, bias, validity, or innocent bystanders? Journal of Educational Psychology, 92, 202–228. doi:10.1037/0022-0663.92.1.202.

    Article  Google Scholar 

  • Martínez-Gómez, M., Sierra, J., Jabaloyes, J., & Zarzo, M. (2011). A multivariate method for analyzing and improving the use of student evaluation of teaching questionnaires: A case study. Quality & Quantity, 45, 1415–1427. doi:10.1007/s11135-010-9345-5.

    Article  Google Scholar 

  • McCallum, L. W. (1984). A meta-analysis of course evaluation data and its use in the tenure decision. Research in Higher Education, 21, 150–158.

    Article  Google Scholar 

  • McKeachie, W. J. (1997). Student ratings: the validity of use. American Psychologist, 52, 1218–1225. doi:10.1037/0003-066X.52.11.1218.

    Article  Google Scholar 

  • Onwuegbuzie, A. J., Daniel, L. G., & Collins, K. M. T. (2009). A meta-validation model for assessing the score validity of student teacher evaluations. Quality & Quantity, 43, 197–209.

    Article  Google Scholar 

  • Orme, B. (2006). sample size issues for conjoint analysis (Chapter 7). Getting started with conjoint analysis: strategies for product design and pricing research. Reprinted from Orme B (2006). Wis: Research Publishers LLC, Madison.

  • Popović, M., Kuzmanović M., & Martić, M. (2012). Using conjoint analysis to elicit employers’ preferences toward key competencies for a business manager position. Management—Journal for Theory and Practice Management, 17(63), 17–26. doi: 10.7595/management.fon.2012.0011.

    Google Scholar 

  • Ryan, M., & Farrar, S. (2000). Using conjoint analysis to elicit preferences for health care. British Medical Journal, 320(7248), 1530–1533.

    Article  Google Scholar 

  • Sohn, S. Y., & Ju, Y. H. (2010). Conjoint analysis for recruiting high quality students for college education. Expert Systems with Applications, 37, 3777–3783.

    Article  Google Scholar 

  • Soutar, G. N., & Turner, J. P. (2002). Students’ preferences for college: A conjoint analysis. International Journal of Educational Management, 16(1), 40–45.

    Article  Google Scholar 

  • Spencer, K. J., & Schmelkin, L. P. (2002). Students’ perspectives on teaching and its evaluation. Assessment & Evaluation in Higher Education, 27, 397–408.

    Article  Google Scholar 

  • Sproule, R. (2000). Student evaluation of teaching: A methodological critique of conventional practices. Education Policy Analysis Archives, 8(50), 125–142.

    Google Scholar 

  • Witte, K. D., & Rogge, N. (2011). Accounting for exogenous influences in performance evaluations of teachers. Economics of Education Review, 30(4), 641–653. doi:10.1016/j.econedurev.2011.02.002.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Organizational Sciences, University of Belgrade, Jove Ilica 154, 11000, Belgrade, Serbia

    Marija Kuzmanovic, Gordana Savic, Milena Popovic & Milan Martic

Authors
  1. Marija Kuzmanovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gordana Savic
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Milena Popovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Milan Martic
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marija Kuzmanovic.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Kuzmanovic, M., Savic, G., Popovic, M. et al. A new approach to evaluation of university teaching considering heterogeneity of students’ preferences. High Educ 66, 153–171 (2013). https://doi.org/10.1007/s10734-012-9596-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10734-012-9596-2

Keywords

  • Students’ evaluation of teaching
  • Conjoint analysis
  • Students’ preferences
  • Criteria importance
  • Heterogeneity
  • Overall rating