Research in Higher Education

, Volume 50, Issue 1, pp 73–100 | Cite as

Profiling Teacher/Teaching Using Descriptors Derived from Qualitative Feedback: Formative and Summative Applications

  • Daphne Pan
  • Gary S. H. Tan
  • Kiruthika Ragupathi
  • Krishna Booluck
  • Rita Roop
  • Yuen K. Ip
Article

Abstract

Considerable work has been done on student evaluation of teaching/teachers, but reservations remain about its use for summative purposes. Student ratings are not universally accepted as being reliable, nor can they provide really meaningful information. Qualitative comments can provide a better understanding but they tend not to be user-friendly from lack of structure and connectedness. This study attempts to devise a method for ‘quantifying’ students’ comments to increase their usefulness in complementing/confirming ratings. The quantified results enable the profile construction of what students regard as an effective/ineffective teacher, and enable identification of strengths and weaknesses. Our findings counter some commonly held assumptions, including those which held that high ratings are dependent on small class size and ‘dumbing down’ of courses and the consequent expectation of high grades. The findings also indicate that students value teaching quality more than teacher characteristics, suggesting their ability to make valid judgments about teaching effectiveness.

Keywords

Profiling teacher/teaching Student ratings Qualitative feedback Students’ written comments Text analysis Formative evaluation Summative assessment 

References

  1. Abrami, P. C., d’Apollonia, S., & Rosenfield, S. (1996). The dimensionality of student ratings of instruction: What we know and what we do not. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (pp. 213–264). New York: Agathon Press.Google Scholar
  2. Abrami, P. C., d’Apollonia, S., & Rosenfield, S. (2007). The dimensionality of student ratings of instruction: What we know and what we do not. In R. P. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 385–456). New York: Springer.CrossRefGoogle Scholar
  3. Abrami, P. C., d’Appolonia, S., & Cohen, P. A. (1990). The validity of student ratings of instruction: what we know and what we don’t. Journal of Educational Psychology, 82, 219–231.CrossRefGoogle Scholar
  4. Abrami, et al. (1980). Do teacher standards for assigning grades affect student evaluations of instruction? Journal of Educational Psychology, 72, 107–118.CrossRefGoogle Scholar
  5. Aleamoni, L. M. (1987). Typical faculty concerns about student evaluation of teaching. In L. M. Aleamoni (Ed.), Techniques for evaluation and improving instruction, new directions for teaching and learning (No. 31, pp. 25–31). San Francisco: Jossey-Bass.Google Scholar
  6. Aleamoni, L. M., & Hexner, P. Z. (1980). A review of the research on student evaluation and a report on the effect of different sets of instructions on student course and instructor evaluation. Instructional Science, 9, 67–84.CrossRefGoogle Scholar
  7. Apodaca, P., & Grad, H. (2005). The dimensionality of student ratings of teaching: integration of uni- and multidimensional models. Studies in Higher Education, 30(6), 723–748.CrossRefGoogle Scholar
  8. Arreola, R. A. (2000). Developing a comprehensive faculty evaluation system (2nd ed.). Bolton, MA: Anker Publishing Company.Google Scholar
  9. Bejar, I. I., & Doyle, K. O. (1976). The effect of prior expectations on the structure of student ratings of instruction. Journal of Educational Measurement, 13(2), 151–154.CrossRefGoogle Scholar
  10. Cashin, W. E. (1992). Student ratings: The need for comparative data. Instructional Evaluation and Faculty Development, 12(2), 1–6.Google Scholar
  11. Centra, J. A. (1981). Determining faculty effectiveness. The Journal of Higher Education, 52(3), 328–330.CrossRefGoogle Scholar
  12. Centra, J. A. (1987). Formative and summative evaluation: Parody or paradox? In L. M. Aleamoni (Ed.), Techniques for evaluation and improving instruction, new directions for teaching and learning (No. 31, pp. 47–55). San Francisco: Jossey-Bass.Google Scholar
  13. Centra, J. A. (1993). Reflective faculty evaluation: Enhancing teaching and determining faculty effectiveness. San Francisco: Jossey-Bass.Google Scholar
  14. Chiu, S. (1999). Use of the unbalanced nested ANOVA to exam factors influencing student ratings of instructional quality, unpublished doctoral dissertation, University of Illinois at Urbana-Champaign.Google Scholar
  15. Emery, C., Kramer, T. & Tian, R. (2003). Return to academic standards: Challenge the student evaluation of teaching effectiveness. Retrieved May 6, 2007, from www.bus.lsu.edu/academics/accounting/faculty/lcrumbley/stu_rat_of_%20instr.htm.
  16. Erdle, S., Murray, H. G., & Rushton, J. P. (1985). Personality, classroom behavior and student ratings of college teaching effectiveness: A path analysis. Journal of Educational Psychology, 77(4), 394–407.CrossRefGoogle Scholar
  17. Feldman, K. A. (1976). Grades and college students’ evaluations of their courses and teachers. Research in Higher Education, 4, 69–111.CrossRefGoogle Scholar
  18. Feldman, K. A. (1997). Identifying exemplary teachers and teaching: Evidence from student ratings. In R. P. Perry & J. C. Smart (Eds.), Effective teaching in higher education: Research and practice (pp. 368–408). New York: Agathon Press.Google Scholar
  19. Feldman, K. A. (2007). Identifying exemplary teachers and teaching: Evidence from student ratings. In R. P. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 93–143). New York: Springer.CrossRefGoogle Scholar
  20. Franklin, J. & Theall, M. (1989). Who reads ratings: knowledge, attitude, and practice of users of student ratings of instruction. Paper presented at the 1988 annual Meeting of the American Education Research association, San Francisco, CA.Google Scholar
  21. Gallagher, T. (2000). Embracing student evaluations of teaching: A case study. Teaching Sociology, 28(2), 140–147.CrossRefGoogle Scholar
  22. Greenwald, A. G., & Gillmore, G. M. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist, 52, 1209–1217.CrossRefGoogle Scholar
  23. Guthrie, E. R. (1954). The evaluation of teaching: A progress report. Seattle: University of Washington.Google Scholar
  24. Harrison, P. D., Douglas, D. K., & Burdsal, C. A. (2004). The relative merits of different types of overall evaluations of teaching effectiveness. Research in Higher Education, 45(3), 311–323.CrossRefGoogle Scholar
  25. Howard, G. S., & Maxwell, S. E. (1980). The correlation between student satisfaction and grades: A case of mistaken causation. Journal of Educational Psychology, 72, 810–820.CrossRefGoogle Scholar
  26. Johnson, T. & Sorenson, L. (2004). Online student ratings of instruction, new directions for teaching and learning (No. 97). San Francisco: Jossey Bass.Google Scholar
  27. Kember, D., & Wong, A. (2000). Implications for evaluation from a study of students’ perceptions of good and poor teaching. Higher Education, 40(1), 69–97.CrossRefGoogle Scholar
  28. Lewis, K. G. (Ed.). (2001). Techniques and strategies for interpreting student evaluations, new directions for teaching and learning (No. 87). San Francisco: Jossey Bass.Google Scholar
  29. Lin, Y., McKeachie, W. J., & Tucker, D. G. (1984). The use of student ratings in promotion decisions. Journal of Higher Education, 55, 583–589.CrossRefGoogle Scholar
  30. Lowman, J. (1996). Characteristics of exemplary teachers, new directions for teaching and learning (No. 65, pp. 33–50). San Francisco: Jossey-Bass.Google Scholar
  31. Marsh, H. W. (1982). SEEQ: A reliable, valid, and useful instrument for collecting students’ evaluations of university teaching. British Journal of Educational Psychology, 52(1), 77–95.Google Scholar
  32. Marsh, H. W. (1983). Multidimensional ratings of teaching effectiveness by students for different academic settings and their relationship to student/course/instructor characteristics. Journal of Educational Psychology, 75(1), 150–166.CrossRefGoogle Scholar
  33. Marsh, H. W. (1984). Students’ evaluations of university teaching; Dimensionality, reliability, validity, potential biases and utility. Journal of Educational Psychology, 76, 707–754.CrossRefGoogle Scholar
  34. Marsh, H. W. (1987). Students’ evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11, 253–388.CrossRefGoogle Scholar
  35. Marsh, H. W. (1991). Multidimensional student’s evaluation of teaching effectiveness: A test of alternative higher-order structures. Journal of Educational Psychology, 83, 285–296.CrossRefGoogle Scholar
  36. Marsh, H. W. (2007). Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness. In R. P. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 319–383). New York: Springer.CrossRefGoogle Scholar
  37. Marsh, H. W., & Bailey, M. (1993). Multidimensionality of students’ evaluations of teaching effectiveness: A profile analysis. Journal of Higher Education, 64, 1–18.CrossRefGoogle Scholar
  38. Marsh, H. W., & Dunkin, M. J. (1997). Students’ evaluation of university teaching: A multidimensional perspective. In R. P. Perry & J. C. Smart (Eds.), Effective teaching in higher education: Research and practice (pp. 241–320). New York: Agathon Press.Google Scholar
  39. McKeachie, W. J. (1979). Student ratings of faculty: A reprise. Academe, 65, 384–397.Google Scholar
  40. Millman, J. (Ed.). (1981). Handbook of teacher evaluation. Beverly Hills, CA: Sage Publications.Google Scholar
  41. Ory, J. C. (2000). Teaching evaluation: Past, present, and future teaching, new directions for teaching & learning (No. 83, pp. 13–18). San Francisco: Jossey-Bass.Google Scholar
  42. Ory, J. C., & Braskamp, L. A. (1981). Faculty perceptions of the quality and usefulness of three types of evaluative information. Research in Higher Education, 15, 271–282.CrossRefGoogle Scholar
  43. Ory, J. C., & Ryan, K. (2001). How do student ratings measure up to a new validity framework? In M. Theall, P. Abrami, & L. Mets (Eds.), The student ratings debate: Are they valid? How can we best use them? New directions for institutional research (No. 109, pp. 27–44). San Francisco: Jossey-Bass.Google Scholar
  44. Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage Publications.Google Scholar
  45. Perry, R. P., Niemi, R. R., & Jones, K. (1974). Effect of prior teaching evaluations and lecture presentation on ratings of teaching performance. Journal of Educational Psychology, 66(6), 851–856.CrossRefGoogle Scholar
  46. Remmers, H. H. (1949). Are student ratings of instructors related to their grades? Student achievement and instructor evaluation in Chemistry. Studies in Higher Evaluation, 66, 18–26.Google Scholar
  47. Ryan, J. M., & Harrison, P. D. (1995). The relationship between individual instructional characteristics and the overall assessment of teaching effectiveness across different instructional contexts. Research in Higher Education, 36, 213–228.CrossRefGoogle Scholar
  48. Theall, M. & Franklin, J. (Eds.). (1990). Student ratings of instruction: issues for improving practice, new directions for teaching and learning (No. 43). San Francisco: Jossey-Bass.Google Scholar
  49. Theall, M., & Franklin, J. (1999). Faculty thinking about the design and evaluation of instruction. In P. Goodyear & N. Hativa (Eds.), Teacher thinking, beliefs, and knowledge in higher education. The Netherlands: Kluwer Academic Publishers.Google Scholar
  50. Wirtz, J. (2004). Student feedback collection tools that can help to continuously improve your teaching. Retrieved May, 6, 2007, from http://www.cdtl.nus.edu.sg/link/mar2004/cover.htm.
  51. Wulff, D. H., & Nyquist, J. D. (2001). Using qualitative methods to generate data for instructional development. In K. G. Lewis & J. P. Lunde (Eds.), Face to face: A sourcebook of individual consultation techniques for faculty/instructional developers (2nd ed.). Stillwater, OK: New forums Press.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Daphne Pan
    • 1
    • 2
  • Gary S. H. Tan
    • 1
    • 3
  • Kiruthika Ragupathi
    • 1
  • Krishna Booluck
    • 1
  • Rita Roop
    • 1
  • Yuen K. Ip
    • 1
    • 4
  1. 1.Centre for Development of Teaching and LearningNational University of SingaporeSingaporeSingapore
  2. 2.Department of English Language and LiteratureNational University of SingaporeSingaporeSingapore
  3. 3.Department of Computer ScienceNational University of SingaporeSingaporeSingapore
  4. 4.Department of Biological SciencesNational University of SingaporeSingaporeSingapore

Personalised recommendations