Advertisement

Multi-Method Evaluation of Instruction in Engineering Classes

  • Ganesh Mohanty
  • John Gretes
  • Claudia Flowers
  • Bob Algozzine
  • Fred Spooner
Article

Abstract

Student evaluation of instruction in college and university courses has been a routine and mandatory part of undergraduate and graduate education for some time. A major shortcoming of the process is that it often relies exclusively on the opinions or qualitative judgments of students rather than the learning or transfer of knowledge that takes place in the classroom. To develop a more objective system of assessment, this research focused on a learning-centered approach to course work and teaching evaluation. Standardized testing tools were developed suitable for measuring the content knowledge of students in a representative group of undergraduate courses. Course evaluations were conducted using two systems of assessment: the traditional student questionnaire feedback system and one based on the learning-centered approach using a computer-based question bank and on-line testing. Significant performance differences were evident in pretest/posttest comparisons of student learning. Favorable ratings of instruction are reflected in opinions on student questionnaires. No relationship was demonstrated between learning and traditional course evaluation outcomes. Our hypothesis that the learning-centered approach provides information that is not available using the traditional student feedback system was supported.

Keywords

evaluation of college teaching faculty teaching evaluation student evaluations of teaching engineering course evaluation 

References

  1. Abrami, P. C. (1993). Using student rating norm groups for summative evaluation. Evaluation and Faculty Development 13, 5–9.Google Scholar
  2. Abrami, P. C., & d'Apollonia, S. (1990). The dimensionality of ratings and their use in personnel decisions. In M. Theall & J. Franklin (Eds.), Student Ratings of Instruction: Issues for Improving Practice (pp. 97–111). San Francisco: Jossey-Bass.Google Scholar
  3. Abrami, P. C., & Mizener, D. A. (1983). Does the attitude similarity of college professors and their students produce “bias” in course evaluations? American Educational Research Journal 20, 123–136.CrossRefGoogle Scholar
  4. Abrami, P. C., Perry, R. P., & Leventhal, L. (1982). The relationship between student personality characteristics, teacher ratings, and student achievement. Journal of Educational Psychology 74, 111–125.CrossRefGoogle Scholar
  5. Abrami, P. C., d'Apollonia, S., & Cohen, P. A. (1990). The validity of student ratings of instruction: what we know and what we do not. Journal of Educational Psychology 82, 219–231.CrossRefGoogle Scholar
  6. Abrami, P. C., d'Apollonia, S., & Rosenfield, S. (1996). The dimensionality of student ratings of instruction: what we know and what we do not. In J. C. Smart (Ed.), Higher Education: Handbook of Theory and Research (Vol. 11, pp. 213–264). New York: Agathon.Google Scholar
  7. Adams, J. V. (1997). Student evaluations: the ratings game. Inquiry 1(2), 10–16.Google Scholar
  8. Aleamoni, L. M. (1987). Typical faculty concerns about student evaluation of teaching. In L. M. Aleamoni (Ed.), Techniques for Evaluating and Improving Instruction. San Francisco: Jossey-Bass.Google Scholar
  9. Aleamoni, L. M. (1999). Student rating myths versus research facts from 1924 to 1998. Journal of Personnel Evaluation in Education 13, 153–166.CrossRefGoogle Scholar
  10. Algozzine, B., Beattie, J., Bray, M., Flowers, C., Gretes, J., Howley, L., Mohanty, G., & Spooner, F. (2004). Student evaluation of college teaching: a practice in search of principles. College Teaching 45(1), 1–7.Google Scholar
  11. Angelo, T. A. (1990, Summer). Classroom assessment: improving learning quality where it matters most. New Directions in Teaching and Learning 42, 71–82.CrossRefGoogle Scholar
  12. Angelo, T. A. (1991, Summer). Ten easy pieces: assessing higher learning in four dimensions. New Directions in Teaching and Learning 46, 17–31.CrossRefGoogle Scholar
  13. Angelo, T. A. (1995). Classroom assessment for critical thinking. Teaching of Psychology 21(1), 6–7.CrossRefGoogle Scholar
  14. Angelo, T. A. (1996, Spring). Relating exemplary teaching to student learning. New Directions in Teaching and Learning 65, 57–64.CrossRefGoogle Scholar
  15. Angelo, T. A., & Cross, K. P. (1989, Fall). Classroom research for teaching assistants. New Directions in Teaching and Learning 39, 99–108.CrossRefGoogle Scholar
  16. Angelo, T. A., & Cross, K. P. (1993). Classroom Assessment Techniques: A Handbook for College Teachers (2nd ed.). San Francisco, California: Jossey-Bass.Google Scholar
  17. Blunt, A. (1991). The effects of anonymity and manipulated grades on student ratings of instructors. Community College Review 18, 48–53.CrossRefGoogle Scholar
  18. Braskamp, L. A., Caulley, D., & Costin, F. (1979). Student ratings and instructor self-ratings and their relationship to student achievement. American Educational Research Journal 16, 295–306.CrossRefGoogle Scholar
  19. Cashin, W. E., & Downey, R. G. (1992). Using global student rating items for summative evaluation. Journal of Educational Psychology 84, 563–572.CrossRefGoogle Scholar
  20. Centra, J. A. (1973). Self-ratings of college teachers: a comparison with student ratings. Journal of Educational Measurement 10, 287–295.CrossRefGoogle Scholar
  21. Centra, J. A. (1977). Student ratings of instruction and their relationship to student learning. American Educational Research Journal 14, 17–24.CrossRefGoogle Scholar
  22. Centra, J. A. (1993). Reflective Faculty Evaluation: Enhancing Teaching and Determining Faculty Effectiveness. San Francisco: Jossey-Bass.Google Scholar
  23. Cohen, P. A. (1981). Student ratings of instruction and student achievement: a meta-analysis of multi-section validity studies. Review of Educational Research 51, 281–309.CrossRefGoogle Scholar
  24. Cranton, P. A., & Smith, R. A. (1986). A new look at the effect of course characteristics on student ratings of instruction. American Educational Research Journal 23, 117–128.CrossRefGoogle Scholar
  25. Cross, K. P. (1990). Making teaching more effective. Journal of the Freshman Year Experience 2(2), 59–74.Google Scholar
  26. Cross, K. P. (2001). Motivation: Er...Will That Be on the Test? Mission Viejo, California: League for Innovation in the Community College.Google Scholar
  27. Damron, J. C. (1995). The Three Faces of Teaching Evaluation. Unpublished manuscript. New Westminster, British Columbia: Douglas College.Google Scholar
  28. d'Apollonia, S., & Abrami, P. (1997). Navigating student ratings of instruction. American Psychologist 52, 1198–1208.CrossRefGoogle Scholar
  29. Doyle, K. O. Jr., & Whitely, S. E. (1974). Student ratings as criteria for effective teaching. American Educational Research Journal 11, 259–274.CrossRefGoogle Scholar
  30. Driscoll, L. A., & Goodwin, W. L. (1979). The effects of varying information about use and disposition of results on university students' evaluations of faculty and courses. American Educational Research Journal 16, 25–37.CrossRefGoogle Scholar
  31. Feldman, K. A. (1989). Instructional effectiveness of college teachers as judged by teachers themselves, current and former students, colleagues, administrators, and external (neutral) observers. Research in Higher Education 30, 137–174.CrossRefGoogle Scholar
  32. Fossey, R. (1999). Personnel evaluation in higher education: a mounting call for accountability. Journal of Personnel Evaluation in Education 13, 115–118.CrossRefGoogle Scholar
  33. Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational Research (6th edn.). White Plains, New York: Longman.Google Scholar
  34. Gigliotti, R. J., & Buchtel, F. S. (1990). Attributional bias and course evaluations. Journal of Educational Psychology 82, 341–351.CrossRefGoogle Scholar
  35. Gillmore, G. (1984). Student ratings as a factor in faculty employment decisions and periodic review. Journal of College and University Law 10, 557–576.Google Scholar
  36. Grant, H. (1998). Academic contests: merit pay in Canadian universities. Relations Industrielles/Industrial Relations 53, 647–664.Google Scholar
  37. Granzin, K. L., & Painter, J. J. (1973). A new explanation for students' course evaluation tendencies. American Educational Research Journal 10, 115–124.CrossRefGoogle Scholar
  38. Greenwald, A., & Gilmore, G. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist 52, 1209–1217.CrossRefPubMedGoogle Scholar
  39. Haskell, R. E. (1997a, February 12). Academic freedom, tenure, and student evaluations of faculty: galloping polls in the 21st century. Education Policy Analysis Archives 5(6).Google Scholar
  40. Haskell, R. E. (1997b, August 25). Academic freedom, promotion, reappointment, tenure, and the administrative use of student evaluation of faculty (SEF): (Part II) views from court. Education Policy Analysis Archives 5(6).Google Scholar
  41. Haskell, R. E. (1997c, August 25). Academic freedom, promotion, reappointment, tenure, and the administrative use of student evaluation of faculty (SEF): (Part III) analysis and implications of views from the court in relation to accuracy and psychometric validity. Education Policy Analysis Archives 5(6).Google Scholar
  42. Haskell, R. E. (1997d, November 25). Academic freedom, promotion, reappointment, tenure, and the administrative use of student evaluation of faculty (SEF): (Part IV) analysis and implications of views from the court in relation to academic freedom, standards, and quality of instruction. Education Policy Analysis Archives 5(6).Google Scholar
  43. Hobson, S. M., & Talbot, D. M. (2001). Understanding student evaluations: what all faculty should know. College Teaching 49, 26–31.CrossRefGoogle Scholar
  44. Hoffmann, R. G. (1978). Variables affecting university student ratings of instructor behavior. American Educational Research Journal 15, 287–299.CrossRefGoogle Scholar
  45. Hogan, T., & Hartley, E. (1972). Some additional factors in student evaluation of courses. American Educational Research Journal 9, 241–250.CrossRefGoogle Scholar
  46. Kirkpartick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and Development Handbook: A Guide to Human Resource Development (2nd edn.) (pp. 18.1–18.27). New York: ASTD-McGraw Hill Book Company.Google Scholar
  47. Lechtreck, R. (1990). College faculty evaluation by students—an opportunity for bias. College Student Journal 24, 297–299.Google Scholar
  48. Marsh, H. W. (1980). The influence of student, course, and instructor characteristics in evaluations of university teaching. American Educational Research Journal 17, 219–237.CrossRefGoogle Scholar
  49. Marsh, H. W. (1982). Factors affecting students' evaluations of the same course by the same instructor on different occasions. American Educational Research Journal 19, 485–497.CrossRefGoogle Scholar
  50. Marsh, H. W. (1983). Multitrait-multimethod analysis: distinguishing between items and traits. Educational and Psychological Measurement 43, 351–358.CrossRefGoogle Scholar
  51. Marsh, H. W. (1984). Students' evaluations of university teaching: dimensionality, reliability, validity, potential biases, and utility. Journal of Educational Psychology 76, 707–764.CrossRefGoogle Scholar
  52. Marsh, H. (1987). Students' evaluations of university teaching: research findings, methodological issues, and directions for future research. International Journal of Educational Research 11, 253–388.CrossRefGoogle Scholar
  53. Marsh, H. (1991a). Multidimensional students' evaluations of teaching effectiveness: a test of alternative higher-order structures. Journal of Educational Psychology 83, 285–296.CrossRefGoogle Scholar
  54. Marsh, H. (1991b). A multidimensional perspective on students' evaluations of teaching effectiveness: Reply to Abrami and d'Apollonia (1991). Journal of Educational Psychology 83, 416–421.CrossRefGoogle Scholar
  55. Marsh, H. W. (1994). Weighting for the right criteria to validate student evaluations of teaching in the IDEA system. Journal of Educational Psychology 86, 631–648.CrossRefGoogle Scholar
  56. Marsh, H. W. (1995). Still weighting for the right criteria to validate student evaluations of teaching in the IDEA system. Journal of Educational Psychology 87, 666–679.CrossRefGoogle Scholar
  57. Marsh, H. W., & Bailey, M. (1993). Multidimensional students' evaluations of teaching effectiveness: A profile analysis. Journal of Higher Education 64, 1–18.CrossRefGoogle Scholar
  58. Marsh, H. W., & Hocevar, D. (1984). The factorial invariance of student evaluations of college teaching. American Educational Research Journal 21, 341–366.CrossRefGoogle Scholar
  59. Marsh, H. W., & Overall, J. U. (1981). The relative influence of course level, course type, and instructor on students' evaluations of college teaching. American Educational Research Journal 18, 103–112.CrossRefGoogle Scholar
  60. Marsh, H., & Roche, L. (1997). Making students' evaluations of teaching effectiveness effective: the central issues of validity, bias, and utility. American Psychologist 52, 1187–1197.CrossRefGoogle Scholar
  61. Marsh, H. W., Overall, J. U., & Kesler, S.P. (1979). Class size, students'evaluations, and instructional effectiveness. American Educational Research Journal 16, 57–69.CrossRefGoogle Scholar
  62. McKeachie, W. (1997). Student ratings: the validity of use. American Psychologist 52, 1218–1225.CrossRefGoogle Scholar
  63. Nevill, D. D., Ware, W. B., & Smith, A. B. (1978). A comparison of student ratings of teaching assistants and faculty members. American Educational Research Journal 15, 25–37.CrossRefGoogle Scholar
  64. Platt, M. (1993). What student evaluations teach. Perspectives in Political Science 22(1), 29–40.CrossRefGoogle Scholar
  65. Ramsden, P. (1991) Performance indicator of teaching quality in higher education: the Course Experience Questionnaire. Studies in Higher Education 16, 129–150.CrossRefGoogle Scholar
  66. Remmers, H. H. (1927). The purdue rating scale for instructors. Educational Administration and Supervision 6, 399–406.Google Scholar
  67. Rifkin, T. (1995). The status and scope of faculty evaluation. ERIC Digest [ED385315].Google Scholar
  68. Rodin, M., & Rodin, B. (1972). Student evaluations of teaching. Science 177, 1164–1166.CrossRefGoogle Scholar
  69. Rolph, P., & Rolph, J. (1989) Evaluating courses. Assessment and Evaluation in Higher Education 14, 117–131.Google Scholar
  70. Ruskai, M. B. (1996). Evaluating student evaluations. Notices of The American Mathematical Society 44(3), 308.Google Scholar
  71. Shingles, R. D. (1977). Faculty ratings: procedures for interpreting student evaluations. American Educational Research Journal 14, 459–470.CrossRefGoogle Scholar
  72. Sproule, R. (2000). Student evaluation of teaching: a methodological critique of evaluation practices. Education Policy Analysis Archives 8(50). [http://epaa.asu.edu/epaa/v8n50.html]
  73. Starry, A. R., Derry, J. O., & Wright, G. L. (1973). An automated instructor and course appraisal system. Educational Technology 13(5), 61–64.Google Scholar
  74. Trujillo, C. M. (1986). A comparative examination of classroom interactions between professors and minority and non-minority students. American Educational Research Journal 23, 629–642.CrossRefGoogle Scholar
  75. Walker, C., & Angelo, T.A. (1998, September). A collective effort classroom assessment technique: Promoting high performance in student teams. New Directions in Teaching and Learning 75, 101–112.CrossRefGoogle Scholar

Copyright information

© Springer Science + Business Media, LLC 2006

Authors and Affiliations

  • Ganesh Mohanty
    • 1
  • John Gretes
    • 1
  • Claudia Flowers
    • 1
  • Bob Algozzine
    • 1
  • Fred Spooner
    • 1
  1. 1.BRIC/EDLD/COEDUniversity of North Carolina at CharlotteCharlotteUSA

Personalised recommendations