Science and Engineering Ethics

, Volume 21, Issue 3, pp 789–807 | Cite as

Validity and Reliability of an Instrument for Assessing Case Analyses in Bioengineering Ethics Education

  • Ilya M. GoldinEmail author
  • Rosa Lynn Pinkus
  • Kevin Ashley
Original Paper


Assessment in ethics education faces a challenge. From the perspectives of teachers, students, and third-party evaluators like the Accreditation Board for Engineering and Technology and the National Institutes of Health, assessment of student performance is essential. Because of the complexity of ethical case analysis, however, it is difficult to formulate assessment criteria, and to recognize when students fulfill them. Improvement in students’ moral reasoning skills can serve as the focus of assessment. In previous work, Rosa Lynn Pinkus and Claire Gloeckner developed a novel instrument for assessing moral reasoning skills in bioengineering ethics. In this paper, we compare that approach to existing assessment techniques, and evaluate its validity and reliability. We find that it is sensitive to knowledge gain and that independent coders agree on how to apply it.


Moral reasoning Assessment Ethics case analysis Mixed methods Validity Reliability 



The authors thank Christian Schunn, Janyce Wiebe, Diane Litman, and our reviewers for valuable feedback; Christos Theodoulou and Scott Nickleach working under the supervision of Dr. Alan Sampson for statistics consulting; Claire Gloeckner and Jessica DiFrancesco for coding and data entry; Mark Sindelar and Kaitlin Jones for coding, and Angela Fortunato for extensive assistance and advice. This work was supported by NSF Engineering and Computing Education Grant #0203307.


  1. Accreditation Board of Engineering and Technology. (2006). Criteria for accrediting engineering programs.Google Scholar
  2. Arras, John D. (1991). Getting down to cases: The revival of casuistry in bioethics. Journal of Medicine and Philosophy, 16, 29–51. doi: 10.1093/jmp/16.1.29.CrossRefGoogle Scholar
  3. Beauchamp, T. L., & Childress, J. F. (2001). Principles of biomedical ethics (Vol. 5). New York, N.Y.: Oxford University Press.Google Scholar
  4. Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46, 423–429.CrossRefGoogle Scholar
  5. Campbell, D.T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. In N. L. Gage (Ed.), Handbook of research on teaching (pp. ix, 84). Chicago: Rand McNally.Google Scholar
  6. Carletta, J. (1996). Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22, 249–254.Google Scholar
  7. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.CrossRefGoogle Scholar
  8. Comunian, A. L. 2002. Structure of the Padua moral judgment scale: A study of young adults in seven countries. In 110th annual conference of the American Psychological Association. Chicago, IL.Google Scholar
  9. Di Eugenio, B., & Glass, M. (2004). The Kappa statistic: A second look. Computational Linguistics, 30(1), 95–101. doi: 10.1162/089120104773633402.
  10. Fisher, F. T., & Peterson, P. L. (2001). A tool to measure adaptive expertise in biomedical engineering students. In American Society for Engineering Education. Mira Digital Publishing, Inc.Google Scholar
  11. Gibbs, J. C., Widaman, K. F., & Colby, A. (1982). Social intelligence: Measuring the development of sociomoral reflection (pp. 191–211). New Jersey: Prentice-Hall.Google Scholar
  12. Goldie, J., Schwartz, L., McConnachie, A., & Morrison, J. (2001). Impact of a new course on students’ potential behaviour on encountering ethical dilemmas. Medical Education, 35, 295–302.CrossRefGoogle Scholar
  13. Goldie, J., Schwartz, L., McConnachie, A., & Morrison, J. (2002). The impact of three years’ ethics teaching, in an integrated medical curriculum, on students’ proposed behaviour on meeting ethical dilemmas. Medical Education, 36, 489–497.CrossRefGoogle Scholar
  14. Goldin, I. M., Ashley, K. D., & Pinkus, R. L. (2006a). Teaching case analysis through framing: Prospects for an ITS in an ill-defined domain. In “Workshop on intelligent tutoring systems for ill-defined domains”, 8th international conference on intelligent tutoring systems. Jhongli, Taiwan.Google Scholar
  15. Goldin, I. M., Ashley, K. D., & Pinkus, R. L. (2006b). Assessing case analyses in bioengineering ethics education: Reliability and training. In Proceedings of the 2006 International Conference on Engineering Education. San Juan, Puerto Rico.Google Scholar
  16. Harris, C. E., Pritchard, M. S., & Rabins, M. J. (2000). Engineering ethics: Concepts and cases (Vol. 2). Belmont, CA: Wadsworth.Google Scholar
  17. Hayes, R. P., Stoudemire, A., Kinlaw, K., Dell, M. L., & Loomis, A. (1999). Qualitative outcome assessment of a medical ethics program for clinical clerkships: A pilot study. General Hospital Psychiatry, 21, 284–295.CrossRefGoogle Scholar
  18. Hébert, P. C., Meslin, E. M., & Dunn, E. V. (1992). Measuring the ethical sensitivity of medical students: A study at the university of Toronto. Journal of Medical Ethics, 18, 142–147.CrossRefGoogle Scholar
  19. Jonsen, A. R., & Toulmin, S. E. (1988). The abuse of casuistry: A history of moral reasoning. Berkeley: University of California Press.Google Scholar
  20. Kipnis, K. (2000). Medical ethics education in a problem-based learning curriculum. American Philosophical Association Newsletter on Philosophy and Medicine, 00(1), 128–132.Google Scholar
  21. Kohlberg, L. (1981). The philosophy of moral development (Vol. 1). San Francisco: Harper & Row.Google Scholar
  22. Kohlberg, L. (1984). The psychology of moral development (Vol. 2). San Francisco: Harper & Row.Google Scholar
  23. Kuczewski, M. G. (1997). Fragmentation and consensus: Communitarian and casuist bioethics. Washington, D.C.: Georgetown University Press.Google Scholar
  24. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.CrossRefGoogle Scholar
  25. Lind, G. (2000). Moral regression in medical students and their learning environment. Revista Brasileira de Educacao Médica, 24, 24–33.Google Scholar
  26. Lynch, D. C., Surdyk, P. M., & Eiser, A. R. (2004). Assessing professionalism: A review of the literature. Medical Teacher, 26, 366–373.CrossRefGoogle Scholar
  27. Martin, T., Rayne, K., Kemp, N. J., Hart, J., & Diller, K. (2005). Teaching adaptive expertise in biomedical engineering ethics. Science and Engineering Ethics, 11, 257–276.CrossRefGoogle Scholar
  28. National Institutes of Health Office of Extramural Research. (2006). Frequently asked questions for the requirement for education on the protection of human subjects. May 9.Google Scholar
  29. Olds, B. M., Moskal, B. M., & Miller, R. L. (2005). Assessment in engineering education: Evolution, approaches and future collaborations. Journal of Engineering Education, 94, 13–25.CrossRefGoogle Scholar
  30. Pence, G. E. (2000). Classic cases in medical ethics: Accounts of cases that have shaped medical ethics, with philosophical, legal, and historical backgrounds (Vol. 3). Boston: McGraw-Hill.Google Scholar
  31. Pinkus, R. L. (1997). Engineering ethics: Balancing cost, schedule, and risk–lessons learned from the space shuttle. New York: Cambridge University Press.Google Scholar
  32. Pinkus, R. L., Chi, M. T. H., McQuaide, J., Ashley, K. D., & Pollack, M. (1999). Some preliminary thoughts on reasoning with cases: A cognitive science approach. Symposium presented at the Association for Moral Education, November 20, Minneapolis, MN.Google Scholar
  33. Pinkus, R. L., Gloeckner, C., & Fortunato, A. (2015). The role of professional knowledge in case-based reasoning in practical ethics. Science and Engineering Ethics. doi: 10.1007/s11948-015-9645-1.Google Scholar
  34. Rest, J. R., Narvaez, D., Bebeau, M. J., & Thoma, S. J. (1999). Postconventional moral thinking: A neo-Kohlbergian approach. New Jersey: Lawrence Erlbaum Associates.Google Scholar
  35. Rudnicka, E. (2004). A review of instruments for measuring moral reasoning/values. In 10th international conference on industry, engineering, and management systems, Cocoa Beach, FL, pp 305–311.Google Scholar
  36. Savulescu, J., Crisp, R., Fulford, K. W. M., & Hope, T. (1999). Evaluating ethics competence in medical education. Journal of Medical Ethics, 25, 367–374.CrossRefGoogle Scholar
  37. Shapiro, J., & Miller, R. (1994). How medical students think about ethical issues. Academic Medicine, 69, 591–593.CrossRefGoogle Scholar
  38. Shuman, L. J., Besterfield-Sacre, M., & McGourty, J. (2005). The ABET “professional skills”—Can they be taught? Can they be assessed? Journal of Engineering Education, 94, 41–55.CrossRefGoogle Scholar
  39. Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy, 85, 257–268.Google Scholar
  40. Sindelar, M., Shuman, L., Besterfield-Sacre, M., Miller, R., Mitcham, C., Olds, B., Pinkus, R. L., & Wolfe, H. (2003). Assessing engineering students’ abilities to resolve ethical dilemmas. In 33rd annual frontiers in education. Boulder, CO: IEEE.Google Scholar
  41. Steneck, N. H. 2007. ORI introduction to the responsible conduct of research. August 3.Google Scholar
  42. Venables, W. N., & Ripley, B. D. (2002). Modern applied statistics with S (4th ed.). New York: Springer.CrossRefGoogle Scholar
  43. Voss, J. F., & Post, T. A. (1988). On the solving of ill-structured problems. In M. T. H. Chi, R. Glaser, & M. J. Farr (Eds.), The nature of expertise (pp. 261–285). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  44. Weil, V. 1990. Engineering ethics in engineering education: Summary report. In conference at the Center for the Study of Ethics in the Professions at the Illinois Institute of Technology. Chicago.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  • Ilya M. Goldin
    • 1
    Email author
  • Rosa Lynn Pinkus
    • 2
  • Kevin Ashley
    • 3
  1. 1.Center for Digital Data, Analytics & Adaptive LearningPearsonPittsburghUSA
  2. 2.Department of BioengineeringUniversity of PittsburghPittsburghUSA
  3. 3.Learning Research and Development CenterUniversity of PittsburghPittsburghUSA

Personalised recommendations