Skip to main content

Ethics Incognito: Detecting Ethically Relevant Courses Across Curricula in Higher Education


As colleges and universities seek to invigorate ethics education, they need methods to identify where and describe how ethics is already present across their curricula. Meeting this need is complicated by the fact that much ethics education occurs in courses not explicitly focused on ethics or morality. In this paper, we review recent methodological advances before presenting a new Ethics Course Identification Tool (ECIT) that combines application of an expert-derived weighted dictionary and natural language processing methods to identify ethics-related courses based on their titles and course catalog descriptions, even when the terms “ethic” or “moral” are not present. Two studies, the second a pre-registered replication, revealed considerable interrater reliability among experts in ethics education regarding the ethical relevance of courses. Critically, both studies revealed strong correlations between expert judgments and ECIT scores. This empirical evidence points to a shared understanding of ethics education among experts, and it supports the valid use of the ECIT to rapidly and reliably identify ethics-related courses. Based on these findings, we propose that the ECIT can be used both to advance research on trends in ethics education and to help target interventions to improve ethics education at colleges and universities.

This is a preview of subscription content, access via your institution.

Fig. 1

Data Availability

Link to OSF data:


  • American Association of Colleges and Universities. (2022, August 8). Essential learning outcomes. AAC&U. Retrieved September 19, 2022, from

  • Araque, O., Gatti, L., & Kalimeri, K. (2020). MoralStrength: Exploiting a moral lexicon and embedding similarity for moral foundations prediction. Knowledge-based Systems, 191, 105184.

    Article  Google Scholar 

  • Bankhead, M., Choi, Y. J., Patil, Y., & Thoma, S. J. (2022). Individual and School Correlates of DIT-2 Scores Using a Multilevel Modeling and Data Mining Analysis. Applied Sciences, 12(9), 4573.

    Article  Google Scholar 

  • Barnwell, D. (1989). ‘Naïve’native speakers and judgements of oral proficiency in Spanish. Language Testing, 6(2), 152–163.

    Article  Google Scholar 

  • Bebeau, M. J., & Monson, V. E. (2008). Guided by theory, grounded in evidence: A way forward for professional ethics education. In Handbook of Moral and Character Education (pp. 573-598). Routledge.

  • Beever, J., Kuebler, S. M., & Collins, J. (2021). Where ethics is taught: An institutional epidemiology. International Journal of Ethics Education, 6(2), 215–238.

    Article  Google Scholar 

  • Berry, J. W. (1989). Imposed etics—emics—derived etics: The operationalization of a compelling idea. International Journal of Psychology, 24(6), 721–735.

    Article  Google Scholar 

  • Boyd, R. L., & Schwartz, H. A. (2021). Natural language analysis and the psychology of verbal behavior: The past, present, and future states of the field. Journal of Language and Social Psychology, 40(1), 21–41.

    Article  Google Scholar 

  • Christensen, L. J., Peirce, E., Hartman, L. P., Hoffman, W. M., & Carrier, J. (2007). Ethics, CSR, and sustainability education in the Financial Times top 50 global business schools: Baseline data and future research directions. Journal of Business Ethics, 73(4), 347–368.

    Article  Google Scholar 

  • Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290.

    Article  Google Scholar 

  • Colby, A., Ehrlich, T., Beaumont E., & Stephens, J. (2003). Educating citizens: Preparing America’s undergraduates for lives of moral responsibility. JosseyBass.

  • Drechsler Sharp, M., Komives, S. R., & Fincher, J. (2011). Learning outcomes in academic disciplines: Identifying common ground. Journal of Student Affairs Research and Practice, 48(4), 481–504.

    Article  Google Scholar 

  • Elliott, D. (2007). Ethics in the First Person: A Guide to Teaching and Learning Practical Ethics. Rowman & Littlefield.

  • Elliott, D., & June, K. (2018). The Evolution of Ethics Education 1980–2015. In Ethics Across the Curriculum—Pedagogical Perspectives (pp. 11-37). Springer, Cham.

  • Finelli, C. J., Holsapple, M. A., Ra, E., Bielby, R. M., Burt, B. A., Carpenter, D. D., Harding, T. S., & Sutkus, J. A. (2012). An assessment of engineering students’ curricular and co-curricular experiences and their ethical development. Journal of Engineering Education, 101(3), 469–494.

    Article  Google Scholar 

  • Fischman, W., & Gardner, H. (2022). The real world of college: What higher education is and what it can be. MIT Press.

  • Garten, J., Hoover, J., Johnson, K. M., Boghrati, R., Iskiwitch, C., & Dehghani, M. (2018). Dictionaries and distributions: Combining expert knowledge and large scale textual data content analysis. Behavior Research Methods, 50(1), 344–361.

    Article  Google Scholar 

  • Glanzer, P. L., & Ream, T. C. (2008). Educating different types of citizens: Identity, tradition, and moral education. Journal of College and Character, 9(4), 1–14.

    Article  Google Scholar 

  • Griffith, S. M., Domenech Rodriguez, M. M., & Anderson, A. J. (2014). Graduate ethics education: A content analysis of syllabi. Training and Education in Professional Psychology, 8(4), 248–252.

    Article  Google Scholar 

  • Hamer, R. M. (1991). A SAS macro for computing intraclass correlation coefficients. Virginia Commonwealth University.

    Google Scholar 

  • Hoover, J., Johnson, K., Boghrati, R., Graham, J., & Dehghani, M. (2018). Moral framing and charitable donation: Integrating exploratory social media analyses and confirmatory experimentation. Collabra: Psychology, 4(1).

  • Hoover, J., Portillo-Wightman, G., Yeh, L., Havaldar, S., Davani, A. M., Lin, Y., ... & Dehghani, M. (2020). Moral foundations twitter corpus: A collection of 35k tweets annotated for moral sentiment. Social Psychological and Personality Science, 11(8), 1057-1071.

  • Jarvis, S. (2017). Grounding lexical diversity in human judgments. Language Testing, 34(4), 537–553.

    Article  Google Scholar 

  • Kidd, D., Miner, J., Schein, M., Blauw, M., & Allen, D. (2020). Ethics across the curriculum: Detecting and describing emergent trends in ethics education. Studies in Educational Evaluation, 67, 100914.

    Article  Google Scholar 

  • Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.

    Article  Google Scholar 

  • Lambeth, E. B., Christians, C. G., Fleming, K., & Lee, S. T. (2004). Media ethics teaching in century 21: Progress, problems, and challenges. Journalism & Mass Communication Educator, 59(3), 239–358.

    Article  Google Scholar 

  • Matchett, N. J. (2008). Ethics across the curriculum. New Directions for Higher Education, 142, 25–38.

    Article  Google Scholar 

  • May, D. R., Luth, M. T., & Schwoerer, C. E. (2014). The influence of business ethics education on moral efficacy, moral meaningfulness, and moral courage: A quasi-experimental study. Journal of Business Ethics, 124(1), 67–80.

    Article  Google Scholar 

  • Mayhew, M. J., Rockenbach, A. N., Bowman, N. A., Seifert, T. A., & Wolniak, G. C. (2016). How college affects students: 21st century evidence that higher education works (Vol. 3). John Wiley & Sons.

  • Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. In Yoshua Bengio et al., editors, 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2- 4, 2013, Workshop Track Proceedings.

  • Mills, A., Sanders, A. K., & Hussain, S. S. (2019). Fitting it all in? A census of undergraduate ethics and leadership courses in accredited US journalism and mass communication programs. Journalism & Mass Communication Educator, 74(3), 265–275.

    Article  Google Scholar 

  • Neill, M. S. (2017). Ethics education in public relations: Differences between stand-alone ethics courses and an integrated approach. Journal of Media Ethics, 32(2), 118–131.

    Article  Google Scholar 

  • Nosek, B. A., & Lindsay, D. S. (2018). Preregistration becoming the norm in psychological science. Observer, 31(3), 19–21.

    Google Scholar 

  • Pike, K. L. (2015). Language in relation to a unified theory of the structure of human behavior (Vol. 24). Walter de Gruyter GmbH & co KG.

  • Sahatjian, Z., MacDougall, A. E., & McIntosh, T. (2022). Benefits beyond service: The facilitative effects of service-learning pedagogy on nuanced reflection and ethical sensemaking. Academy of Management Learning & Education, 21(1), 7–34.

    Article  Google Scholar 

  • Schouten, G. (2022). The case for egalitarian consciousness raising in higher education. Philosophical Studies, 1-24.

  • Triandis, H. C., Malpass, R. S., & Davidson, A. R. (1973). Psychology and culture. Annual Review of Psychology, 24(1), 355–378.

    Article  Google Scholar 

  • Van’t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12.

    Article  Google Scholar 

  • Wang, L. C., & Calvano, L. (2015). Is business ethics education effective? An analysis of gender, personal ethical perspectives, and moral judgment. Journal of Business Ethics, 126(4), 591–602.

    Article  Google Scholar 

  • Weston, A. (2006). A Practical Companion to Ethics (3rd ed.). Oxford University Press.

    Google Scholar 

  • You, D., Ruiz, A., & Warchal, J. (2018). The presentation of ethics in undergraduate psychology syllabi: A review. Teaching of Psychology, 45(4), 346–350.

    Article  Google Scholar 

  • Zou, G. Y. (2012). Sample size formulas for estimating intraclass correlation coefficients with precision and assurance. Statistics in Medicine, 31(29), 3972–3981.

    Article  Google Scholar 

Download references


We are grateful to Ka Ya Lee and Drew Chambers for their careful review of the manuscript and for their valuable insight. We also extend our thanks to Cara Biasucci and Deni Elliott for assistance in recruiting participants. Their contributions were invaluable to the success of this project. This work was generously supported by the Edmond J. Safra Philanthropic Foundation and Eugene P. Beard.

Author information

Authors and Affiliations


Corresponding authors

Correspondence to Martino Ongis or David Kidd.

Ethics declarations

Conflict of Interests

We have no conflicts of interest to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ongis, M., Kidd, D. & Miner, J. Ethics Incognito: Detecting Ethically Relevant Courses Across Curricula in Higher Education. J Acad Ethics (2023).

Download citation

  • Accepted:

  • Published:

  • DOI: