Skip to main content

Using theory-informed data science methods to trace the quality of dental student reflections over time

Advances in Health Sciences Education volume 27, pages 23–48 (2022)Cite this article

Abstract

This study describes a theory-informed application of data science methods to analyze the quality of reflections made in a health professions education program over time. One thousand five hundred reflections written by a cohort of 369 dental students over 4 years of academic study were evaluated for an overall measure of reflection depth (No, Shallow, Deep) and the presence of six theoretically-indicated elements of reflection quality (Description, Analysis, Feeling, Perspective, Evaluation, Outcome). Machine learning models were then built to automatically detect these qualities based on linguistic features in the reflections. Results showed a dramatic increase from No to Shallow reflections from the start to end of year one (20%  →  66%), but only a limited gradual rise in Deep reflections across all four years (2%  →  26%). The presence of all six reflection elements increased over time, but inclusion of Feelings and Analysis remained relatively low even at the end of year four (found in 44% and 60% of reflections respectively). Models were able to reliably detect the presence of Description (κTEST = 0.70) and Evaluation (κTEST = 0.65) in reflections; models to detect the presence of Analysis (κTEST = 0.50), Feelings (κTEST = 0.54), and Perspectives (κTEST = 0.53) showed moderate performance; the model to detect Outcomes suffered from overfitting (κTRAIN = 0.90, κTEST = 0.53). A classifier for overall depth built on the reflection elements showed moderate performance across all time periods (κTEST > 0.60) but relied almost exclusively on the presence of Description. Implications for the conceptualization of reflection quality and providing personalized learning support to help students develop reflective skills are discussed.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  • Alzahem, A. M., Van der Molen, H. T., Alaujan, A. H., Schmidt, H. G., & Zamakhshary, M. H. (2011). Stress amongst dental students: A systematic review. European Journal of Dental Education, 15(1), 8–18.

    Article  Google Scholar 

  • Artstein, R., & Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555–596.

    Article  Google Scholar 

  • Asadoorian, J., Schönwetter, D. J., & Lavigne, S. E. (2011). Developing reflective health care practitioners: Learning from experience in dental hygiene education. Journal of Dental Education, 75(4), 472–484.

    Article  Google Scholar 

  • Boud, D., Keogh, R., & Walker, D. (Eds.). (1985). Reflection: Turning experience into learning. Kogan Page.

    Google Scholar 

  • Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32.

    Article  Google Scholar 

  • Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-centred learning analytics. Journal of Learning Analytics, 6(2), 1–9.

    Article  Google Scholar 

  • Bush, H., & Bissell, V. (2008). The evaluation of an approach to reflective learning in the undergraduate dental curriculum. European Journal of Dental Education, 12(2), 103–110.

    Article  Google Scholar 

  • Cantor, A. B. (1996). Sample-size calculations for cohen’s kappa. Psychological Methods, 1(2), 150–153.

    Article  Google Scholar 

  • Chawla, N., Bowyer, K., Hall, L., & Kegelmeyer, W. (2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321–357.

    Article  Google Scholar 

  • Chirema, K. D. (2007). The use of reflective journals in the promotion of reflection and learning in post-registration nursing students. Nurse Education Today, 27(3), 192–202.

    Article  Google Scholar 

  • Cotton, A. H. (2001). Private thoughts in public spheres: Issues in reflection and reflective practices in nursing. Journal of Advanced Nursing, 36(4), 512–519.

    Article  Google Scholar 

  • Crossley, S. A., Kyle, K., & McNamara, D. S. (2017). Sentiment analysis and social cognition engine (SEANCE): An automatic tool for sentiment, social cognition, and social-order analysis. Behavior Research Methods, 49(3), 803–821.

    Article  Google Scholar 

  • Cui, Y., Wise, A. F., & Allen, K. L. (2019). Developing reflection analytics for health professions education: A multi-dimensional framework to align critical concepts with data features. Computers in Human Behavior, 100, 305–324.

    Article  Google Scholar 

  • Davis, E. A. (2000). Scaffolding students’ knowledge integration: Prompts for reflection in KIE. International Journal of Science Education, 22(8), 819–837.

    Article  Google Scholar 

  • Dewey, J. (1933). How we think: A restatement of the relation of reflective thinking to the educative process. D.C. Heath.

    Google Scholar 

  • Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71.

    Article  Google Scholar 

  • Gibbs, G. (1988). Learning by doing: A guide to teaching and learning methods. Oxford, UK.

    Google Scholar 

  • Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C., & Knight, S. (2017). Reflective writing analytics for actionable feedback. In Proceedings of the seventh international learning analytics & knowledge conference (pp. 153–162). New York, NY: ACM.

  • Gibson, A., Kitto, K., & Bruza, P. (2016). Towards the discovery of learner metacognition from reflective writing. Journal of Learning Analytics, 3(2), 22–36.

    Article  Google Scholar 

  • Hanson, K., & Alexander, S. (2010). The influence of technology on reflective learning in dental hygiene education. Journal of Dental Education, 74(6), 644–653.

    Article  Google Scholar 

  • Heeneman, S., & Driessen, E. W. (2017). The use of a portfolio in postgraduate medical education–reflect, assess and account, one for each or all in one? GMS Journal for Medical Education, 34(5), 1–12.

    Google Scholar 

  • Herrera, F., Charte, F., Rivera, A. J., & Del Jesus, M. J. (2016). Multilabel classification. Springer.

    Book  Google Scholar 

  • Hulsman, R. L., Harmsen, A. B., & Fabriek, M. (2009). Reflective teaching of medical communication skills with DiViDU: Assessing the level of student reflection on recorded consultations with simulated patients. Patient Education and Counseling, 74(2), 142–149.

    Article  Google Scholar 

  • Ip, W. Y., Lui, M. H., Chien, W. T., Lee, I. F., Lam, L. W., & Lee, D. (2012). Promoting self- reflection in clinical practice among Chinese nursing undergraduates in Hong Kong. Contemporary Nurse, 41(2), 253–262.

    Article  Google Scholar 

  • James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. Springer.

    Book  Google Scholar 

  • Knight, S., Shum, S. B., Ryan, P., Sándor, Á., & Wang, X. (2018). Designing academic writing analytics for civil law student self-assessment. International Journal of Artificial Intelligence in Education, 28(1), 1–28.

    Article  Google Scholar 

  • Koole, S., Dornan, T., Aper, L., Scherpbier, A., Valcke, M., Cohen-Schotanus, J., & Derese, A. (2011). Factors confounding the assessment of reflection: A critical review. BMC Medical Education, 11(1), 104.

    Article  Google Scholar 

  • Kovanović, V., Joksimović, S., Mirriahi, N., Blaine, E., Gašević, D., Siemens, G., and Dawson, S. (2018). Understand students' self-reflections through learning analytics. In Proceedings of the 8th international conference on learning analytics & knowledge (pp. 389–398). New York, NY, USA: ACM.

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

    Article  Google Scholar 

  • Liu, M., Shum, S. B., Mantzourani, E., & Lucas, C. (2019). Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflective Statements. In Proceedings of International Conference on Artificial Intelligence in Education (pp. 220–230). Cham, Switzerland: Springer.

  • Mann, K., Gordon, J., & MacLeod, A. (2009). Reflection and reflective practice in health professions education: A systematic review. Advances in Health Sciences Education, 14(4), 595.

    Article  Google Scholar 

  • Mezirow, J. (1991). Transformative dimensions of adult learning. Jossey- Bass.

    Google Scholar 

  • Moon, J. A. (2013). Reflection in learning and professional development: Theory and practice. Routledge.

    Book  Google Scholar 

  • Ochoa, X., Domínguez, F., Guamán, B., Maya, R., Falcones, G., & Castells, J. (2018). The rap system: automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors. In Proceedings of the 8th international conference on learning analytics and knowledge (pp. 360–364). New York, NY, USA: ACM.

  • Ochoa, X., & Duval, E. (2009). Automatic evaluation of metadata quality in digital repositories. International Journal on Digital Libraries, 10(2–3), 67–91.

    Article  Google Scholar 

  • Pennebaker, J. W., Boyd, R. L., Jordan, K., & Blackburn, K. (2015). The development and psychometric properties of LIWC2015. University of Texas at Austin.

    Google Scholar 

  • Read, J., Pfahringer, B., Holmes, G., & Frank, E. (2011). Classifier chains for multi-label classification. Machine Learning, 85(3), 333.

    Article  Google Scholar 

  • Schön, D. A. (1983). The reflective practitioner. Jossey-Bass.

    Google Scholar 

  • Shibani, A., Knight, S., & Shum, S. B. (2019). Contextualizable learning analytics design: A generic model and writing analytics evaluations. In Proceedings of the 9th international conference on learning analytics and knowledge (pp. 210–219). New York, NY, USA: ACM.

  • Tolsgaard, M. G., Boscardin, C. K., Park, Y. S., Cuddy, M. M., & Sebok-Syer, S. S. (2020). The role of data science and machine learning in Health Professions Education: practical applications, theoretical contributions, and epistemic beliefs. Advances in Health Sciences Education. https://doi.org/10.1007/s10459-020-10009-8

    Article  Google Scholar 

  • Tsingos, C., Bosnic-Anticevich, S., Lonie, J. M., & Smith, L. (2015). A model for assessing reflective practices in pharmacy education. American Journal of Pharmaceutical Education, 79(8), 124.

    Article  Google Scholar 

  • Ullmann, T. D. (2019). Automated analysis of reflection in writing: validating machine learning approaches. International Journal of Artificial Intelligence in Education, 29(2), 217–257.

    Article  Google Scholar 

  • Wise, A., & Cui, Y. (2018). Envisioning a learning analytics for the learning sciences. In Proceedings of the 13th international conference of the learning sciences (pp. 1799–1806). London, UK: International Society of the Learning Sciences.

  • Wise, A.F., Reza, S. & Han, R. J. (2020). Becoming a dentist: Tracing professional identity development through mixed-methods data mining of student reflections. In Proceedings of the 13th international conference of the learning sciences. Nashville, TN: International Society of the Learning Sciences.

  • Wise, A. F., & Shaffer, D. W. (2015). Why theory matters more than ever in the age of big data. Journal of Learning Analytics, 2(2), 5–13.

    Article  Google Scholar 

  • Wong, F. K., Kember, D., Chung, L. Y., & Yan, L. (1995). Assessing the level of student reflection from reflective journals. Journal of Advanced Nursing, 22(1), 48–57.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Sarita Xinying Liu for her assistance with the process of content analysis and the NYU Dental Informatics team for their help in accessing the data. This article expands on preliminary work presented at the International Conference on Learning Analytics and Knowledge in 2020.

Author information

Authors and Affiliations

  1. Learning Analytics Research Network (NYU-LEARN), New York University, 370 Jay Street, 5th Floor, Brooklyn, NY, 11201, USA

    Yeonji Jung & Alyssa Friend Wise

  2. Department of Cariology and Comprehensive Care, College of Dentistry, New York University, 137 E. 25th Street, 6th Floor, New York, NY, 10010, USA

    Kenneth L. Allen

Authors
  1. Yeonji Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alyssa Friend Wise
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kenneth L. Allen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yeonji Jung.

Ethics declarations

Conflict of interest

None.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Appendix A: model performance of the classifiers

See Tables 3, 4, 5, 6, 7 and 8.

Table 3 Model performance of the reflection elements using linguistic features
Full size table
Table 4 Model performance of the depth classifier using linguistic features
Full size table
Table 5 Confusion matrix of the depth classifier using linguistic features on the test data
Full size table
Table 6 Model performance of the depth classifier using coded and predicted values for reflection elements as features
Full size table
Table 7 Confusion matrix of the depth classifier using coded values for reflection elements on the test data
Full size table
Table 8 Confusion matrix of the depth classifier using predicted values for reflection elements on the test data
Full size table

Appendix B: lists of top 10 predictive features for each of the classifiers

See Tables 9, 10 and 11.

Table 9 Reflection elements classifiers: Top 10 features (with MDG mean index)
Full size table
Table 10 Depth classifier using linguistic feature: Top 10 features (with MDG mean index)
Full size table
Table 11 Depth classifier using coded reflection elements: Top 10 features (with MDG mean index)
Full size table

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jung, Y., Wise, A.F. & Allen, K.L. Using theory-informed data science methods to trace the quality of dental student reflections over time. Adv in Health Sci Educ 27, 23–48 (2022). https://doi.org/10.1007/s10459-021-10067-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-021-10067-6

Keywords

  • Reflection
  • Health professions education
  • Educational data sciences
  • Classification