Abstract
Introduction
In order to fulfill the enormous potential of digital health in the healthcare sector, digital health must become an integrated part of medical education. We aimed to investigate which knowledge, skills and attitudes should be included in a digital health curriculum for medical students through a scoping review and Delphi method study.
Methods
We conducted a scoping review of the literature on digital health relevant for medical education. Key topics were split into three sub-categories: knowledge (facts, concepts, and information), skills (ability to carry out tasks) and attitudes (ways of thinking or feeling). Thereafter, we used a modified Delphi method where experts rated digital health topics over two rounds based on whether topics should be included in the curriculum for medical students on a scale from 1 (strongly disagree) to 5 (strongly agree). A predefined cut-off of ≥4 was used to identify topics that were critical to include in a digital health curriculum for medical students.
Results
The scoping review resulted in a total of 113 included articles, with 65 relevant topics extracted and included in the questionnaire. The topics were rated by 18 experts, all of which completed both questionnaire rounds. A total of 40 (62%) topics across all three sub-categories met the predefined rating cut-off value of ≥4.
Conclusion
An expert panel identified 40 important digital health topics within knowledge, skills, and attitudes for medical students to be taught. These can help guide medical educators in the development of future digital health curricula.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Background
Digital health, an umbrella term broadly defined as the use of digital technologies for health, [1] has progressively become an integrated part of healthcare practice. Spanning a wide range of fields, including electronic health records (EHRs), telehealth, mobile and wearable health technology (mHealth), and artificial intelligence (AI), digital health has been heralded as a means by which to increase the delivery of and access to healthcare [2, 3]. AI is already being used increasingly in clinical practice, and has been validated for e.g. image recognition, tumor identification, respiratory syndrome differentiation, etc. [4,5,6] Smartphone apps and other mHealth technologies have also been introduced as screening and monitoring tools for a wide range of ailments, including eye health, mental health, and a range of chronic diseases [7,8,9]. As such, digital health technology is playing an increasingly important role in clinical practice.
The Covid-19 pandemic has accelerated the digital health trend, specifically with regards to telehealth and telemonitoring of patients unable to access their typical healthcare providers [10]. While the implementation of digital health has so far been highly heterogeneous, the potential of digital health is enormous [11]. However, the fulfillment of this potential is dependent upon future healthcare workforces being competent and comfortable with its use [12,13,14].
Doctors are particularly critical in the process of implementing digital health in the healthcare sector. Given the role of doctors in guiding patient treatment, digital health is unlikely to be routinely implemented unless they are engaged in its use. For this to happen, digital health must become an integrated part of medical education. While there have been calls for its inclusion into medical curricula, digital health is not routinely included in medical education - although digital health courses prior to and during medical school do exist [15,16,17,18]. In order for a systematic integration to occur, consensus regarding the topics to include, as well as those not to include, in such a curriculum need to be established. To contribute to this knowledge base, we conducted a Delphi study among Danish digital health experts to define the most important topics to be included in a digital health curriculum for medical students.
Method
Delphi method and scoping review
A modified Delphi method was used to identify key topics to include in a digital health curriculum for medical students. The modified Delphi method is a common method of achieving consensus within a group that is based on six stages: agreeing on a research question, an initial literature search, the development of a questionnaire, multiple iterative rounds of questionnaires, feedback to participants between rounds and consequently a summary of the findings [19, 20]. The four key features of consensus methods are anonymity, iteration, statistical group response and controlled feedback [21]. The benefits of the Delphi method include the large number of potential participants, anonymity in the expert panel group (avoiding undue intra-group influence between participants) and that questionnaires can be filled remotely [22].
In order to identify relevant topics for inclusion in the questionnaire for the expert panel, we conducted a scoping review based on the methodology developed by Arksey and O’Malley [23]. The scoping review was based on the following research question: “What knowledge, skills and attitudes (KSAs) within digital health are essential for future doctors?”. Relevant information pertaining to the research question was found through MEDLINE with the assistance of a medical librarian. We searched the following MeSH terms: digital health, biohack*, biomedical engineering, biomedical technology, medical informatics, telemedicine*, electronic health record*, wearable electronic devices*, algorithm*, medical student*, and medical education*. The search string used was: ((digital health*[Text Word]) OR (biohack*[Text Word]) OR (biomedical engineering*[MeSH Terms]) OR (biomedical technology*[MeSH Terms]) OR (medical informatics*[MeSH Terms]) OR (telemedicine*[MeSH Terms]) OR (electronic health record*[MeSH Terms])) OR (wearable electronic devices*[MeSH Terms]) OR (algorithm**[MeSH Terms]) AND ((medical student*[MeSH Terms]) OR (medical education*[MeSH Terms])) AND (2015:2020[pdat]).
Multiple reviewers (DRP, MPK, JSB) reviewed the literature, with each article being reviewed by two reviewers. In the first round, 4972 titles were scanned. Those relevant to either medical education or digital health that were published between 2015 and 2020 in the English language were included. Hereafter, 1899 abstracts from relevant articles were scanned to identify those that were directly relevant to both digital health and medical education. 190 articles were read in full and 113 were included in the final review (Fig. 1). Topics were extracted if they were deemed to fall within the broad definition of digital health, namely the use of digital technologies for health, using the mutually exclusive and collectively exhaustive principle [1].
Expert panel
Potential participants for the Delphi expert panel were identified through authors and references from the scoping review (8 experts), referrals from academic directors (3 experts), and referrals from academic community members (7 experts). To ensure diversity and representativeness in the participant group, we utilized a purposive sampling methodology. We selected participants to achieve diversity across several domains, including educational background (e.g., medical professionals, healthcare technology experts, social scientists), medical specialization, geography, seniority, and institutional settings. Convenience sampling was used until sources of potential experts were exhausted. All participants were based in Denmark and were employed by public institutions. We defined an expert as an individual who satisfied one of the following two criteria: 1) works professionally with digital health research/implementation or 2) is active in the development of medical education curricula with knowledge of digital health. A total of 35 participants were invited by e-mail to participate. Participants were informed of the purpose of the study with full anonymity between experts. Participants were not offered any remuneration for completion of the surveys.
Questionnaire development and completion
The primary method for developing the questionnaire was the inclusion of topics identified during the scoping review; several other reports not published as articles were also reviewed for digital health topics (Additional file 1). Key topics were collated into a master list. The keywords were split into three commonly utilized sub-categories: knowledge, skills, and attitudes. Knowledge is defined as the understanding of facts, concepts, and information. Skills are defined as abilities to carry out particular tasks. Attitudes are settled ways of thinking or feeling about something. A total of 63 items were included in the first-round questionnaire, and 65 items were included in the final round questionnaire since experts could suggest additional topics following the first questionnaire round.
Questionnaires were developed using Google Forms, and invitations were sent to participants by email. Experts rated the topics in the questionnaire based on whether they believed the topic should be integrated into a digital health curriculum for medical students. Topics were rated from 1 to 5 (1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree and 5 = Strongly agree). Additional topics not included in the first round could be suggested by participants to be added in the second round. After the first round, participants were provided average ratings for each topic from the previous round. We completed two questionnaire rounds and used a predefined rating cut-off of ≥4 to identify topics that were critical to include in a digital health curriculum for medical students.
Results
Characteristics of the expert panel
In total, of the 35 invited digital health experts, 18 (51%) completed both the first and second rounds of questionnaires (Fig. 2). All experts (18/18) that completed the first round subsequently also completed the second round. 12 of the 18 (67%) participants were male, 7 of the 18 experts were physicians (39%) (Additional file 2) and 77% of the participants had a PhD or a higher doctorate, with roles divided between directors, head of departments, vice-deans, associate professors, and a civil servant (Table 1).
Rating of topics
Of the knowledge topics identified during the scoping review, 22 met the pre-defined cut-off of ≥4 in the second round of questionnaires (Fig. 3). Nine topics were ranked between 3 and 4, with nine rated < 3. Two new knowledge topics were included in the second round after being suggested by participants in the first round: “Principles of virtual clinical trials (e.g., clinical trials run remotely)” and “Advantages and limitations of clinical decision support systems”. Broad introductions to topics, such as health data infrastructures, digital health terminology and basic concepts of artificial intelligence were rated highly, whereas highly technical knowledge such as mathematical modeling, robotics and 3D reconstruction and printing were ranked lower. A full list of topics with average ratings, median values, and inter-quartile ranges (IQR) can be found in Additional file 3.
Among the skill topics included, five met the pre-defined cut-off of ≥4 in the second round of questionnaires (Fig. 4). Four topics were ranked between 3 and 4, with two rated < 3. No new topics were added in the second round. The three highest rated topics were “Working with clinical decision support systems”, “Using electronic health records in practice” and “Conducting telemedicine in practice”. One expert noted that “The skills section has many interesting points where obtaining skills are maybe a high bar to set, but where an understanding of the dynamics behind is essential. E.g., the use of electronic health records - no need for practical teaching in the curriculum, they should be taught in the current programs of the department at their clinical internships. But they need a basic understanding of how EHRs work, how they exchange data with other systems and why it is challenging to make a proper EHR [ …] The same with designing digital health services. Of course, all doctors should not be able to do that. But they need a basic understanding of what goes on in the design, innovation, and implementation processes so they can relevantly contribute to the process of systems they are to use and promote in the future. The more they understand, the more relevant system they can demand.”
For the digital health topics on attitudes, twelve met the pre-defined cut-off of ≥4 in the second round of questionnaires (Fig. 5). Only two topics were ranked between 3 and 4. The three highest rated topics were “Digital ethics”, “Recognition of how digital health impacts the patient-provider relationship” and “Acknowledgement of the advantages and disadvantages of electronic health records”.
Discussion
The digital health expert panel found that among the digital health topics identified in the medical literature, approximately half should be included in the medical curriculum. Conversely, they believed a large proportion of digital health topics within knowledge, skills, and attitudes are currently not relevant enough for future doctors to learn.
Interestingly, the proportion of topics that were important to include was highest in the attitudes section and approximately equal for the knowledge and skills sections, whereas absolute numbers were highest for knowledge (22 topics) followed by attitudes (12 topics) and skills (6 topics). Generally, the results suggested that attitudes towards digital health, and a basic understanding of digital health’s uses and limitations, were significantly more important than practical skills within digital health. This implies that collaborative skills are likely to be crucial in maximizing the potential of digital health, since physicians must be able to work in interdisciplinary environments with other specialists that have digital health skills not expected to be possessed by physicians themselves. This distinction highlights the panel’s view that the role of doctors is to understand the applications of digital health without necessarily developing solutions themselves. The higher absolute number of topics within knowledge and attitudes compared to skills identified during the scoping review further supports this insight. In addition, this is evident within each knowledge, skills, and attitudes section, where highly technical competencies were considered less important than digital health tools relating to clinical practice, such as competency with the use of EHRs, clinical decision support tools, and telemedicine solutions.
Some medical schools have implemented digital health courses, although their integration thus far has been heterogenous. In a recent scoping review of digital health courses in medical schools, the authors found that the majority of courses were elective and mainly focused on medical informatics [24]. They are thus not routinely integrated into the curriculum and tend to focus on narrow topics within digital health. In Germany, following the implementation of a 3-week curricular module on digital health at a medical school, the module was well-received by both faculty and students, both of which highlighted the importance of digital health for clinical care and its underrepresentation in the curriculum [25]. From an implementation perspective, however, there are justifiable concerns regarding the already packed medical curriculum, with fears of information overload [26, 27]. Identifying and removing outdated topics, which may follow directly from the automatization of certain clinical functions as digital health solutions become more commonplace, is therefore equally important in developing an optimal medical curriculum [28].
Several limitations should be acknowledged when interpreting the findings of the study. Firstly, the topics included from the scoping review were all articles published in English, and the expert panel was composed of experts living in Denmark. The generalizability of the results may thus primarily apply to English-speaking settings or those with similar healthcare structures and educational systems to Denmark. Another limitation of the Delphi study process is that the expert panel members may represent leading universities and are therefore not fully representative of all medical schools and institutions. Given that experts are likely to be older than non-experts, an age bias may also affect the topics that are deemed important to include, possibly skewing towards less novel but more established topics. Secondly, topics were stratified during the scoping review based on the mutually exclusive and collectively exhaustive principle, where topics were assumed not to overlap - despite the practical challenges associated with achieving this. Some topics may have overlapped, making them more challenging for the expert panel to rate, which was noted by two experts during the first questionnaire round. Additionally, ratings from the survey indicate which topics are important to prioritize in future curricula, but do not indicate how big a part of the curriculum each topic should comprise. Lastly, the field of digital health is rapidly evolving, [29] and the medical curriculum should ideally be modified to reflect this development. As such, the findings in this study represent a snapshot of current knowledge, skills, and attitudes, and should therefore be adapted regularly going forward.
The primary strength of this study is that we conducted a large scoping review prior to the Delphi process to include the most up-to-date digital health topics. We also allowed experts to include additional important topics not identified during the scoping review. The expert panel was large and diverse, with multiple clinical specialties represented as well as experts spanning from directors to civil servants to professors (Additional file 2). Additionally, the follow-up response rate was high between the first and second questionnaire rounds.
Conclusion
An expert panel identified 40 important digital health topics within knowledge (22 topics), skills (6 topics), and attitudes (12 topics) to be taught during medical school. The average rating of each topic can be understood to represent its relative importance for inclusion in the curriculum. Given the growing role of digital tools in health, curriculum changes are needed to keep pace with this development. The insights from this study can help guide medical and digital health educators in the development of future digital health curricula.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
References
World Health Organization. WHO guideline: recommendations on digital interventions for health system strengthening. Executive summary. Geneva: World Health Organization; 2019. (WHO/RHR/19.8). Available from: https://apps.who.int/iris/bitstream/handle/10665/311977/WHO-RHR-19.8-eng.pdf?ua=1.
Senbekov M, Saliev T, Bukeyeva Z, Almabayeva A, Zhanaliyeva M, Aitenova N, et al. The recent progress and applications of digital technologies in Healthcare: a review. Fayn J, editor. Int J Telemed Appl. 2020;2020:1–18. Available from: https://www.hindawi.com/journals/ijta/2020/8830200/.
Smuck M, Odonkor CA, Wilt JK, Schmidt N, Swiernik MA. The emerging clinical role of wearables: factors for successful implementation in healthcare. NPJ Digit Med. 2021;4(1):45. Available from: http://www.nature.com/articles/s41746-021-00418-3.
Sjoding MW, Taylor D, Motyka J, Lee E, Co I, Claar D, et al. Deep learning to detect acute respiratory distress syndrome on chest radiographs: a retrospective study with external validation. Lancet Digit Heal. 2021;3(6):e340–8 Available from: https://linkinghub.elsevier.com/retrieve/pii/S258975002100056X.
Jiang Y, Liang X, Han Z, Wang W, Xi S, Li T, et al. Radiographical assessment of tumour stroma and treatment outcomes using deep learning: a retrospective, multicohort study. Lancet Digit Heal. 2021;3(6):e371–82. Available from: https://linkinghub.elsevier.com/retrieve/pii/S2589750021000650.
Shamout FE, Shen Y, Wu N, Kaku A, Park J, Makino T, et al. An artificial intelligence system for predicting the deterioration of COVID-19 patients in the emergency department. NPJ Digit Med. 2021;4(1):80. Available from: http://www.nature.com/articles/s41746-021-00453-0.
Ernsting C, Dombrowski SU, Oedekoven M, O’Sullivan JL, Kanzler M, Kuhlmey A, et al. Using smartphones and health apps to change and manage health behaviors: a population-based survey. J Med Internet Res. 2017;19(4):e101. Available from: http://www.jmir.org/2017/4/e101/.
Rono H, Bastawrous A, Macleod D, Mamboleo R, Bunywera C, Wanjala E, et al. Effectiveness of an mHealth system on access to eye health services in Kenya: a cluster-randomised controlled trial. Lancet Digit Heal. 2021;3(7):e414–24. Available from: https://linkinghub.elsevier.com/retrieve/pii/S2589750021000832.
Bauer M, Glenn T, Geddes J, Gitlin M, Grof P, Kessing LV, et al. Smartphones in mental health: a critical review of background issues, current status and future concerns. Int J Bipolar Disord. 2020;8(1):2. Available from: https://journalbipolardisorders.springeropen.com/articles/10.1186/s40345-019-0164-x.
Monaghesh E, Hajizadeh A. The role of telehealth during COVID-19 outbreak: a systematic review based on current evidence. BMC Public Health. 2020;20(1):1193 Available from: https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-020-09301-4.
Makri A. Bridging the digital divide in health care. Lancet Digit Heal. 2019;1(5):e204–5. Available from: https://linkinghub.elsevier.com/retrieve/pii/S2589750019301116.
Wong BLH, Khurana MP, Smith RD, El-Omrani O, Pold A, Lotfi A, et al. Harnessing the digital potential of the next generation of health professionals. Hum Resour Health. 2021;19(1):50. Available from: https://human-resources-health.biomedcentral.com/articles/10.1186/s12960-021-00591-2.
Chen Y, Banerjee A. Improving the digital health of the workforce in the COVID-19 context: an opportunity to future-proof medical training. Futur Healthc J. 2020;7(3):189–92. Available from: https://www.rcpjournals.org/lookup/doi/10.7861/fhj.2020-0162.
The Topol review: Preparing the healthcare workforce to deliver the digital future. An independent report on behalf of the secretary of state for health and social care. 2019. Available from: https://topol.hee.nhs.uk/wp-content/uploads/HEE-Topol-Review-2019.pdf.
Chandrashekar PA. Digital Health Preclinical Requirement for Medical Students. Acad Med. 2019;94(6):749. Available from: http://journals.lww.com/00001888-201906000-00009.
Khurana M. Keeping pace: the need for digital health education in medical schools. Acad Med. 2020;95(11):1629–30. Available from: https://journals.lww.com/10.1097/ACM.0000000000003672.
Rosenman D, Swanson WS. “Digital health” in medical school. Acad Med. 2020;95(7):973–3. Available from: https://journals.lww.com/10.1097/ACM.0000000000003359.
Aungst TD, Patel R. Integrating digital health into the curriculum—considerations on the current landscape and future developments. J Med Educ Curric Dev. 2020;20(7):238212051990127. Available from: http://journals.sagepub.com/doi/10.1177/2382120519901275.
Jones J, Hunter D. Qualitative research: consensus methods for medical and health services research. BMJ. 1995;311(7001):376–80. Available from: https://www.bmj.com/lookup/doi/10.1136/bmj.311.7001.376.
Dalkey N, Helmer O. An experimental application of the DELPHI method to the use of experts. Manag Sci. 1963;9(3):458–67 Available from: http://pubsonline.informs.org/doi/abs/10.1287/mnsc.9.3.458.
Grime MM, Wright G. Delphi Method. In: Wiley StatsRef: Statistics Reference Online: Wiley; 2016. p. 1–6. https://doi.org/10.1002/9781118445112.stat07879.
de Villiers MR, de Villiers PJT, Kent AP. The Delphi technique in health sciences education research. Med Teach. 2005;27(7):639–43. Available from: https://www.tandfonline.com/doi/full/10.1080/13611260500069947.
Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32. Available from: http://www.tandfonline.com/doi/abs/10.1080/1364557032000119616.
Tudor Car L, Kyaw BM, Nannan Panday RS, van der Kleij R, Chavannes N, Majeed A, et al. Digital health training programs for medical students: scoping review. JMIR Med Educ. 2021;7(3):e28275. Available from: https://mededu.jmir.org/2021/3/e28275.
Poncette A-S, Glauert DL, Mosch L, Braune K, Balzer F, Back DA. Undergraduate medical competencies in digital health and curricular module development: mixed methods study. J Med Internet Res. 2020;22(10):e22161. Available from: http://www.jmir.org/2020/10/e22161/.
Smith R. Strategies for coping with information overload. BMJ. 2010;341(dec15 2):c7126–6. Available from: https://www.bmj.com/lookup/doi/10.1136/bmj.c7126.
Wartman SA, Combs CD. Reimagining medical education in the age of AI. AMA J Ethics. 2019;21(2):E146–52. Available from: https://journalofethics.ama-assn.org/article/reimagining-medical-education-age-ai/2019-02.
McCoy LG, Nagaraj S, Morgado F, Harish V, Das S, Celi LA. What do medical students actually need to know about artificial intelligence? NPJ Digit Med. 2020;3(1):86. Available from: http://www.nature.com/articles/s41746-020-0294-7.
Guo C, Ashrafian H, Ghafur S, Fontana G, Gardner C, Prime M. Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches. NPJ Digit Med. 2020;3(1):110. Available from: https://www.nature.com/articles/s41746-020-00314-2.
National Videnskabsetisk Komité. Hvad skal jeg anmelde. 2020. Available from: https://www.nvk.dk/forsker/naar-du-anmelder/hvilke-projekter-skal-jeg-anmelde.
Bekendtgørelse af lov om videnskabsetisk behandling af sundhedsvidenskabelige forskningsprojekter og sundhedsdatavidenskabelige forskningsprojekter. CN001359 Denmark; 2020. Available from: https://www.retsinformation.dk/eli/lta/2020/1338.
Acknowledgements
We would like to thank Dr. Signe Jervelund for her guidance regarding Delphi method studies. The authors would also like to thank the expert panel members for their time.
Funding
None.
Author information
Authors and Affiliations
Contributions
All authors contributed equally to the design of the study. MPK, DRP, and JSB carried out the scoping review and organized questionnaire rounds. The author(s) read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
All participants provided written informed consent prior to study commencement in accordance with the 1964 Declaration of Helsinki. Ethical approval was not required by regulatory authorities in Denmark due to the nature of study (questionnaire) and full anonymization of study participants [30, 31].
Consent for publication
Not applicable.
Competing interests
MPK: None to declare.
DRP: None to declare.
JK: None to declare.
JB: None to declare.
SO: None to declare.
JSB: None to declare.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Additional file 1.
Reports included for digital health topic selection.
Additional file 2.
Demographic characteristics between invited experts and participating experts.
Additional file 3.
Tables with average ratings, median values, and interquartile ranges for knowledge, skills, and attitudes.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Khurana, M.P., Raaschou-Pedersen, D.E., Kurtzhals, J. et al. Digital health competencies in medical school education: a scoping review and Delphi method study. BMC Med Educ 22, 129 (2022). https://doi.org/10.1186/s12909-022-03163-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12909-022-03163-7