Advertisement

Perspectives on Medical Education

, Volume 7, Issue 3, pp 219–223 | Cite as

The Massive Online Needs Assessment (MONA) to inform the development of an emergency haematology educational blog series

  • Teresa M. Chan
  • David Jo
  • Andrew W. Shih
  • Vinai C. Bhagirath
  • Lana A. Castellucci
  • Calvin Yeh
  • Brent Thoma
  • Eric K. Tseng
  • Kerstin de Wit
Open Access
Show and Tell

Abstract

Background

Online educational resources are criticized as being teacher-centred, failing to address learner’s needs. Needs assessments are an important precursor to inform curriculum development, but these are often overlooked or skipped by developers of online educational resources due to cumbersome measurement tools. Novel methods are required to identify perceived and unperceived learning needs to allow targeted development of learner-centred curricula.

Objectives

To evaluate the feasibility of performing a novel technique dubbed the Massive Online Needs Assessment (MONA) for the purpose of emergency haematology online educational curricular planning, within an online learning community (affiliated with the Free Open Access Medical education movement).

Methods

An online survey was launched on CanadiEM.org using an embedded Google Forms survey. Participants were recruited using the study website and a social media campaign (utilizing Twitter, Facebook, Blogs, and a poster) targeting a specific online community. Web analytics were used to monitor participation rates in addition to survey responses.

Results

The survey was open from 20 September to 10 December 2016 and received 198 complete responses representing 6 medical specialties from 21 countries. Most survey respondents identified themselves as staff physicians (n = 109) and medical trainees (n = 75). We identified 17 high-priority perceived needs, 17 prompted needs, and 10 topics with unperceived needs through our MONA process.

Conclusions

A MONA is a feasible, novel method for collecting data on perceived, prompted, and unperceived learning needs to inform an online emergency haematology educational blog. This methodology could be useful to the developers of other online education resources.

Keywords

Needs assessment Online educational resources Free open access medical education (FOAM) 

Introduction

The Free Open Access Medical education (FOAM) movement encompasses the grassroots explosion of online medical educational and knowledge translation resources. FOAM seeks to minimize barriers to providing education and continued competency in medical care [1, 2, 3]. However, the FOAM movement has been criticized for being ad hoc in terms of generation of content. Critics cite an imbalance of resources focusing on trendy topics in emergency medicine and critical care [4]. Topics focussing on airway techniques, electrocardiography interpretation, resuscitation, and ultrasonography are overrepresented, while other topics such as haematological disorders (<0.6% of total content) are scarce [4]. Imbalanced representation of topics may lead to a disconnect between online curricula and clinical practice. Curriculum design principles commonly list needs assessment as a core planning procedure [5]. Yet, few FOAM producers engage in learner-centred strategies for discerning their audience’s needs.

CanadiEM (www.canadiem.org) is a multi-author educational emergency medicine blog, which provides free-access educational content for practising emergency physicians, residents, medical students, and other emergency healthcare providers. In 2016, CanadiEM was approached by a multidisciplinary team of haematology experts to host a practical, novel and accessible curriculum aimed at educating students, residents and physicians on emergency bleeding and clotting scenarios. We had noticed a lack of accessible and up-to-date material for practising clinicians, partly because of rapid developments in the clinical field. To inform the design of a learner-centred FOAM curriculum that addresses gaps in current curricula, we embarked on an online innovation that sought to engage our readership in a needs assessment [6, 7], we dubbed a Massive Online Needs Assessment (MONA). By sharing our process and findings, we hope that other FOAM producers may similarly engage their readership to identify priority educational content.

Methods

Setting

CanadiEM is a multi-author blog with a volunteer corps of approximately 50 editors who work collaboratively online to generate novel content for emergency providers. The site receives roughly a quarter of a million unique international site visits per year.

Ethics

This study received approval from our institutional review board (HIREB: #2016-1954-GRA).

Materials

Our needs assessment was created on Google Forms (Mountainview, CA, USA) and embedded on a CanadiEM.org blog post. The needs assessment included:
  1. 1.

    a section on the reader’s perceived needs;

     
  2. 2.

    a section which prompted them to identify needs (e. g. via story-telling around difficult cases); and then

     
  3. 3.

    a section with multiple-choice questions (designed to discern unperceived knowledge gaps).

     

The full survey can be found in the Online Electronic Supplementary Material, Appendix A & B. Our needs assessment was informed by previous literature, with the exception of the ‘prompted’ needs assessment part (i. e. difficult case description, Part 2) which is a novel innovation in our MONA survey [8].

Participant recruitment

We recruited participants on social media via three promotional avenues, namely the CanadiEM blog, Twitter, and Facebook. Our 30-minute survey was available online from 20 September to 10 December 2016. Participants who completed the survey had an opportunity to enter a draw for gift cards.

Outcomes

Topics endorsed by 30–50% of the participants were considered moderate priority while topics with >50% participant endorsement were considered high priority. The topics where <50% of participants answered questions correctly were identified as knowledge gaps. A misperception was defined as an incorrect response that >60% of participants selected.

Analysis

Feasibility of our new MONA technique was defined a priori as 150 responses from at least 4 specialties in 4 or more countries. We felt this sample size was required to identify areas of knowledge gaps (defined a priori as questions where <50% of readers gave the correct responses). Participant demographics were analyzed with descriptive statistics using Microsoft Excel for Mac 2015 (Microsoft Corporation, Redmond, WA). A thematic analysis of data from the qualitative comments about possible topics and the difficult scenarios were analyzed by two investigators (TC, ET) and agreed upon by consensus with iterative rounds of discussion. Responses were coded by each author, in a constant comparison fashion.

Results

During the study period, the CanadiEM webpage which hosted the link to the survey was visited 866 times by visitors at unique internet protocol addresses according to Google Analytics (Mountainview, CA, USA). A total of 198 participants from 6 specialty areas and 21 countries completed the MONA. This exceeded our feasibility goal of 150 participants, from at least 4 specialties and 4 countries. The participants comprised 109 physicians (55% respondents), 46 residents (23%), 29 medical students (15%), and other healthcare practitioners (n = 14). Of the respondents, 57% were identified as male (n = 113). The majority of respondents were from Canada (n = 115, 58%), and the United States (n = 51, 26%), with the remaining 32 respondents from a wide variety of 19 other countries. Most were Emergency Medicine providers (n = 118, 60%), with a strong showing from Internal Medicine (n = 41, 20%), and the remainder (n = 39) identifying with Primary Care, Surgical Specialties, Critical Care, Anaesthesia, or other groups.

Part 1—Perceived needs results

Risk of thrombosis for reversal agents, adjunct treatments for acute bleeding, and reversal of anticoagulants were identified as topics of highest perceived need. We also identified 17 other high priority topics and 6 moderate priority topics. There were no additional topics via free-entry text responses.

Part 2—Prompted needs results

Our thematic analysis revealed 17 unique topics. Most cases were focused on patients with a high bleeding risk and high risk of clotting (for example, acute venous thrombosis or mechanical heart valve). Interestingly, none of these topics were written within the optional free-text responses of Part 1.

Part 3—Unperceived needs results

Knowledge gaps were identified in 10 of 15 questions. Most were associated with cases regarding perioperative reversal of anticoagulants, coagulopathic trauma patients, and deep vein thrombosis management. Only one question on the topic of pulmonary embolism diagnosis was identified as problematic, as participants correctly answered these questions with a high frequency. In 8 of 10 questions where a knowledge gap was identified, a misperception (one incorrect response selected >60% of the time) was responsible.

Fig. 1 summarizes the contributions of each part of the needs assessment. Appendices C & D in the Online Electronic Supplementary Material detail our findings in full.
Fig. 1

The contributions of each part of our Massive Online Needs Assessment (MONA)

Discussion

As shown previously with the METRIQ study [9, 10], it is possible to engage a learning community to assist with online surveys. We achieved our feasibility mark (i.e. >150 participants from more than 4 countries) in less than 4 months. Assuming each unique visit was from distinct individuals, the survey completion rate was 22% (198/866), which is similar to rates reported for other online surveys of physicians [11].

Asking participants to describe their needs through storytelling was a novel way to triangulate their learning needs (Part 2). We identified 17 additional areas of need, which the participants did not self-identify in Part 1 (i. e. when they could write in ‘other’ within the drop-down list). We feel that the most interesting lesson learned was that the ‘difficult case’ descriptions provided a unique window into the needs of our audience. In addition to the perceived needs, the descriptions also provided insight that more complex situations were important to address (e. g. when a patient who required antiplatelet drugs for a diagnosis such as new cardiac stents presented with a bleeding problem such as an intracranial haemorrhage). Furthermore, the multiple choice questions (Part 3) also helped to identify areas of unperceived needs, since this revealed where there was substantive variation and/or misperception on various topic areas.

When developing our curriculum, we anticipate that we will utilize Parts 1 and 3 to derive individual topics, and Part 2 results have led to the consideration of case-based discussions around complex scenarios and non-medical expert topics. This MONA has allowed us to gather useful information to create a new online curriculum.

Limitations

We acknowledge that this has been a singular attempt at using a technique to establish the needs of an online learning community, and is restricted to a specific topic and specific online context. As this is a new tool, future studies will focus on the effect of MONA on educational outcomes. Further replication studies are required to validate MONA for other FOAM contexts and for other topics. We believe that multidisciplinary teams are not only key for content generation, but also for outreach. Two of the investigators (TC, BT) have been active within the online learning community [12], in part due to their involvement with CanadiEM, and this improved our ability to reach and/or influence members of this learning community to participate. Furthermore, we do not know whether the readers who chose not to complete the needs assessment are active online learners, or whether they differ in their needs from the people who completed the survey. This is a common problem for survey methodology and not unique to this MONA. We had 198 completed responses. Unfortunately, Google Forms does not record partially completed surveys and it is likely that data from potential participants were lost.

Conclusion

Our MONA has demonstrated that it is possible to recruit members of an online learning community to assist educators to determine both perceived and unperceived learning needs in an emergency haematology blog. The data gathered from our MONA will help us create a targeted curriculum. Online educators may choose to use such a technique to better quantify and qualify the needs of their audiences.

Notes

Funding

We thank the McMaster University Continuing Health Sciences Education for supporting us via the Research and Innovation Fund. We also thank the Canadian Blood Services Centre For Innovation for funding the development of online resources that resulted from this needs assessment through the Canadian Blood Services BloodTechNet competition.

Conflict of interest

D. Jo, A.W. Shih, V.C. Bhagirath, L.A. Castellucci, C. Yeh, E.K. Tseng and K. de Wit declare that they have no competing interests. B. Thoma and T.M. Chan are volunteer editors for the CanadiEM.org website, but they receive no financial remuneration for these positions.

Ethical standards

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study.

Supplementary material

40037_2018_406_MOESM1_ESM.pdf (375 kb)
Appendix A: Social Media for Knowledge Translation and Education 2 Needs Assessment Survey
40037_2018_406_MOESM2_ESM.pdf (76 kb)
Appendix B: Results of a case-based multiple-choice test created to identify unperceived needs
40037_2018_406_MOESM3_ESM.pdf (50 kb)
Appendix C: Topics in area of perceived needs within the domains of thrombosis and clotting, acute bleeding, and treatment and therapy of thrombosis and bleeding
40037_2018_406_MOESM4_ESM.pdf (56 kb)
Appendix D: Topics in areas of perceived needs from the difficult scenario descriptions

References

  1. 1.
    Choo EK, Ranney ML, Chan TM, et al. Twitter as a tool for communication and knowledge exchange in academic medicine: a guide for skeptics and novices. Med Teach. 2015;37:411–6.CrossRefGoogle Scholar
  2. 2.
    Thoma B, Mohindra R, Artz JD, Chan TM. CJEM and the changing landscape of medical education and knowledge translation. Can J Emerg Med [internet]. 2015;17:184–7.Google Scholar
  3. 3.
    Chan T, Trueger NS, Roland D, Thoma B. Evidence-based medicine in the era of social media: Scholarly engagement through participation and online interaction. Can J Emerg Med. 2017;  https://doi.org/10.1017/cem.2016.407.Google Scholar
  4. 4.
    Stuntz R, Clontz R. An evaluation of emergency medicine core content covered by free open access medical education resources. Ann Emerg Med. 2016;67(5):649.  https://doi.org/10.1016/j.annemergmed.2015.12.020.CrossRefGoogle Scholar
  5. 5.
    Kern DE. Curriculum Development for Medical Education: A Six-Step Approach. Baltimore: The Johns Hopkins University Press; 2009. p. 253.Google Scholar
  6. 6.
    Lave J, Wenger E. Situated learning: Legitimate peripheral participation (learning in doing: Social, cognitive and computational perspectives). New York: Cambridge University Press; 1991. p. 138.CrossRefGoogle Scholar
  7. 7.
    Dube L, Bourhis A, Jacob R. Towards a typology of virtual communities of practice. Interdiscip J Inform Knowl Manag. 2006;1:69–93.Google Scholar
  8. 8.
    Ratnapalan S, Hilliard R. Needs assessment in postgraduate medical education: a review. Med Educ Online. 2002;7:1–8. Available from: http://www.med-ed-online.org.CrossRefGoogle Scholar
  9. 9.
    Thoma B, Paddock M, Purdy E, et al. Leveraging a virtual community of practice to participate in a survey-based study: A description of the METRIQ Study Methodology. Acad Emerg Med Educ Train. 2017;1:110–3.Google Scholar
  10. 10.
    Thoma B, Sebok-Syer SS, Krishnan K, et al. Individual gestalt is unreliable for the evaluation of quality in medical education Blogs: a METRIQ study. Ann Emerg Med. 2017;  https://doi.org/10.1016/j.annemergmed.2016.12.025.Google Scholar
  11. 11.
    Cunningham CT, Quan H, Hemmelgarn B, et al. Exploring physician specialist response rates to web-based surveys. BMC Med Res Methodol. 2015;15:32.  https://doi.org/10.1186/s12874-015-0016-z.CrossRefGoogle Scholar
  12. 12.
    Riddell J, Brown A, Kovic I, Jauregui J. Who are the most influential Emergency Medicine Physicians on Twitter? West J Emerg Med. 2017;18(2):281–7.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2018

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Teresa M. Chan
    • 1
    • 2
  • David Jo
    • 3
  • Andrew W. Shih
    • 4
  • Vinai C. Bhagirath
    • 5
  • Lana A. Castellucci
    • 6
  • Calvin Yeh
    • 7
  • Brent Thoma
    • 8
  • Eric K. Tseng
    • 5
  • Kerstin de Wit
    • 1
    • 5
  1. 1.Division of Emergency Medicine, Department of MedicineMcMaster UniversityHamiltonCanada
  2. 2.McMaster Education Research, Innovation, and Theory (MERIT)HamiltonCanada
  3. 3.Michael G. DeGroote School of MedicineMcMaster UniversityHamiltonCanada
  4. 4.Department of Pathology and Molecular MedicineUniversity of British ColumbiaVancouverCanada
  5. 5.Division of Hematology and Thromboembolism, Department of MedicineMcMaster UniversityHamiltonCanada
  6. 6.Division of Hematology, Department of MedicineUniversity of OttawaOttawaCanada
  7. 7.Royal College of Physicians and Surgeons of Canada’s Emergency Medicine Training ProgramUniversity of TorontoTorontoCanada
  8. 8.Department of Emergency MedicineUniversity of SaskatchewanSaskatoonCanada

Personalised recommendations