Abstract
Conducting mixed methods research is critical for healthcare researchers to understand attitudes, behaviors, and experiences on health-related topics, such as vaccine acceptance. As the COVID-19 pandemic has made it difficult to employ traditional, face-to-face qualitative methodologies, this paper describes the use of a virtual platform to conduct person-centered research. To overcome these challenges and better understand the attitudes and behaviors of vaccine-eligible individuals in the United States, an online health community called the Virtual Engagement Research Community (VERC) was designed and implemented. Using the Health Belief Model as a framework, the VERC employed a mixed methods approach to elicit insights, which included discussion topics, rapid polls, and surveys. Throughout the initial enrollment period of April–October 2021, continuous improvement efforts were made to bolster recruitment and member engagement. This agile research strategy was successful in utilizing mixed methods to capture community sentiments regarding vaccines. While this community focused on vaccination, the methodology holds promise for other areas of health research such as obesity, HIV, mental health disorders, and diabetes.
Plain English summary
The COVID-19 pandemic has made it difficult to conduct research in person, but qualitative and quantitative data remain critical for developing research design. To gather and understand vaccine perceptions, opinions, and behaviors in a real time manner, The Virtual Engagement Research Community (VERC) was launched. As of October 2021, the VERC had 72 members who had the opportunity to participate in research events such as surveys, polls, discussion topics, and moderated chats in a private, invitation-only virtual community. The online format offered researchers the opportunity to adapt the types and frequency of engagements, fine-tune the research questions based on community responses and current events, and adjust the analysis and reporting methodology as needed to increase participation and maximize the value of the VERC. The success of the VERC and the lessons learned support the use of online communities to enhance mixed methods research as a valuable strategy for other health researchers.
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Background
During a pandemic there are barriers to carrying out in-person research methodologies, especially in healthcare [1, 2]. Stakeholders (e.g., public health authorities, healthcare providers, research teams, and potential participants) may be preoccupied with providing care, dealing with resource constraints, or struggling with the personal impact of a pandemic. Furthermore, carrying out fieldwork during a pandemic could expose researchers or participants to infection, especially if close contact with affected communities or health care facilities occurs. The progression of the COVID-19 pandemic created a rapidly evolving landscape, which required research methodologies to quickly adapt to real-time needs. It challenged researchers to identify new ways to recruit and engage participants, for both qualitative and quantitative research [3]. The pandemic also revealed new research topics, such as how and why individuals are adopting safety behavior recommendations and mandates, attitudes towards vaccines in development, as well as what information sources are deemed credible. Lastly, given the heightened sensitivity in the environment adding to the complexity of the vaccine decision-making process, research to understand attitudes and behaviors towards the COVID-19 vaccine became critical [4].
Main text
Determining a solution
The US Medical Affairs vaccine team at Janssen, the pharmaceutical company of Johnson & Johnson, has a commitment to conduct person-centered research to better understand vaccine perceptions, opinions, and behaviors, and how these may change over time given the dynamic environment. To achieve this after the COVID-19 pandemic began, the team partnered with HealthiVibe, the patient experience unit of CorEvitas LLC, to employ a methodology to continue its commitment to patient engagement research. The following criteria were considered necessary for methodological success:
-
Engagement with the same individuals over time
-
Easily accessible Internet platform so participants can engage from any computer or digital device anywhere
-
Demographic and geographic diversity among participants
-
Agility to adapt research topics in a changing landscape
-
Integration of both qualitative and quantitative research methods
-
Interactivity of participants with researchers and other participants
-
Anonymity of participants to engage with researchers and other participants openly
These elements are all characteristic of online health communities (OHCs), which offer a virtual environment to capture longitudinal insights from a diverse group of people. In these communities, individuals can be asked to participate anonymously and asynchronously in discussion topics and moderated chats, and to complete surveys and polls. These capabilities allow researchers to employ a mixed methods approach to generate a robust data set that other methodologies, such as telephone or video interviews, may not offer [5,6,7,8]. While numerous studies and health promotion efforts have been conducted to understand the factors associated with vaccine acceptance and/or hesitancy prior to the COVID-19 pandemic [9, 10], there is a paucity of data regarding utilizing OHCs to capture individuals’ attitudes, opinions, and experiences regarding vaccines. Therefore, the objective of our intervention was to understand the potential strengths and limitations of a private OHC as a nimble approach to generate evidence.
The OHC is called the “Virtual Engagement Research Community” (VERC) and is hosted on CorEvitas's social network for health, HealthUnlocked (HU; www.healthunlocked.com). HealthUnlocked is an accessible, established social network platform for health, comprising more than 1.5 million members and 315 public peer-to-peer support communities. The VERC is the only vaccine community within HealthUnlocked and is an invitation-only space. The homepage of the VERC (Fig. 1) has several features, including space to respond to discussion topics, vote in rapid polls, and search for previous posts via keywords. Chat functionality enables participants to have live, direct discussions with moderators.
Developing and launching the online health community
Inclusion and exclusion criteria were developed based on the publicly available guidance for the COVID-19 vaccines provided by the Advisory Committee on Immunization Practices (ACIP) of the Centers for Disease Control and Prevention (CDC). Table 1 outlines the inclusion and exclusion criteria, and Table 2 shows the target and current demographic mix, which prioritized individuals across different race/ethnic groups, various age ranges, and employment status. A recruitment plan was developed as well as an online screener, which was sent to current members of the HealthUnlocked platform. Individuals who expressed interest in joining the VERC completed the screener and invitations to join the VERC were sent based on the screener responses against the criteria and target demographics. Recruitment began in April 2021 and by October 2021, there were 72 members enrolled and engaged in the VERC. Table 3 shows the recruitment funnel detailing the number of screeners completed, number of invitations sent, and members enrolled as of October 31, 2021.
Participants electronically signed a consent document during the screening process. Because all data collected were de-identified, no ethics board review was required. Community guidelines were created and posted in the community with a disclaimer that the forum was not intended to discuss a specific company’s products or treatments, medical conditions, clinical trials, or financial results. HealthUnlocked’s Terms of Use and Privacy policy were readily available to members.
The VERC was designed to provide ongoing opportunities for community members to engage in research activities. Several different methodologies were used, which were categorized broadly as discussion topics and research events. Discussion topics were weekly qualitative questions posted on the VERC’s homepage. Members could reply to these questions as well comment on the replies of other members. The threads were moderated by research team members who would follow-up with questions and comments. Research events took place monthly and employed both qualitative and quantitative methodologies, including: rapid polls, surveys, live chats, and participant panels. The details of each engagement methodology can be seen in Table 4. The activities were intentionally scheduled to be more frequent at the start of the community to build engagement and then decrease slightly over time. Data were monitored on a weekly basis to capture the number of members who participated in discussion topics and research events.
Research content development
The VERC was conceptualized as a research platform to understand vaccine knowledge, attitudes, and behaviors. The initial research content was developed in early spring 2021, when the first COVID-19 vaccines were authorized for emergency use by the US Food and Drug Administration and vaccine recommendations were rolling out in phases throughout the United States. Therefore, initial priority topics were related to COVID-19 vaccine availability and willingness, vaccination experience, knowledge/information sources as well as general experiences during a novel pandemic (e.g., safety behaviors and mandates); topics later evolved to include boosters, variants, as well as impacts on healthcare and the workplace. Topics that garnered the most engagement typically were those that coincided with specific, real-time events. For example, discussion topics focused on potential boosters prompted the most responses from members as recommendations for these were being considered.
The development of the research content was grounded in the Health Belief Model (HBM), a widely accepted theoretical model that helps explain individuals’ willingness to adopt health interventions, including vaccines [11]. The key constructs within the HBM are perceived susceptibility, perceived severity, perceived benefits, barriers, self-efficacy, and cues to action. These are the beliefs an individual has that then drive their health behavior(s) [12,13,14,15]. Table 5 shows example of discussion topics and/or research events and the HBM construct they were designed to explore.
Several strategies were employed to optimize the agility of the online community platform in response to the ever-changing environment. First, the community was moderated daily to ensure that emerging themes were identified. For example, research content was developed in response to member-generated discussion around COVID-19 boosters prior to formal recommendation from the CDC. Second, qualitative discussion topics were used to inform research events and vice versa. In one topic, several members shared their experiences with close family and friends who were vaccine hesitant. Those members were invited to a live chat and had an in-depth discussion that provided valuable insight into hesitancy within their communities. Finally, research content was adapted based on community members’ engagement patterns; some members responded to discussion topics, but not research events. Therefore, community members were asked about key topics in both formats to collect comprehensive quantitative and qualitative data.
Discussion
Implementing any intervention requires careful planning, and OHCs are no exception. This OHC is not modeled directly on any previous research, but it was designed to build on prior experience with virtual research methods, focused on one diagnosis or health condition. No patients or members of the public were consulted as part of the design of this research or its analysis. This experience showed the importance for researchers to consider the circumstances of the population they are trying to reach. The VERC accomplished this by leveraging technology to gather insights into vaccine attitudes and behaviors at a time when other, more traditional methodologies would have proven difficult. Additionally, some key strengths of the OHC methodology allowed the researchers to gather these insights effectively, including:
-
Allowing community members to engage anywhere and asynchronously, reducing logistical or geographic limitations and member burden
-
Maintaining the anonymity of community members, which may increase their comfort sharing information and encourage honest participation, especially with politically charged and potentially sensitive topics
-
Creating an agile and adaptive methodology that allows for learning throughout the process and implementation of continuous improvement
-
Building diversity in types of engagements and topics, which can appeal to a broad audience and maintain interest
-
Utilizing a mixed methods approach, which provides the advantages of both qualitative and quantitative research
-
Granting members the opportunity to reply to other members’ comments and create their own posts, which provides opportunities for community building and organically emerging themes
-
Providing ongoing engagement over a period of time, allowing the community to be leveraged for new topics (e.g. vaccines for other diseases) and creating opportunities to revisit topics for longitudinal comparison
There are, however, some important limitations to the OHC methodology to consider. For example:
-
Lack of member compensation, relying instead on members’ interest in the topics and satisfaction/engagement with the platform
-
Member comfort with technology and access to a computer and internet was necessary, which could introduce sample bias
-
Communication is entirely written, which required members to have a higher degree of literacy
-
The platform allows members to edit or wordsmith their comments before posting, which could allow them to alter their true response and make it more “socially acceptable”
The advantages to this person-centered research strategy outweigh the limitations; in particular, the agile platform allows for rapid response to the community sentiment and the ability to employ a mixed methods approach. Researchers interested in replicating this methodology should be prepared to evolve their strategies and thinking, but doing so allows the OHCs to be continually used throughout a longer period of time. While this paper focuses on the implementation of the VERC methodology and key lessons learned between April–October 2021, the platform is still active and research is ongoing.
Lessons learned
The need for person-centered, qualitative and quantitative research was critical before the COVID-19 pandemic, but the circumstances since March 2020 have required nimble and evidence-based approaches [16, 17]. Developing and implementing an OHC during this time proved to be a successful method to better understand vaccine attitudes and behaviors. As the VERC progressed, three key lessons emerged from the methodology: recruitment, engagement, and data analysis/reporting.
The goal was to recruit 100 participants into a demographically diverse community and progress toward this goal was actively monitored. As of October 31, 2021, 72 members were enrolled in the community. The initial recruitment strategy focused on outreach to existing HealthUnlocked users who matched the eligibility criteria. This led to the community skewing towards mostly white individuals with underlying conditions who were greater than 50 years of age and with similar types of employment (e.g., healthcare providers or non-medical essential workers). To recruit younger members from different racial/ethnic and employment backgrounds, the HealthUnlocked team pursued outreach on other social media channels, sent personalized emails to racially/ethnically diverse people active in other HealthUnlocked communities, and leveraged strong relationships within the communities of interest. As a result, the number of racial/ethnic minority groups increased from less than 20% in April to 42% by October. It may, however, have resulted in sampling bias as members of HealthUnlocked may be more likely to be willing to discuss health related topics, limiting the generalizability of findings.
The VERC was designed to capture attitudes of those who were recommended a COVID-19 vaccine and may potentially choose to be vaccinated at the time of roll out. Therefore, certain exclusion criteria were decided upon. For example, potential members were excluded if they were actively being treated for cancer because there was no recommendation for immunocompromised populations at the time of enrollment. Other exclusion criteria were also applied. Members who reported themselves as ‘not-at-all’ or ‘somewhat’ comfortable with using technology were excluded from participating due to the online nature of the community and the different digital methods employed. Additionally, individuals who reported themselves as being ‘completely opposed to vaccination’ were excluded. It was hypothesized that excluding these potential members would allow the research team to more accurately assess the nuance of opposition to specific vaccines, such as the COVID-19 vaccine, and facilitate a continuous dialogue related to vaccines in general. Also this group would likely not be a group that would accept the COVID-19 vaccines during the roll out. While this resulted in a cohort that included individuals who were, at most, ‘somewhat opposed’ to vaccines, the initial results still indicate that the community members have a wide variety of attitudes towards vaccines. To correct for this potential bias, questions were asked related to vaccine opposition. For example, members were asked about opposition to vaccines among their families, social circles, and community. Opportunities may exist for future research to use OHCs to explore individuals’ vaccine opposition.
Community member engagement was initially strong throughout April and May, but began to wane in late June and July. It is not believed that this is due to drop out, as only 5 members had left the community between April and October 2021. This may have been multifactorial: posting weekly discussion topics may have been too frequent and/or members chose to log in less often; some members also may have felt uncomfortable commenting on politically charged topics or providing a minority opinion. Additionally, while beneficial to ask similar questions in both the discussion topics and the research events, community members may have felt this was redundant or that they had previously contributed similar information.
To re-engage members, several approaches were employed, including sending members email reminders and personal direct messages within HealthUnlocked, ‘tagging’ specific community members in discussion threads, increasing the length of time members could participate in research events, and conducting the research events that previously garnered more engagement (e.g., surveys). These adjustments successfully increased engagement on discussion topics and research events. Researchers interested in replicating this methodology could employ similar tactics to maintain engagement. While the VERC was able to be accessed through both computers and digital devices, data is not available to understand which platforms members used to access the community. Understanding how the platform is used is an opportunity for future research and this insight could help design more effective studies in the future.
Ongoing data analysis and reporting were performed to respond to the changing environment and data trends within the VERC. Initially, research events and discussion topics were reported on individually and at different cadences. Because community members’ contributions to the discussion topics were thoughtful and thorough, and members were responsive to moderators’ follow-up questions, it became clear that discussion topics and research events could inform and complement one another. For example, surveys were conducted to learn about trusted sources of information sources, while members provided more detailed insights about these sources in the discussion topics. Therefore, the analysis and reporting approach pivoted to focus more on emerging themes across all the data sources and compiled into a monthly report for review, discussion, and action. Researchers should consider analyzing all data sources regularly to generate robust and meaningful reporting to more effectively guide the development of research content.
Conclusion
Online health communities such as the VERC appear to be effective for conducting vaccine-related research and may be a potential platform to conduct health research in other areas of public health. They provide an anonymous method to collect longitudinal insights from individuals who have conditions linked to an increased risk for stigmatization [18] including, but not limited to, obesity [19, 20], mental health disorders [21,22,23], HIV [24, 25], and both type 2 [19, 26] and juvenile diabetes [27, 28]. This is because the anonymous nature of OHCs may provide a’safe’ space for patients to share their experiences openly. Additionally, some conditions may make it harder for patients to engage in traditional qualitative work and this method allows researchers to pursue real-time research activities that could improve understanding of these patients’ experiences and help inform interventions designed to improve health outcomes among these populations. Further research is needed into the effectiveness of OHCs overall, which medical conditions or health behaviors could benefit most from this approach, and how to sustain and grow an engaged community over a longer period of time.
Availability of data and materials
Not applicable.
Abbreviations
- ACIP:
-
Advisory Committee on Immunization Practices
- CDC:
-
Centers for Disease Control and Prevention
- COPD:
-
Chronic obstructive pulmonary disease
- COVID-19:
-
Coronavirus disease 2019
- HBM:
-
Health belief model
- HCP:
-
Healthcare provider
- OHC:
-
Online health community
- VERC:
-
Virtual Engagement Research Community
References
Pocock T, Smith M, Wiles J. Recommendations for virtual qualitative health research during a pandemic. Qual Health Res. 2021;31(13):2403–13.
Graetz DE, Sniderman E, Villegas C, et al. Utilizing multilingual methods and rapid analysis for global qualitative research during a pandemic. Glob Qual Nurs Res. 2022;26(9):23333936221080970. https://doi.org/10.1177/23333936221080969.
Tremblay S, Castiglione AL, et al. Conducting qualitative research to respond to COVID-19 challenges: reflections for the present and beyond. J Qual Methods. 2021;20:1–8.
Rane MS, Kochhar S, Poehlein E, et al. Chasing Covid Cohort Study Team FT: determinants and trends of COVID-19 vaccine hesitancy and vaccine uptake in a national cohort of US adults: a longitudinal study. Am J Epidemiol. 2022;191:570.
Makai P, Perry M, Robben SH, Schers H, Heinen M, Olde Rikkert MG, Melis RJ. Which frail older patients use online health communities and why? A mixed methods process evaluation of use of the Health and Welfare portal. J Med Internet Res. 2014;16(12):e278.
Barbabella F, Poli A, Andréasson F, Salzmann B, et al. A web-based psychosocial intervention for family caregivers of older people: results from a mixed-methods study in three European countries. JMIR Res Protoc. 2016;5(4):e196.
Liang OS, Chen Y, Bennett DS, Yang CC. Identifying self-management support needs for pregnant women with opioid misuse in online health communities: mixed methods analysis of web posts. J Med Internet Res. 2021;23(2):e18296.
Cohen J, Patterson P, Noke M, Clarke K, Husson O. Development and evaluation of the canteen connect online health community: using a participatory design approach in meeting the needs of young people impacted by cancer. Cancers (Basel). 2021;14(1):50.
Trogen B, Pirofski LA. Understanding vaccine hesitancy in COVID-19. Med (N Y). 2021;2(5):498–501.
de Figueiredo A, Simas C, Karafillakis E, Paterson P, Larson HJ. Mapping global trends in vaccine confidence and investigating barriers to vaccine uptake: a large-scale retrospective temporal modelling study. Lancet. 2020;396(10255):898–908.
Carpenter CJ. A meta-analysis of the effectiveness of health belief model variables in predicting behavior. Health Commun. 2010;25(8):661–9.
Youssef D, Abou-Abbas L, Berry A, Youssef J, Hassan H. Determinants of acceptance of Coronavirus disease-2019 (COVID-19) vaccine among Lebanese health care workers using health belief model. PLoS ONE. 2022;17(2):e0264128.
Reindl D, Catma S. A pre-vaccine analysis using the Health Belief Model to explain parents’ willingness to vaccinate (WTV) their children in the United States: implications for vaccination programs. Expert Rev Pharmacoecon Outcomes. 2022. https://doi.org/10.1080/14737167.2022.2045957.
Coe AB, Gatewood SB, Moczygemba LR, Goode JV, Beckner JO. The use of the health belief model to assess predictors of intent to receive the novel H1N1 influenza vaccine. Innov Pharm. 2012;3(2):1–11.
Shmueli L. Predicting intention to receive COVID-19 vaccine among the general population using the health belief model and the theory of planned behavior model. BMC Public Health. 2021;21(1):804.
Roberts J, Pavlakis A, Richards M. It’s more complicated than it seems: virtual qualitative research in the COVID-19 era. Int J Qual Methods. 2021;20:201–13.
Salmon D, Opel DJ, Dudley MZ, Brewer J, Breiman R. Reflections on governance, communication, and equity: challenges and opportunities in covid-19 vaccination. Health Aff (Millwood). 2021;40(3):419–25.
Engebretson J. Understanding stigma in chronic health conditions: implications for nursing. J Am Assoc Nurse Pract. 2013;25(10):545–50.
Rubino F, Puhl RM, Cummings DE, et al. Joint international consensus statement for ending stigma of obesity. Nat Med. 2020;26(4):485–97.
Puhl RM, Himmelstein MS, Speight J. Weight stigma and diabetes stigma: implications for weight-related health behaviors in adults with type 2 diabetes. Clin Diabetes. 2022;40(1):51–61.
Latalova K, Ociskova M, Prasko J, Kamaradova D, Jelenova D, Sedlackova Z. Self-stigmatization in patients with bipolar disorder. Neuro Endocrinol Lett. 2013;34(4):265–72 (PMID: 23803872).
Townsend M, Pareja K, Buchanan-Hughes A, et al. aAntipsychotic-related stigma and the impact on treatment choices: a systematic review and framework synthesis. Patient Prefer Adher. 2022;16(16):373–401. https://doi.org/10.2147/PPA.S343211.
Conceição V, Rothes I, Gusmão R. The association between stigmatizing attitudes towards depression and help seeking attitudes in college students. PLoS One. 2022;17(2):1.
Armoon B, Higgs P, Fleury MJ, et al. Socio-demographic, clinical and service use determinants associated with HIV related stigma among people living with HIV/AIDS: a systematic review and meta-analysis. BMC Health Serv Res. 2021;21:1004.
Greenwood GL, Wilson A, Bansal GP, et al. HIV-related stigma research as a priority at the national institutes of health. AIDS Behav. 2022;26(Suppl 1):5–26.
Abdoli S, Doosti Irani M, Hardy LR, Funnell M. A discussion paper on stigmatizing features of diabetes. Nurs Open. 2018;5(2):113–9.
Montali L, Zulato E, Cornara M, Ausili D, Luciani M. Barriers and facilitators of type 1 diabetes self-care in adolescents and young adults. J Pediatr Nurs. 2022;62:136–43.
Browne JL, Ventura A, Mosely K, Speight J. “I’m not a druggie, I’m just a diabetic”: a qualitative study of stigma from the perspective of adults with type 1 diabetes. Br Med J Open. 2014;4(7):e005625.
Acknowledgements
Not applicable.
Funding
This project was funded by the US Medical Affairs vaccine team at Janssen, the pharmaceutical company of Johnson & Johnson.
Author information
Authors and Affiliations
Contributions
ST, PG, LS, GG, JB, NA, JL developed the VERC from its inception, including the development of materials, research content, recruitment and engagement strategy. JS was a key contributor in writing the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
All data collected within the VERC were de-identified; thus, no ethics board review was required. Participants electronically signed a consent document when completing the screener. Within the community, there was also a prominent disclaimer that the forum was not intended to discuss Janssen or other company’s products or treatments, medical conditions, clinical trials, or financial results. The purpose of this study was to collect participants’ personal perspectives and qualitative insights. The study was also conducted in accordance with the Helsinki Declaration of 1964 and its later amendments.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Shea, L., Bushen, J., Ahmad, N. et al. Development and implementation of an online community as a strategy for mixed methods research during a pandemic. Res Involv Engagem 8, 47 (2022). https://doi.org/10.1186/s40900-022-00383-5
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s40900-022-00383-5