Design Considerations for Implementing eHealth Behavioral Interventions for HIV Prevention in Evolving Sociotechnical Landscapes
Purpose of review
Despite tremendous potential for public health impact and continued investments in development and evaluation, it is rare for eHealth behavioral interventions to be implemented broadly in practice. Intervention developers may not be planning for implementation when designing technology-enabled interventions, thus creating greater challenges for real-world deployment following a research trial. To facilitate faster translation to practice, we aimed to provide researchers and developers with an implementation-focused approach and set of design considerations as they develop new eHealth programs.
Using the Accelerated Creation-to-Sustainment model as a lens, we examined challenges and successes experienced during the development and evaluation of four diverse eHealth HIV prevention programs for young men who have sex with men: Keep It Up!, Harnessing Online Peer Education, Guy2Guy, and HealthMindr. HIV is useful for studying eHealth implementation because of the substantial proliferation of diverse eHealth interventions with strong evidence of reach and efficacy and the responsiveness to rapid and radical disruptions in the field. Rather than locked-down products to be disseminated, eHealth interventions are complex sociotechnical systems that require continual optimization, vigilance to monitor and troubleshoot technological issues, and decision rules to refresh content and functionality while maintaining fidelity to core intervention principles. Platform choice and sociotechnical relationships (among end users, implementers, and the technology) heavily influence implementation needs and challenges. We present a checklist of critical implementation questions to address during intervention development.
In the absence of a clear path forward for eHealth implementation, deliberate design of an eHealth intervention’s service and technological components in tandem with their implementation plans is critical to mitigating barriers to widespread use. The design considerations presented can be used by developers, evaluators, reviewers, and funders to prioritize the pragmatic scalability of eHealth interventions in research.
KeywordseHealth mHealth Implementation Intervention development Scalability Sustainability HIV Young men who have sex with men
(adolescent/young) men who have sex with men
Accelerated Creation-to-Sustainment (model)
Community Popular Opinion Leader (model)
human immunodeficiency virus
Harnessing Online Peer Education
Information–Motivation–Behavioral Skills (model)
Keep It Up!
randomized controlled trial
sexually transmitted infection
We thank all of our Ce-PIM colleagues and community advisors for their contributions.
Availability of Data and Material
Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.
DL led the analysis and writing of the manuscript. BM, SY, CG, and PS provided details about the four interventions and initial drafts of those sections. CHB provided guidance on implementation science. BM provided guidance on eHealth. EM provided review and feedback. All authors contributed significantly to conceptually framing the manuscript and its conclusions, and all authors read and approved the final manuscript.
This study is funded by the National Institutes of Health and other sponsors. Development, evaluation, and implementation of Keep It Up! was previously sponsored by the National Institute of Mental Health (R34MH079714, Mustanski), National Institute on Drug Abuse (R01DA035145, Mustanski), Chicago Department of Public Health, and ViiV Healthcare. Harnessing Online Peer Education was previously sponsored by the National Institute of Mental Health (K01MH090884, Young) and UCLA Center for AIDS Research (P30AI028697, Zack), with ongoing sponsorship from the former (R01MH106415, Young). Guy2Guy was previously sponsored by the National Institute of Mental Health (R01MH096660, Mustanski & Ybarra), with ongoing work supported by the Third Coast Center for AIDS Research (P30AI117943, D’Aquila) and the American Foundation for Suicide Prevention (SRG-0-110-15, Pisani). HealthMindr was previously sponsored by the MAC AIDS Fund, Emory Center for AIDS Research (P30AI050409, Del Rio), and National Institute on Drug Abuse (R01DA045612, Sullivan).
This manuscript was a product of a workgroup on eHealth interventions formed through the Center for Prevention Implementation Methodology for Drug Abuse and HIV (Ce-PIM), which is supported by the National Institute on Drug Abuse (P30DA027828, Brown & Mustanski). This work was also supported by the Keep It Up! 3.0 grant from the National Institute of Mental Health, National Institute on Drug Abuse, Office of Behavioral and Social Sciences Research, and Office of Disease Prevention at the National Institutes of Health (R01MH118213, Mustanski), as well as an individual postdoctoral fellowship grant from the National Institute on Drug Abuse (F32DA046313, Morgan). The sponsors had no involvement in the conduct of the research or the preparation of the manuscript. The content is the sole responsibility of the authors and does not necessarily represent the official views of the funders.
Compliance with Ethical Standards
Conflict of Interest
Dr. Sullivan reports grants and personal fees from NIH, grants and personal fees from CDC, grants and personal fees from Gilead Sciences outside the submitted work.
Dr. Li reports grants from National Institutes of Health, during the conduct of the study.
Dr. Brown reports grants from National Institute on Drug Abuse/NIH, during the conduct of the study.
Dr. Mustanski reports grants from National Institutes of Health, during the conduct of the study.
Dr. Morgan reports grants from National Institutes of Health, during the conduct of the study.
Dr. Gallo has nothing to disclose.
Dr. Young has nothing to disclose.
Human and Animal Rights and Informed Consent
This article does not contain any studies with human or animal subjects performed by any of the authors.
The authors declare that they have no competing interests.
Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance
- 2.Bennett GG, Glasgow RE. The delivery of public health interventions via the Internet: actualizing their potential. Annu Rev Public Health. 2009;30:273–92. https://doi.org/10.1146/annurev.publhealth.031308.100235.CrossRefPubMedGoogle Scholar
- 3.Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. Gen Hosp Psychiatry. 2013;35(4):332–8. https://doi.org/10.1016/j.genhosppsych.2013.03.008.CrossRefPubMedGoogle Scholar
- 4.Bailey J, Mann S, Wayal S, Hunter R, Free C, Abraham C, et al. Sexual health promotion for young people delivered via digital media: a scoping review. Public Health Res. 2015;3(13). https://doi.org/10.3310/phr03130.
- 8.Brown CH, Mohr DC, Gallo CG, Mader C, Palinkas L, Wingood G, et al. A computational future for preventing HIV in minority communities: how advanced technology can improve implementation of effective programs. J Acquir Immune Defic Syndr. 2013;63(Suppl 1):S72–84. https://doi.org/10.1097/QAI.0b013e31829372bd.CrossRefPubMedPubMedCentralGoogle Scholar
- 11.Muessig KE, Nekkanti M, Bauermeister J, Bull S, Hightow-Weidman LB. A systematic review of recent smartphone, Internet and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep. 2015;12(1):173–90. https://doi.org/10.1007/s11904-014-0239-3.CrossRefPubMedPubMedCentralGoogle Scholar
- 16.• Mohr DC, Lyon AR, Lattie EG, Reddy M, Schueller SM. Accelerating digital mental health research from early design and creation to successful implementation and sustainment. J Med Internet Res. 2017;19(5):e153. https://doi.org/10.2196/jmir.7725 This paper describes the Accelerated Creation-to-Sustainment (ACTS) model, a process model for producing functioning technology-enabled services that are sustainable in a real-world treatment settings. Its deconstruction of eHealth interventions into service, technology, and implementation components forms the framework for the current study.CrossRefPubMedPubMedCentralGoogle Scholar
- 17.Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform. 2000;1:65–70.Google Scholar
- 18.Allison S, Bauermeister JA, Bull S, Lightfoot M, Mustanski B, Shegog R, et al. The intersection of youth, technology, and new media with sexual health: moving the research agenda forward. J Adolesc Health. 2012;51(3):207–12. https://doi.org/10.1016/j.jadohealth.2012.06.012.CrossRefPubMedPubMedCentralGoogle Scholar
- 26.Centers for Disease Control and Prevention. HIV among youth. 2016. http://www.cdc.gov/hiv/group/age/youth/. Accessed 18 Nov 2018.
- 27.Centers for Disease Control and Prevention. CDC Fact Sheet: HIV among Gay and Bisexual Men. 2017. https://www.cdc.gov/nchhstp/newsroom/docs/factsheets/cdc-msm-508.pdf. Accessed 18 Nov 2018.
- 28.Centers for Disease Control and Prevention. HIV prevention in the United States: New opportunities, new expectations. Atlanta, GA: CDC; 2015.Google Scholar
- 37.Centers for Disease Control and Prevention. HIV infection risk, prevention, and testing behaviors among men who have sex with men - National HIV Behavioral Surveillance, 20 U.S. cities, 2014. 2016.Google Scholar
- 38.Harper GW, Bruce D, Serrano P, Jamil OB. The role of the Internet in the sexual identity development of gay and bisexual male adolescents. In: Hammack PL, Cohler BJ, editors. The story of sexual identity: narrative perspectives on the gay and lesbian life course. New York, NY: Oxford University Press; 2009. p. 297–326.CrossRefGoogle Scholar
- 39.Macapagal K, Moskowitz DA, Li DH, Carrion A, Bettin E, Fisher CB, et al. Hookup app use, sexual behavior, and sexual health among adolescent men who have sex with men in the United States. J Adolesc Health. 2018;62(6):708–15. https://doi.org/10.1016/j.jadohealth.2018.01.001.CrossRefPubMedPubMedCentralGoogle Scholar
- 40.Lenhart A. Teen, social media and technology overview 2015. Pew Research Center; 2015.Google Scholar
- 46.Bauermeister JA, Pingel ES, Jadwin-Cakmak L, Harper GW, Horvath K, Weiss G, et al. Acceptability and preliminary efficacy of a tailored online HIV/STI testing intervention for young men who have sex with men: the Get Connected! program. AIDS Behav. 2015;19(10):1860–74. https://doi.org/10.1007/s10461-015-1009-y.CrossRefPubMedPubMedCentralGoogle Scholar
- 47.Mustanski B, Garofalo R, Monahan C, Gratzer B, Andrews R. Feasibility, acceptability, and preliminary efficacy of an online HIV prevention program for diverse young men who have sex with men: the Keep It Up! intervention. AIDS Behav. 2013;17(9):2999–3012. https://doi.org/10.1007/s10461-013-0507-z.CrossRefPubMedGoogle Scholar
- 52.Rotheram-Borus MJ, Swendeman D, Chovnick G. The past, present, and future of HIV prevention: integrating behavioral, biomedical, and structural intervention strategies for the next generation of HIV prevention. Annu Rev Clin Psychol. 2009;5:143–67. https://doi.org/10.1146/annurev.clinpsy.032408.153530.CrossRefPubMedPubMedCentralGoogle Scholar
- 54.Centers for Disease Control and Prevention. Supported activities: prioritizing high impact HIV prevention. 2015. https://www.cdc.gov/hiv/programresources/healthdepartments/supportedactivities.html. Accessed 18 Nov 2018.
- 55.Kaiser Family Foundation. U.S. Federal Funding for HIV/AIDS: Trends Over Time. 2017. https://www.kff.org/global-health-policy/fact-sheet/u-s-federal-funding-for-hivaids-trends-over-time/. Accessed 18 Nov 2018.
- 56.•• Aarons GA, Sklar M, Mustanski B, Benbow N, Brown CH. “Scaling-out” evidence-based interventions to new populations or new health care delivery systems. Implement Sci. 2017;12(1):111. https://doi.org/10.1186/s13012-017-0640-6 This paper provides a conceptual framework for determining what new empirical evidence is required for an intervention to retain its evidence-based standard when adapting to a new population, new delivery context, or both. It also describes when an adaptation can “borrow strength” from prior evidence of impact.CrossRefPubMedPubMedCentralGoogle Scholar
- 60.Fisher JD, Fisher WA. The information-motivation-behavioral skills model. In: DiClemente RJ, Crosby RA, Kegler MC, editors. Emerging theories in health promotion practice and research: strategies for improving public health. San Francisco: Jossey-Bass; 2002. p. 40–70.Google Scholar
- 61.Mustanski B, Madkins K, Greene GJ, Parsons JT, Johnson BA, Sullivan P, et al. Internet-based HIV prevention with at-home sexually transmitted infection testing for young men having sex with men: study protocol of a randomized controlled trial of Keep It Up! 2.0. JMIR Res Protoc. 2017;6(1):e1. https://doi.org/10.2196/resprot.5740.CrossRefPubMedPubMedCentralGoogle Scholar
- 63.Centers for Disease Control and Prevention. Evidence of HIV treatment and viral suppression in preventing the sexual transmission of HIV. 2018. https://www.cdc.gov/hiv/pdf/risk/art/cdc-hiv-art-viral-suppression.pdf. Accessed 18 Nov 2018.
- 65.Turner-McGrievy GM, Hales SB, Schoffman DE, Valafar H, Brazendale K, Weaver RG, et al. Choosing between responsive-design websites versus mobile apps for your mobile behavioral intervention: presenting four case studies. Transl Behav Med. 2017;7(2):224–32. https://doi.org/10.1007/s13142-016-0448-y.CrossRefPubMedGoogle Scholar
- 67.Brown CH, Curran G, Palinkas LA, Aarons GA, Wells KB, Jones L, et al. An overview of research and evaluation designs for dissemination and implementation. Annu Rev Public Health. 2017;38:1–22. https://doi.org/10.1146/annurev-publhealth-031816-044215.CrossRefPubMedPubMedCentralGoogle Scholar
- 68.Rogers EM. Diffusion of innovations. 5th ed. New York, NY: Free Press; 2003.Google Scholar
- 73.Young SD, Koussa M, Lee J, Gill K, Perez H, Gelberg L, Heinzerling K. Feasibility of a social media/online community support group intervention among chronic pain patients on opioid therapy. J Addict Dis. in press.Google Scholar
- 75.Young SD. Stick with it: a scientifically proven process for changing your life--for good. New York, NY: Harper; 2017.Google Scholar
- 78.Ybarra ML, Prescott TL, Phillips GL, 2nd, Bull SS, Parsons JT, Mustanski B. Pilot RCT results of an mHealth HIV prevention program for sexual minority male adolescents. Pediatrics. 2017;140(1). https://doi.org/10.1542/peds.2016-2999.
- 79.Gilead Sciences Inc. U.S. Food and Drug Administration approves expanded indication for Truvada® (emtricitabine and tenofovir disoproxil fumarate) for reducing the risk of acquiring HIV-1 in adolescents. Business Wire; 2018.Google Scholar
- 80.Gallo C, Moran K, Brown CH, Mustanski B. Optimizing HIV prevention mHealth interventions by measuring linguistic alignment. 10th Annual Conference on the Science of Dissemination and Implementation; Arlington, VA. 2017.Google Scholar
- 81.Sullivan PS, Driggers R, Stekler JD, Siegler A, Goldenberg T, McDougal SJ, et al. Usability and acceptability of a mobile comprehensive HIV prevention app for men who have sex with men: a pilot study. JMIR Mhealth Uhealth. 2017;5(3):e26. https://doi.org/10.2196/mhealth.7199.CrossRefPubMedPubMedCentralGoogle Scholar
- 82.Bandura A. Social foundations of thought and action: a social cognitive theory. Prentice Hall: Engelwood Cliffs, NJ; 1986.Google Scholar
- 83.Goldenberg T, McDougal SJ, Sullivan PS, Stekler JD, Stephenson R. Building a mobile HIV prevention app for men who have sex with men: an iterative and community-driven process. JMIR Public Health Surveill. 2015;1(2):e18. https://doi.org/10.2196/publichealth.4449.CrossRefPubMedPubMedCentralGoogle Scholar
- 85.Sullivan PS, Sineath C, Kahle EST. Awareness, willingness, and use of oral pre-exposure prophylaxis (PrEP) among a national sample of US men who have sex with men. Amsterdam: AIDS Impact; 2015.Google Scholar
- 86.Siegler AJ, Wirtz S, Weber S, Sullivan PS. Developing a web-based geolocated directory of HIV pre-exposure prophylaxis-providing clinics: the PrEP locator protocol and operating procedures. JMIR Public Health Surveill. 2017;3(3):e58. https://doi.org/10.2196/publichealth.7902.CrossRefPubMedPubMedCentralGoogle Scholar
- 87.Mustanski B, Feinstein BA, Madkins K, Sullivan P, Swann G. Prevalence and risk factors for rectal and urethral sexually transmitted infections from self-collected samples among Young men who have sex with men participating in the Keep It Up! 2.0 randomized controlled trial. Sex Transm Dis. 2017;44(8):483–8. https://doi.org/10.1097/OLQ.0000000000000636.CrossRefPubMedPubMedCentralGoogle Scholar
- 89.Whittaker R. Key issues in mobile health and implications for New Zealand. Health Care and Informatics Review Online. 2012;16(2):2–7.Google Scholar
- 90.Centers for Disease Control and Prevention. Compendium of evidence-based interventions and best practices for HIV prevention. 2019. http://www.cdc.gov/hiv/prevention/research/compendium/rr/complete.html. Accessed 12 Mar 2019.
- 94.Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med. 2007;32(5 Suppl):S112–8. https://doi.org/10.1016/j.amepre.2007.01.022.CrossRefPubMedPubMedCentralGoogle Scholar
- 99.• Agarwal S, Lefevre AE, Labrique AB. A call to digital health practitioners: new guidelines can help improve the quality of digital health evidence. JMIR Mhealth Uhealth. 2017;5(10):e136. https://doi.org/10.2196/mhealth.6640 This paper presents the World Health Organization’s mHealth Evidence Reporting and Assessment (mERA) checklist of 16 core mHealth items and 29 methodology items to ensure comprehensiveness and quality of reporting on critical aspects of digital health program effectiveness research.CrossRefPubMedPubMedCentralGoogle Scholar
- 100.Bartholomew Eldredge LK, Parcel GS, Kok G, Gottlieb NH, Fernandez ME. Planning health promotion programs: an intervention mapping approach. Fourth edition. Ed. San Francisco, CA: Jossey-Bass & Pfeiffer Imprints, Wiley; 2016.Google Scholar