Skip to main content
SpringerLink
Log in

Methodological Issues in Research on Web-Based Behavioral Interventions

  • Original Article
  • Published:
Annals of Behavioral Medicine

Abstract

Background

Web-based behavioral intervention research is rapidly growing.

Purpose

We review methodological issues shared across Web-based intervention research to help inform future research in this area.

Methods

We examine measures and their interpretation using exemplar studies and our research.

Results

We report on research designs used to evaluate Web-based interventions and recommend newer, blended designs. We review and critique methodological issues associated with recruitment, engagement, and social validity.

Conclusions

We suggest that there is value to viewing this burgeoning realm of research from the broader context of behavior change research. We conclude that many studies use blended research designs, that innovative mantling designs such as the Multiphase Optimization Strategy and Sequential Multiple Assignment Randomized Trial methods hold considerable promise and should be used more widely, and that Web-based controls should be used instead of usual care or no-treatment controls in public health research. We recommend topics for future research that address participant recruitment, engagement, and social validity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Onken LS, Blaine JD, Battjes RJ. Behavioral therapy research: A conceptualization of a process. In: Henngler SW, Amentos R, eds. Innovative Approaches From Difficult-To-Treat Populations. Washington: American Psychiatric Press; 1997: 477–485.

    Google Scholar 

  2. Eng TR, Gustafson DH, Henderson J, Jimison H, Patrick K. Introduction to evaluation of interactive health communication applications. Science Panel on Interactive Communication and Health. Am J Prev Med. 1999; 16(1): 10–15.

    Article  PubMed  CAS  Google Scholar 

  3. Rounsaville BJ, Carroll KM, Onken LS. A stage model of behavioral therapies research: Getting started and moving on from Stage I. Clin Psychol Sci Pract. 2001; 8(133): 142.

    Google Scholar 

  4. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: The RE-AIM framework. Am J Public Health. 1999; 89(9): 1322–1327.

    Article  PubMed  CAS  Google Scholar 

  5. Dansky KH, Thompson D, Sanner T. A framework for evaluating eHealth research. Eval Program Plann. 2006; 29: 397–404.

    Article  PubMed  Google Scholar 

  6. Carroll KM, Rounsaville BJ. Bridging the gap: A hybrid model to link efficacy and effectiveness research in substance abuse treatment. Psychiatr Serv. 2003; 54(3): 333–339.

    Article  PubMed  Google Scholar 

  7. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: Increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003; 290(12): 1624–1632.

    Article  PubMed  CAS  Google Scholar 

  8. Hotopf M, Churchill R, Lewis G. Pragmatic randomised controlled trials in psychiatry. Br J Psychiatry. 1999; 175: 217–223.

    Article  PubMed  CAS  Google Scholar 

  9. Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med. 2007; 32(5 Suppl): S119–S126.

    Article  PubMed  Google Scholar 

  10. Harvey-Berino J, Pintauro SJ, Gold EC. The feasibility of using Internet support for the maintenance of weight loss. Behav Modif. 2002; 26(1): 103–116.

    Article  PubMed  Google Scholar 

  11. Lorig KR, Ritter PL, Laurent DD, Plant K. Internet-based chronic disease self-management: A randomized trial. Med Care. 2006; 44(11): 964–971.

    Article  PubMed  Google Scholar 

  12. Ljotsson B, Lundin C, Mitsell K, et al. Remote treatment of bulimia nervosa and binge eating disorder: A randomized trial of Internet-assisted cognitive behavioural therapy. Behav Res Ther. 2007; 45(4): 649–661.

    Article  PubMed  CAS  Google Scholar 

  13. Strecher VJ, Shiffman S, West R. Randomized controlled trial of a web-based computer-tailored smoking cessation program as a supplement to nicotine patch therapy. Addiction. 2005; 100(5): 682–688.

    Article  PubMed  Google Scholar 

  14. Tate DF, Wing RR, Winett RA. Using Internet technology to deliver a behavioral weight loss program. JAMA. 2001; 285(9): 1172–1177.

    Article  PubMed  CAS  Google Scholar 

  15. Bond GE, Burr R, Wolf FM, et al. Preliminary findings of the effects of comorbidities on a web-based intervention on self-reported blood sugar readings among adults age 60 and older with diabetes. Telemed J E Health. 2006; 12(6): 707–710.

    Article  PubMed  Google Scholar 

  16. Gustafson DH, Hawkins R, Boberg E, et al. Impact of a patient-centered, computer-based health information/support system. Am J Prev Med. 1999; 16(1): 1–9.

    Article  PubMed  CAS  Google Scholar 

  17. Gerber BS, Solomon MC, Shaffer TL, Quinn MT, Lipton RB. Evaluation of an internet diabetes self-management training program for adolescents and young adults. Diabetes Technol Ther. 2007; 9(1): 60–67.

    Article  PubMed  Google Scholar 

  18. Glasgow RE, Davidson KW, Dobkin PL, Ockene J, Spring B. Practical behavioral trials to advance evidence-based behavioral medicine. Ann Behav Med. 2006; 31(1): 5–13.

    Article  PubMed  Google Scholar 

  19. Wangberg SC. An Internet-based diabetes self-care intervention tailored to self-efficacy. Health Educ Res. 2008; 23(1): 170–179.

    Article  PubMed  Google Scholar 

  20. Tate DF, Jackvony EH, Wing RR. Effects of Internet behavioral counseling on weight loss in adults at risk for type 2 diabetes: A randomized trial. JAMA. 2003; 289(14): 1833–1836.

    Article  PubMed  Google Scholar 

  21. Thompson D, Baranowski T, Cullen K, et al. Food, Fun and Fitness Internet program for girls: Influencing log-on rate. Health Educ Res. 2008; 23(2): 228–237.

    Article  PubMed  Google Scholar 

  22. Clarke G, Eubanks D, Reid E, et al. Overcoming Depression on the Internet (ODIN) (2): A randomized trial of a self-help depression skills program with reminders. J Med Internet Res. 2005; 7(2): e16.

    Article  PubMed  Google Scholar 

  23. Lenert L, Muñoz RF, Perez JE, Bansod A. Automated e-mail messaging as a tool for improving quit rates in an Internet smoking cessation intervention. J Am Med Inform Assoc. 2004; 11(4): 235–240.

    Article  PubMed  Google Scholar 

  24. Collins LM, Murphy SA, Strecher V. The Multiphase Optimization Strategy (MOST) and the Sequential Multiple Assignment Randomized Trial (SMART) new methods for more potent ehealth interventions. Am J Prev Med. 2007; 32(5 Suppl): S112–S118.

    Article  PubMed  Google Scholar 

  25. Strecher VJ, McClure JB, Alexander GL, et al. Web-based smoking-cessation programs: Results of a randomized trial. Am J Prev Med. 2008; 34(5): 373–381.

    Article  PubMed  Google Scholar 

  26. Norman GJ. Answering the “what works?” Question in health behavior change. Am J Prev Med. 2008; 34(5): 449–450.

    Article  PubMed  Google Scholar 

  27. Street LL, Luoma JB. Control groups in psychosocial intervention research: Ethical and methodological issues. Ethics Behav. 2002; 12(1): 1–30.

    Article  PubMed  CAS  Google Scholar 

  28. Mangunkusumo R, Brug J, Duisterhout J, de Koning H, Raat H. Feasibility, acceptability, and quality of Internet-administered adolescent health promotion in a preventive-care setting. Health Educ Res. 2007; 22(1): 1–13.

    Article  PubMed  Google Scholar 

  29. Celio AA, Winzelberg AJ, Wilfley DE, et al. Reducing risk factors for eating disorders: Comparison of an Internet- and a classroom-delivered psychoeducational program. J Consult Clin Psychol. 2000; 68(4): 650–657.

    Article  PubMed  CAS  Google Scholar 

  30. Patten CA, Croghan IT, Meis TM, et al. Randomized clinical trial of an Internet-based versus brief office intervention for adolescent smoking cessation. Patient Educ Couns. 2006; 64: 249–258.

    Article  PubMed  Google Scholar 

  31. Steele RM, Mummery WK, Dwyer T. Examination of program exposure across intervention delivery modes: Face-to-face versus internet. Int J Behav Nutr Phys Activ. 2007; 4:7.

    Article  Google Scholar 

  32. Carlbring P, Nilsson-Ihrfelt E, Waara J, et al. Treatment of panic disorder: Live therapy vs. self-help via the Internet. Behav Res Ther. 2005; 43(10): 1321–1333.

    Article  PubMed  Google Scholar 

  33. Klein B, Richards JC, Austin DW. Efficacy of Internet therapy for panic disorder. J Behav Ther Exp Psychiatry. 2005; 37: 213–238.

    Article  PubMed  Google Scholar 

  34. Orbach G, Lindsay S, Grey S. A randomised placebo-controlled trial of a self-help Internet-based intervention for test anxiety. Behav Res Ther. 2006; 45: 483–496.

    Article  PubMed  Google Scholar 

  35. Rothert K, Strecher VJ, Doyle LA, et al. Web-based weight management programs in an integrated health care setting: A randomized, controlled trial. Obesity. 2006; 14(2): 266–272.

    Article  PubMed  Google Scholar 

  36. Severson HH, Gordon JS, Danaher BG, Akers L. ChewFree.com: Evaluation of a Web-based cessation program for smokeless tobacco users. Nicotine Tob Res. 2008; 10(2): 381–391.

    Article  PubMed  Google Scholar 

  37. Danaher BG, Smolkowski K, Seeley JR, Severson HH. Mediators of a successful Web-based smokeless tobacco cessation program. Addiction. 2008; 103: 1706–1712.

    Article  PubMed  Google Scholar 

  38. Ström L, Pettersson R, Andersson G. A controlled trial of self-help treatment of recurrent headache conducted via the Internet. J Consult Clin Psychol. 2000; 68(4): 722–727.

    Article  PubMed  Google Scholar 

  39. Spittaels H, De Bourdeaudhuij I, Vandelanotte C. Evaluation of a website-delivered computer-tailored intervention for increasing physical activity in the general population. Prev Med. 2007; 44(3): 209–217.

    Article  PubMed  Google Scholar 

  40. Jones M, Luce KH, Osborne MI, et al. Randomized, controlled trial of an internet-facilitated intervention for reducing binge eating and overweight in adolescents. Pediatrics. 2008; 121(3): 453–462.

    Article  PubMed  Google Scholar 

  41. Borkovec TD, Miranda J. Between-group psychotherapy outcome research and basic science. J Clin Psychol. 1999; 55(2): 147–158.

    Article  PubMed  CAS  Google Scholar 

  42. Hursey KG, Rains JC, Penzien DB, Nash JM, Nicholson RA. Behavioral headache research: Methodologic considerations and research design alternatives. Headache. 2005; 45(5): 466–478.

    Article  PubMed  Google Scholar 

  43. Koo M, Skinner H. Challenges of internet recruitment: A case study with disappointing results. J Med Internet Res. 2005; 7(1): e6.

    Article  PubMed  Google Scholar 

  44. Watson JM, Torgerson DJ. Increasing recruitment to randomised trials: A review of randomised controlled trials. BMC Med Res Methodol. 2006; 6: 34.

    Article  PubMed  Google Scholar 

  45. Gordon JS, Akers L, Severson HH, Danaher BG, Boles SM. Successful participant recruitment strategies for an online smokeless tobacco cessation program. Nicotine Tob Res. 2006; 8: S35–S41.

    Article  PubMed  Google Scholar 

  46. McKay HG, Danaher BG, Seeley JR, Lichtenstein E, Gau JM. Comparing two Web-based smoking cessation programs: Randomized controlled trial. J Med Internet Res. 2008; 10(5): e40.

    Article  PubMed  Google Scholar 

  47. Glasgow RE, Nelson CC, Kearney KA, et al. Reach, engagement, and retention in an Internet-based weight loss program in a multi-site randomized controlled trial. J Med Internet Res. 2007; 9(2): e11.

    Article  PubMed  Google Scholar 

  48. Horrigan JB, Rainie L, Allen K et al. The ever-shifting Internet population: A new look at Internet access and the digital divide. Pew Internet & American Life, 2003. Retrieved July 23, 2009 from http://www.pewinternet.org/pdfs/PIP_Shifting_Net_Pop_Report.pdf

  49. Nielsen J. Digital divide: The three stages. useit com, 2006. Retrieved July 22, 2009 from http://www.useit.com/alertbox/digital-divide.html

  50. Kalichman SC, Weinhardt L, Benotsch E, Cherry C. Closing the digital divide in HIV/AIDS care: Development of a theory-based intervention to increase Internet access. AIDS Care. 2002; 14(4): 523–537.

    PubMed  CAS  Google Scholar 

  51. Kotler P, Roberto N, Lee NR. Social Marketing: Improving the Quality of Life. Thousand Oaks: Sage; 2002.

    Google Scholar 

  52. Christensen H, Mackinnon A. The law of attrition revisited. J Med Internet Res. 2006; 8(3): e20.

    Article  PubMed  Google Scholar 

  53. Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: Toward a unified view. MIS Quarterly. 2003; 27(3): 425-478.

    Google Scholar 

  54. Porter CS, Donthu N. Using the Technology Acceptance Model to explain how attitudes determine Internet usage: The role of perceived access barriers and demographics. J Bus Res. 2006; 59(9): 999-1007.

    Article  Google Scholar 

  55. Ritterband LM, Thorndike F. Internet interventions or patient education web sites? J Med Internet Res. 2006; 8(3): e18.

    Article  PubMed  Google Scholar 

  56. Stanford J, Tauber ER, Fogg BJ, Marable L. Experts vs. online consumers: A comparative credibility study of health and finance Web sites. Consumer Reports Webwatch, 2002. Retrieved July 21, 2009 from http://www.consumerwebwatch.org/pdfs/expert-vs-online-consumers.pdf

  57. Graham AL, Bock BC, Cobb NK, Niaura R, Abrams DB. Characteristics of smokers reached and recruited to an internet smoking cessation trial: A case of denominators. Nicotine Tob Res. 2006; 8(Suppl 1): S43–S48.

    Article  PubMed  Google Scholar 

  58. Alexander GL, Divine GW, Couper MP, et al. Effect of incentives and mailing features on online health program enrollment. Am J Prev Med. 2008; 34(5): 382–388.

    Article  PubMed  Google Scholar 

  59. Danaher BG, McKay HG, Seeley JR. The information architecture of behavior change websites. J Med Internet Res. 2005; 7(2): e12.

    Article  PubMed  Google Scholar 

  60. Eysenbach G. The law of attrition. J Med Internet Res. 2005; 7(1): e11.

    Article  PubMed  Google Scholar 

  61. Frenn M, Malin S, Villarruel AM, et al. Determinants of physical activity and low-fat diet among low income African American and Hispanic middle school students. Public Health Nurs. 2005; 22(2): 89–97.

    Article  PubMed  Google Scholar 

  62. Wikipedia: Web analytics. Wikipedia, 2008. Retrieved July 21, 2009 from http://en.wikipedia.org/wiki/Web_analytics

  63. Danaher BG, Boles SB, Akers L, Gordon JS, Severson HH. Defining participant exposure measures in Web-based health behavior change programs. J Med Internet Res. 2006; 8(3): e15.

    Article  PubMed  Google Scholar 

  64. Associated Press. Nielsen revises its gauge of Web page rankings. Associated Press, 2008. Retrieved July 22, 2009 from http://www.nytimes.com/2007/07/10/business/media/10online.html?ex=1341720000&en=7ad7f7ae739cb786&ei=5088&partner=rssnyt&emc=rss

  65. Cobb NK, Graham AL, Bock BC, Papandonatos G, Abrams DB. Initial evaluation of a real-world Internet smoking cessation system. Nicotine Tob Res. 2005; 7(2): 207–216.

    Article  PubMed  Google Scholar 

  66. McCoy MR, Couch D, Duncan ND, Lynch GS. Evaluating an Internet weight loss program for diabetes prevention. Health Promot Int. 2005; 20: 221–228.

    Article  PubMed  Google Scholar 

  67. Danaher PJ, Mullarkey GW, Essegaier S. Factors affecting website visit duration: A cross-domain analysis. J Mark Res. 2006; 43(2): 182–194.

    Article  Google Scholar 

  68. Danaher BG, Jazdzewski SA, McKay HG, Hudson CR. Bandwidth constraints to using video and other rich media in behavior change websites. J Med Internet Res. 2005; 7(4): e49.

    Article  PubMed  Google Scholar 

  69. Peterson ET. Web Site Measurement Hacks. Sebastopol: O'Reilly Media; 2005.

    Google Scholar 

  70. Williamson DA, Martin PD, White MA, et al. Efficacy of an internet-based behavioral weight loss program for overweight adolescent African-American girls. Eat Weight Disord. 2005; 10(3): 193–203.

    PubMed  CAS  Google Scholar 

  71. comScore: Media metrix. Comscore, 2009. Retrieved July 23, 2009 from http://www.comscore.com/metrix/

  72. Farvolden P, Denisoff E, Selby P, Bagby RM, Rudy L. Usage and longitudinal effectiveness of a Web-based self-help cognitive behavioral therapy program for panic disorder. J Med Internet Res. 2005; 7(1): e7.

    Article  PubMed  Google Scholar 

  73. Andersson G, Carlbring P, Holmstrom A, et al. Internet-based self-help with therapist feedback and in vivo group exposure for social phobia: A randomized controlled trial. J Consult Clin Psychol. 2006; 74(4): 677–686.

    Article  PubMed  Google Scholar 

  74. Hurling R, Catt M, Boni MD, et al. Using internet and mobile phone technology to deliver an automated physical activity program: Randomized controlled trial. J Med Internet Res. 2007; 9(2): e7.

    Article  PubMed  Google Scholar 

  75. Brendryen H, Kraft P. Happy ending: A randomized controlled trial of a digital multi-media smoking cessation intervention. Addiction. 2008; 103(3): 478–484.

    Article  PubMed  Google Scholar 

  76. Reynolds B, Dallery J, Shroff P, Patak M, Leraas K. A Web-based contingency management program with adolescent smokers. J Appl Behav Anal. 2008; 41(4): 597–601.

    Article  PubMed  Google Scholar 

  77. Strecher VJ, Marcus A, Bishop K, et al. A randomized controlled trial of multiple tailored messages for smoking cessation among callers to the cancer information service. J Health Commun. 2005; 10(Suppl 1): 105–118.

    Article  PubMed  Google Scholar 

  78. Preacher KJ, Hayes AF. Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behav Res Methods. 2008; 40(3): 879–891.

    Article  PubMed  Google Scholar 

  79. Christensen H, Griffiths K, Groves C, Korten A. Free range users and one hit wonders: Community users of an Internet-based cognitive behaviour therapy program. Aust NZ J Psychiatry. 2006; 40(1): 59–62.

    Article  Google Scholar 

  80. Lenert L. Automating smoking cessation on the web. WATI: Web-assisted Tobacco Intervention, 2005. Retrieved July 22, 2009 from http://www.wati.net/presentations/watiII.lenert.pres.20050607.pdf

  81. Strecher VJ, Shiffman S, West R. Moderators and mediators of a Web-based computer-tailored smoking cessation program among nicotine patch users. Nicotine Tob Res. 2006; 8(S1): S95–S101.

    Article  PubMed  Google Scholar 

  82. Tullis T, Albert W. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. San Francisco: Morgan Kaufmann; 2008.

    Google Scholar 

  83. Tullis TS, Stetson JN. A comparison of questionnaires for assessing website usability. Usability Professionals Association Conference, Minneapolis, MN, June 7–11, 2004. Retrieved July 22, 2009 from http://home.comcast.net/%7Etomtullis/publications/UPA2004TullisStetson.pdf

Download references

Acknowledgments

This work was supported, in part, by grants CA118575 and CA84225 from the National Cancer Institute. We would like to extend our thanks to the following individuals: H. Garth McKay, who made important contributions to an earlier version of this paper; Edward Lichtenstein, who reviewed multiple drafts and participated in brainstorming sessions; and both Deborah Toobert and Stephen Boyd, who reviewed earlier report drafts.

Author information

Authors and Affiliations

  1. Oregon Research Institute, 1715 Franklin Boulevard, Eugene, OR, 97403, USA

    Brian G. Danaher Ph.D. & John R. Seeley Ph.D.

Authors
  1. Brian G. Danaher Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John R. Seeley Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Brian G. Danaher Ph.D..

About this article

Cite this article

Danaher, B.G., Seeley, J.R. Methodological Issues in Research on Web-Based Behavioral Interventions. ann. behav. med. 38, 28–39 (2009). https://doi.org/10.1007/s12160-009-9129-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12160-009-9129-0

Keywords

Search

Navigation