Methodological Issues in Research on Web-Based Behavioral Interventions

Original Article

Abstract

Background

Web-based behavioral intervention research is rapidly growing.

Purpose

We review methodological issues shared across Web-based intervention research to help inform future research in this area.

Methods

We examine measures and their interpretation using exemplar studies and our research.

Results

We report on research designs used to evaluate Web-based interventions and recommend newer, blended designs. We review and critique methodological issues associated with recruitment, engagement, and social validity.

Conclusions

We suggest that there is value to viewing this burgeoning realm of research from the broader context of behavior change research. We conclude that many studies use blended research designs, that innovative mantling designs such as the Multiphase Optimization Strategy and Sequential Multiple Assignment Randomized Trial methods hold considerable promise and should be used more widely, and that Web-based controls should be used instead of usual care or no-treatment controls in public health research. We recommend topics for future research that address participant recruitment, engagement, and social validity.

Keywords

Internet Web-based Interventions Design Measurement Recruitment Engagement 

References

  1. 1.
    Onken LS, Blaine JD, Battjes RJ. Behavioral therapy research: A conceptualization of a process. In: Henngler SW, Amentos R, eds. Innovative Approaches From Difficult-To-Treat Populations. Washington: American Psychiatric Press; 1997: 477–485.Google Scholar
  2. 2.
    Eng TR, Gustafson DH, Henderson J, Jimison H, Patrick K. Introduction to evaluation of interactive health communication applications. Science Panel on Interactive Communication and Health. Am J Prev Med. 1999; 16(1): 10–15.CrossRefPubMedGoogle Scholar
  3. 3.
    Rounsaville BJ, Carroll KM, Onken LS. A stage model of behavioral therapies research: Getting started and moving on from Stage I. Clin Psychol Sci Pract. 2001; 8(133): 142.Google Scholar
  4. 4.
    Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: The RE-AIM framework. Am J Public Health. 1999; 89(9): 1322–1327.CrossRefPubMedGoogle Scholar
  5. 5.
    Dansky KH, Thompson D, Sanner T. A framework for evaluating eHealth research. Eval Program Plann. 2006; 29: 397–404.CrossRefPubMedGoogle Scholar
  6. 6.
    Carroll KM, Rounsaville BJ. Bridging the gap: A hybrid model to link efficacy and effectiveness research in substance abuse treatment. Psychiatr Serv. 2003; 54(3): 333–339.CrossRefPubMedGoogle Scholar
  7. 7.
    Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: Increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003; 290(12): 1624–1632.CrossRefPubMedGoogle Scholar
  8. 8.
    Hotopf M, Churchill R, Lewis G. Pragmatic randomised controlled trials in psychiatry. Br J Psychiatry. 1999; 175: 217–223.CrossRefPubMedGoogle Scholar
  9. 9.
    Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med. 2007; 32(5 Suppl): S119–S126.CrossRefPubMedGoogle Scholar
  10. 10.
    Harvey-Berino J, Pintauro SJ, Gold EC. The feasibility of using Internet support for the maintenance of weight loss. Behav Modif. 2002; 26(1): 103–116.CrossRefPubMedGoogle Scholar
  11. 11.
    Lorig KR, Ritter PL, Laurent DD, Plant K. Internet-based chronic disease self-management: A randomized trial. Med Care. 2006; 44(11): 964–971.CrossRefPubMedGoogle Scholar
  12. 12.
    Ljotsson B, Lundin C, Mitsell K, et al. Remote treatment of bulimia nervosa and binge eating disorder: A randomized trial of Internet-assisted cognitive behavioural therapy. Behav Res Ther. 2007; 45(4): 649–661.CrossRefPubMedGoogle Scholar
  13. 13.
    Strecher VJ, Shiffman S, West R. Randomized controlled trial of a web-based computer-tailored smoking cessation program as a supplement to nicotine patch therapy. Addiction. 2005; 100(5): 682–688.CrossRefPubMedGoogle Scholar
  14. 14.
    Tate DF, Wing RR, Winett RA. Using Internet technology to deliver a behavioral weight loss program. JAMA. 2001; 285(9): 1172–1177.CrossRefPubMedGoogle Scholar
  15. 15.
    Bond GE, Burr R, Wolf FM, et al. Preliminary findings of the effects of comorbidities on a web-based intervention on self-reported blood sugar readings among adults age 60 and older with diabetes. Telemed J E Health. 2006; 12(6): 707–710.CrossRefPubMedGoogle Scholar
  16. 16.
    Gustafson DH, Hawkins R, Boberg E, et al. Impact of a patient-centered, computer-based health information/support system. Am J Prev Med. 1999; 16(1): 1–9.CrossRefPubMedGoogle Scholar
  17. 17.
    Gerber BS, Solomon MC, Shaffer TL, Quinn MT, Lipton RB. Evaluation of an internet diabetes self-management training program for adolescents and young adults. Diabetes Technol Ther. 2007; 9(1): 60–67.CrossRefPubMedGoogle Scholar
  18. 18.
    Glasgow RE, Davidson KW, Dobkin PL, Ockene J, Spring B. Practical behavioral trials to advance evidence-based behavioral medicine. Ann Behav Med. 2006; 31(1): 5–13.CrossRefPubMedGoogle Scholar
  19. 19.
    Wangberg SC. An Internet-based diabetes self-care intervention tailored to self-efficacy. Health Educ Res. 2008; 23(1): 170–179.CrossRefPubMedGoogle Scholar
  20. 20.
    Tate DF, Jackvony EH, Wing RR. Effects of Internet behavioral counseling on weight loss in adults at risk for type 2 diabetes: A randomized trial. JAMA. 2003; 289(14): 1833–1836.CrossRefPubMedGoogle Scholar
  21. 21.
    Thompson D, Baranowski T, Cullen K, et al. Food, Fun and Fitness Internet program for girls: Influencing log-on rate. Health Educ Res. 2008; 23(2): 228–237.CrossRefPubMedGoogle Scholar
  22. 22.
    Clarke G, Eubanks D, Reid E, et al. Overcoming Depression on the Internet (ODIN) (2): A randomized trial of a self-help depression skills program with reminders. J Med Internet Res. 2005; 7(2): e16.CrossRefPubMedGoogle Scholar
  23. 23.
    Lenert L, Muñoz RF, Perez JE, Bansod A. Automated e-mail messaging as a tool for improving quit rates in an Internet smoking cessation intervention. J Am Med Inform Assoc. 2004; 11(4): 235–240.CrossRefPubMedGoogle Scholar
  24. 24.
    Collins LM, Murphy SA, Strecher V. The Multiphase Optimization Strategy (MOST) and the Sequential Multiple Assignment Randomized Trial (SMART) new methods for more potent ehealth interventions. Am J Prev Med. 2007; 32(5 Suppl): S112–S118.CrossRefPubMedGoogle Scholar
  25. 25.
    Strecher VJ, McClure JB, Alexander GL, et al. Web-based smoking-cessation programs: Results of a randomized trial. Am J Prev Med. 2008; 34(5): 373–381.CrossRefPubMedGoogle Scholar
  26. 26.
    Norman GJ. Answering the “what works?” Question in health behavior change. Am J Prev Med. 2008; 34(5): 449–450.CrossRefPubMedGoogle Scholar
  27. 27.
    Street LL, Luoma JB. Control groups in psychosocial intervention research: Ethical and methodological issues. Ethics Behav. 2002; 12(1): 1–30.CrossRefPubMedGoogle Scholar
  28. 28.
    Mangunkusumo R, Brug J, Duisterhout J, de Koning H, Raat H. Feasibility, acceptability, and quality of Internet-administered adolescent health promotion in a preventive-care setting. Health Educ Res. 2007; 22(1): 1–13.CrossRefPubMedGoogle Scholar
  29. 29.
    Celio AA, Winzelberg AJ, Wilfley DE, et al. Reducing risk factors for eating disorders: Comparison of an Internet- and a classroom-delivered psychoeducational program. J Consult Clin Psychol. 2000; 68(4): 650–657.CrossRefPubMedGoogle Scholar
  30. 30.
    Patten CA, Croghan IT, Meis TM, et al. Randomized clinical trial of an Internet-based versus brief office intervention for adolescent smoking cessation. Patient Educ Couns. 2006; 64: 249–258.CrossRefPubMedGoogle Scholar
  31. 31.
    Steele RM, Mummery WK, Dwyer T. Examination of program exposure across intervention delivery modes: Face-to-face versus internet. Int J Behav Nutr Phys Activ. 2007; 4:7.CrossRefGoogle Scholar
  32. 32.
    Carlbring P, Nilsson-Ihrfelt E, Waara J, et al. Treatment of panic disorder: Live therapy vs. self-help via the Internet. Behav Res Ther. 2005; 43(10): 1321–1333.CrossRefPubMedGoogle Scholar
  33. 33.
    Klein B, Richards JC, Austin DW. Efficacy of Internet therapy for panic disorder. J Behav Ther Exp Psychiatry. 2005; 37: 213–238.CrossRefPubMedGoogle Scholar
  34. 34.
    Orbach G, Lindsay S, Grey S. A randomised placebo-controlled trial of a self-help Internet-based intervention for test anxiety. Behav Res Ther. 2006; 45: 483–496.CrossRefPubMedGoogle Scholar
  35. 35.
    Rothert K, Strecher VJ, Doyle LA, et al. Web-based weight management programs in an integrated health care setting: A randomized, controlled trial. Obesity. 2006; 14(2): 266–272.CrossRefPubMedGoogle Scholar
  36. 36.
    Severson HH, Gordon JS, Danaher BG, Akers L. ChewFree.com: Evaluation of a Web-based cessation program for smokeless tobacco users. Nicotine Tob Res. 2008; 10(2): 381–391.CrossRefPubMedGoogle Scholar
  37. 37.
    Danaher BG, Smolkowski K, Seeley JR, Severson HH. Mediators of a successful Web-based smokeless tobacco cessation program. Addiction. 2008; 103: 1706–1712.CrossRefPubMedGoogle Scholar
  38. 38.
    Ström L, Pettersson R, Andersson G. A controlled trial of self-help treatment of recurrent headache conducted via the Internet. J Consult Clin Psychol. 2000; 68(4): 722–727.CrossRefPubMedGoogle Scholar
  39. 39.
    Spittaels H, De Bourdeaudhuij I, Vandelanotte C. Evaluation of a website-delivered computer-tailored intervention for increasing physical activity in the general population. Prev Med. 2007; 44(3): 209–217.CrossRefPubMedGoogle Scholar
  40. 40.
    Jones M, Luce KH, Osborne MI, et al. Randomized, controlled trial of an internet-facilitated intervention for reducing binge eating and overweight in adolescents. Pediatrics. 2008; 121(3): 453–462.CrossRefPubMedGoogle Scholar
  41. 41.
    Borkovec TD, Miranda J. Between-group psychotherapy outcome research and basic science. J Clin Psychol. 1999; 55(2): 147–158.CrossRefPubMedGoogle Scholar
  42. 42.
    Hursey KG, Rains JC, Penzien DB, Nash JM, Nicholson RA. Behavioral headache research: Methodologic considerations and research design alternatives. Headache. 2005; 45(5): 466–478.CrossRefPubMedGoogle Scholar
  43. 43.
    Koo M, Skinner H. Challenges of internet recruitment: A case study with disappointing results. J Med Internet Res. 2005; 7(1): e6.CrossRefPubMedGoogle Scholar
  44. 44.
    Watson JM, Torgerson DJ. Increasing recruitment to randomised trials: A review of randomised controlled trials. BMC Med Res Methodol. 2006; 6: 34.CrossRefPubMedGoogle Scholar
  45. 45.
    Gordon JS, Akers L, Severson HH, Danaher BG, Boles SM. Successful participant recruitment strategies for an online smokeless tobacco cessation program. Nicotine Tob Res. 2006; 8: S35–S41.CrossRefPubMedGoogle Scholar
  46. 46.
    McKay HG, Danaher BG, Seeley JR, Lichtenstein E, Gau JM. Comparing two Web-based smoking cessation programs: Randomized controlled trial. J Med Internet Res. 2008; 10(5): e40.CrossRefPubMedGoogle Scholar
  47. 47.
    Glasgow RE, Nelson CC, Kearney KA, et al. Reach, engagement, and retention in an Internet-based weight loss program in a multi-site randomized controlled trial. J Med Internet Res. 2007; 9(2): e11.CrossRefPubMedGoogle Scholar
  48. 48.
    Horrigan JB, Rainie L, Allen K et al. The ever-shifting Internet population: A new look at Internet access and the digital divide. Pew Internet & American Life, 2003. Retrieved July 23, 2009 from http://www.pewinternet.org/pdfs/PIP_Shifting_Net_Pop_Report.pdf
  49. 49.
    Nielsen J. Digital divide: The three stages. useit com, 2006. Retrieved July 22, 2009 from http://www.useit.com/alertbox/digital-divide.html
  50. 50.
    Kalichman SC, Weinhardt L, Benotsch E, Cherry C. Closing the digital divide in HIV/AIDS care: Development of a theory-based intervention to increase Internet access. AIDS Care. 2002; 14(4): 523–537.PubMedGoogle Scholar
  51. 51.
    Kotler P, Roberto N, Lee NR. Social Marketing: Improving the Quality of Life. Thousand Oaks: Sage; 2002.Google Scholar
  52. 52.
    Christensen H, Mackinnon A. The law of attrition revisited. J Med Internet Res. 2006; 8(3): e20.CrossRefPubMedGoogle Scholar
  53. 53.
    Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: Toward a unified view. MIS Quarterly. 2003; 27(3): 425-478.Google Scholar
  54. 54.
    Porter CS, Donthu N. Using the Technology Acceptance Model to explain how attitudes determine Internet usage: The role of perceived access barriers and demographics. J Bus Res. 2006; 59(9): 999-1007.CrossRefGoogle Scholar
  55. 55.
    Ritterband LM, Thorndike F. Internet interventions or patient education web sites? J Med Internet Res. 2006; 8(3): e18.CrossRefPubMedGoogle Scholar
  56. 56.
    Stanford J, Tauber ER, Fogg BJ, Marable L. Experts vs. online consumers: A comparative credibility study of health and finance Web sites. Consumer Reports Webwatch, 2002. Retrieved July 21, 2009 from http://www.consumerwebwatch.org/pdfs/expert-vs-online-consumers.pdf
  57. 57.
    Graham AL, Bock BC, Cobb NK, Niaura R, Abrams DB. Characteristics of smokers reached and recruited to an internet smoking cessation trial: A case of denominators. Nicotine Tob Res. 2006; 8(Suppl 1): S43–S48.CrossRefPubMedGoogle Scholar
  58. 58.
    Alexander GL, Divine GW, Couper MP, et al. Effect of incentives and mailing features on online health program enrollment. Am J Prev Med. 2008; 34(5): 382–388.CrossRefPubMedGoogle Scholar
  59. 59.
    Danaher BG, McKay HG, Seeley JR. The information architecture of behavior change websites. J Med Internet Res. 2005; 7(2): e12.CrossRefPubMedGoogle Scholar
  60. 60.
    Eysenbach G. The law of attrition. J Med Internet Res. 2005; 7(1): e11.CrossRefPubMedGoogle Scholar
  61. 61.
    Frenn M, Malin S, Villarruel AM, et al. Determinants of physical activity and low-fat diet among low income African American and Hispanic middle school students. Public Health Nurs. 2005; 22(2): 89–97.CrossRefPubMedGoogle Scholar
  62. 62.
    Wikipedia: Web analytics. Wikipedia, 2008. Retrieved July 21, 2009 from http://en.wikipedia.org/wiki/Web_analytics
  63. 63.
    Danaher BG, Boles SB, Akers L, Gordon JS, Severson HH. Defining participant exposure measures in Web-based health behavior change programs. J Med Internet Res. 2006; 8(3): e15.CrossRefPubMedGoogle Scholar
  64. 64.
    Associated Press. Nielsen revises its gauge of Web page rankings. Associated Press, 2008. Retrieved July 22, 2009 from http://www.nytimes.com/2007/07/10/business/media/10online.html?ex=1341720000&en=7ad7f7ae739cb786&ei=5088&partner=rssnyt&emc=rss
  65. 65.
    Cobb NK, Graham AL, Bock BC, Papandonatos G, Abrams DB. Initial evaluation of a real-world Internet smoking cessation system. Nicotine Tob Res. 2005; 7(2): 207–216.CrossRefPubMedGoogle Scholar
  66. 66.
    McCoy MR, Couch D, Duncan ND, Lynch GS. Evaluating an Internet weight loss program for diabetes prevention. Health Promot Int. 2005; 20: 221–228.CrossRefPubMedGoogle Scholar
  67. 67.
    Danaher PJ, Mullarkey GW, Essegaier S. Factors affecting website visit duration: A cross-domain analysis. J Mark Res. 2006; 43(2): 182–194.CrossRefGoogle Scholar
  68. 68.
    Danaher BG, Jazdzewski SA, McKay HG, Hudson CR. Bandwidth constraints to using video and other rich media in behavior change websites. J Med Internet Res. 2005; 7(4): e49.CrossRefPubMedGoogle Scholar
  69. 69.
    Peterson ET. Web Site Measurement Hacks. Sebastopol: O'Reilly Media; 2005.Google Scholar
  70. 70.
    Williamson DA, Martin PD, White MA, et al. Efficacy of an internet-based behavioral weight loss program for overweight adolescent African-American girls. Eat Weight Disord. 2005; 10(3): 193–203.PubMedGoogle Scholar
  71. 71.
    comScore: Media metrix. Comscore, 2009. Retrieved July 23, 2009 from http://www.comscore.com/metrix/
  72. 72.
    Farvolden P, Denisoff E, Selby P, Bagby RM, Rudy L. Usage and longitudinal effectiveness of a Web-based self-help cognitive behavioral therapy program for panic disorder. J Med Internet Res. 2005; 7(1): e7.CrossRefPubMedGoogle Scholar
  73. 73.
    Andersson G, Carlbring P, Holmstrom A, et al. Internet-based self-help with therapist feedback and in vivo group exposure for social phobia: A randomized controlled trial. J Consult Clin Psychol. 2006; 74(4): 677–686.CrossRefPubMedGoogle Scholar
  74. 74.
    Hurling R, Catt M, Boni MD, et al. Using internet and mobile phone technology to deliver an automated physical activity program: Randomized controlled trial. J Med Internet Res. 2007; 9(2): e7.CrossRefPubMedGoogle Scholar
  75. 75.
    Brendryen H, Kraft P. Happy ending: A randomized controlled trial of a digital multi-media smoking cessation intervention. Addiction. 2008; 103(3): 478–484.CrossRefPubMedGoogle Scholar
  76. 76.
    Reynolds B, Dallery J, Shroff P, Patak M, Leraas K. A Web-based contingency management program with adolescent smokers. J Appl Behav Anal. 2008; 41(4): 597–601.CrossRefPubMedGoogle Scholar
  77. 77.
    Strecher VJ, Marcus A, Bishop K, et al. A randomized controlled trial of multiple tailored messages for smoking cessation among callers to the cancer information service. J Health Commun. 2005; 10(Suppl 1): 105–118.CrossRefPubMedGoogle Scholar
  78. 78.
    Preacher KJ, Hayes AF. Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behav Res Methods. 2008; 40(3): 879–891.CrossRefPubMedGoogle Scholar
  79. 79.
    Christensen H, Griffiths K, Groves C, Korten A. Free range users and one hit wonders: Community users of an Internet-based cognitive behaviour therapy program. Aust NZ J Psychiatry. 2006; 40(1): 59–62.CrossRefGoogle Scholar
  80. 80.
    Lenert L. Automating smoking cessation on the web. WATI: Web-assisted Tobacco Intervention, 2005. Retrieved July 22, 2009 from http://www.wati.net/presentations/watiII.lenert.pres.20050607.pdf
  81. 81.
    Strecher VJ, Shiffman S, West R. Moderators and mediators of a Web-based computer-tailored smoking cessation program among nicotine patch users. Nicotine Tob Res. 2006; 8(S1): S95–S101.CrossRefPubMedGoogle Scholar
  82. 82.
    Tullis T, Albert W. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. San Francisco: Morgan Kaufmann; 2008.Google Scholar
  83. 83.
    Tullis TS, Stetson JN. A comparison of questionnaires for assessing website usability. Usability Professionals Association Conference, Minneapolis, MN, June 7–11, 2004. Retrieved July 22, 2009 from http://home.comcast.net/%7Etomtullis/publications/UPA2004TullisStetson.pdf

Copyright information

© The Society of Behavioral Medicine 2009

Authors and Affiliations

  1. 1.Oregon Research InstituteEugeneUSA

Personalised recommendations