Advertisement

Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss?

  • Evan M. Forman
  • Stephanie G. Kerrigan
  • Meghan L. Butryn
  • Adrienne S. Juarascio
  • Stephanie M. Manasse
  • Santiago Ontañón
  • Diane H. Dallal
  • Rebecca J. Crochiere
  • Danielle Moskow
Article
  • 113 Downloads

Abstract

Behavioral weight loss (WL) trials show that, on average, participants regain lost weight unless provided long-term, intensive—and thus costly—intervention. Optimization solutions have shown mixed success. The artificial intelligence principle of “reinforcement learning” (RL) offers a new and more sophisticated form of optimization in which the intensity of each individual’s intervention is continuously adjusted depending on patterns of response. In this pilot, we evaluated the feasibility and acceptability of a RL-based WL intervention, and whether optimization would achieve equivalent benefit at a reduced cost compared to a non-optimized intensive intervention. Participants (n = 52) completed a 1-month, group-based in-person behavioral WL intervention and then (in Phase II) were randomly assigned to receive 3 months of twice-weekly remote interventions that were non-optimized (NO; 10-min phone calls) or optimized (a combination of phone calls, text exchanges, and automated messages selected by an algorithm). The Individually-Optimized (IO) and Group-Optimized (GO) algorithms selected interventions based on past performance of each intervention for each participant, and for each group member that fit into a fixed amount of time (e.g., 1 h), respectively. Results indicated that the system was feasible to deploy and acceptable to participants and coaches. As hypothesized, we were able to achieve equivalent Phase II weight losses (NO = 4.42%, IO = 4.56%, GO = 4.39%) at roughly one-third the cost (1.73 and 1.77 coaching hours/participant for IO and GO, versus 4.38 for NO), indicating strong promise for a RL system approach to weight loss and maintenance.

Keywords

Artificial intelligence Behavioral treatment Lifestyle modification Weight loss Obesity Optimization 

Notes

Acknowledgements

This project was funded by a Drexel University DARE (Drexel Areas of Research Excellence) Award to Evan Forman. We would also like to acknowledge the assistance of Pedrag Krasnja whom we consulted during the planning of this study.

Compliance with ethical standards

Conflict of interest

Evan M. Forman, Stephanie G. Kerrigan, Meghan L. Butryn, Adrienne S. Juarascio, Stephanie M. Manasse, Santiago Ontañón, Diane H. Dallal, Rebecca J. Crochiere and Danielle Moskow declare that they have no conflict of interest.

Human and animal rights and Informed consent

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors. Informed consent was obtained from all individual participants included in the study.

References

  1. Abbeel, P., & Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. Paper presented at the Proceedings of the twenty-first international conference on Machine learning.Google Scholar
  2. Anderson, J. W., Vichitbandra, S., Qian, W., & Kryscio, R. J. (1999). Long-term weight maintenance after an intensive weight-loss program. Journal of the American College of Nutrition, 18, 620–627.CrossRefPubMedGoogle Scholar
  3. Association of Behavioral and Cognitive Therapies. (2018). Find a CBT Therapist. Retrieved May 23, 2018, from http://www.findcbt.org.
  4. Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47, 235–256.CrossRefGoogle Scholar
  5. Aydin, M. E., & Öztemel, E. (2000). Dynamic job-shop scheduling using reinforcement learning agents. Robotics and Autonomous Systems, 33, 169–178.CrossRefGoogle Scholar
  6. Brindal, E., Freyne, J., Saunders, I., Berkovsky, S., Smith, G., & Noakes, M. (2012). Features predicting weight loss in overweight or obese participants in a web-based intervention: Randomized trial. Journal of medical Internet research, 14, e173.CrossRefPubMedPubMedCentralGoogle Scholar
  7. Butryn, M. L., Forman, E. M., Lowe, M. R., Gorin, A. A., Zhang, F., & Schaumberg, K. (2017). Efficacy of environmental and acceptance-based enhancements to behavioral weight loss treatment: The ENACT trial. Obesity, 25, 866–872.CrossRefPubMedGoogle Scholar
  8. Butryn, M. L., Zhang, F., Remmert, J. E., Roberts, S. R., & Forman, E. M. (2018). Baseline Executive Functioning Predicts Weight Loss and Physical Activity Outcomes in A Lifestyle Modification Program. Paper presented at the 39th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine, New Orleans, LA.Google Scholar
  9. Carels, R. A., Darby, L., Cacciapaglia, H. M., Douglass, O. M., Harper, J., Kaplar, M. E., et al. (2005). Applying a stepped-care approach to the treatment of obesity. Journal of Psychosomatic Research, 59, 375–383.CrossRefPubMedGoogle Scholar
  10. Carels, R. A., Darby, L., Cacciapaglia, H. M., Konrad, K., Coit, C., Harper, J., et al. (2007). Using motivational interviewing as a supplement to obesity treatment: A stepped-care approach. Health Psychology, 26, 369.CrossRefPubMedGoogle Scholar
  11. Carels, R. A., Hoffmann, D. A., Hinman, N., Burmeister, J. M., Koball, A., Ashrafioun, L., et al. (2013). Step-down approach to behavioural weight loss treatment: A pilot of a randomised clinical trial. Psychology & Health, 28, 1121–1134.CrossRefGoogle Scholar
  12. Carels, R. A., Selensky, J. C., Rossi, J., Solar, C., & Hlavka, R. (2017). A novel stepped-care approach to weight loss: The role of self-monitoring and health literacy in treatment outcomes. Eating Behaviors, 26, 76–82.CrossRefPubMedGoogle Scholar
  13. Centers for Disease Control and Prevention (CDC). (2014). National Health and Nutrition Examination Survey Data. Hyattsville, MD. Retrieved May 13, 2018, from https://www.cdc.gov/nchs/fastats/obesity-overweight.htm.
  14. Diabetes Prevention Program Research Group. (2003). Costs associated with the primary prevention of type 2 diabetes mellitus in the diabetes prevention program. Diabetes Care, 26, 36–47.CrossRefGoogle Scholar
  15. Forman, E. M., Butryn, M. L., Manasse, S. M., Crosby, R. D., Goldstein, S. P., Wyckoff, E. P., et al. (2016). Acceptance-based versus standard behavioral treatment for obesity: Results from the mind your health randomized controlled trial. Obesity, 24, 2050–2056.CrossRefPubMedGoogle Scholar
  16. Jakicic, J. M., Tate, D. F., Lang, W., Davis, K. K., Polzien, K., Rickman, A. D., et al. (2012). Effect of a stepped-care intervention approach on weight loss in adults: A randomized clinical trial. JAMA, 307, 2617–2626.CrossRefPubMedPubMedCentralGoogle Scholar
  17. Jeffery, R. W., Epstein, L. H., Wilson, G. T., Drewnowski, A., Stunkard, A. J., & Wing, R. R. (2000). Long-term maintenance of weight loss: Current status. Health Psychology, 19, 5.CrossRefPubMedGoogle Scholar
  18. Joo, N.-S., & Kim, B.-T. (2007). Mobile phone short message service messaging for behaviour modification in a community-based weight control programme in Korea. Journal of Telemedicine and Telecare, 13, 416–420.CrossRefPubMedGoogle Scholar
  19. Korinek, E. V., Phatak, S. S., Martin, C. A., Freigoun, M. T., Rivera, D. E., Adams, M. A., et al. (2018). Adaptive step goals and rewards: A longitudinal growth model of daily steps for a smartphone-based walking intervention. Journal of Behavioral Medicine, 41, 74–86.CrossRefPubMedGoogle Scholar
  20. Kramer, F. M., Jeffery, R. W., Forster, J. L., & Snell, M. K. (1989). Long-term follow-up of behavioral treatment for obesity: Patterns of weight regain among men and women. International Journal of Obesity, 13, 123–136.PubMedGoogle Scholar
  21. Krukowski, R. A., Tilford, J. M., Harvey-Berino, J., & West, D. S. (2011). Comparing behavioral weight loss modalities: Incremental cost-effectiveness of an internet-based versus an in-person condition. Obesity, 19, 1629–1635.CrossRefPubMedGoogle Scholar
  22. Kuderer, M., Gulati, S., & Burgard, W. (2015). Learning driving styles for autonomous vehicles from demonstration. Paper presented at the 2015 IEEE International Conference on Robotics and Automation (ICRA).Google Scholar
  23. McGraa, K. L. K. (2010). The effects of persuasive motivational text messaging on adherence to diet and exercise programs across different personality traits. Santa Barbara: Fielding Graduate University.Google Scholar
  24. Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., et al. (2006). Autonomous inverted helicopter flight via reinforcement learning. In M. H. Ang & O. Khatib (Eds.), Experimental Robotics IX (pp. 363–372). Berlin: Springer.CrossRefGoogle Scholar
  25. Ontanón, S. (2013). The combinatorial multi-armed bandit problem and its application to real-time strategy games. Paper presented at the Ninth Artificial Intelligence and Interactive Digital Entertainment Conference.Google Scholar
  26. Ontanón, S. (2017). Combinatorial multi-armed bandits for real-time strategy games. Journal of Artificial Intelligence Research, 58, 665–702.CrossRefGoogle Scholar
  27. Paredes, P., Gilad-Bachrach, R., Czerwinski, M., Roseway, A., Rowan, K., & Hernandez, J. (2014). PopTherapy: Coping with stress through pop-culture. Paper presented at the Proceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare.Google Scholar
  28. Patrick, K., Raab, F., Adams, M. A., Dillon, L., Zabinski, M., Rock, C. L., et al. (2009). A text message–based intervention for weight loss: Randomized controlled trial. Journal of medical Internet research, 11, e1.CrossRefPubMedPubMedCentralGoogle Scholar
  29. Phatak, S. S., Freigoun, M. T., Martín, C. A., Rivera, D. E., Korinek, E. V., Adams, M. A., et al. (2018). Modeling individual differences: A case study of the application of system identification for personalizing a physical activity intervention. Journal of Biomedical Informatics, 79, 82–97.CrossRefPubMedGoogle Scholar
  30. Piette, J. D., Farris, K. B., Newman, S., An, L., Sussman, J., & Singh, S. (2014). The potential impact of intelligent systems for mobile health self-management support: Monte Carlo simulations of text message support for medication adherence. Annals of Behavioral Medicine, 49, 84–94.CrossRefPubMedCentralGoogle Scholar
  31. Piette, J. D., Krein, S. L., Striplin, D., Marinec, N., Kerns, R. D., Farris, K. B., et al. (2016). Patient-centered pain care using artificial intelligence and mobile health tools: Protocol for a randomized study funded by the US Department of Veterans Affairs Health Services Research and Development Program. JMIR Research Protocols, 5, e53.CrossRefPubMedPubMedCentralGoogle Scholar
  32. Poirier, J., Bennett, W. L., Jerome, G. J., Shah, N. G., Lazo, M., Yeh, H.-C., et al. (2016). Effectiveness of an activity tracker-and internet-based adaptive walking program for adults: A randomized controlled trial. Journal of Medical Internet Research, 18, e34.CrossRefPubMedPubMedCentralGoogle Scholar
  33. Rabbi, M., Aung, M. H., Zhang, M., & Choudhury, T. (2015). MyBehavior: Automatic personalized health feedback from user behaviors and preferences using smartphones. Paper presented at the Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing.Google Scholar
  34. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529, 484–489.CrossRefPubMedGoogle Scholar
  35. Snook, K. R., Hansen, A. R., Duke, C. H., Finch, K. C., Hackney, A. A., & Zhang, J. (2017). Change in percentages of adults with overweight or obesity trying to lose weight, 1988–2014. JAMA, 317, 971–973.CrossRefPubMedGoogle Scholar
  36. Society of Clinical Psychology. (2018). Therapists Near Me. Retrieved May 23, 2018, from https://www.div12.org/therapist-search/.
  37. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Vol. 1). Cambridge: MIT press Cambridge.Google Scholar
  38. Wadden, T. A., Butryn, M. L., & Wilson, C. (2007). Lifestyle modification for the management of obesity. Gastroenterology, 132, 2226–2238.CrossRefPubMedGoogle Scholar
  39. Wadden, T. A., Sternberg, J., Letizia, K., Stunkard, A., & Foster, G. (1989). Treatment of obesity by very low calorie diet, behavior therapy, and their combination: A five-year perspective. International Journal of Obesity, 13, 39–46.PubMedGoogle Scholar
  40. Webber, K. H., Tate, D. F., & Bowling, J. M. (2008). A randomized comparison of two motivationally enhanced Internet behavioral weight loss programs. Behaviour Research and Therapy, 46, 1090–1095.CrossRefPubMedGoogle Scholar
  41. Wilson, G. T. (1994). Behavioral treatment of obesity: Thirty years and counting. Advances in Behaviour Research and Therapy, 16, 31–75.CrossRefGoogle Scholar
  42. Wilson, G. T., & Brownell, K. D. (2002). Behavioral treatment for obesity. In C. G. Fairburn & K. D. Brownell (Eds.), Eating disorders and obesity. New York: Guilford Press.Google Scholar
  43. Yom-Tov, E., Feraru, G., Kozdoba, M., Mannor, S., Tennenholtz, M., & Hochberg, I. (2017). Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of Medical Internet Research, 19, e338.CrossRefPubMedPubMedCentralGoogle Scholar
  44. Zhou, M., Fukuoka, Y., Mintz, Y., Goldberg, K., Kaminsky, P., Flowers, E., et al. (2018). Evaluating machine learning-based automated personalized daily step goals delivered through a Mobile Phone App: Randomized controlled trial. JMIR mHealth and uHealth, 6, e28.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Evan M. Forman
    • 1
    • 2
  • Stephanie G. Kerrigan
    • 1
    • 2
  • Meghan L. Butryn
    • 1
    • 2
  • Adrienne S. Juarascio
    • 1
    • 2
  • Stephanie M. Manasse
    • 2
  • Santiago Ontañón
    • 3
  • Diane H. Dallal
    • 1
    • 2
  • Rebecca J. Crochiere
    • 1
    • 2
  • Danielle Moskow
    • 1
    • 2
  1. 1.Department of Psychology, WELL CenterDrexel UniversityPhiladelphiaUSA
  2. 2.Center for Weight, Eating, and Lifestyle Sciences (WELL Center)Drexel UniversityPhiladelphiaUSA
  3. 3.Department of Computer ScienceDrexel UniversityPhiladelphiaUSA

Personalised recommendations