Optimization of Just-in-Time Adaptive Interventions Using Reinforcement Learning

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10868)


Momentary context data is an important source for intelligent decision making towards personalization of mobile phone notifications. We propose a reinforcement learning based personalized notification delivery algorithm, reasoning over momentary context data. Beyond the state of the art, we propose new approaches for faster convergence of the algorithm and jump start of learning performance at the beginning of the learning process. We test our approach in both simulated and real settings trying to optimize the timing of the notifications. Our eventual, practical aim is to make office workers more physically active during the work time. We compare the results obtained for standard and improved algorithms in both testbeds where improved versions yield better results.



The research leading to these results has received funding partially from the European Community’s H2020 Programme under grant agreement no H2020-PHC-689444, POWER2DM project (Predictive Model-Based Decision Support for Diabetes Patient Empowerment) and partially from The Scientific and Technological Research Council of Turkey (TÜBİTAK).


  1. 1.
    Yardley, L., Morrison, L., Bradbury, K., Muller, I.: The person-based approach to intervention development: application to digital health-related behavior change interventions. J. Med. Internet Res. 17(1), 30 (2015)CrossRefGoogle Scholar
  2. 2.
    Ramachandran, A., Snehalatha, C., Ram, J., Selvam, S., Simon, M., Nanditha, A., Oliver, N., Shetty, A.S., Godsland, I.F., Chaturvedi, N., Majeed, A., Toumazou, C., Alberti, K.G., Johnston, D.G.: Effectiveness of mobile phone messaging in prevention of type 2 diabetes by lifestyle modification in men in India: a prospective, parallel-group, randomised controlled trial. Lancet Diabetes Endocrinol. 1(3), 191–198 (2013)CrossRefGoogle Scholar
  3. 3.
    Eaton, C.B., Hartman, S.J., Perzanowski, E., Pan, G., Roberts, M.B., Risica, P.M., Gans, K.M., Jakicic, J.M., Marcus, B.H.: A randomized clinical trial of a tailored lifestyle intervention for obese, sedentary, primary care patients. Ann. Fam. Med. 14(4), 311–319 (2016)CrossRefGoogle Scholar
  4. 4.
    Nahum-Shani, I., Smith, S.N., Tewari, A., Witkiewitz, K., Collins, L.M., Spring, B., Murphy, S.: Just in time adaptive interventions (JITAIs): an organizing framework for ongoing health behavior support. Methodology Center Technical report (14-126) (2014)Google Scholar
  5. 5.
    Rummery, G.A., Niranjan, M.: On-line Q-learning using connectionist systems, vol. 37. University of Cambridge, Department of Engineering (1994)Google Scholar
  6. 6.
    Pellegrini, C.A., Pfammatter, A.F., Conroy, D.E., Spring, B.: Smartphone applications to support weight loss: current perspectives. Adv. Health Care Technol. 1, 13 (2015)CrossRefGoogle Scholar
  7. 7.
    Dantzig, S., Geleijnse, G., Halteren, A.T.: Toward a persuasive mobile application to reduce sedentary behavior. Pers. Ubiquit. Comput. 17(6), 1237–1246 (2013)CrossRefGoogle Scholar
  8. 8.
    Kelly, J., Gooding, P., Pratt, D., Ainsworth, J., Welford, M., Tarrier, N.: Intelligent real-time therapy: harnessing the power of machine learning to optimise the delivery of momentary cognitive-behavioural interventions. J. Ment. Health 21(4), 404–414 (2012)CrossRefGoogle Scholar
  9. 9.
    Hammond, R.A., Ornstein, J.T., Fellows, L.K., Dubé, L., Levitan, R., Dagher, A.: A model of food reward learning with dynamic reward exposure. Front. Comput. Neurosci. 6, 82 (2012). Scholar
  10. 10.
    Mohan, S., Venkatakrishnan, A., Silva, M., Pirolli, P.: On designing a social coach to promote regular aerobic exercise. In: AAAI, pp. 4721–4727 (2017)Google Scholar
  11. 11.
    Navarro-Barrientos, J.E., Rivera, D.E., Collins, L.M.: A dynamical model for describing behavioural interventions for weight loss and body composition change. Math. Comput. Model. Dyn. Syst. 17(2), 183–203 (2011)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Pielot, M., Cardoso, B., Katevas, K., Serrà, J., Matic, A., Oliver, N.: Beyond interruptibility: predicting opportune moments to engage mobile phone users. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 1(3), 91 (2017)CrossRefGoogle Scholar
  13. 13.
    Oh, H., Jalali, L., Jain, R.: An intelligent notification system using context from real-time personal activity monitoring. In: IEEE International Conference on 2015 Multimedia and Expo (ICME), pp. 1–6. IEEE (2015)Google Scholar
  14. 14.
    Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10, 1633–1685 (2009)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer EngineeringMiddle East Technical UniversityAnkaraTurkey
  2. 2.SRDC Software Research and Development and Consultancy Ltd.AnkaraTurkey

Personalised recommendations