A Flexible Toolkit for Evaluating Person-Centred Digital Health and Wellness at Scale

  • Marilyn McGee-LennonEmail author
  • Matt-Mouley BouamraneEmail author
  • Eleanor GrieveEmail author
  • Catherine A. O’Donnell
  • Siobhan O’ConnorEmail author
  • Ruth AgbakobaEmail author
  • Alison M. DevlinEmail author
  • Sarah BarryEmail author
  • Annemieke BikkerEmail author
  • Tracy Finch
  • Frances S. MairEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 482)


The Delivering Assisted Living Lifestyles at Scale (dallas) program was a large-scale, nationwide deployment of digital health and wellbeing products and services in the UK. Telehealth, telecare, mobile apps, personal health records, and assisted living technology were implemented by four large multi-stakeholder consortia and a multidimensional evaluation was carried out across the lifecycle from examining co-design and redesign of services through to rolling out services via statutory, private and consumer routes. A flexible toolkit of descriptive, process and outcome measures was developed and iteratively refined throughout the program. This approach enabled a longitudinal mixed-methods evaluation, underpinned by a robust social theory of implementation called ‘Normalization Process Theory’. There remains uncertainty about the best approaches to real world digital health evaluation. This program provided a unique opportunity to develop the knowledge base and toolkit of qualitative and quantitative methods necessary to evaluate person-centered digital health technologies deployed at scale.


Health informatics eHealth Digital health Telemedicine Implementation Evaluation 


  1. 1.
    Barnett, K., Mercer, S.W., Norbury, M., Watt, G., Wyke, S., Guthrie, B.: Epidemiology of multimorbidity and implications for health care, research, and medical education: a cross-sectional study. The Lancet 380, 37–43 (2012)CrossRefGoogle Scholar
  2. 2.
    Davis, M., Freeman, M., Kaye, J., Vuckovic, N., Buckley, D.: A systematic review of clinician and staff views on the acceptability of incorporating remote monitoring technology into primary care. Telemedicine e-Health 20(5), 428–438 (2014)CrossRefGoogle Scholar
  3. 3.
    Robinson, L., Gibson, G., Kingston, A., Newton, L., Pritchard, G., Finch, T., Brittain, K.: Assistive technologies in caring for the oldest old: a review of current practice and future directions. Aging Health 9(4), 365–375 (2013)CrossRefGoogle Scholar
  4. 4.
    Finkelstein, S., Speedie, S., Potthoff, S.: Home telehealth improves clinical outcomes at lower cost for home healthcare. Telemedicine e-Health 12(2), 128–136 (2006)CrossRefGoogle Scholar
  5. 5.
    Archer, N., Fevrier-Thomas, U., Lokker, C., McKibbon, K.A., Straus, S.E.: Personal health records: a scoping review. J. Am. Med. Inf. Assoc. 18, 515–522 (2011)CrossRefGoogle Scholar
  6. 6.
    Mosa, A.S.M., Yoo, I., Sheets, L.: A systematic review of healthcare applications for smartphones. BMC Med. Inform. Decis. Mak. 12, 1–31 (2012)CrossRefGoogle Scholar
  7. 7.
    Orwat, C., Graefe, A., Faulwasser, T.: Towards pervasive computing in health care—a literature review. BMC Med. Inform. Decis. Mak. 8, 26 (2008)CrossRefGoogle Scholar
  8. 8.
    Linskell, J., Bouamrane, M.-M.: Assisted-living spaces for end-users with complex needs: a proposed implementation and delivery model. Health Inform. J. 18(3), 159–170 (2012)CrossRefGoogle Scholar
  9. 9.
    Steventon, A., Bardsley, M., Billings, J., Dixon, J., et al.: Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ 344, e3874 (2012)CrossRefGoogle Scholar
  10. 10.
    Dansky, K.H., Thompson, D., Sanner, T.: A framework for evaluating eHealth research. Eval. Program Plann. 29(4), 397–404 (2006)CrossRefGoogle Scholar
  11. 11.
    Lilford, R., Foster, J., Pringle, M.: Evaluating eHealth: how to make evaluation more methodologically robust. PLoS Med. 6(11), e1000186 (2009)CrossRefGoogle Scholar
  12. 12.
    Baker, T.B., Gustafson, D.H., Shah, D.: How can research keep up with eHealth? Ten strategies for increasing the timeliness and usefulness of eHealth research. J. Med. Internet Res. 16(2), e36 (2014)CrossRefGoogle Scholar
  13. 13.
    Greenhalgh, T., Russell, J.: Why do evaluations of eHealth Programs Fail? An alternative set of guiding principles. PLoS Med. 7, e1000360 (2010)CrossRefGoogle Scholar
  14. 14.
    Finch, T., May, C., Mair, F., Mort, M., Gask, L.: Integrating service development with evaluation in telehealthcare: an ethnographic study. BMJ 327, 1205–1209 (2003)CrossRefGoogle Scholar
  15. 15.
    McGee-Lennon, M., Bouamrane, M-M., Barry, S. et al.: Evaluating the delivery of assisted living lifestyles at scale (dallas). In: Proceedings of HCI 2012—People and Computers XXVI, The 26th BCS Conference on Human Computer Interaction, DIILT’12, Birmingham, UK, 12–14 Sept 2012Google Scholar
  16. 16.
    Agbakoba, R., McGee-Lennon, M., Bouamrane, M.-M., Watson, N., Mair, F.: Implementation factors affecting the large-scale deployment of digital health and well-being technologies: a qualitative study of the initial phases of the ‘Living-It-Up’ program. Health Inform. J. 1460458215594651 (2015)Google Scholar
  17. 17.
    Agbakoba, R., McGee-Lennon, M., Bouamrane, M.-M., Watson, N., Mair, F.: Implementing a national scottish digital health and wellbeing service at scale: a qualitative study of stakeholders’ views. Stud. Health Technol. Inform. 216, 487–491 (2015)Google Scholar
  18. 18.
    Devlin, A.M., McGee-Lennon, M., O’Donnell, C.A., Bouamrane, M.M., et al.: Delivering digital health and wellbeing at scale: lessons learned during the implementation of the dallas program in the UK. J. Am. Med. Inform. Assoc. 23(1), 48–59 (2015) Google Scholar
  19. 19.
    May, C., Mair, F., Finch, T., MacFarlane, A., et al.: Development of a theory of implementation and integration: normalization process theory. Implement Sci. 4, 9p (2009)CrossRefGoogle Scholar
  20. 20.
    May, C., Finch, T.: Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology 43(3), 535–554 (2009)CrossRefGoogle Scholar
  21. 21.
    McGee-Lennon, M., Bouamrane, M-M., Devlin, A., O’Connor, S., et al:. Readiness for digital health: findings from dallas, a National Digital Health Innovation Program. Under Peer-Rev. (2016)Google Scholar
  22. 22.
    Mathieson, K.: Predicting user intentions: comparing the technology acceptance model with the theory of planned behavior. Inf. Syst. Res. 2, 173–191 (1992)CrossRefGoogle Scholar
  23. 23.
    Osborne, R.H., Elsworth, G.R., Whitfield, K.: The health education impact questionnaire (heiQ): an outcomes and evaluation measure for patient education and self-management interventions for people with chronic conditions. Patient Educ. Couns. 66, 192–201 (2007)CrossRefGoogle Scholar
  24. 24.
    EuroQol Group: EuroQol: a new facility for the measurement of health related quality of life. Health Policy 16, 199–208 (1990)CrossRefGoogle Scholar
  25. 25.
    McIntosh, E., Clarke, P., Frew, E., et al.: Applied Methods of Cost-Benefit Analysis in Health Care, Vol 4, 119–138. Oxford, UK: Oxford University Press (2010)Google Scholar
  26. 26.
    Räsänen, P., Roine, E., Sintonen, H., Semberg-Konttinen, V., Ryynänen, O.P., Roine, R.: Use of quality-adjusted life years for the estimation of effectiveness of health care: a systematic literature review. Int. J. Technol. Assess. Health Care 22, 235–241 (2006)Google Scholar
  27. 27.
    Lupton, D.: Critical perspectives on digital health technologies. Sociol. Compass 8, 1344–1359 (2014)CrossRefGoogle Scholar
  28. 28.
    Sands, D.Z., Wald, J.S.: Transforming health care delivery through consumer engagement, health data transparency, and patient-generated health information. Yearb. Med. Inf. 9(1), 170–176 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  1. 1.Computer and Information SciencesUniversity of StrathclydeGlasgowUK
  2. 2.Institute of Health and WellbeingUniversity of GlasgowGlasgowUK
  3. 3.Institute of Health and SocietyNewcastle UniversityNewcastleUK

Personalised recommendations