Supportive Care in Cancer

, Volume 27, Issue 5, pp 1835–1844 | Cite as

Usability testing of EirV3—a computer-based tool for patient-reported outcome measures in cancer

  • Hilde KrogstadEmail author
  • Stine Marie Sundt-Hansen
  • Marianne Jensen Hjermstad
  • Liv Ågot Hågensen
  • Stein Kaasa
  • Jon Håvard Loge
  • Sunil X. Raj
  • Aslak Steinsbekk
  • Kari Sand
Original Article



Eir version 3 (V3) is an electronic tool for administration of patient-reported outcome measures (Eir-Patient) that immediately presents patient scores on the physician’s computer (Eir-Doctor). Perceived usability is an important determinant for successful implementation. The aim of this study was to answer the following research question evaluated at the cancer outpatient clinics, in the patients’ home, and at general practitioners’ (GPs) offices: What are the number, type, and severity of usability issues evaluated by the patient (Eir-Patient module) and by the physician (Eir-Doctor module)?


A usability evaluation using observations, think-aloud sessions, individual interviews and focus group interviews in cancer patients and their physicians was conducted. Identified usability issues were graded on a severity scale from 1 (irritant) to 4 (unusable).


Overall, 73 Eir registrations were performed by 37 patients, and used by 17 physicians in clinical consultations. All patients were able to complete the Eir-Patient symptom registration. Seventy-two usability issues were identified. None of them were graded as unusable. For the Eir-Patient module, 62% of the identified usability issues was graded as irritant (grade 1), 18% as moderate (grade 2), and 20% as severe (grade 3). For the Eir-Doctor module, 46% of the identified usability issues were graded as irritant, 36% as moderate and 18% as severe.


In the updated Eir version, issues in the severe and moderate categories have been changed, to optimize the usability of using real-time PROMs in clinical practice.


Patient-reported outcome measures PROMs Usability Feasibility Electronic patient-reported outcomes 



We would like to thank all participating patients and physicians in this study, as well as the Coordination Unit Orkdal Region for their project management contribution.

Funding information

The study was funded by the Regional Research Funds in Norway, Mid-Norway, and the European Palliative Care Research Centre, Department of Clinical and Molecular Medicine, NTNU, Norwegian University of Science and Technology, and St. Olavs Hospital, Trondheim University Hospital.

Compliance with ethical standards

Disclosure of potential conflicts of interests

Krogstad, Sundt-Hansen, Hjermstad, Hågensen, Raj, Steinsbekk, and Sand have declared no conflicts of interests. Eir Solutions AS was established in 2015 with Kaasa, Loge, and NTNU technology Transfer AS/Andersen as shareholders. No income, dividend, or financial benefits are related to the work presented here nor in relation to Eir in any way.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.


  1. 1.
    Stacey D, Legare F, Col NF, Bennett CL, Barry MJ, Eden KB, Holmes-Rovner M, Llewellyn-Thomas H, Lyddiatt A, Thomson R, Trevena L, Wu JH (2014) Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev (1):Cd001431.
  2. 2.
    Richards T, Coulter A, Wicks P (2015) Time to deliver patient centred care. BMJ 350:h530. CrossRefGoogle Scholar
  3. 3.
  4. 4.
    Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, Rogak L, Bennett AV, Dueck AC, Atkinson TM, Chou JF, Dulko D, Sit L, Barz A, Novotny P, Fruscione M, Sloan JA, Schrag D (2016) Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol Off J Am Soc Clin Oncol 34(6):557–565. CrossRefGoogle Scholar
  5. 5.
    Atkinson TM, Li Y, Coffey CW, Sit L, Shaw M, Lavene D, Bennett AV, Fruscione M, Rogak L, Hay J, Gonen M, Schrag D, Basch E (2012) Reliability of adverse symptom event reporting by clinicians. Qual Life Res 21(7):1159–1164. CrossRefGoogle Scholar
  6. 6.
    Strasser F, Blum D, von Moos R, Cathomas R, Ribi K, Aebi S, Betticher D, Hayoz S, Klingbiel D, Brauchli P, Haefner M, Mauri S, Kaasa S, Koeberle D (2016) The effect of real-time electronic monitoring of patient-reported symptoms and clinical syndromes in outpatient workflow of medical oncologists: E-MOSAIC, a multicenter cluster-randomized phase III study (SAKK 95/06). Ann Oncol 27(2):324–332. CrossRefGoogle Scholar
  7. 7.
    Basch E, Deal AM, Dueck AC, Scher HI, Kris MG, Hudis C, Schrag D (2017) Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 318(2):197–198. CrossRefGoogle Scholar
  8. 8.
    Food and Drug Administration USDoHaHS (2009) Guidance for industry. Patient-reported outcome measures: use in medical product development to support labeling claims.
  9. 9.
    Zikos E, Ghislain I, Coens C, Ediebah DE, Sloan E, Quinten C, Koller M, van Meerbeeck JP, Flechtner HH, Stupp R, Pallis A, Czimbalmos A, Sprangers MA, Bottomley A (2014) Health-related quality of life in small-cell lung cancer: a systematic review on reporting of methods and clinical issues in randomised controlled trials. Lancet Oncol 15(2):e78–e89. CrossRefGoogle Scholar
  10. 10.
    Gravis G, Marino P, Joly F, Oudard S, Priou F, Esterni B, Latorzeff I, Delva R, Krakowski I, Laguerre B, Rolland F, Theodore C, Deplanque G, Ferrero JM, Pouessel D, Mourey L, Beuzeboc P, Zanetta S, Habibian M, Berdah JF, Dauba J, Baciuchka M, Platini C, Linassier C, Labourey JL, Machiels JP, El Kouri C, Ravaud A, Suc E, Eymard JC, Hasbini A, Bousquet G, Soulie M, Fizazi K (2014) Patients’ self-assessment versus investigators’ evaluation in a phase III trial in non-castrate metastatic prostate cancer (GETUG-AFU 15). Eur J Cancer 50(5):953–962. CrossRefGoogle Scholar
  11. 11.
    Kroenke K, Monahan PO, Kean J (2015) Pragmatic characteristics of patient-reported outcome measures are important for use in clinical practice. J Clin Epidemiol 68(9):1085–1092. CrossRefGoogle Scholar
  12. 12.
    McGrail K, Bryan S, Davis J (2011) Let’s all go to the PROM: the case for routine patient-reported outcome measurement in Canadian healthcare. Healthc Pap 11(4):8–18 discussion 55-18CrossRefGoogle Scholar
  13. 13.
    Benze G, Nauck F, Alt-Epping B, Gianni G, Bauknecht T, Ettl J, Munte A, Kretzschmar L, Gaertner J (2017) PROutine: a feasibility study assessing surveillance of electronic patient reported outcomes and adherence via smartphone app in advanced cancer. Ann Palliat Med 6:705. CrossRefGoogle Scholar
  14. 14.
    Hjermstad MJ, Lie HC, Caraceni A, Currow DC, Fainsinger RL, Gundersen OE, Haugen DF, Heitzer E, Radbruch L, Stone PC, Strasser F, Kaasa S, Loge JH (2012) Computer-based symptom assessment is feasible in patients with advanced cancer: results from an international multicenter study, the EPCRC-CSA. J Pain Symptom Manag 44(5):639–654. CrossRefGoogle Scholar
  15. 15.
    Krogstad H, Brunelli C, Sand K, Andersen E, Garresori H, Halvorsen T, Haukland EC, Jordal F, Kaasa S, Loge JH, Løhre ET, Raj SX, Hjermstad MJ (2017) Development of EirV3: a computer-based tool for patient-reported outcome measures in cancer. JCO Clin Cancer Inf.
  16. 16.
    Wintner LM, Giesinger JM, Zabernigg A, Rumpold G, Sztankay M, Oberguggenberger AS, Gamper EM, Holzner B (2015) Evaluation of electronic patient-reported outcome assessment with cancer patients in the hospital and at home. BMC Med Inf Decis Mak 15:110. CrossRefGoogle Scholar
  17. 17.
    International Organization for Standardization/International Electrotehcnical Commission (ISO/IEC) (1998) Ergonomic requirements for office work with visual display terminals (VDTs)-Part 11- Guidance on usability 9241-11Google Scholar
  18. 18.
    Stinson J, Gupta A, Dupuis F, Dick B, Laverdiere C, LeMay S, Sung L, Dettmer E, Gomer S, Lober J, Chan CY (2015) Usability testing of an online self-management program for adolescents with cancer. J Pediatr Oncol Nurs 32(2):70–82. CrossRefGoogle Scholar
  19. 19.
    Corrao NJ, Robinson AG, Swiernik MA, Naeim A (2010) Importance of testing for usability when selecting and implementing an electronic health or medical record system. J Oncol Pract 6(3):120–124. CrossRefGoogle Scholar
  20. 20.
    European Palliative Care Research Centre (PRC), .
  21. 21.
    Sigurdardottir KR, Kaasa S, Rosland JH, Bausewein C, Radbruch L, Haugen DF (2014) The European Association for palliative care basic dataset to describe a palliative care cancer population: results from an international Delphi process. Palliat Med 28(6):463–473. CrossRefGoogle Scholar
  22. 22.
    Bauer J, Capra S, Ferguson M (2002) Use of the scored patient-generated subjective global assessment (PG-SGA) as a nutrition assessment tool in patients with cancer. Eur J Clin Nutr 56(8):779–785. CrossRefGoogle Scholar
  23. 23.
    Kroenke K, Spitzer RL, Williams JB (2001) The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 16(9):606–613CrossRefGoogle Scholar
  24. 24.
    Kroenke K, Spitzer RL, Williams JB, Monahan PO, Lowe B (2007) Anxiety disorders in primary care: prevalence, impairment, comorbidity, and detection. Ann Intern Med 146(5):317–325CrossRefGoogle Scholar
  25. 25.
    Morin CM (1996) Insomnia: psychological assessment and managementGoogle Scholar
  26. 26.
    Common Terminology Criteria for Adverse Events (CTCAE) Version 4.0 (2009)Google Scholar
  27. 27.
    Rubin J, Chisnell D (2008) Handbook of usability testing: how to plan, design, and conduct effective tests. WileyGoogle Scholar
  28. 28.
    Lewis C (1982) Using the “think aloud” method in cognitive interface design. IBM: Technical report RC-9265, Yorktown HeightsGoogle Scholar
  29. 29.
    Stinson J, McGrath P, Hodnett E, Feldman B, Duffy C, Huber A, Tucker L, Hetherington R, Tse S, Spiegel L, Campillo S, Gill N, White M (2010) Usability testing of an online self-management program for adolescents with juvenile idiopathic arthritis. J Med Internet Res 12(3):e30. CrossRefGoogle Scholar
  30. 30.
    Wolpin SE, Halpenny B, Whitman G, McReynolds J, Stewart M, Lober WB, Berry DL (2015) Development and usability testing of a web-based cancer symptom and quality-of-life support intervention. Health Informatics J 21(1):10–23. CrossRefGoogle Scholar
  31. 31.
    Bradbury K, Watts S, Arden-Close E, Yardley L, Lewith G (2014) Developing digital interventions: a methodological guide. Evid Based Complement Alternat Med 2014:561320–561327. CrossRefGoogle Scholar
  32. 32.
    Dawson J, Doll H, Fitzpatrick R, Jenkinson C, Carr AJ (2010) The routine use of patient reported outcome measures in healthcare settings. BMJ 340:c186. CrossRefGoogle Scholar
  33. 33.
    Walker J, Bohnke JR, Cerny T, Strasser F (2010) Development of symptom assessments utilising item response theory and computer-adaptive testing—a practical method based on a systematic review. Crit Rev Oncol Hematol 73(1):47–67. CrossRefGoogle Scholar
  34. 34.
    Breakey VR, Warias AV, Ignas DM, White M, Blanchette VS, Stinson JN (2013) The value of usability testing for internet-based adolescent self-management interventions: “managing hemophilia online”. BMC Med Inform Decis Mak 13:113. CrossRefGoogle Scholar
  35. 35.
    Korzeniowski M, Kalyvas M, Mahmud A, Shenfield C, Tong C, Zaza K, Howell D, Brundage M (2016) Piloting prostate cancer patient-reported outcomesin clinical practice. Support Care Cancer 24(5):1983–1990. CrossRefGoogle Scholar
  36. 36.
    Mirkovic J, Kaufman DR, Ruland CM (2014) Supporting cancer patients in illness management: usability evaluation of a mobile app. JMIR Mhealth Uhealth 2(3):e33. CrossRefGoogle Scholar
  37. 37.
    Jordan K, Aapro M, Kaasa S, Ripamonti CI, Scotté F, Strasser F, Young A, Bruera E, Herrstedt J, Keefe D, Laird B, Walsh D, Douillard JY, Cervantes A (2018) European Society for Medical Oncology (ESMO) position paper on supportive and palliative care. Ann Oncol 29(1):36–43. CrossRefGoogle Scholar
  38. 38.
    Jensen RE, Snyder CF, Abernethy AP, Basch E, Potosky AL, Roberts AC, Loeffler DR, Reeve BB (2014) Review of electronic patient-reported outcomes systems used in cancer clinical care. J Oncol Pract 10(4):e215–e222. CrossRefGoogle Scholar
  39. 39.
    Blum D, Raj SX, Oberholzer R, Riphagen II, Strasser F, Kaasa S (2015) Computer-based clinical decision support systems and patient-reported outcomes: a systematic review. Patient 8(5):397–409. CrossRefGoogle Scholar
  40. 40.
    Palliative Care Integrated in Oncology (PALLiON) (2017)
  41. 41.
    Virzi RA (1992) Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors 34(4):457–468. CrossRefGoogle Scholar
  42. 42.
    Brooke J (1996) SUS: a “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL (eds) Usability evaluation in industry. Taylor & Francis, London, pp 189–194Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Hilde Krogstad
    • 1
    • 2
    Email author
  • Stine Marie Sundt-Hansen
    • 1
    • 3
  • Marianne Jensen Hjermstad
    • 4
  • Liv Ågot Hågensen
    • 5
  • Stein Kaasa
    • 4
    • 6
  • Jon Håvard Loge
    • 4
    • 7
  • Sunil X. Raj
    • 1
    • 2
  • Aslak Steinsbekk
    • 8
  • Kari Sand
    • 1
    • 2
  1. 1.European Palliative Care Research Centre (PRC), Department of Clinical and Molecular Medicine, Faculty of Medicine and Health Sciences, NTNUNorwegian University of Science and Technology, and St. Olavs Hospital, Trondheim University HospitalTrondheimNorway
  2. 2.Cancer Clinic, St. Olavs HospitalTrondheim University HospitalTrondheimNorway
  3. 3.Department of Medicine, St. Olavs HospitalTrondheim University HospitalTrondheimNorway
  4. 4.European Palliative Care Research Centre (PRC), Department of OncologyOslo University Hospital and Institute of Clinical Medicine, University of OsloOsloNorway
  5. 5.Coordination Unit Orkdal RegionOrkdalNorway
  6. 6.Department of OncologyOslo University HospitalOsloNorway
  7. 7.Institute of Basic Medical Sciences, Faculty of MedicineUniversity of OsloOsloNorway
  8. 8.Department of Public Health and Nursing, Faculty of Medicine and Health Sciences, NTNUNorwegian University of Science and TechnologyTrondheimNorway

Personalised recommendations