How to Undertake Outcomes Research in Oncology

  • Monika K. KrzyzanowskaEmail author
  • Melanie Powis


Outcomes research is a commonly used term to describe research that focuses on evaluating the quality and effectiveness of care delivery in the routine care setting. However, there is no universally accepted definition or consensus on what this area of inquiry entails. The terms “outcomes research”, “health services research”, and “comparative effectiveness research” are often used interchangeably; however, each seeks to address slightly different research questions. Herein, we examine the evolution of outcomes research, from measuring and reporting on aspects of cancer care delivered using retrospective chart review and administrative data to driving quality improvement via measuring quality, defining performance targets, and evaluating alternative treatments in routine practice through to observational and prospective, pragmatic evaluations of interventions and new models of care.


Outcomes Administrative data Quality measures Indicators Quality improvement Effectiveness Patient-centered outcomes 


List of Technical Terms and Abbreviations

Administrative data 

Data collected routinely for billing purposes.


American Society of Clinical Oncology.


American Society of Hematology.

Case-control study 

A type of observational study that compares a group with an existing outcome (“cases”) with a similar group without the outcome (“controls”) with respect to an exposure.

Cohort study 

A study in which researchers compare what happens to a group that has been exposed to a particular variable with a group who have not been exposed.

Comparative effectiveness research 

Utilizes observational data to compare the benefits and harms of two or more alternative treatments in a real-world, routine care setting to fill in gaps in data derived from randomized trials, such as uptake, long-term complications, and resource utilization, or where such data is missing from randomized trials.

Delphi panel 

A structured, systematic consensus process that utilizes iterative rounds of evaluation, performed by a panel of experts, to converge on an answer.

Health services research 

Descriptive research that is conducted on a population-based cohort or at the system level to address policy-related questions to inform the organization, funding, and/or the delivery of health care.

Outcomes research 

Emphasizes the use of endpoints to examine effectiveness and improve patient care.


Plan-Do-Study-Act; iterative cycles of quality improvement.

process measure 

Measures that evaluate actions or components of care delivery.

Quality indicators/measures 

Mathematical constructs, consisting of a numerator and denominator, that are used to quantify, evaluate, and compare the quality of structures, processes, or outcomes of care being delivered; usually expressed as the proportion of patients receiving a service relative to the number who were eligible to receive that service.

Run chart 

Simple line graph of a measure plotted against time; used to evaluate and visualize trends, patterns, and variation in data in response to process improvement efforts.

structure measure 

This indicator measures the physical and human capital resources available to deliver care.


  1. 1.
    In H, Rosen JE. Primer on outcomes research. J Surg Oncol. 2014;110(5):489–93.CrossRefPubMedGoogle Scholar
  2. 2.
    Lohr KN, Steinwachs DM. Health services research: an evolving definition of the field. Health Serv Res. 2002;37(1):7–9.PubMedGoogle Scholar
  3. 3.
    Marko NF, Weil RJ. The role of observational investigations in comparative effectiveness research. Value Health. 2010;13(8):989–97.CrossRefPubMedGoogle Scholar
  4. 4.
    Hershman DL, Wright JD. Comparative effectiveness research in oncology methodology: observational data. J Clin Oncol. 2012;30(34):4215–22.CrossRefPubMedGoogle Scholar
  5. 5.
    Wilkin D. Outcomes research in primary health care. Fam Pract. 1985;2(4):253–4.CrossRefPubMedGoogle Scholar
  6. 6.
    Lipscomb J, Donaldson MS. Outcomes research at the National Cancer Institute: measuring, understanding, and improving the outcomes of cancer care. Clin Ther. 2003;25(2):699–712.CrossRefPubMedGoogle Scholar
  7. 7.
    Karakiewicz PI, Briganti A, Chun FK, Valiquette L. Outcomes research: a methodologic review. Eur Urol. 2006;50(2):218–24.CrossRefPubMedGoogle Scholar
  8. 8.
    Codman EA. The product of a hospital. Surg Gynecol Obstet. 1914;18:491–6.Google Scholar
  9. 9.
    Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q. 1966;44:166–206.CrossRefGoogle Scholar
  10. 10.
    Cochrane AL. Archie Cochrane in his own words. Selections arranged from his 1972 introduction to “Effectiveness and efficiency: random reflections on health services” 1972. Control Clin Trials. 1989;10:428–33.CrossRefPubMedGoogle Scholar
  11. 11.
    Wennberg J, Gittelsohn A. Small area variations in health care delivery. Science. 1973;182:1102–8.CrossRefPubMedGoogle Scholar
  12. 12.
    Hanks GE, Kramer S, Diamond JJ, Herring DF. Patterns of care outcome survey: national outcome data for six disease sites. Am J Clin Oncol. 1982;5:349–53.CrossRefPubMedGoogle Scholar
  13. 13.
    Kramer S, Hanks GE, Diamond JJ, MacLean CJ. The study of the patterns of clinical care in radiation therapy in the United States. CA Cancer J Clin. 1984;34:75–85.CrossRefPubMedGoogle Scholar
  14. 14.
    Ellwood PM. Shattuck lecture- outcomes management: a technology of patient experience. N Engl J Med. 1988;318:1549–56.CrossRefPubMedGoogle Scholar
  15. 15.
    Institute of Medicine (US) Committee on Potential Conflicts of Interest in Patient Outcomes Research Teams. In: Donaldson MS, Capron AM, editors. Patient outcomes research teams: managing conflict of interest. Washington (DC): National Academies Press (US); 1991.Google Scholar
  16. 16.
    Epstein AM. The outcomes movement- will it get us where we want to go? N Engl J Med. 1990;323:266–7.CrossRefPubMedGoogle Scholar
  17. 17.
    van Dam PA, Tomatis M, Marotti L, Heil J, Wilson R, Rosselli Del Turco M, et al. The effect of EUSOMA certification on quality of breast cancer care. Eur J Surg Oncol. 2015;41(10):1423–9.CrossRefPubMedGoogle Scholar
  18. 18.
    Prince RM, Atenafu EG, Krzyzanowska MK. Hospitalizations during systemic therapy for metastatic lung cancer: a systematic review of real world vs clinical trial outcomes. JAMA Oncol. 2015;1(9):1333–9.CrossRefPubMedGoogle Scholar
  19. 19.
    Fojo T, Mailankody S, Lo A. Unintended consequences of expensive cancer therapeutics—the pursuit of marginal indications and a me-too mentality that stifles innovation and creativity: the John Conley lecture. JAMA Otolaryngol Head Neck Surg. 2014;140:1225–36.CrossRefPubMedGoogle Scholar
  20. 20.
    Parry C, Kent EE, Mariotto AB, Alfano CM, Rowland JH. Cancer survivors: a booming population. Cancer Epidemiol Biomark Prev. 2011;20(10):1996–2005.CrossRefGoogle Scholar
  21. 21.
    Mooney K, Beck SL, Wong B, Dunson WA, Wujcik D. An IT-integrated, computer-based telephone system for monitoring patient-reported symptoms: result of two trials. J Clin Oncol. 2013;30(suppl 34):abst 2.Google Scholar
  22. 22.
    Kearney N, McCann L, Norrie J. Evaluation of a mobile phone-based, advanced symptom management system (ASyMS) in the management of chemotherapy-related toxicity. Support Care Cancer. 2009;17(4):437–44.CrossRefPubMedGoogle Scholar
  23. 23.
    Lee SJ, Earle CC, Weeks JC. Outcomes research in oncology: history, conceptual framework, and trends in the literature. J Natl Cancer Inst. 2000;92(3):196–204.CrossRefGoogle Scholar
  24. 24.
    Schnipper LE, Lyman GH, Blayney DW. American Society of Clinical Oncology 2013 top five list in oncology. J Clin Oncol. 2013;31:4362–70.CrossRefPubMedGoogle Scholar
  25. 25.
    Hicks LK, Bering H, Carson KR, Kleinerman J, Kukreti V, Ma A, et al. The ASH cChoosing Wisely® campaign: five hematologic tests and treatments to question. Blood. 2013;122(24):3879–83.CrossRefPubMedGoogle Scholar
  26. 26.
    Biagi JJ, Raphael MJ, Mackillop WJ, Kong W, King WD, Booth CM. Association between time to initiation of adjuvant chemotherapy and survival in colorectal cancer: a systematic review and meta-analysis. JAMA. 2011;305(22):2335–42.CrossRefPubMedGoogle Scholar
  27. 27.
    Juarez JE, Choi J, St John M, Abemayor E, TenNapel M, Chen AM. Patterns of care for elderly patients with locally advanced head and neck cancer. Int J Radiat Oncol Biol Phys. 2017;98:767–74. pii: S0360-3016(17)30271-7 [Epub ahead of print]CrossRefPubMedGoogle Scholar
  28. 28.
    Fountzilas C, Chang K, Hernandez B, Michalek J, Crownover R, Floyd J, Mahalingam D. Clinical characteristics and treatment outcomes of patients with colorectal cancer who develop brain metastasis: a single institution experience. J Gastrointest Oncol. 2017;8(1):55–63.CrossRefPubMedPubMedCentralGoogle Scholar
  29. 29.
    Murthy VH, Krumholtz HM, Gross CP. Participating in cancer clinical trials: race-, sex-, and age-based disparities. JAMA. 2004;291:2720–6.CrossRefPubMedGoogle Scholar
  30. 30.
    Simon MS, Du W, Flaherty L, Philip PA, Lorusso P, Miree C, et al. Factors associated with breast cancer clinical trials participation and enrollment at a large academic medical centre. J Clin Oncol. 2004;22:2046–52.CrossRefPubMedGoogle Scholar
  31. 31.
    Enright KA, Krzyzanowska MK. Benefits and pitfalls of using administrative data to study hospitalization patterns in patients with cancer treated with chemotherapy. J Oncol Pract. 2016;12(2):140–1.CrossRefPubMedGoogle Scholar
  32. 32.
    Williams J, Young W. A summary of studies on the quality of healthcare administrative databases in Canada. In: Goel V, Williams JI, Anderson GM, Blackstien-Hirsch P, Fooks C, Naylor CD, editors. Patterns of healthcare in Ontario: the ICES practice atlas. Ottawa: Canadian Medical Association; 1996. p. 339–45.Google Scholar
  33. 33.
    Institute of Medicine. Ensuring Quality of Cancer Care. The National Academies of Science and Engineering. Accessed 22 April 2017.
  34. 34.
    Kiefe CI, Weissman NW, Allison JJ, Farmer R, Weaver M, Williams OD. Identifying achievable benchmarks of care: concepts and methodology. Int J Qual Health Care. 1998;10:443–7.CrossRefPubMedGoogle Scholar
  35. 35.
    Shen S, Krzyzanowska MK. A decade of research on the quality of systemic cancer therapy in routine care: what aspects of quality are we measuring? J Oncol Pract. 2015;11(1):55–61.CrossRefPubMedGoogle Scholar
  36. 36.
    Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Information & Management. 2004;42(1):15–29.CrossRefGoogle Scholar
  37. 37.
    In H, Neville BA, Lipsitz SR, Corso KA, Weeks JC, Greenberg CC. The role of National Cancer Institute-designated cancer center status: observed variation in surgical care depends on the level of evidence. Ann Surg. 2012;255(5):890–5.CrossRefPubMedPubMedCentralGoogle Scholar
  38. 38.
    American Society of Clinical Oncology: quality oncology practice initiative. Accessed 16 May 2017.
  39. 39.
    Duvalko KM, Sherar M, Sawka C. Creating a system for performance improvement in cancer care: Cancer Care Ontario's clinical governance framework. Cancer Control. 2009;16(4):293–302.CrossRefPubMedGoogle Scholar
  40. 40.
    Hassett MJ, Neville BA, Weeks JC. The relationship between quality, spending and outcomes among women with breast cancer. J Natl Cancer Inst. 2014;106(10):dju242.CrossRefPubMedGoogle Scholar
  41. 41.
    Simunovic M, Rempel E, Thériault ME, Coates A, Whelan T, Holowaty E, et al. Influence of hospital characteristics on operative death and survival of patients after major cancer surgery in Ontario. Can J Surg. 2006;49(4):251–8.PubMedPubMedCentralGoogle Scholar
  42. 42.
    Mandelblatt JS, Huang K, Makgoeng SB, Luta G, Song JX, Tallarico M, et al. Preliminary development and evaluation of an algorithm to identify breast cancer chemotherapy toxicities using electronic medical records and administrative data. J Oncol Pract. 2015;11(1):e1–8.CrossRefPubMedGoogle Scholar
  43. 43.
    Earle CC, Neville BA, Landrum MB, Souza JM, Weeks JC, Block SD, et al. Evaluating claims-based indicators of the intensity of end-of-life cancer care. Int J Qual Health Care. 2005;17(6):505–9.CrossRefPubMedGoogle Scholar
  44. 44.
    Hibbard JH, Stockard J, Tusler M. Hospital performance reports: impact on quality, market share, and reputation. Health Aff (Millwood). 2005;24(4):1150–60.CrossRefGoogle Scholar
  45. 45.
    Grossbart SR. What's the return? Assessing the effect of "pay-for-performance" initiatives on the quality of care delivery. Med Care Res Rev. 2006;63(1 Suppl):29S–48S.CrossRefPubMedGoogle Scholar
  46. 46.
    Aiello Bowles EJ, Tuzzio L, Wiese CJ, Kirlin B, Greene SM, Clauser SB, Wagner EH. Understanding high-quality cancer care: a summary of expert perspectives. Cancer. 2008;112(4):934–42.CrossRefPubMedGoogle Scholar
  47. 47.
    Howard DH. A better understanding of variation in cancer care. Med Care. 2012;50(5):363–5.CrossRefPubMedGoogle Scholar
  48. 48.
    Morgan J, Richards P, Ward S, Francis M, Lawrence G, Collins K, et al. Case-mix analysis and variation in rates of non-surgical treatment of older women with operable breast cancer. Br J Surg. 2015;102(9):1056–63.CrossRefPubMedGoogle Scholar
  49. 49.
    Selby JV, Schmittdiel JA, Lee J, Fung V, Thomas S, Smider N, et al. Meaningful variation in performance: what does variation in quality tell us about improving quality? Med Care. 2010;48(2):133–9.CrossRefPubMedGoogle Scholar
  50. 50.
    Hassett MJ, Hughes ME, Niland JC, Ottesen R, Edge SB, Bookman MA, et al. Selecting high priority quality measures for breast cancer quality improvement. Med Care. 2008;46(8):762–70.CrossRefPubMedPubMedCentralGoogle Scholar
  51. 51.
    Enright KA, Taback NA, Powis M, Gonzalez A, Yun L, Sutradhar R, et al. Setting quality improvement priorities for women receiving systemic therapy for early stage breast cancer using population level administrative data. J Clin Oncol. 2017;35:3207–14.CrossRefPubMedGoogle Scholar
  52. 52.
    Thonon F, Watson J, Saghatchian M. Benchmarking facilities providing care: an international overview of initiatives. SAGE Open Med. 2015;3:2050312115601692.CrossRefPubMedPubMedCentralGoogle Scholar
  53. 53.
    van Dam PA, Verkinderen L, Hauspy J, Vermeulen P, Dirix L, Huizing M, et al. Benchmarking and audit of breast units improves quality of care. Facts Views Vis Obgyn. 2013;5(1):26–32.PubMedPubMedCentralGoogle Scholar
  54. 54.
    Brucker SY, Schumacher C, Sohn C, Rezai M, Bamberg M, Wallwiener D. Steering Committee. Benchmarking the quality of breast cancer care in a nationwide voluntary system: the first five-year results (2003-2007) from Germany as a proof of concept. BMC Cancer. 2008;8:358.CrossRefPubMedPubMedCentralGoogle Scholar
  55. 55.
    Palmer RH. Process-based measures of quality: the need for detailed clinical data in large health care databases. Ann Intern Med. 1997;127(8 Pt 2):733–8.CrossRefPubMedGoogle Scholar
  56. 56.
    Weissman NW, Allison JJ, Kiefe CI, Farmer RM, Weaver MT, Williams OD, et al. Achievable benchmarks of care: the ABCs of benchmarking. J Eval Clin Pract. 1999;5(3):269–81.CrossRefPubMedGoogle Scholar
  57. 57.
    Powis M, Sutradhar R, Gonzalez A, Enright KA, Taback NA, Booth CM, et al. Establishing achievable benchmarks for quality improvement in systemic therapy for early-stage breast cancer. Cancer. 2017;123:3772–80.CrossRefPubMedGoogle Scholar
  58. 58.
    Deming WE. Out of the crisis. Cambridge, MA: Center for Advanced Engineering Study; 1986.Google Scholar
  59. 59.
    Langley GJ, Nolan KM, Nolan TW, Norman CL, Provost LP. The improvement guide: a practical approach to enhancing organizational performance. San Francisco, CA: Jossey-Bass; 1996.Google Scholar
  60. 60.
    Plsek PE. Quality improvement methods in clinical medicine. Pediatrics. 1999;103(1 Suppl E):203–14.PubMedGoogle Scholar
  61. 61.
    Perla RJ, Provost LP, Murray SK. The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Qual Saf. 2011;20(1):46–51.CrossRefPubMedGoogle Scholar
  62. 62.
    Gross CP, Filardo G, Mayne ST, Krumholz HM. The impact of socioeconomic status and race on trial participation for older women with breast cancer. Cancer. 2005;103:483–91.CrossRefPubMedGoogle Scholar
  63. 63.
    Crew KD, Neugut AI, Wang X, Jacobson JS, Grann VR, Raptis G, Hershman DL. Racial disparities in treatment and survival of male breast cancer. J Clin Oncol. 2007;25(9):1089–98.CrossRefPubMedGoogle Scholar
  64. 64.
    Barzi A, Lenz HJ, Quinn DI, Sadeghi S. Comparative effectiveness of screening strategies for colorectal cancer. Cancer. 2017;123(9):1516–27.CrossRefPubMedGoogle Scholar
  65. 65.
    Lee L, Crump M, Khor S, Hoch JS, Luo J, Bremner K, et al. Impact of rituximab on treatment outcomes of patients with diffuse large b-cell lymphoma: a population-based analysis. Br J Haematol. 2012;158(4):481–8.CrossRefPubMedGoogle Scholar
  66. 66.
    Patel MI, Periyakoil VS, Blayney DW, Moore D, Nevedal A, Asch S, et al. Redesigning cancer care delivery: views from patients and caregivers. J Oncol Pract. 2017;13(4):e291–302.CrossRefPubMedPubMedCentralGoogle Scholar
  67. 67.
    Jensen RE, Snyder CF, Abernethy AP, Basch E, Potosky AL, Roberts AC, et al. Review of electronic patient-reported outcomes systems used in cancer clinical care. J Oncol Pract. 2014;10(4):e215–22.CrossRefPubMedGoogle Scholar
  68. 68.
    Spandio JD. Oncology patient–centered medical home. J Oncol Pract. 2012;8(3 Suppl):47s–9s.CrossRefGoogle Scholar
  69. 69.
    Enright K, Grunfeld E, Yun L, Moineddin R, Ghannam M, Dent S, et al. Population-based assessment of emergency room visits and hospitalizations among women receiving adjuvant chemotherapy for early breast cancer. J Oncol Pract. 2015;11(2):126–32.CrossRefPubMedGoogle Scholar
  70. 70.
    Hassett MJ, O'Malley AJ, Pakes JR, Newhouse JP, Earle CC. Frequency and cost of chemotherapy-related serious adverse effects in a population sample of women with breast cancer. J Natl Cancer Inst. 2006;98(16):1108–17.CrossRefPubMedGoogle Scholar
  71. 71.
    Kearney N, Miller M, Maguire R, Dolan S, MacDonald R, McLeod J, et al. WISECARE+: results of a European study of a nursing intervention for the management of chemotherapy-related symptoms. Eur J Oncol Nurs. 2008;12(5):443–8.CrossRefPubMedGoogle Scholar
  72. 72.
    Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol. 2016;34(6):557–65.CrossRefPubMedGoogle Scholar
  73. 73.
    Committee on Improving the Quality of Cancer Care. Addressing the challenges of an aging population; Board on Health Care Services. In: Institute of Medicine, Levit L, Balogh E, Nass S, Ganz PA, editors. Delivering high-quality cancer care: charting a new course for a system in crisis. Washington (DC): National Academies Press (US); 2013.Google Scholar
  74. 74.
    Goulart BHL. Value: the next frontier in cancer care. Oncologist. 2016;21:651–3.CrossRefPubMedPubMedCentralGoogle Scholar
  75. 75.
    Schnipper LE, Davidson NE, Wollins DS, Tyne C, Blayney DW, Blum D, et al. American Society of Clinical Oncology Statement: a conceptual framework to assess the value of cancer treatment options. J Clin Oncol. 2015;33(23):2563–77.CrossRefPubMedPubMedCentralGoogle Scholar
  76. 76.
    Cherny NI, Sullivan R, Dafni U, Kerst JM, Sobrero A, Zielinski C, et al. A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS). Ann Oncol. 2015;26(8):1547–73.CrossRefPubMedGoogle Scholar
  77. 77.
    Mandelblatt JS, Ramsey SD, Lieu TA, Phelps CE. Evaluating frameworks that provide value measures for health care interventions. Value Health. 2017;20(2):185–92.CrossRefPubMedPubMedCentralGoogle Scholar
  78. 78.
    Miriovsky BJ, Shulman LN, Abernethy AP. Importance of health information technology, electronic health records, and continuously aggregating data to comparative effectiveness research and learning health care. J Clin Oncol. 2012;30(34):4243–8.CrossRefPubMedGoogle Scholar
  79. 79.
    Gu D, Li J, Li X, Liang C. Visualizing the knowledge structure and evolution of big data research in healthcare informatics. Int J Med Inform. 2017;98:22–32.CrossRefPubMedGoogle Scholar
  80. 80.
    Shah A, Stewart AK, Kolacevski A, Michels D, Miller R. Building a rapid learning health care system for oncology: why CancerLinQ collects identifiable health information to achieve its vision. J Clin Oncol. 2016;34(7):756–63.CrossRefPubMedGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Division of Medical Oncology & HematologyPrincess Margaret Cancer CentreTorontoCanada

Personalised recommendations