, Volume 32, Issue 4, pp 327–334 | Cite as

Methods for Identifying the Cost-effective Case Definition Cut-Off for Sequential Monitoring Tests: An Extension of Phelps and Mushlin

  • Roberta Longo
  • Paul Baxter
  • Peter Hall
  • Jenny Hewison
  • Mehran Afshar
  • Geoff Hall
  • Christopher McCabe
Leading Article


The arrival of personalized medicine in the clinic means that treatment decisions will increasingly rely on test results. The challenge of limited healthcare resources means that the dissemination of these technologies will be dependent on their value in relation to their cost, i.e., their cost effectiveness. Phelps and Mushlin have described how to optimize tests to meet a cost-effectiveness target. However, when tests are applied repeatedly the case mix of the patients tested changes with each administration, and this impacts upon the value of each subsequent test administration. In this article, we present a modification of Phelps and Mushlin’s framework for diagnostic tests; to identify the cost-effective cut-off for monitoring tests. Using the Ca125 test monitoring for relapse in ovarian cancer, we show how the repeated use of the initial cut-off can lead to a substantially increased false-negative rate compared with the monitoring cut-off—over 4 % higher than in this example—with the associated harms for individual and population health.


Ovarian Cancer Case Definition Receiver Operator Curve Sick Person Monitoring Test 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This article presents independent research partially funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research Programme (RP-PG-0707-10101: Evaluating the benefits for patients and the NHS of new and existing biological fluid biomarkers in liver and renal disease). Additional support was received through the Medical Technologies Innovation and Knowledge Centre funded by the UK Research Councils, EPSRC, and BBSRC, and the Technology Strategy Board. We wish to acknowledge helpful discussions with colleagues on the NIHR Programme Grant award, especially Doug Altman, Jon Deeks, Walter Gregory, and Peter Selby. Prof. McCabe’s research is made possible by the Capital Health Research Chair Endowment to the University of Alberta Faculty of Medicine and Dentistry. He also received funding from the Genome Canada Large Scale Applied Research Program on Personalized Medicine for some of this work. The work reported in this paper and any errors remain the responsibility of the authors.

All individuals who made substantive contributions to the ideas presented in this article are included as authors. The initial concepts for this article were proposed by Christopher McCabe. Roberta Longo and Paul Baxter undertook the modeling work to provide the example and prepared the draft for the manuscript. Peter Hall, Geoff Hall, and Mehran Afshar provided the clinical data and contributed to the manuscript writing. Christopher McCabe and Jenny Hewison supervised the preparation of the draft and contributed to the manuscript writing.

Conflicts of interest

Roberta Longo has no conflicts of interest.

Paul Baxter has no conflicts of interest.

Peter Hall has no conflicts of interest.

Jenny Hewison has no conflicts of interest.

Mehran Afshar has no conflicts of interest.

Geoff Hall has no conflicts of interest.

Christopher McCabe has no conflicts of interest.


The views and opinions expressed by the authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, the Department of Health, or Genome Canada.


  1. 1.
    National Institute for Health and Clinical Excellence. Diagnostics Assessment Programme. Accessed 29th July 2013.
  2. 2.
    Department of Health and Aging. Co-dependent and Hybrid Technologies. Accessed 29th July 2013.
  3. 3.
    Phelps CE, Mushlin A. Focussing technology assessment using decision theory. Med Decis Mak. 1988;8(4):279–89.CrossRefGoogle Scholar
  4. 4.
    Guttmacher AE, Collins FS. Genomic Medicine: a primer. New Engl J Med. 2002;347(19):1512–20.PubMedCrossRefGoogle Scholar
  5. 5.
    Carroll PR, Whitson JM, Cooperberg MR. Serum prostate-specific antigen for the early detection of prostate cancer: always, never, or only sometimes? J Clin Oncol. 2011;28(4):345–6.CrossRefGoogle Scholar
  6. 6.
    Rustin GJ, van der Burg ME, Griffin CL, Guthrie D, Lamont A, Jayson GC, Kristensen G, Mediola C, Coens C, Qian W, Parmar MK. Swart AM; MRC OV05; EORTC 55955 investigators. Early versus delayed treatment of relapsed ovarian cancer (MRC OV05/EORTC 55955): a randomised trial. Lancet. 2010;376(9747):1155–63.PubMedCrossRefGoogle Scholar
  7. 7.
    Parkes J, Guha IN, Roderick P, Harris S, Cross R, Manos MM, Irving W, Zaitoun A, Wheatley M, Ryder S, Rosenberg W. Enhanced liver fibrosis (ELF) test accurately identifies liver fibrosis in patients with chronic hepatitis C. J Viral Hepat. 2011;18(1):23–31. doi: 10.1111/j.1365-2893.2009.01263.x.PubMedCrossRefGoogle Scholar
  8. 8.
    Hosmer DW, Lemeshow S. Applied logistic regression. 2nd ed. New York: Wiley; 2000. p. 156–164.Google Scholar
  9. 9.
    McCabe C, Claxton K, Culyer AJ. The NICE cost effectiveness threshold: what it is and what that means? PharmacoEconomics. 2008;26(9):733–44.PubMedCrossRefGoogle Scholar
  10. 10.
    NICE. Guide to the Methods of Technology Appraisal. London; 2008.Google Scholar
  11. 11.
    Pesce LL, Metz CE, Berbaum KS. On the convexity of ROC curves estimated from radiological test results. Acad Radiol. 2010;17(8):960–8.PubMedCentralPubMedCrossRefGoogle Scholar
  12. 12.
    Grey DR, Morgan BJT. Some aspects of ROC curve-fitting: normal and logistic models. J Math Psychol. 1972;9:128–39.CrossRefGoogle Scholar
  13. 13.
    Gu W, Pepe MS. Estimating the diagnostic likelihood ratio of a continuous marker. Biostatistics. 2011;12(1):87–101.PubMedCentralPubMedCrossRefGoogle Scholar
  14. 14.
    Hanley JS. The robustness of the binormal assumptions used in fitting ROC curves. Med Decis Mak. 1988;8:197–203.CrossRefGoogle Scholar
  15. 15.
    Metz CE, Herman BA, Shen J-H. Maximum-likelihood estimation of ROC curves from continuously-distributed data. Stat Med. 1998;17:1033–53.PubMedCrossRefGoogle Scholar
  16. 16.
    Metz CE, Pan X. Proper binormal ROC curves: theory and maximum-likelihood estimation. J Math Psychol. 1999;43:1–33.PubMedCrossRefGoogle Scholar
  17. 17.
    Burden RL, Faires JD. Numerical analysis. 7th ed. Conn: Brooks/Cole; 2000.Google Scholar
  18. 18.
    Pesce LL, Metz CE. Reliable and computationally efficient maximum-likelihood estimation of “proper” binormal ROC curves. Acad Radiol. 2007;14:814–29.PubMedCentralPubMedCrossRefGoogle Scholar
  19. 19.
    National Institute for Health and Clinical Excellence. Ovarian cancer: the recognition and initial management of ovarian cancer. London: NICE Full Guideline; 2011.Google Scholar
  20. 20.
    Kind P, Dolan P, Gudex C, Williams A. Variations in population health status: results from a United Kingdom national questionnaire survey. BMJ. 1998;316:736–41.PubMedCentralPubMedCrossRefGoogle Scholar
  21. 21.
    Mason H, Jones-Lee M, Donaldson C. Modelling the monetary value of a QALY: a new approach based on UK data. Health Econ. 2009;18(8):933–50.PubMedCrossRefGoogle Scholar
  22. 22.
    Towse A. Should NICE’s threshold range for cost per QALY be raised? Yes. BMJ. 2009;338:268–9.CrossRefGoogle Scholar
  23. 23.
    Claxton K, Martin P, Soares M, et al. Methods for the estimation of the NICE cost effectiveness threshold. Draft Final Report October 2012.
  24. 24.
    Novielli N, Cooper NJ, Sutton AJ. Evaluating the cost-effectiveness of diagnostic tests in combination: is it important to allow for performance dependency? Value Health. 2013;16(4):536–41.PubMedCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Roberta Longo
    • 1
  • Paul Baxter
    • 2
  • Peter Hall
    • 1
  • Jenny Hewison
    • 1
  • Mehran Afshar
    • 3
  • Geoff Hall
    • 3
  • Christopher McCabe
    • 4
  1. 1.Leeds Institute of Health SciencesUniversity of LeedsLeedsUK
  2. 2.Leeds Institute of Genetics, Health & TherapeuticsUniversity of LeedsLeedsUK
  3. 3.Leeds Teaching Hospitals NHS TrustLeedsUK
  4. 4.Capital Health Endowed Research Chair, Department of Emergency MedicineUniversity of AlbertaEdmontonCanada

Personalised recommendations