Methodological challenges in the evaluation of prognostic factors in breast cancer

  • Douglas G. Altman
  • Gary H. Lyman


Many studies are carried out in an effort to find factors that help explain the large unexplained variation in prognosis of breast cancer patients. The principles of good study design and analysis are less well appreciated for prognostic factor studies than for therapeutic trials. The oncology literature is full of results from studies of varying quality, often with conflicting findings. As a consequence, despite the large number of studies, there is still uncertainty about the importance of most prognostic factors. Few recently proposed prognostic factors for breast cancer have become widely accepted. This paper reviews the important methodological issues underlying such research. These issues are illustrated with examples from published studies and recent reviews of papers published in cancer journals. Guidelines are proposed for conducting and evaluating prognostic factor studies which should improve the quality of research in this important area.

Key words

prognostic studies study design sample size multiple regression analysis cutpoints continuous variables interaction analysis tests of predictiveness model validation meta-analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    McGuire WL: Breast cancer prognostic factors: evaluation guidelines. J Natl Cancer Inst 83:154–155, 1991PubMedCrossRefGoogle Scholar
  2. 2.
    Gasparini G, Pozza F, Harris AL: Evaluating the potential usefulness of new prognostic and predictive indicators in node negative breast cancer patients. J Natl Cancer Inst 85:1206–1219, 1993PubMedCrossRefGoogle Scholar
  3. 3.
    Clark GM, Wenger CR, Beardslee S, et al: How to integrate steroid hormone receptor, flow cytometric, and other prognostic information in regard to primary breast cancer. Cancer 71:2157–2162, 1993PubMedCrossRefGoogle Scholar
  4. 4.
    Simon R, Altman DG: Statistical aspects of prognostic factor studies in oncology. Br J Cancer 6:979–985, 1994CrossRefGoogle Scholar
  5. 5.
    Gray-Donald K, Kramer MS: Causality inference in observational vs. experimental studies. An empirical comparison. Am J Epidemiol 127:885–892, 1988PubMedGoogle Scholar
  6. 6.
    Tukey JW: Some thoughts on clinical trials, especially problems of multiplicity. Science 198:679–684, 1977PubMedCrossRefGoogle Scholar
  7. 7.
    Fayers PM, Machin D: Sample size: how many patients are necessary? Br J Cancer 72:1–9, 1995PubMedCrossRefGoogle Scholar
  8. 8.
    Machin D, Campbell MJ, Fayers PM, Pinol APY: Sample Size Tables for Clinical Studies, 2nd edition. Blackwell, Oxford, 1997Google Scholar
  9. 9.
    Harrell FE, Lee KL, Matchar DB, Reichert TA: Regression models for prognostic prediction: advantages, problems, and suggested solutions. Cancer Treat Rep 69:1071–1077, 1985PubMedGoogle Scholar
  10. 10.
    Peduzzi P, Concato J, Feinstein AR, Holford TR: The importance of events per independent variable (EPV) in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. J Clin Epidemiol 48:1503–1510, 1995PubMedCrossRefGoogle Scholar
  11. 11.
    Altman DG, De Stavola BL, Love SB, Stepniewska KA: Review of survival analyses published in cancer journals. Br J Cancer 72:511–518, 1995PubMedCrossRefGoogle Scholar
  12. 12.
    Axelsson K, Ljung B-ME, Moore DH, et al: Tumor angiogenesis as a prognostic assay for invasive ductal breast carcinoma. J Natl Cancer Inst 87:997–1008, 1995PubMedCrossRefGoogle Scholar
  13. 13.
    Press MF, Hung G, Godolphin W, Slamon DJ: Sensitivity of HER-2/neu antibodies in archival tissue samples: potential source of error in immunohistochemical studies of oncogene expression. Cancer Res 54:2771–2777, 1994PubMedGoogle Scholar
  14. 14.
    Romero H, Schneider J: Different detection rates of HER-2/NEU overexpression in ovarian carcinoma using two different commercially available detection kits. Eur J Cancer 31 A: 1020–1021, 1995CrossRefGoogle Scholar
  15. 15.
    Molino A, Micciolo R, Turazza M, et al: Prognostic significance of estrogen receptors in 405 primary breast cancers: a comparison of immunohistochemical and biochemical methods. Breast Cancer Res Treat 345:241–249, 1997CrossRefGoogle Scholar
  16. 16.
    Bland JM, Altman DG: Statistical methods for comparing two methods of measurement. Lancet i:307–310, 1986CrossRefGoogle Scholar
  17. 17.
    Cox DR: Regression models and life-tables. J R Stat Soc B 34:187–220, 1972Google Scholar
  18. 18.
    Fielding LP, Fenoglio-Preiser CM, Freedman LS: The future of prognostic factors in outcome prediction for patients with cancer. Cancer 70:2367–2377, 1992PubMedCrossRefGoogle Scholar
  19. 19.
    Chen C-H, George SL: The bootstrap and identification of prognostic factors via Cox’s proportional hazards regression model. Stat Med 4:39–46, 1985PubMedCrossRefGoogle Scholar
  20. 20.
    Altman DG, Andersen PK: Bootstrap investigation of the stability of a Cox regression model. Stat Med 8:771–783, 1989PubMedCrossRefGoogle Scholar
  21. 21.
    Sauerbrei W, Schumacher M: A bootstrap resampling procedure for model building: application to the Cox regression model. Stat Med 11:2093–2109, 1992PubMedCrossRefGoogle Scholar
  22. 22.
    Schumacher M, Höllander, N, Sauerbrei W: Resampling and cross-validation techniques: a tool to reduce bias caused by model building? Stat Med 16:2813–2827, 1997PubMedCrossRefGoogle Scholar
  23. 23.
    Gamel JW, McCurdy JB, McLean IW: A comparison of prognostic covariates for uveal melanoma. Invest Ophthalmol Vis Sci 33:1919–1922, 1992PubMedGoogle Scholar
  24. 24.
    Jenks S, Volkers N: Razors and refrigerators and reindeer — oh my. J Natl Cancer Inst 84:1863, 1992PubMedCrossRefGoogle Scholar
  25. 25.
    Peters JM, Preston-Martin S, London SJ, et al: Processed meats and risk of childhood leukemia (California, USA). Cancer Causes Control 5:195–202, 1994PubMedCrossRefGoogle Scholar
  26. 26.
    Wyatt JC, Altman DG: Prognostic models: clinically useful or quickly forgotten? Br Med J 311:1539–1541, 1995CrossRefGoogle Scholar
  27. 27.
    Altman DG, Royston P: What do we mean by validating a prognostic model? Stat Med, in pressGoogle Scholar
  28. 28.
    Vach W: Some issues in estimating the effect of prognostic factors from incomplete covariate information. Stat Med 16:57–72, 1997PubMedCrossRefGoogle Scholar
  29. 29.
    Sagman U, Maki E, Evans WK, et al: Small-cell carcinoma of the lung: derivation of a prognostic staging system. J Clin Oncol 9:1639–1649, 1991PubMedGoogle Scholar
  30. 30.
    Thor A, Benz C, Moore D, et al: Stress response protein (srp-27) determination in primary human breast carcinomas: clinical, histologic, and prognostic correlations. J Natl Cancer Inst 83:170–178, 1991PubMedCrossRefGoogle Scholar
  31. 31.
    Hart A, Wyatt J: Evaluating black boxes as medical decision-aids: issues arising from a study of neural networks. Med Informatics 15:229–236, 1990CrossRefGoogle Scholar
  32. 32.
    Ohno-Machado L: A comparison of Cox proportional hazards and artificial network models for medical prognosis. Comput Biol Med 27:55–65, 1997PubMedCrossRefGoogle Scholar
  33. 33.
    Schwarzer G, Vach W, Schumacher M: On the misuses of artificial neural networks for prognostic factor and diagnostic classification in oncology. University of Freiburg Technical Report No. 46, 1997Google Scholar
  34. 34.
    Morgan TM, Elashoff RM: Effect of categorizing a continuous covariate on the comparison of survival time. J Am Stat Assoc 81:917–921, 1986CrossRefGoogle Scholar
  35. 35.
    Hilsenbeck SG, Clark GM, McGuire WL: Why do so many prognostic factors fail to pan out? Breast Cancer Res Treat 22:197–206, 1992PubMedCrossRefGoogle Scholar
  36. 36.
    Altman DG, Lausen B, Sauerbrei W, Schumacher M: Dangers of using ‘optimal’ cutpoints in the evaluation of prognostic factors. J Natl Cancer Inst 86:829–835, 1994PubMedCrossRefGoogle Scholar
  37. 37.
    Altman DG: Suboptimal analysis using ‘optimal’ cut-points. Br J Cancer 78:556–557, 1998PubMedCrossRefGoogle Scholar
  38. 38.
    Jänicke F, Schmitt M, Ulm K, et al: Urokinase-type plasminogen activator antigen and early relapse in breast cancer. Lancet 2:1049, 1989PubMedCrossRefGoogle Scholar
  39. 39.
    Jänicke F, Schmitt M, Pache L, et al: Urokinase (uPA) and its inhibitor PAI-1 are strong and independent prognostic factors in node negative breast cancer. Breast Cancer Res Treat 24:195–208, 1993PubMedCrossRefGoogle Scholar
  40. 40.
    Knoop A, Andreasen PA, Andersen JA, et al: Prognostic significance of urokinase-type plasminogen activator and plasminogen activator inhibitor-1 in primary breast cancer. Br J Cancer 77:932–940, 1998PubMedCrossRefGoogle Scholar
  41. 41.
    Buettner P, Garbe C, Guggenmoos-Holzmann I: Problems in defining cutoff points of continuous prognostic factors: example of tumor thickness in primary cutaneous melanoma. J Clin Epidemiol 50:1201–1210, 1997PubMedCrossRefGoogle Scholar
  42. 42.
    Budinha M, Skrk J, Zakotnik B, et al: Prognostic value of total cathepsin B in invasive ductal carcinoma of the breast. Eur J Cancer 31A:661–664, 1995CrossRefGoogle Scholar
  43. 43.
    Durrleman S, Simon R: Flexible regression models with cubic splines. Stat Med 8:551–561, 1989PubMedCrossRefGoogle Scholar
  44. 44.
    Hastie T, Sleeper L, Tibshirani R: Flexible covariate effects in the proportional hazards model. Breast Cancer Res Treat 22:241–250, 1992PubMedCrossRefGoogle Scholar
  45. 45.
    Royston P, Altman DG: Regression using fractional polynomials of continuous covariates: parsimonious parametric modelling. Appl Stat 43:429–467, 1994CrossRefGoogle Scholar
  46. 46.
    Knorr KL, Hilsenbeck SG, Wenger CR, et al: Making the most of your prognostic factors: presenting a more accurate survival model for breast cancer patients. Breast Cancer Res Treat 22:251–262, 1992PubMedCrossRefGoogle Scholar
  47. 47.
    Buyse M: Analysis of clinical trial outcomes: some comments on subgroup analyses. Controlled Clin Trials 10:187S–194S, 1989PubMedCrossRefGoogle Scholar
  48. 48.
    Henry JA, McCarthy AL, Angus B, et al: Prognostic significance of the estrogen-regulated protein, cathepsin D, in breast cancer. An immunohistochemical study. Cancer 65:265–271, 1990PubMedCrossRefGoogle Scholar
  49. 49.
    Simon R: Confidence limits for reporting results of clinical trials. Ann Intern Med 105:429–435, 1986PubMedGoogle Scholar
  50. 50.
    Gardner MJ, Altman DG (eds) Statistics with Confidence. British Medical Journal, London, 1989Google Scholar
  51. 51.
    Simon R: Patient subsets and variation in therapeutic efficacy. Br J Clin Pharmacol 14:473–482, 1982PubMedCrossRefGoogle Scholar
  52. 52.
    Aubele N, Auer G, Falkmer U, et al: Improved prognostication in small (pT1) breast cancers by image cytometry. Breast Cancer Res Treat 36:83–91, 1995PubMedCrossRefGoogle Scholar
  53. 53.
    Harrell FE, Lee KL, Mark DB: Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 15:361–387, 1996PubMedCrossRefGoogle Scholar
  54. 54.
    Haybittle JL, Blarney RW, Elston CW, et al: A prognostic index in primary breast cancer. Br J Cancer 45:361–366, 1982PubMedCrossRefGoogle Scholar
  55. 55.
    Todd JH, Dowle C, Williams MR, et al: Confirmation of a prognostic index in primary breast cancer. Br J Cancer 56:489–492, 1987PubMedCrossRefGoogle Scholar
  56. 56.
    Brown JM, Benson EA, Jones M: Confirmation of a long-term prognostic index in breast cancer. Breast 2:144–147, 1993CrossRefGoogle Scholar
  57. 57.
    Stern JM, Simes RJ: Publication bias: evidence of delayed publication in a cohort study of clinical research projects. Br Med J 315:640–645, 1997CrossRefGoogle Scholar
  58. 58.
    Ferrandina G, Scambia G, Bardelli F, et al: Relationship between cathepsin-D content and disease-free survival in node-negative breast cancer patients: a metaanalysis. Br J Cancer 76:661–666, 1997PubMedCrossRefGoogle Scholar
  59. 59.
    Fox SB, Smith K, Hollyer J, et al: The epidermal growth factor receptor as a prognostic marker: results of 370 patients and review of 3009 patients. Breast Cancer Res Treat 29:41–49, 1994PubMedCrossRefGoogle Scholar
  60. 60.
    Rawson NSB, Peto J: An overview of prognostic factors in small cell lung cancer. Br J Cancer 61:597–604, 1990PubMedCrossRefGoogle Scholar
  61. 61.
    The International Non-Hodgkin’s Lymphoma Prognostic Factors Project: A predictive model for aggressive lymphoma. N Engl J Med 329:987–994, 1993CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 1998

Authors and Affiliations

  • Douglas G. Altman
    • 1
    • 2
  • Gary H. Lyman
    • 3
    • 4
  1. 1.Imperial Cancer Research Fund Medical Statistics GroupCentre for Statistics in Medicine, Institute of Health SciencesUK
  2. 2.ICRF Medical Statistics GroupCentre for Statistics in Medicine, Institute of Health SciencesHeadingtonUK
  3. 3.Medical Statistics Unit, Department of Epidemiology and Population HealthLondon School of Hygiene and Tropical MedicineUK
  4. 4.H Lee Moffitt Cancer Center and Research Institute at the University of South FloridaTampaUSA

Personalised recommendations