Skip to main content
SpringerLink
Log in

Surgical Outcomes Research Based on Administrative Data: Inferior or Complementary to Prospective Randomized Clinical Trials?

  • SURGICAL OUTCOMES
  • Published:
World Journal of Surgery Aims and scope Submit manuscript

Abstract

The importance of surgical research has gained new prominence over the past decades as the relevance of well designed and well conducted studies has become increasingly evident. There are two basic but diametrically different methods of conducting research: the prospective randomized clinical trial and the retrospective surgical outcomes study based on administrative data. Administrative databases contain data that were initially collected for purposes other than scientific research. Whereas the prospective randomized clinical trial is familiar to most surgeons, surgical outcomes research based on administrative data constitutes a genre of investigation that is often unfamiliar to and even disparaged by the surgical community. In the present article, the strengths and weaknesses of both prospective randomized clinical trials and retrospective surgical outcomes research are discussed. Specifically, the advantages and limitations of investigations based on large administrative databases are outlined. Because both study designs play an important role in surgical research, carefully designed and implemented surgical outcomes research based on administrative data should be viewed as being complementary and not inferior to prospective randomized clinical trials.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Porter GA, Skibber JM. Outcomes research in surgical oncology. Ann Surg Oncol 2000;7:367–75

    CAS  PubMed  Google Scholar 

  2. Brenneman FD, Wright JG, Kennedy ED, et al. Outcomes research in surgery. World J Surg 1999;23:1220–1223

    CAS  PubMed  Google Scholar 

  3. Epstein AM. The outcomes movement—will it get us where we want to go? N Engl J Med 1990;323:266–270

    CAS  PubMed  Google Scholar 

  4. Iezzoni LI. Risk Adjustment for Measuring Health Care Outcomes. Chicago, Health Administrative Press, Foundation of the American College of Excecutives, 1997

    Google Scholar 

  5. Best AE. Secondary data bases and their use in outcomes research: a review of the area resource file and the Healthcare Cost and Utilization Project. J Med Syst 1999;23:175–181

    Article  CAS  PubMed  Google Scholar 

  6. Iezzoni LI. Using risk-adjusted outcomes to assess clinical practice: an overview of issues pertaining to risk adjustment. Ann Thorac Surg 1994; 58:1822–1826

    CAS  PubMed  Google Scholar 

  7. Lewis NJ, Patwell JT, Briesacher BA. The role of insurance claims databases in drug therapy outcomes research. Pharmacoeconomics 1993;4:323–330

    CAS  PubMed  Google Scholar 

  8. Wennberg JE, Roos N, Sola L, et al. Use of claims data systems to evaluate health care outcomes: mortality and reoperation following prostatectomy. JAMA 1987;257:933–936

    Article  CAS  PubMed  Google Scholar 

  9. Coleman AL, Morgenstern H. Use of insurance claims databases to evaluate the outcomes of ophthalmic surgery. Surv Ophthalmol 1997;42:271–278

    CAS  PubMed  Google Scholar 

  10. Armstrong EP, Manuchehri F. Ambulatory care databases for managed care organizations. Am J Health Syst Pharm 1997;54:1973–2005

    CAS  PubMed  Google Scholar 

  11. Wennberg J, Gittelsohn. Small area variations in health care delivery. Science 1973;182:1102–1108

    CAS  PubMed  Google Scholar 

  12. Farrow DC, Hunt WC, Samet JM. Geographic variation in the treatment of localized breast cancer. N Engl J Med 1992;326:1097–1101

    CAS  PubMed  Google Scholar 

  13. Sainsbury R, Rider L, Smith A, et al. Does it matter where you live? Treatment variation for breast cancer in Yorkshire; The Yorkshire Breast Cancer Group. Br J Cancer 1995;71:1275–1278

    CAS  PubMed  Google Scholar 

  14. Nilasena DS, Vaughn RJ, Mori M, et al. Surgical trends in the treatment of diseases of the lumbar spine in Utah’s Medicare population, 1984 to 1990. Med Care 1995;33:585–597

    CAS  PubMed  Google Scholar 

  15. Cooper GS, Yuan Z, Stange KC, et al. Use of Medicare claims data to measure county-level variations in the incidence of colorectal carcinoma. Cancer 1998;83:673–678

    CAS  PubMed  Google Scholar 

  16. Skinner J, Weinstein JN, Sporer SM, et al. Racial, ethnic, and geographic disparities in rates of knee arthroplasty among Medicare patients. N Engl J Med 2003;349:1350–1359

    Article  CAS  PubMed  Google Scholar 

  17. McPherson K, Wennberg JE, Hovind OB, et al. Small-area variations in the use of common surgical procedures: an international comparison of New England, England, and Norway. N Engl J Med 1982;307:1310–1314

    CAS  PubMed  Google Scholar 

  18. Nattinger AB, Gottlieb MS, Veum J, et al. Geographic variation in the use of breast-conserving treatment for breast cancer. N Engl J Med 1992;326:1102–1107

    CAS  PubMed  Google Scholar 

  19. Wright JG, Hawker GA, Bombardier C, et al. Physician enthusiasm as an explanation for area variation in the utilization of knee replacement surgery. Med Care 1999;37:946–956

    CAS  PubMed  Google Scholar 

  20. Gilligan MA, Kneusel RT, Hoffmann RG, et al. Persistent differences in sociodemographic determinants of breast conserving treatment despite overall increased adoption. Med Care 2002;40:181–189

    PubMed  Google Scholar 

  21. Jerome-D’Emilia B, Begun JW. Diffusion of breast conserving surgery in medical communities. Soc Sci Med 2005;60:143–151

    PubMed  Google Scholar 

  22. Birkmeyer JD, Siewers AE, Finlayson EV, et al. Hospital volume and surgical mortality in the United States. N Engl J Med 2002;346:1128–1137

    Article  PubMed  Google Scholar 

  23. Begg CB, Cramer LD, Hoskins WJ, et al. Impact of hospital volume on operative mortality for major cancer surgery. JAMA 1998;280:1747–1751

    Article  CAS  PubMed  Google Scholar 

  24. Glasgow RE, Mulvihill SJ. Hospital volume influences outcome in patients undergoing pancreatic resection for cancer. West J Med 1996;165:294–300

    CAS  PubMed  Google Scholar 

  25. Lieberman MD, Kilburn H, Lindsey M, et al. Relation of perioperative deaths to hospital volume among patients undergoing pancreatic resection for malignancy. Ann Surg 1995;222:638–645

    CAS  PubMed  Google Scholar 

  26. Romano PS, Mark DH. Patient and hospital characteristics related to in-hospital mortality after lung cancer resection. Chest 1992;101:1332–1337

    CAS  PubMed  Google Scholar 

  27. Harmon JW, Tang DG, Gordon TA, et al. Hospital volume can serve as a surrogate for surgeon volume for achieving excellent outcomes in colorectal resection. Ann Surg 1999;230:404–413

    Article  CAS  PubMed  Google Scholar 

  28. Yao SL, Lu-Yao G. Population-based study of relationships between hospital volume of prostatectomies, patient outcomes, and length of hospital stay. J Natl Cancer Inst 1999;91:1950–1956

    Article  CAS  PubMed  Google Scholar 

  29. Gordon TA, Burleyson GP, Tielsch JM, et al. The effects of regionalization on cost and outcome for one general high-risk surgical procedure. Ann Surg 1995;221:43–49

    CAS  PubMed  Google Scholar 

  30. Purves H, Pietrobon R, Hervey S, et al. Relationship between surgeon caseload and sphincter preservation in patients with rectal cancer. Dis Colon Rectum 2005;48:195–204

    Article  PubMed  Google Scholar 

  31. Schulman KA, Berlin JA, Harless W, et al. The effect of race and sex on physicians’ recommendations for cardiac catheterization. N Engl J Med 1999;340:618–626

    Article  CAS  PubMed  Google Scholar 

  32. Greenwald HP, Polissar NL, Borgatta EF, et al. Social factors, treatment, and survival in early-stage non-small cell lung cancer. Am J Public Health 1998;88:1681–1684

    CAS  PubMed  Google Scholar 

  33. Richards RJ, Reker DM. Racial differences in use of colonoscopy, sigmoidoscopy, and barium enema in Medicare beneficiaries. Dig Dis Sci 2002;47:2715–2719

    PubMed  Google Scholar 

  34. Eley JW, Hill HA, Chen VW, et al. Racial differences in survival from breast cancer: results of the National Cancer Institute Black/White Cancer Survival Study. JAMA 1994;272:947–954

    Article  CAS  PubMed  Google Scholar 

  35. Dayal HH, Polissar L, Dahlberg S. Race, socioeconomic status, and other prognostic factors for survival from prostate cancer. J Natl Cancer Inst 1985;74:1001–1006

    CAS  PubMed  Google Scholar 

  36. Dayal H, Polissar L, Yang CY, et al. Race, socioeconomic status, and other prognostic factors for survival from colo-rectal cancer. J Chronic Dis 1987;40:857–864

    Article  CAS  PubMed  Google Scholar 

  37. Cooper GS, Yuan Z, Landefeld CS, et al. Surgery for colorectal cancer: race-related differences in rates and survival among Medicare beneficiaries. Am J Public Health 1996;86:582–586

    CAS  PubMed  Google Scholar 

  38. Bach PB, Cramer LD, Warren JL, et al. Racial differences in the treatment of early-stage lung cancer. N Engl J Med 1999;341:1198–1205

    Article  CAS  PubMed  Google Scholar 

  39. Guller U, Jain N, Curtis L, et al. Insurance status and race represent independent predictors of undergoing laparoscopic surgery for appendicitis: secondary data analysis of 145,546 patients. J Am Coll Surg 2004;199:567–577

    PubMed  Google Scholar 

  40. Flum DR, Morris A, Koepsell T, et al. Has misdiagnosis of appendicitis decreased over time? A population-based analysis. JAMA 2001;286:1748–1753

    Article  CAS  PubMed  Google Scholar 

  41. Guller U, Hervey S, Purves H, et al. Laparoscopic versus open appendectomy: outcomes comparison based on a large administrative database. Ann Surg 2004;239:43–52

    PubMed  Google Scholar 

  42. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med 2000;342:1878–1886

    Article  CAS  PubMed  Google Scholar 

  43. Lloyd SS, Rissing JP. Physician and coding errors in patient records. JAMA 1985;254:1330–1336

    Article  CAS  PubMed  Google Scholar 

  44. Steiner C, Elixhauser A, Schnaier J. The healthcare cost and utilization project: an overview. Eff Clin Pract 2002;5:143–151

    PubMed  Google Scholar 

  45. Losina E, Barrett J, Baron JA, et al. Accuracy of Medicare claims data for rheumatologic diagnoses in total hip replacement recipients. J Clin Epidemiol 2003;56:515–519

    Article  PubMed  Google Scholar 

  46. Romano PS, Roos LL, Luft HS, et al. A comparison of administrative versus clinical data: coronary artery bypass surgery as an example; Ischemic Heart Disease Patient Outcomes Research Team. J Clin Epidemiol 1994;47:249–260

    Article  CAS  PubMed  Google Scholar 

  47. Jollis JG, Ancukiewicz M, DeLong ER, et al. Discordance of databases designed for claims payment versus clinical information systems: implications for outcomes research. Ann Intern Med 1993;119:844–850

    CAS  PubMed  Google Scholar 

  48. Cooper GS, Yuan Z, Stange KC, et al. The sensitivity of Medicare claims data for case ascertainment of six common cancers. Med Care 1999;37:436–444

    CAS  PubMed  Google Scholar 

  49. Cooper GS, Yuan Z, Stange KC, et al. The utility of Medicare claims data for measuring cancer stage. Med Care 1999;37:706–711

    CAS  PubMed  Google Scholar 

  50. Cooper GS, Yuan Z, Stange KC, et al. Agreement of Medicare claims and tumor registry data for assessment of cancer-related treatment. Med Care 2000;38:411–421

    Article  CAS  PubMed  Google Scholar 

  51. Hsia DC, Krushat WM, Fagan AB, et al. Accuracy of diagnostic coding for Medicare patients under the prospective-payment system. N Engl J Med 1988;318:352–355

    CAS  PubMed  Google Scholar 

  52. Guller U, DeLong ER. Interpreting statistics in medical literature: a vade mecum for surgeons. J Am Coll Surg 2004;198:441–458

    Article  PubMed  Google Scholar 

  53. Bender R, Lange S. Adjusting for multiple testing—when and how? J Clin Epidemiol 2001;54:343–349

    Article  CAS  PubMed  Google Scholar 

  54. Katz MH. What are confounders and how does multivariate analysis help me to deal with them. In: Multivariable Analysis. Cambridge University Press, Cambridge, 1999

  55. Klungel OH, Martens EP, Psaty BM, et al. Methods to assess intended effects of drug treatment in observational studies are reviewed. J Clin Epidemiol 2004;57:1223–1231

    Article  PubMed  Google Scholar 

  56. Byar DP, Simon RM, Friedewald WT, et al. Randomized clinical trials: perspectives on some recent ideas. N Engl J Med 1976;295:74–80

    CAS  PubMed  Google Scholar 

  57. Stewart LA, Parmar MK. Bias in the analysis and reporting of randomized controlled trials. Int J Technol Assess Health Care 1996;12:264–275

    CAS  PubMed  Google Scholar 

  58. Bailey KR. Generalizing the results of randomized clinical trials. Control Clin Trials 1994;15:15–23

    CAS  PubMed  Google Scholar 

  59. Abel U, Koch A. The role of randomization in clinical studies: myths and beliefs. J Clin Epidemiol 1999;52:487–497

    CAS  PubMed  Google Scholar 

  60. Altman DG. Better reporting of randomised controlled trials: the CONSORT statement. BMJ 1996;313:570–571

    CAS  PubMed  Google Scholar 

  61. McLeod RS. Issues in surgical randomized controlled trials. World J Surg 1999;23:1210–1214

    CAS  PubMed  Google Scholar 

  62. Young J, Harrison J, White G, et al. Developing measures of surgeons’ equipoise to assess the feasibility of randomized controlled trials in vascular surgery. Surgery 2004;136:1070–1076

    Article  PubMed  Google Scholar 

  63. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000;342:1887–1892

    Article  CAS  PubMed  Google Scholar 

  64. Kennedy WA, Laurier C, Malo JL, et al. Does clinical trial subject selection restrict the ability to generalize use and cost of health services to “real life” subjects? Int J Technol Assess Health Care 2003;19:8–16

    PubMed  Google Scholar 

  65. Crawford ED. On the importance of clinical trials. J Urol 1990;143:787

    CAS  PubMed  Google Scholar 

  66. Cummings SR, Grady D, Hulley SB. Designing an Experiment: Clinical Trials I. Designing Clinical Research. Philadelphia, Lippincott Williams & Wilkins, 2001, pp 143–155

    Google Scholar 

  67. Helms PJ. ‘Real world’ pragmatic clinical trials: what are they and what do they tell us? Pediatr Allergy Immunol 2002;13:4–9

    Article  PubMed  Google Scholar 

  68. Roland M, Torgerson DJ. What are pragmatic trials? BMJ 1998; 316:285

    CAS  PubMed  Google Scholar 

  69. Clarke CE. A “cure” for Parkinson’s disease: can neuroprotection be proven with current trial designs? Mov Disord 2004;19:491–498

    Article  PubMed  Google Scholar 

  70. Fleming TR, DeMets DL. Surrogate end points in clinical trials: are we being misled? Ann Intern Med 1996;125:605–613

    CAS  PubMed  Google Scholar 

  71. Guller U, Blumenstein BA. Trends in clinical trials in surgical oncology: implications for outcomes research. Clin Ther 2003;25(2):684–698

    Article  PubMed  Google Scholar 

  72. Ellenberg S, Hamilton JM. Surrogate endpoints in clinical trials: cancer. Stat Med 1989;8:405–413

    CAS  PubMed  Google Scholar 

  73. D’Agostino RB Jr. Debate: the slippery slope of surrogate outcomes. Curr Control Trials Cardiovasc Med 2000;1:76–78

    PubMed  Google Scholar 

  74. Guller U, Oertli D. Sample size matters: a guide for surgeons. World J Surg 2005;29:601–605

    PubMed  Google Scholar 

  75. Berwick DM. Experimental power: the other side of the coin. Pediatrics 1980;65:1043–1045

    CAS  PubMed  Google Scholar 

  76. Moher D, Dulberg CS, Wells GA. Statistical power, sample size, and their reporting in randomized controlled trials. JAMA 1994;272:122–124

    Article  CAS  PubMed  Google Scholar 

  77. Vandekerckhove P, O’Donovan PA, Lilford RJ, et al. Infertility treatment: from cookery to science: the epidemiology of randomised controlled trials. Br J Obstet Gynaecol 1993;100:1005–1036

    CAS  PubMed  Google Scholar 

  78. Freiman JA, Chalmers TC, Smith H Jr, et al. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial: survey of 71 “negative” trials. N Engl J Med 1978;299:690–694

    CAS  PubMed  Google Scholar 

  79. Lilford R, Braunholtz D, Harris J, Gill T. Trials in surgery. Br J Surg 2004;91:6–16

    Article  CAS  PubMed  Google Scholar 

  80. Montori VM, Kleinbart J, Newman TB, et al. Tips for learners of evidence-based medicine. 2. Measures of precision (confidence intervals). Can Med Assoc J 2004;171:611–615

    Google Scholar 

  81. Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA 1983;249:1743–1745

    Article  CAS  PubMed  Google Scholar 

  82. Moseley JB, O’Malley K, Petersen NJ, et al. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med 2002;347:81–88

    Article  PubMed  Google Scholar 

  83. Albin RL. Sham surgery controls: intracerebral grafting of fetal tissue for Parkinson’s disease and proposed criteria for use of sham surgery controls. J Med Ethics 2002;28:322–325

    Article  CAS  PubMed  Google Scholar 

  84. Macklin R. The ethical problems with sham surgery in clinical research. N Engl J Med 1999;341:992–996

    Article  CAS  PubMed  Google Scholar 

  85. Devereaux PJ, Bhandari M, Clarke M, et al. Need for expertise based randomised controlled trials. BMJ 2005;330:88

    Article  CAS  PubMed  Google Scholar 

  86. Bais JE, Bartelsman JF, Bonjer HJ, et al. Laparoscopic or conventional Nissen fundoplication for gastro-oesophageal reflux disease: randomised clinical trial: The Netherlands Antireflux Surgery Study Group. Lancet 2000;355:170–174

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

I thank Mr. Jonathan McCall for carefully reading the manuscript and making many valuable suggestions.

Author information

Authors and Affiliations

  1. Department of Surgery, Divisions of General Surgery and Surgical Research, University Hospital Basel, Basel, CH-4031, Switzerland

    Ulrich Guller MD, MHS

Authors
  1. Ulrich Guller MD, MHS
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ulrich Guller MD, MHS.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Guller, U. Surgical Outcomes Research Based on Administrative Data: Inferior or Complementary to Prospective Randomized Clinical Trials?. World J. Surg. 30, 255–266 (2006). https://doi.org/10.1007/s00268-005-0156-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00268-005-0156-0

Keywords

Search

Navigation