Clinical Orthopaedics and Related Research®

, Volume 471, Issue 11, pp 3496–3503 | Cite as

Challenges in Outcome Measurement: Clinical Research Perspective

  • Daniel P. O’ConnorEmail author
  • Mark R. Brinker
Symposium: ABJS Carl T. Brighton Workshop on Outcome Measures



Comparative effectiveness research evaluates treatments as actually delivered in routine clinical practice, shifting research focus from efficacy and internal validity to effectiveness and external validity (“generalizability”). Such research requires accurate assessments of the numbers of patients treated and the completeness of their followup, their clinical outcomes, and the setting in which their care was delivered. Choosing measures and methods for clinical outcome research to produce meaningful information that may be used to improve patient care presents a number of challenges.

Where Are We Now?

Orthopaedic surgery research has many stakeholders, including patients, providers, payers, and policy makers. A major challenge in orthopaedic surgery outcome measurement and clinical research is providing all of these users with valid information for their respective decision making. At present, no plan exists for capturing data on such a broad scale and scope.

Where Do We Need to Go?

Practical challenges include identifying and obtaining resources for widespread data collection and merging multiple data sources. Challenges of study design include sampling to obtain representative data, timing of data collection in the episode of care, and minimizing missing data and study dropout.

How Do We Get There?

Resource limitations may be addressed by repurposing existing clinical resources and capitalizing on technologic advances to increase efficiencies. Increasing use of rigorous, well-designed observational research designs can provide information that may be unattainable in clinical trials. Such study designs should incorporate methods to minimize missing data, to sample multiple providers, facilities, and patients, and to include evaluation of potential confounding variables to minimize bias and allow generalization to broad populations.


Routine Clinical Practice Comparative Effectiveness Research Practical Barrier Clinical Outcome Research Data Collection Interval 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Ahn H, Court-Brown CM, McQueen MM, Schemitsch EH. The use of hospital registries in orthopaedic surgery. J Bone Joint Surg Am. 2009;91(suppl 3):68–72.PubMedCrossRefGoogle Scholar
  2. 2.
    Ayers DC, Zheng H, Franklin PD. Integrating Patient-reported Outcomes Into Orthopaedic Clinical Practice: Proof of Concept From FORCE-TJR. Clin Orthop Relat Res. 2013. DOI  10.1007/s11999-013-3143-z.
  3. 3.
    Barrack RL. The results of TKA: what the registries don’t tell us. Orthopedics. 2011;34:e485–e487.PubMedGoogle Scholar
  4. 4.
    Bryant DM, Willits K, Hanson BP. Principles of designing a cohort study in orthopaedics. J Bone Joint Surg Am. 2009;91(suppl 3):10–14.PubMedCrossRefGoogle Scholar
  5. 5.
    Cadarette SM, Dickson L, Gignac MA, Beaton DE, Jaglal SB, Hawker GA. Predictors of locating women six to eight years after contact: internet resources at recruitment may help to improve response rates in longitudinal research. BMC Med Res Methodol. 2007;7:22.PubMedCrossRefGoogle Scholar
  6. 6.
    Cook JA, Bruckner T, MacLennan GS, Seiler CM. Clustering in surgical trials—database of intracluster correlations. Trials. 2012;13:2.PubMedCrossRefGoogle Scholar
  7. 7.
    Cook KF, Roddey TS, O’Malley KJ, Gartsman GM. Development of a Flexilevel Scale for use with computer-adaptive testing for assessing shoulder function. J Shoulder Elbow Surg. 2005;14:90S–94S.PubMedCrossRefGoogle Scholar
  8. 8.
    Franklin PD, Allison JJ, Ayers DC. Beyond joint implant registries: a patient-centered research consortium for comparative effectiveness in total joint replacement. JAMA. 2012;308:1217–1218.PubMedCrossRefGoogle Scholar
  9. 9.
    Franklin PD, Harrold L, Ayers DC. Incorporating Patient-reported Outcomes in Total Joint Arthroplasty Registries: Challenges and Opportunities. Clin Orthop Relat Res. 2013. DOI  10.1007/s11999-013-3193-2.
  10. 10.
    Goldstein J. Private practice outcomes: validated outcomes data collection in private practice. Clin Orthop Relat Res. 2010;468:2640–2645.PubMedCrossRefGoogle Scholar
  11. 11.
    Hoppe DJ, Schemitsch EH, Morshed S, Tornetta P 3rd, Bhandari M. Hierarchy of evidence: where observational studies fit in and why we need them. J Bone Joint Surg Am. 2009;91(suppl 3):2–9.PubMedCrossRefGoogle Scholar
  12. 12.
    Hung M, Clegg DO, Greene T, Weir C, Saltzman CL. A lower extremity physical function computerized adaptive testing instrument for orthopaedic patients. Foot Ankle Int. 2012;33:326–335.PubMedCrossRefGoogle Scholar
  13. 13.
    Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. Washington, DC: The National Academies Press; 2009.Google Scholar
  14. 14.
    King PJ, Malin AS, Scott RD, Thornhill TS. The fate of patients not returning for follow-up five years after total knee arthroplasty. J Bone Joint Surg Am. 2004;86:897–901.PubMedGoogle Scholar
  15. 15.
    Kozlowski AJ, Pretz CR, Dams-O’Connor K, Kreider S, Whiteneck G. Applying individual growth curve models to evaluate change in rehabilitation: a National Institute on Disability and Rehabilitation Research Traumatic Brain Injury Model Systems Report. Arch Phys Med Rehabil. 2012;94:589–596.PubMedCrossRefGoogle Scholar
  16. 16.
    Little RJ, Cohen ML, Dickersin K, Emerson SS, Farrar JT, Neaton JD, Shih W, Siegel JP, Stern H. The design and conduct of clinical trials to limit missing data. Stat Med. 2012;31:3433–3443.PubMedCrossRefGoogle Scholar
  17. 17.
    Little RJ, D’Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, Frangakis C, Hogan JW, Molenberghs G, Murphy SA, Neaton JD, Rotnitzky A, Scharfstein D, Shih WJ, Siegel JP, Stern H. The prevention and treatment of missing data in clinical trials. N Engl J Med. 2012;367:1355–1360.PubMedCrossRefGoogle Scholar
  18. 18.
    Louie DL, Earp BE, Blazar PE. Finding orthopedic patients lost to follow-up for long-term outcomes research using the Internet: an update for 2012. Orthopedics. 2012;35:595–599.PubMedCrossRefGoogle Scholar
  19. 19.
    Lubowitz JH, Smith PA. Current concepts in clinical research: web-based, automated, arthroscopic surgery prospective database registry. Arthroscopy. 2012;28:425–428.PubMedCrossRefGoogle Scholar
  20. 20.
    Marjoua Y, Butler CA, Bozic KJ. Public reporting of cost and quality information in orthopaedics. Clin Orthop Relat Res. 2012;470:1017–1026.PubMedCrossRefGoogle Scholar
  21. 21.
    Martin BI, Mirza SK, Franklin GM, Lurie JD, MacKenzie TA, Deyo RA. Hospital and surgeon variation in complications and repeat surgery following incident lumbar fusion for common degenerative diagnoses. Health Serv Res. 2013;48:1–25.PubMedCrossRefGoogle Scholar
  22. 22.
    Morshed S, Tornetta P 3rd, Bhandari M. Analysis of observational studies: a guide to understanding statistical methods. J Bone Joint Surg Am. 2009;91(suppl 3):50–60.PubMedCrossRefGoogle Scholar
  23. 23.
    National Research Council. The Prevention and Treatment of Missing Data in Clinical Trials. Panel on Handling Missing Data in Clinical Trials. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press; 2010.Google Scholar
  24. 24.
    Randsborg PH, Sivertsen EA, Skramm I, Saltyt Benth Jr, Gulbrandsen P. The need for better analysis of observational studies in orthopedics: a retrospective study of elbow fractures in children. Acta Orthop. 2010;81:377–381.PubMedCrossRefGoogle Scholar
  25. 25.
    Roberts C, Roberts SA. Design and analysis of clinical trials with clustering effects due to treatment. Clin Trials. 2005;2:152–162.PubMedCrossRefGoogle Scholar
  26. 26.
    Scharfstein DO, Hogan J, Herman A. On the prevention and analysis of missing data in randomized clinical trials: the state of the art. J Bone Joint Surg Am. 2012;94(suppl 1):80–84.PubMedGoogle Scholar
  27. 27.
    Shah J, Rajgor D, Pradhan S, McCready M, Zaveri A, Pietrobon R. Electronic data capture for registries and clinical trials in orthopaedic surgery: open source versus commercial systems. Clin Orthop Relat Res. 2010;468:2664–2671.PubMedCrossRefGoogle Scholar
  28. 28.
    Siebens H, Andres PL, Pengsheng N, Coster WJ, Haley SM. Measuring physical function in patients with complex medical and postsurgical conditions: a computer adaptive approach. Am J Phys Med Rehabil. 2005;84:741–748.PubMedCrossRefGoogle Scholar
  29. 29.
    Smith JS, Watts HG. Methods for locating missing patients for the purpose of long-term clinical studies. J Bone Joint Surg Am. 1998;80:431–438.PubMedGoogle Scholar
  30. 30.
    Sox HC, Goodman SN. The methods of comparative effectiveness research. Annu Rev Public Health. 2012;33:425–445.PubMedCrossRefGoogle Scholar
  31. 31.
    Stewart JI, Moyle S, Criner GJ, Wilson C, Tanner R, Bowler RP, Crapo JD, Zeldin RK, Make BJ, Regan EA, For The COPDGene Investigators. Automated telecommunication to obtain longitudinal follow-up in a multicenter cross-sectional COPD study. COPD. 2012;9:466–472.PubMedCrossRefGoogle Scholar
  32. 32.
    Swiontkowski MF, Buckwalter JA, Keller RB, Haralson R. The outcomes movement in orthopaedic surgery: where we are and where we should go. J Bone Joint Surg Am. 1999;81:732–740.PubMedCrossRefGoogle Scholar
  33. 33.
    Vandenbroucke JP, von Elm E, Altman DG, Gotzsche PC, Mulrow CD, Pocock SJ, Poole C, Schlesselman JJ, Egger M; STROBE Initiative. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Ann Intern Med. 2007;147:W163–W194.PubMedCrossRefGoogle Scholar
  34. 34.
    VanLare JM, Conway PH, Sox HC. Five next steps for a new national program for comparative-effectiveness research. N Engl J Med. 2010;362:970–973.PubMedCrossRefGoogle Scholar
  35. 35.
    von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;147:573–577.CrossRefGoogle Scholar
  36. 36.
    Vorhies JS, Wang Y, Herndon JH, Maloney WJ, Huddleston JI. Decreased length of stay after TKA is not associated with increased readmission rates in a national Medicare sample. Clin Orthop Relat Res. 2012;470:166–171.PubMedCrossRefGoogle Scholar
  37. 37.
    Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. 2013;20:144–151.PubMedCrossRefGoogle Scholar
  38. 38.
    Weng C, Appelbaum P, Hripcsak G, Kronish I, Busacca L, Davidson KW, Bigger JT. Using EHRs to integrate research with patient care: promises and challenges. J Am Med Inform Assoc. 2012;19:684–687.PubMedCrossRefGoogle Scholar

Copyright information

© The Association of Bone and Joint Surgeons® 2013

Authors and Affiliations

  1. 1.Department of Health and Human PerformanceUniversity of HoustonHoustonUSA
  2. 2.Fondren Orthopedic GroupTexas Orthopedic HospitalHoustonUSA

Personalised recommendations