Abstract
Purpose
A ratio of observed difference (OD) over the 95% confidence interval (CI) has been shown to be strongly associated with the perceived clinical relevance (CR) of medical research results. The purpose of this study was to evaluate the association between the OD/CI ratio and perceived CR in orthopaedic research.
Methods
Sixty-seven orthopaedic surgeons completed a survey with 15 study outcomes (mean difference and CI) and were asked if they perceived the findings as clinically relevant. The interobserver reliability of perceived CR and the association between CR and the OD/CI ratio and p-value were assessed.
Results
The interobserver reliability of CR between respondents was moderate (kappa = 0.46, CI 0.45 to 0.48). P-values did not differ between results with and without CR (median difference (MD) − 0.12, CI − 0.74 to 0.0009, p = 0.07). The OD/CI ratio, however, was greater for results with CR (MD 1.01, CI 0.3 to 3.9, p = 0.004). The area under the curve (AUC) for the p-value and OD/CI ratio receiver operating characteristic (ROC) curves was 0.80 (p = 0.01) and 0.97 (p = 0.0003). The cutoff p -value and OD/CI ratio that maximized the sensitivity (SN) and specificity (SP) for CR were 0.001 (SN 80%, SP 80%) and 0.84 (SN 100%, SP 90%). The SN and SP of a p-value cutoff of 0.05 was 100% and 50%.
Conclusion
The interobserver reliability of the perceived CR of orthopaedic research findings was moderate. The OD/CI ratio, in contrast to the p-value, was strongly associated with perceived CR making it a potentially useful measure to evaluate research results.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Fayaz HC, Haas N, Kellam J et al (2013) Improvement of research quality in the fields of orthopaedics and trauma: a global perspective. Int Orthop 37:1205–1212. https://doi.org/10.1007/s00264-013-1897-2
Houle TT (2007) Statistical reporting for current and future readers. Anesthesiology 107:193–194
Spreckelsen TF (2018) Editorial: changes in the field: banning p values (or not), transparency, and the opportunities of a renewed discussion on rigorous (quantitative) research. Child Adolesc Ment Health 23:61–62
Kelley K, Preacher KJ (2012) On effect size. Psychol Methods 17:137–152. https://doi.org/10.1037/a0028086
Strage K, Stacey S, Mauffrey C, Parry JA (2022) The interobserver reliability of clinical relevance in medical research. Injury. https://doi.org/10.1016/j.injury.2021.12.044
Tirefort J, Schwitzguebel AJ, Collin P et al (2019) Postoperative mobilization after superior rotator cuff repair: sling versus no sling: a randomized prospective study. J Bone Jt Surg Am 101:494–503. https://doi.org/10.2106/JBJS.18.00773
Kelly MP, Lurie JD, Yanik EL et al (2019) Operative versus nonoperative treatment for adult symptomatic lumbar scoliosis. J Bone Jt Surg Am 101:338–352. https://doi.org/10.2106/JBJS.18.00483
McKearney DA, Stender CJ, Cook BK et al (2019) Altered range of motion and plantar pressure in anterior and posterior malaligned total ankle arthroplasty: a cadaveric gait study. J Bone Jt Surg Am 101:e93. https://doi.org/10.2106/JBJS.18.00867
Norvell DC, Ledoux WR, Shofer JB et al (2019) Effectiveness and safety of ankle arthrodesis versus arthroplasty: a prospective multicenter study. J Bone Jt Surg Am 101:1485–1494. https://doi.org/10.2106/JBJS.18.01257
Jules-Elysee KM, Tseng A, Sculco TP et al (2019) Comparison of topical and intravenous tranexamic acid for total knee replacement: a randomized double-blinded controlled study of effects on tranexamic acid levels and thrombogenic and inflammatory marker levels. J Bone Jt Surg Am 101:2120–2128. https://doi.org/10.2106/JBJS.19.00258
Weekes DG, Campbell RE, Shi WJ et al (2019) Prevalence of clinical depression among patients after shoulder stabilization: a prospective study. J Bone Jt Surg Am 101:1628–1635. https://doi.org/10.2106/JBJS.18.01460
Sepucha K, Bedair H, Yu L et al (2019) Decision support strategies for hip and knee osteoarthritis: less is more: a randomized comparative effectiveness trial (DECIDE-OA Study). J Bone Jt Surg Am 101:1645–1653. https://doi.org/10.2106/JBJS.19.00004
Keeney BJ, Austin DC, Jevsevar DS (2019) Preoperative weight loss for morbidly obese patients undergoing total knee arthroplasty: determining the necessary amount. J Bone Jt Surg Am 101:1440–1450. https://doi.org/10.2106/JBJS.18.01136
Sattler LN, Hing WA, Vertullo CJ (2019) Pedaling-based protocol superior to a 10-exercise, non-pedaling protocol for postoperative rehabilitation after total knee replacement: a randomized controlled trial. J Bone Jt Surg Am 101:688–695. https://doi.org/10.2106/JBJS.18.00898
Stepan JG, Lovecchio FC, Premkumar A et al (2019) Development of an institutional opioid prescriber education program and opioid-prescribing guidelines. J Bone Jt Surg 101:5–13. https://doi.org/10.2106/JBJS.17.01645
Wang D, Wang H-Y, Luo Z-Y et al (2019) Finding the optimal regimen for oral tranexamic acid administration in primary total hip arthroplasty: a randomized controlled trial. J Bone Jt Surg Am 101:438–445. https://doi.org/10.2106/JBJS.18.00128
Ferle M, Guo R, Hurschler C (2019) The laxity of the native knee: a meta-analysis of in vitro studies. J Bone Jt Surg Am 101:1119–1131. https://doi.org/10.2106/JBJS.18.00754
Lian J, Diermeier T, Meghpara M et al (2020) Rotatory knee laxity exists on a continuum in anterior cruciate ligament injury. J Bone Jt Surg Am 102:213–220. https://doi.org/10.2106/JBJS.19.00502
McHugh ML (2012) Interrater reliability: the kappa statistic. Biochem Medica 22:276–282. https://doi.org/10.11613/bm.2012.031
Ma H, Bandos AI, Gur D (2015) On the use of partial area under the ROC curve for comparison of two diagnostic tests. Biom J 57:304–320. https://doi.org/10.1002/bimj.201400023
Martínez-Camblor P, Pardo-Fernández JC (2019) The youden index in the generalized receiver operating characteristic curve context. Int J Biostat 15:20180060. https://doi.org/10.1515/ijb-2018-0060
Staffa SJ, Zurakowski D (2020) Calculation of confidence intervals for differences in medians between groups and comparison of methods. Anesth Analg 130:542–546. https://doi.org/10.1213/ANE.0000000000004535
Lee DK (2016) Alternatives to p value: confidence interval and effect size. Korean J Anesthesiol 69:555–562. https://doi.org/10.4097/kjae.2016.69.6.555
Sullivan GM, Feinn R (2012) Using effect size-or why the p value is not enough. J Grad Med Educ 4:279–282. https://doi.org/10.4300/JGME-D-12-00156.1
Acknowledgements
We would like to thank all of the surgeons who participated in this survey.
Funding
This study had no funding.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors have no conflict of interest to declare.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was not obtained due to the nature of study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Strage, K.E., Stacey, S.C., Mauffrey, C. et al. The interobserver reliability of clinical relevance in orthopaedic research. Eur J Orthop Surg Traumatol 33, 1721–1725 (2023). https://doi.org/10.1007/s00590-022-03346-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00590-022-03346-4