Abstract
Background
There is a need to improve the prediction of fracture risk for patients with metastatic bone disease. CT-based rigidity analysis (CTRA) is a sensitive and specific method, yet its influence on clinical decision-making has never been quantified.
Questions/purposes
What is the influence of CTRA on providers’ perceived risk of fracture? (2) What is the influence of CTRA on providers’ treatment recommendations in simulated clinical scenarios of metastatic bone disease of the femur? (3) Does CTRA improve interobserver agreement regarding treatment recommendations?
Methods
We conducted a survey among 80 academic physicians (orthopaedic oncologists, musculoskeletal radiologists, and radiation oncologists) using simulated vignettes of femoral lesions presented as three separate scenarios: (1) no CTRA input (baseline); (2) CTRA input suggesting increased risk of fracture (CTRA+); and (3) CTRA input suggesting decreased risk of fracture (CTRA−). Participants were asked to rate the patient’s risk of fracture on a scale of 0% to 100% and to provide a treatment recommendation. Overall response rate was 62.5% (50 of 80).
Results
When CTRA suggested an increased risk of fracture, physicians perceived the fracture risk to be slightly greater (37% ± 3% versus 42% ± 3%, p < 0.001; mean difference [95% confidence interval {CI}] = 5% [4.7%–5.2%]) and were more prone to recommend surgical stabilization (46% ± 9% versus 54% ± 9%, p < 0.001; mean difference [95% CI] = 9% [7.9–10.1]). When CTRA suggested a decreased risk of fracture, physicians perceived the risk to be slightly decreased (37% ± 25% versus 35% ± 25%, p = 0.04; mean difference [95% CI] = 2% [2.74%–2.26%]) and were less prone to recommend surgical stabilization (46% ± 9% versus 42% ± 9%, p < 0.03; mean difference [95% CI] = 4% [3.9–5.1]). The effect size of the influence of CTRA on physicians’ perception of fracture risk and treatment planning varied with lesion severity and specialty of the responders. CTRA did not increase interobserver agreement regarding treatment recommendations when compared with the baseline scenario (κ = 0.41 versus κ = 0.43, respectively).
Conclusions
Based on this survey study, CTRA had a small influence on perceived fracture risk and treatment recommendations and did not increase interobserver agreement. Further work is required to properly introduce this technique to physicians involved in the care of patients with metastatic lesions. Given the number of preclinical and clinical studies outlining the efficacy of this technique, better education through presentations at seminars/webinars and symposia will be the first step. This should be followed by clinical trials to establish CTRA-based clinical guidelines based on evidence-based medicine. Increased exposure of clinicians to CTRA, including its underlying methodology to study bone structural characteristics, may establish CTRA as a uniform guideline to assess fracture risk.
Level of Evidence
Level III, economic and decision analyses.
Similar content being viewed by others
Introduction
The skeleton is the third most common site of metastatic cancer, and nearly half of all cancers metastasize to bone [10]. As a result of new and aggressive treatments, the mean survival of patients with disseminated disease has improved substantially [12]. Nevertheless, pathologic fractures occur at sites of metastatic bone disease in up to 35% of affected bones after minimal trauma [14]. Lesions with a high risk of fracture may require surgical stabilization, whereas low-risk lesions can be managed nonoperatively [2]. Effective prevention of pathologic fractures depends on accurately assessing risk of fracture and is critical to avoid debilitating complications.
The Mirels’ score is the most commonly cited approach for classifying fracture risk of the appendicular skeleton when metastatic bone disease is present [9]. Mirels’ classification is useful as a reproducible screening tool for impending pathologic fractures, but because of its poor specificity (13%–35%) [3, 17], reliance on Mirels’ rating system may result in unnecessary prophylactic fixation in two-thirds of the patients [3]. Additionally, Mirels’ score can be unreliable when used by medical and radiation oncologists, who systematically underestimate the risk of fracture of metastatic bone disease [3]. Finally, there is a dilemma at a total score of 8: the probability of fracture is 15% and Mirels recommended the use of clinical judgment in such cases [6].
Metastatic cancer alters both the material and geometric properties of bone; failure to account for changes in both of these parameters limits the accuracy of current fracture risk assessments. Rigidity, defined as the product of the material modulus (which is in turn a function of bone density) and cross-sectional geometry of the structure, can be used as a mechanical assay of cancer-induced changes to the structural competency of the bone. Using this principle, CT-based rigidity analysis (CTRA) was developed. CTRA has been previously used to predict the risk of fracture in children with benign appendicular skeletal lesions and it has been demonstrated to be highly sensitive and specific [8, 16]. A 35% reduction in structural rigidity discriminated fracture from nonfracture cases with 100% sensitivity and 94% specificity. In contrast, plain radiographic criteria demonstrated 28% to 83% sensitivity and 6% to 78% specificity. CTRA is also superior to Mirels’ score in adult patients with metastatic bone disease: a recent prospective study showed that the former is 100% sensitive and 90% specific in predicting pathologic fractures, whereas the latter is 71% sensitive and 50% specific [13].
There is an unmet need to accurately and uniformly assess fracture risk in patients with metastatic bone disease. A standardized risk assessment method would improve patient care by facilitating communication between physicians involved in the treatment of patients with metastatic bone disease. The limitations associated with existing fracture prediction paradigms and the promise of CTRA as an accurate, yet not fully vetted, method to assess fracture risk prompted us to characterize the fracture risk assessment and decision-making process among physicians involved in the care of this patient population. Therefore, our aim is to establish what the credibility is that CTRA holds among physicians by measuring its influence on perceived risk of fracture and treatment recommendations in simulated clinical scenarios of metastatic bone disease of the femur. We hypothesize that providing CTRA results that indicate a rigidity reduction of greater than 35% in simulated clinical scenarios of metastatic bone disease will increase the perceived risk of fracture among individual physicians and the proportion of physicians advocating for prophylactic surgery. Conversely, providing CTRA results that indicate a rigidity reduction of less than 35% will decrease the perceived risk of fracture among individual physicians and the proportion of physicians advocating for prophylactic surgery. We further hypothesize that providing CTRA results (either positive or negative) will increase interobserver agreement for treatment recommendations when compared with the Mirels’ score.
Accordingly, we asked: (1) What is the influence of CTRA on providers’ perceived risk of fracture? (2) What is the influence of CTRA on providers’ treatment recommendations in simulated clinical scenarios of metastatic bone disease of the femur? (3) Does CTRA improve interobserver agreement regarding treatment recommendations for metastatic bone disease?
Materials and Methods
A web-based survey (Survey Monkey, Portland, OR, USA) was developed based on information from a focus group of orthopaedic oncologists, interviews with individual physicians, and a review of the literature. A pilot study was carried out on a sample of 10 orthopaedic oncologists, radiation oncologists, and musculoskeletal radiologists to validate the instrument and to reduce ambiguity in the survey based on their feedback.
Participants were asked to assess the risk of fracture and to give a treatment recommendation for 18 simulated clinical vignettes of metastatic bone disease. Each physician was provided with academic publications on fracture risk assessment methodologies for metastatic bone disease [11, 15, 16]. No other instructions were supplied nor were the participants encouraged to use a particular method while answering the survey to avoid bias or undue influence on the decision-making process.
This study was considered exempt from institutional review board approval by the Institutional Committee on Clinical Investigations at Beth Israel Deaconess Medical Center. All participants provided informed consent for the use of their responses for research purposes.
Physicians from the specialties of orthopaedic oncology, musculoskeletal radiology, and radiation oncology were identified, using convenience sampling, from the web sites of 18 academic medical centers in the United States and Canada. Those with no listed email addresses or those who requested not to be contacted were excluded from the study. This yielded a sample of 80 eligible physicians, all of whom were contacted by email. The overall response rate was 62.5% (50 of 80) (Table 1).
Eighteen clinical vignettes were developed to simulate commonly encountered metastatic lesions of the femur. The vignettes were created by orthopaedic oncologists based on real radiographs of patients with metastatic bone disease to provide scenarios with low, moderate, and high severity as defined by the Mirels’ score: cases with a score of 6 or less were considered low severity, cases with a score of 10 or more were considered high severity, and those in between as moderate severity. Upper extremity lesions were not included in the clinical vignettes as a result of their lower incidence and lower risk of pathologic fracture when compared with the femur [1].
Each vignette provided representative biplanar radiographs of the lesions as well as a concise description of the patient’s age, gender, primary source of metastasis, and pain. Although other factors are considered important in the clinical decision-making process, the concise history was selected to allow minimal time commitment from the participants while providing enough information to use the Mirels’ score.
Because the purpose of this study was to evaluate the influence of CTRA on fracture risk assessments and treatment recommendations, each of the 18 vignettes was presented in three separate scenarios: (1) clinical vignette without additional information (baseline scenario); (2) clinical vignette with input from CTRA suggesting high risk of fracture (positive [+] CTRA results); and (3) clinical vignette with input from CTRA suggesting low risk of fracture (negative [−] CTRA results). The 54 scenarios were presented in random order to avoid carryover effects from analyzing the three scenarios sequentially.
In total, each participant was presented with 54 clinical scenarios (three scenarios for each of the 18 clinical vignettes). A sample of two clinical vignettes with the three scenarios is available. (Appendix 1 [Supplemental materials are available with the online version of CORR®.])
To determine the physicians’ fracture risk assessments, participants were asked to rate the patient’s risk of fracture on a continuous scale of 0% to 100% after each scenario. Additionally, physicians were asked to provide a treatment recommendation; the possible answers were observation, radiotherapy with or without chemotherapy, surgical stabilization, and surgical stabilization with radiotherapy. Two post hoc categories of treatment recommendation were created. The first category was nonoperative treatment, which included observation and radiotherapy with or without chemotherapy. The second category was prophylactic surgery, which included surgical stabilization with or without radiotherapy.
To assess whether the CTRA has a significant influence on fracture risk assessment and treatment recommendations, survey data were analyzed using repeated-measures analysis of variance (ANOVA) with a generalized estimating equations (GEE) approach [5] to account for the same physicians evaluating each of the 18 vignettes based on three different scenarios using F-tests, which performs well in small sample sizes, with a Bonferroni correction for comparisons of CTRA+ and CTRA− to baseline. Physicians’ fracture risk assessment (0%–100%) and treatment recommendation (surgery versus no surgery) were used as the outcome variables. A linear model was used for fracture risk assessment and a binomial distribution was fit for the binary treatment recommendation (each using the GEE strategy and based on a logistic link function where appropriate) with the appropriate repeated measures as nested terms. Multirater κ was used to quantify the interrater agreement for treatment recommendation based on CTRA results.
Power analysis performed a priori determined that a minimum of 16 physicians per specialty would provide 80% power (β = 0.20, α = 0.05) to capture a 5% change in fracture risk assessment and decision-making when confronted with positive or negative CTRA results stratified according to Mirels’ criteria. In addition, the ANOVA included a two-way interaction test to assess whether the potential effects regarding risk assessment and decision-making depend on severity of the cases as classified using Mirels’ scoring system and the specialty of the individual responder. We did not find any evidence of these interaction effects. However, clinically there seems to be clear evidence that the magnitude of the effect was related to both severity and specialty, and therefore we performed statistical testing under these conditions.
Summary data are reported and displayed graphically as means for the population of vignettes and standard errors using the 0% to 100% scale for risk assessment and recommendation for surgery. Two-tailed values of p < 0.05 were considered statistically significant with adjustment for multiple post hoc comparisons based on the Bonferroni procedure.
Results
Fracture Risk Assessments
When CTRA suggested an increased risk of fracture, physicians perceived the fracture risk to be slightly greater (37% ± 3% versus 42% ± 3%, p < 0.001; mean difference [95% confidence interval {CI}] = 5% [4.7%–5.2%]). Similarly, when CTRA suggested a decreased risk of fracture, physicians perceived the risk to be slightly decreased (37% ± 3% versus 35% ± 3%, p = 0.04; mean difference [95% CI] = 2% [2.74%–2.26%]; Fig. 1). However, the significance and magnitude of the influence of CTRA on physicians’ perception of fracture risk varied with the severity of the lesion and the specialty of the responders (Table 2). For instance, only positive CTRA results affected assessments made by musculoskeletal radiologists across low, moderate, and high severity cases. Positive and negative CTRA results affected assessments made by orthopaedic oncologists across moderate and high severity cases, whereas in low severity cases, only positive CTRA results had an influence. For radiation oncologists, negative CTRA results affected fracture risk assessments in high severity cases, whereas positive CTRA results affected fracture risk assessments in low and moderate severity cases. Orthopaedic oncologists were the most influenced group (F = 26.4; p < 0.001), and musculoskeletal radiologists were the least influenced (F = 5.5; p = 0.004). Post hoc analysis of fracture risk assessments indicated that physicians in all three specialties were more influenced by positive rather than negative CTRA results.
Treatment Recommendations
When CTRA suggested an increased risk of fracture, physicians were more prone to recommend surgical stabilization (46% ± 9% versus 54% ± 9%, p <0.001; mean difference [95% CI] = 9% [7.9–10.1]). Similarly, when CTRA suggested a decreased risk of fracture, physicians were less prone to recommend surgical stabilization (46% ± 9% versus 42% ± 9%, p < 0.03; mean difference [95% CI] = 4% [3.9–5.1]; Fig. 2). Like with fracture risk perception, the influence of CTRA on physicians’ treatment recommendation varied with severity and specialty (Table 3). For example, the treatment recommendations of musculoskeletal radiologists were only influenced by negative results in low severity. The treatment recommendations of orthopaedic oncologists in moderate severity cases were influenced by both negative and positive CTRA results, whereas in low severity cases, only positive CTRA results affected their recommendations. The only effect of CTRA on the treatment recommendations of radiation oncologists was found in low severity cases, where both negative and positive CTRA results had an influence.
Effect of CTRA on Interobserver Agreement
CTRA did not increase interobserver agreement regarding treatment recommendations when compared with the baseline scenario (κ = 0.41 versus κ = 0.43, respectively). When provided with CTRA results, either positive or negative, interobserver agreement ranged from fair to moderate (Table 4) [7].
Discussion
With the increased survival of patients with cancer and the growing demand for higher quality of life, physicians are involved in the care of a larger population of patients with metastatic bone disease who are at risk for pathologic fractures. However, there is little consensus regarding the most appropriate method to assess risk of fracture in this patient population and treatment planning is subject to high variability between medical specialties [4]. The Mirels’ score is the most popular method to determine risk of fracture in metastatic bone disease. However, its development was based on retrospective studies with small sample sizes and it has been shown to be poorly specific and highly variable. A survey study by Damron and Ward [3] demonstrated that medical and radiation oncologists systematically underscore Mirels’ criteria and provide the least accurate fracture predictions based on their clinical judgment and Mirels’ score. CTRA is a sensitive and specific method for predicting pathologic fractures in benign and malignant bone disease [8, 13, 15] and could potentially be used as a screening tool, yet its influence on clinical decision-making has never been quantified.
The present study had limitations. Participants were recruited from academic medical centers in North America using nonprobability sampling. Therefore, this sample may not be representative of the population of physicians involved in the care of patients with metastatic bone disease. Additionally, the 62.5% response rate may indicate that the responders may be somewhat different from our intended sample. However, we believe that the characteristics of the responders in this study provide an adequate distribution of specialty, degree of experience, and expertise in the treatment of metastatic bone disease. Finally, the survey consisted of simulated clinical scenarios instead of real case histories, and the actual risk of fracture of the lesions presented in the survey was unknown, limiting our ability to quantify the diagnostic performance of the Mirels’ score.
In this study, we have demonstrated that CTRA had a small influence on overall fracture risk assessments and treatment recommendations. However, CTRA consistently had an influence on treatment recommendations for moderate severity scenarios (Mirels’ score 7–9). This is an encouraging finding, because this is the most prevalent risk category, and patients within this range are the most challenging to properly assign to either surgical or nonoperative care. On the other hand, CTRA had variable or no influence on perceived fracture risk or treatment recommendations in those cases with low or high risk. This suggests that physicians are unwilling to change their treatment recommendations, when there is a clear indication, according to Mirels’ criteria, for or against surgery despite evidence that this particular method has low sensitivity and specificity for pathologic fractures and therefore is not well suited to be used as a screening tool. Furthermore, its use may result in a high number of unnecessary procedures in up to two-thirds of patients with lesions classified as high risk for fracture [6]. CTRA, on the other hand, has shown to be 100% sensitive and 90% specific for predicting pathologic fractures in patients with metastatic bone disease and as such could potentially replace Mirels’ as the preferred fracture risk assessment method [13]. The low influence in perceived fracture risk and treatment recommendations among specialists involved in the treatment of patients with metastatic bone disease can be attributed to lack of exposure to the technique as a relative newcomer to the field, the unwillingness of physicians to adopt new techniques, and lack of evidence-based clinical guidelines for the field.
The largest effect of CTRA on fracture risk assessments and treatment recommendations was observed during subgroup analyses in orthopaedic oncologists. This group included a small number of physicians who have participated in CTRA clinical trials as well as others who have been exposed to the technique in numerous scientific meetings and conferences as opposed to other specialists whose only exposure has been through the scientific literature presented to them before participating in the survey study. This leads us to believe that alternative means of exposure to CTRA such as seminars/webinars and symposia could be more effective to disseminate the improved efficacy of CTRA and could lead to a larger influence on fracture risk assessments and treatment recommendations among other specialists.
The ideal classification system should accurately and reliably estimate the risk of pathologic fracture, provide clarity in communication, and guide treatment planning. Furthermore, as a result of the multidisciplinary nature of care for patients with metastatic bone disease, it should have high interobserver reliability. Currently, CTRA is an improvement on conventional fracture risk assessment methods and could potentially replace Mirels’ score as the preferred tool when evaluating patients at risk for pathologic fractures. However, although traditional scientific studies demonstrating the benefits of CTRA have been published, the technique exerts little to no influence on specialists’ perceived fracture risk and their treatment recommendations. The majority of the physicians who participated in this survey study were unaware of the CTRA technique and were only presented with papers to review before taking the survey. The decision to read any or all of the papers and to better understand the technique before taking the survey is unknown to us. This could be an important source for the outcome of the study. During planning phases, the authors considered this potential outcome but ultimately chose to not promote CTRA heavily to the cohort so as not to bias them in favor of the technique.
As a result, further work is warranted to properly introduce this technique to physicians involved in the treatment of patients with metastatic bone disease. Given the number of preclinical and clinical studies outlining the efficacy of this technique, better education through presentations at seminars/webinars and symposia will be the first step to undertake. This should be followed by undertaking clinical trials to establish CTRA-centered clinical guidelines based on evidence-based medicine. Establishment of the said guidelines will provide the physicians with a vetted tool to plan treatment and followup for their patients. Given the consistent reporting of very high sensitivity and specificity of this technique, it is hoped that all patients (low, moderate, and high risk) with metastatic bone disease will benefit from CTRA-based guidelines for their care.
References
Carnesale P, ed. Malignant Tumours of Bone. Amsterdam, The Netherlands: Elsevier; 2003.
Costa L, Major PP. Effect of bisphosphonates on pain and quality of life in patients with bone metastases. Nat Clin Pract Oncol. 2009;6:163–174.
Damron TA, Ward WG. Risk of pathologic fracture: assessment. Clin Orthop Relat Res. 2003;415(Suppl):208–211.
El-Husseiny M, Coleman N. Inter- and intra-observer variation in classification systems for impending fractures of bone metastases. Skeletal Radiol. 2010;39:155–160.
Fahrmeir L, Tutz G. Models for Multicategorical Responses: Multivariate Extensions of Generalized Linear Models. New York, NY, USA: Springer; 2001.
Jawad MU, Scully SP. In brief: classifications in brief: Mirels’ classification: metastatic disease in long bones and impending pathologic fracture. Clin Orthop Relat Res. 2010;468:2825–2827.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.
Leong NL, Anderson ME, Gebhardt MC, Snyder BD. Computed tomography-based structural analysis for predicting fracture risk in children with benign skeletal neoplasms: comparison of specificity with that of plain radiographs. J Bone Joint Surg Am. 2010;92:1827–1833.
Mac Niocaill RF, Quinlan JF, Stapleton RD, Hurson B, Dudeney S, O’Toole GC. Inter- and intra-observer variability associated with the use of the Mirels’ scoring system for metastatic bone lesions. Int Orthop. 2011;35:83–86.
Michaeli DA, Inoue K, Hayes WC, Hipp JA. Density predicts the activity-dependent failure load of proximal femora with defects. Skeletal Radiol. 1999;28:90–95.
Mirels H. Metastatic disease in long bones. Clin Orthop Relat Res. 1989;249:256–264.
Nathan SS, Healey JH, Mellano D, Hoang B, Lewis I, Morris CD, Athanasian EA, Boland PJ. Survival in patients operated on for pathologic fracture: implications for end-of-life orthopedic care. J Clin Oncol. 2005;23:6072–6082.
Nazarian A, Entezari V, Zurakowski D, Calderon N, Hipp JA, Villa-Camacho JC, Lin PP, Cheung FH, Aboulafia AJ, Turcotte R, Anderson ME, Gebhardt MC, Cheng EY, Terek RM, Yaszemski M, Damron TA, Snyder BD. Treatment planning and fracture prediction in patients with skeletal metastasis with CT-based rigidity analysis. Clin Cancer Res. 2015 Feb 27 [Epub ahead of print].
Plunkett TA, Smith P, Rubens RD. Risk of complications from bone metastases in breast cancer. implications for management. Eur J Cancer. 2000;36:476–482.
Snyder BD, Cordio MA, Nazarian A, Kwak SD, Chang DJ, Entezari V, Zurakowski D, Parker LM. Noninvasive prediction of fracture risk in patients with metastatic cancer to the spine. Clin Cancer Res. 2009;15:7676–7683.
Snyder BD, Hauser-Kara DA, Hipp JA, Zurakowski D, Hecht AC, Gebhardt MC. Predicting fracture through benign skeletal lesions with quantitative computed tomography. J Bone Joint surg Am. 2006;88:55–70.
Van der Linden YM, Dijkstra PD, Kroon HM, Lok JJ, Noordijk EM, Leer JW, Marijnen CA. Comparative analysis of risk factors for pathological fracture with femoral metastases. J Bone Joint Surg Br. 2004;86:566–573.
Acknowledgments
We would like to acknowledge the Musculoskeletal Tumor Society and the National Institutes of Health T32 COMET Program (AN) (AR055885) and LRP (AN) (L30 AR056606) for providing financial support towards this project. We would also like to thank the orthopaedic oncologists, radiation oncologists, and radiologists who participated in this study.
Author information
Authors and Affiliations
Corresponding author
Additional information
One of the authors (AN) has received research support funding from the Musculoskeletal Tumor Society.
All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research® editors and board members are on file with the publication and can be viewed on request.
Each author certifies that his or her institution approved or waived approval for the human protocol for this investigation and that all investigations were conducted in conformity with ethical principles of research.
This work was performed at Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA.
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Nazarian, A., Entezari, V., Villa-Camacho, J.C. et al. Does CT-based Rigidity Analysis Influence Clinical Decision-making in Simulations of Metastatic Bone Disease?. Clin Orthop Relat Res 474, 652–659 (2016). https://doi.org/10.1007/s11999-015-4371-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11999-015-4371-1