Introduction

The skeleton is the third most common site of metastatic cancer, and nearly half of all cancers metastasize to bone [10]. As a result of new and aggressive treatments, the mean survival of patients with disseminated disease has improved substantially [12]. Nevertheless, pathologic fractures occur at sites of metastatic bone disease in up to 35% of affected bones after minimal trauma [14]. Lesions with a high risk of fracture may require surgical stabilization, whereas low-risk lesions can be managed nonoperatively [2]. Effective prevention of pathologic fractures depends on accurately assessing risk of fracture and is critical to avoid debilitating complications.

The Mirels’ score is the most commonly cited approach for classifying fracture risk of the appendicular skeleton when metastatic bone disease is present [9]. Mirels’ classification is useful as a reproducible screening tool for impending pathologic fractures, but because of its poor specificity (13%–35%) [3, 17], reliance on Mirels’ rating system may result in unnecessary prophylactic fixation in two-thirds of the patients [3]. Additionally, Mirels’ score can be unreliable when used by medical and radiation oncologists, who systematically underestimate the risk of fracture of metastatic bone disease [3]. Finally, there is a dilemma at a total score of 8: the probability of fracture is 15% and Mirels recommended the use of clinical judgment in such cases [6].

Metastatic cancer alters both the material and geometric properties of bone; failure to account for changes in both of these parameters limits the accuracy of current fracture risk assessments. Rigidity, defined as the product of the material modulus (which is in turn a function of bone density) and cross-sectional geometry of the structure, can be used as a mechanical assay of cancer-induced changes to the structural competency of the bone. Using this principle, CT-based rigidity analysis (CTRA) was developed. CTRA has been previously used to predict the risk of fracture in children with benign appendicular skeletal lesions and it has been demonstrated to be highly sensitive and specific [8, 16]. A 35% reduction in structural rigidity discriminated fracture from nonfracture cases with 100% sensitivity and 94% specificity. In contrast, plain radiographic criteria demonstrated 28% to 83% sensitivity and 6% to 78% specificity. CTRA is also superior to Mirels’ score in adult patients with metastatic bone disease: a recent prospective study showed that the former is 100% sensitive and 90% specific in predicting pathologic fractures, whereas the latter is 71% sensitive and 50% specific [13].

There is an unmet need to accurately and uniformly assess fracture risk in patients with metastatic bone disease. A standardized risk assessment method would improve patient care by facilitating communication between physicians involved in the treatment of patients with metastatic bone disease. The limitations associated with existing fracture prediction paradigms and the promise of CTRA as an accurate, yet not fully vetted, method to assess fracture risk prompted us to characterize the fracture risk assessment and decision-making process among physicians involved in the care of this patient population. Therefore, our aim is to establish what the credibility is that CTRA holds among physicians by measuring its influence on perceived risk of fracture and treatment recommendations in simulated clinical scenarios of metastatic bone disease of the femur. We hypothesize that providing CTRA results that indicate a rigidity reduction of greater than 35% in simulated clinical scenarios of metastatic bone disease will increase the perceived risk of fracture among individual physicians and the proportion of physicians advocating for prophylactic surgery. Conversely, providing CTRA results that indicate a rigidity reduction of less than 35% will decrease the perceived risk of fracture among individual physicians and the proportion of physicians advocating for prophylactic surgery. We further hypothesize that providing CTRA results (either positive or negative) will increase interobserver agreement for treatment recommendations when compared with the Mirels’ score.

Accordingly, we asked: (1) What is the influence of CTRA on providers’ perceived risk of fracture? (2) What is the influence of CTRA on providers’ treatment recommendations in simulated clinical scenarios of metastatic bone disease of the femur? (3) Does CTRA improve interobserver agreement regarding treatment recommendations for metastatic bone disease?

Materials and Methods

A web-based survey (Survey Monkey, Portland, OR, USA) was developed based on information from a focus group of orthopaedic oncologists, interviews with individual physicians, and a review of the literature. A pilot study was carried out on a sample of 10 orthopaedic oncologists, radiation oncologists, and musculoskeletal radiologists to validate the instrument and to reduce ambiguity in the survey based on their feedback.

Participants were asked to assess the risk of fracture and to give a treatment recommendation for 18 simulated clinical vignettes of metastatic bone disease. Each physician was provided with academic publications on fracture risk assessment methodologies for metastatic bone disease [11, 15, 16]. No other instructions were supplied nor were the participants encouraged to use a particular method while answering the survey to avoid bias or undue influence on the decision-making process.

This study was considered exempt from institutional review board approval by the Institutional Committee on Clinical Investigations at Beth Israel Deaconess Medical Center. All participants provided informed consent for the use of their responses for research purposes.

Physicians from the specialties of orthopaedic oncology, musculoskeletal radiology, and radiation oncology were identified, using convenience sampling, from the web sites of 18 academic medical centers in the United States and Canada. Those with no listed email addresses or those who requested not to be contacted were excluded from the study. This yielded a sample of 80 eligible physicians, all of whom were contacted by email. The overall response rate was 62.5% (50 of 80) (Table 1).

Table 1 Characteristics of 50 survey respondents
Full size table

Eighteen clinical vignettes were developed to simulate commonly encountered metastatic lesions of the femur. The vignettes were created by orthopaedic oncologists based on real radiographs of patients with metastatic bone disease to provide scenarios with low, moderate, and high severity as defined by the Mirels’ score: cases with a score of 6 or less were considered low severity, cases with a score of 10 or more were considered high severity, and those in between as moderate severity. Upper extremity lesions were not included in the clinical vignettes as a result of their lower incidence and lower risk of pathologic fracture when compared with the femur [1].

Each vignette provided representative biplanar radiographs of the lesions as well as a concise description of the patient’s age, gender, primary source of metastasis, and pain. Although other factors are considered important in the clinical decision-making process, the concise history was selected to allow minimal time commitment from the participants while providing enough information to use the Mirels’ score.

Because the purpose of this study was to evaluate the influence of CTRA on fracture risk assessments and treatment recommendations, each of the 18 vignettes was presented in three separate scenarios: (1) clinical vignette without additional information (baseline scenario); (2) clinical vignette with input from CTRA suggesting high risk of fracture (positive [+] CTRA results); and (3) clinical vignette with input from CTRA suggesting low risk of fracture (negative [−] CTRA results). The 54 scenarios were presented in random order to avoid carryover effects from analyzing the three scenarios sequentially.

In total, each participant was presented with 54 clinical scenarios (three scenarios for each of the 18 clinical vignettes). A sample of two clinical vignettes with the three scenarios is available. (Appendix 1 [Supplemental materials are available with the online version of CORR®.])

To determine the physicians’ fracture risk assessments, participants were asked to rate the patient’s risk of fracture on a continuous scale of 0% to 100% after each scenario. Additionally, physicians were asked to provide a treatment recommendation; the possible answers were observation, radiotherapy with or without chemotherapy, surgical stabilization, and surgical stabilization with radiotherapy. Two post hoc categories of treatment recommendation were created. The first category was nonoperative treatment, which included observation and radiotherapy with or without chemotherapy. The second category was prophylactic surgery, which included surgical stabilization with or without radiotherapy.

To assess whether the CTRA has a significant influence on fracture risk assessment and treatment recommendations, survey data were analyzed using repeated-measures analysis of variance (ANOVA) with a generalized estimating equations (GEE) approach [5] to account for the same physicians evaluating each of the 18 vignettes based on three different scenarios using F-tests, which performs well in small sample sizes, with a Bonferroni correction for comparisons of CTRA+ and CTRA− to baseline. Physicians’ fracture risk assessment (0%–100%) and treatment recommendation (surgery versus no surgery) were used as the outcome variables. A linear model was used for fracture risk assessment and a binomial distribution was fit for the binary treatment recommendation (each using the GEE strategy and based on a logistic link function where appropriate) with the appropriate repeated measures as nested terms. Multirater κ was used to quantify the interrater agreement for treatment recommendation based on CTRA results.

Power analysis performed a priori determined that a minimum of 16 physicians per specialty would provide 80% power (β = 0.20, α = 0.05) to capture a 5% change in fracture risk assessment and decision-making when confronted with positive or negative CTRA results stratified according to Mirels’ criteria. In addition, the ANOVA included a two-way interaction test to assess whether the potential effects regarding risk assessment and decision-making depend on severity of the cases as classified using Mirels’ scoring system and the specialty of the individual responder. We did not find any evidence of these interaction effects. However, clinically there seems to be clear evidence that the magnitude of the effect was related to both severity and specialty, and therefore we performed statistical testing under these conditions.

Summary data are reported and displayed graphically as means for the population of vignettes and standard errors using the 0% to 100% scale for risk assessment and recommendation for surgery. Two-tailed values of p < 0.05 were considered statistically significant with adjustment for multiple post hoc comparisons based on the Bonferroni procedure.

Results

Fracture Risk Assessments

When CTRA suggested an increased risk of fracture, physicians perceived the fracture risk to be slightly greater (37% ± 3% versus 42% ± 3%, p < 0.001; mean difference [95% confidence interval {CI}] = 5% [4.7%–5.2%]). Similarly, when CTRA suggested a decreased risk of fracture, physicians perceived the risk to be slightly decreased (37% ± 3% versus 35% ± 3%, p = 0.04; mean difference [95% CI] = 2% [2.74%–2.26%]; Fig. 1). However, the significance and magnitude of the influence of CTRA on physicians’ perception of fracture risk varied with the severity of the lesion and the specialty of the responders (Table 2). For instance, only positive CTRA results affected assessments made by musculoskeletal radiologists across low, moderate, and high severity cases. Positive and negative CTRA results affected assessments made by orthopaedic oncologists across moderate and high severity cases, whereas in low severity cases, only positive CTRA results had an influence. For radiation oncologists, negative CTRA results affected fracture risk assessments in high severity cases, whereas positive CTRA results affected fracture risk assessments in low and moderate severity cases. Orthopaedic oncologists were the most influenced group (F = 26.4; p < 0.001), and musculoskeletal radiologists were the least influenced (F = 5.5; p = 0.004). Post hoc analysis of fracture risk assessments indicated that physicians in all three specialties were more influenced by positive rather than negative CTRA results.

Fig. 1
figure 1

Fracture risk assessments made by all physicians are presented based on case severity. Errors bars represent robust standard errors using the “Sandwich” estimator. Asterisks (*) indicate a significant difference between baseline and CTRA scenarios.

Full size image
Table 2 Fracture risk assessment based on case severity and medical specialty
Full size table

Treatment Recommendations

When CTRA suggested an increased risk of fracture, physicians were more prone to recommend surgical stabilization (46% ± 9% versus 54% ± 9%, p <0.001; mean difference [95% CI] = 9% [7.9–10.1]). Similarly, when CTRA suggested a decreased risk of fracture, physicians were less prone to recommend surgical stabilization (46% ± 9% versus 42% ± 9%, p < 0.03; mean difference [95% CI] = 4% [3.9–5.1]; Fig. 2). Like with fracture risk perception, the influence of CTRA on physicians’ treatment recommendation varied with severity and specialty (Table 3). For example, the treatment recommendations of musculoskeletal radiologists were only influenced by negative results in low severity. The treatment recommendations of orthopaedic oncologists in moderate severity cases were influenced by both negative and positive CTRA results, whereas in low severity cases, only positive CTRA results affected their recommendations. The only effect of CTRA on the treatment recommendations of radiation oncologists was found in low severity cases, where both negative and positive CTRA results had an influence.

Fig. 2
figure 2

Treatment recommendations made by all physicians are presented based on case severity. Errors bars represent robust standard errors using the “Sandwich” estimator. Asterisks (*) indicate a significant difference between baseline and CTRA scenarios.

Full size image
Table 3 Treatment recommendation based on case severity and medical specialty
Full size table

Effect of CTRA on Interobserver Agreement

CTRA did not increase interobserver agreement regarding treatment recommendations when compared with the baseline scenario (κ = 0.41 versus κ = 0.43, respectively). When provided with CTRA results, either positive or negative, interobserver agreement ranged from fair to moderate (Table 4) [7].

Table 4 Multirater kappa statistics for interobserver agreement regarding treatment recommendations
Full size table

Discussion

With the increased survival of patients with cancer and the growing demand for higher quality of life, physicians are involved in the care of a larger population of patients with metastatic bone disease who are at risk for pathologic fractures. However, there is little consensus regarding the most appropriate method to assess risk of fracture in this patient population and treatment planning is subject to high variability between medical specialties [4]. The Mirels’ score is the most popular method to determine risk of fracture in metastatic bone disease. However, its development was based on retrospective studies with small sample sizes and it has been shown to be poorly specific and highly variable. A survey study by Damron and Ward [3] demonstrated that medical and radiation oncologists systematically underscore Mirels’ criteria and provide the least accurate fracture predictions based on their clinical judgment and Mirels’ score. CTRA is a sensitive and specific method for predicting pathologic fractures in benign and malignant bone disease [8, 13, 15] and could potentially be used as a screening tool, yet its influence on clinical decision-making has never been quantified.

The present study had limitations. Participants were recruited from academic medical centers in North America using nonprobability sampling. Therefore, this sample may not be representative of the population of physicians involved in the care of patients with metastatic bone disease. Additionally, the 62.5% response rate may indicate that the responders may be somewhat different from our intended sample. However, we believe that the characteristics of the responders in this study provide an adequate distribution of specialty, degree of experience, and expertise in the treatment of metastatic bone disease. Finally, the survey consisted of simulated clinical scenarios instead of real case histories, and the actual risk of fracture of the lesions presented in the survey was unknown, limiting our ability to quantify the diagnostic performance of the Mirels’ score.

In this study, we have demonstrated that CTRA had a small influence on overall fracture risk assessments and treatment recommendations. However, CTRA consistently had an influence on treatment recommendations for moderate severity scenarios (Mirels’ score 7–9). This is an encouraging finding, because this is the most prevalent risk category, and patients within this range are the most challenging to properly assign to either surgical or nonoperative care. On the other hand, CTRA had variable or no influence on perceived fracture risk or treatment recommendations in those cases with low or high risk. This suggests that physicians are unwilling to change their treatment recommendations, when there is a clear indication, according to Mirels’ criteria, for or against surgery despite evidence that this particular method has low sensitivity and specificity for pathologic fractures and therefore is not well suited to be used as a screening tool. Furthermore, its use may result in a high number of unnecessary procedures in up to two-thirds of patients with lesions classified as high risk for fracture [6]. CTRA, on the other hand, has shown to be 100% sensitive and 90% specific for predicting pathologic fractures in patients with metastatic bone disease and as such could potentially replace Mirels’ as the preferred fracture risk assessment method [13]. The low influence in perceived fracture risk and treatment recommendations among specialists involved in the treatment of patients with metastatic bone disease can be attributed to lack of exposure to the technique as a relative newcomer to the field, the unwillingness of physicians to adopt new techniques, and lack of evidence-based clinical guidelines for the field.

The largest effect of CTRA on fracture risk assessments and treatment recommendations was observed during subgroup analyses in orthopaedic oncologists. This group included a small number of physicians who have participated in CTRA clinical trials as well as others who have been exposed to the technique in numerous scientific meetings and conferences as opposed to other specialists whose only exposure has been through the scientific literature presented to them before participating in the survey study. This leads us to believe that alternative means of exposure to CTRA such as seminars/webinars and symposia could be more effective to disseminate the improved efficacy of CTRA and could lead to a larger influence on fracture risk assessments and treatment recommendations among other specialists.

The ideal classification system should accurately and reliably estimate the risk of pathologic fracture, provide clarity in communication, and guide treatment planning. Furthermore, as a result of the multidisciplinary nature of care for patients with metastatic bone disease, it should have high interobserver reliability. Currently, CTRA is an improvement on conventional fracture risk assessment methods and could potentially replace Mirels’ score as the preferred tool when evaluating patients at risk for pathologic fractures. However, although traditional scientific studies demonstrating the benefits of CTRA have been published, the technique exerts little to no influence on specialists’ perceived fracture risk and their treatment recommendations. The majority of the physicians who participated in this survey study were unaware of the CTRA technique and were only presented with papers to review before taking the survey. The decision to read any or all of the papers and to better understand the technique before taking the survey is unknown to us. This could be an important source for the outcome of the study. During planning phases, the authors considered this potential outcome but ultimately chose to not promote CTRA heavily to the cohort so as not to bias them in favor of the technique.

As a result, further work is warranted to properly introduce this technique to physicians involved in the treatment of patients with metastatic bone disease. Given the number of preclinical and clinical studies outlining the efficacy of this technique, better education through presentations at seminars/webinars and symposia will be the first step to undertake. This should be followed by undertaking clinical trials to establish CTRA-centered clinical guidelines based on evidence-based medicine. Establishment of the said guidelines will provide the physicians with a vetted tool to plan treatment and followup for their patients. Given the consistent reporting of very high sensitivity and specificity of this technique, it is hoped that all patients (low, moderate, and high risk) with metastatic bone disease will benefit from CTRA-based guidelines for their care.