Introduction

Sarcomas are a rare and heterogeneous group of cancers that represent < 1% of all malignancies [24]. However, as a result of their clinical behavior, multidisciplinary management, and complex multimodal treatment, the impact to patients is significant and the cost of care is substantial [7, 13, 27]. Moreover, the skeleton is the third most common organ affected by metastatic cancer [36]. In the advent of improved medical treatment of many cancers, bone metastases are becoming increasingly prevalent because patients with cancer are living longer with their disease [1, 2]. Many critical questions surrounding the surgical management of patients with bone and soft tissue tumors and metastatic bone disease remain unanswered, but funding to support prospective clinical research is meager in comparison to basic science research [34]. Furthermore, a recent qualitative study determined that orthopaedic oncology collaborators are enthusiastic to conduct further research, provided that the research questions are feasible and address clinically relevant endpoints [28]. Therefore, it is important to identify research priorities through a systematic and thoughtful process.

An effective research system should address health issues, interventions, and outcomes of concern within a given field that are considered important by both clinicians and patients. The Delphi method is one approach that can be utilized to amass the expertise and opinions from various stakeholders in an area and use it to determine a suitable set of research priorities through consensus. This method has been previously used in the development of research agendas in other specialties. For example, the Acute Kidney Injury Network (AKIN) utilized a Delphi process to construct a research agenda to guide future clinical and translational acute kidney injury (AKI) research [17, 21]. This collaborative network has since conducted multiple studies that address many of the identified priorities and has generated evidence to improve the care of patients with AKI [11, 16, 25]. Their work also exemplifies the impact a formal research agenda can have on clinical care in a given field. However, research priorities have yet to be identified in the field of orthopaedic oncology.

The purpose of this study was to use a consensus-based approach to prioritize research questions within orthopaedic oncology so that efforts and available resources can then be streamlined toward addressing these priorities.

Materials and Methods

The protocol for this initiative was previously reported elsewhere in further detail [29]. Briefly, a three-step modified Delphi process was conducted between April and November 2016, consisting of a qualitative assessment (Phase I), a rating evaluation (Phase II), and a consensus panel meeting (Phase III) (Fig. 1). During the qualitative assessment, participants were asked to identify a maximum of three research questions that they believed most urgently needed answering to guide patient care. In the rating evaluation, participants were asked to individually rate each candidate research question based on the definitions provided of five criteria that are considered crucial to the development of a realistic yet compelling research question: scientific merit, clinical significance, innovation, relevance, and feasibility (Table 1). Research questions from the rating evaluation that met the a priori consensus thresholds outlined in detail subsequently were brought forward to the consensus panel meeting for facilitated discussion. Thereafter, each consensus panel member anonymously scored each candidate research question; those that met the predetermined criteria were then brought forward for final ranking. The consensus panel members were then asked to rank their top three research questions, which were then distributed to all participants for validation.

Fig. 1
figure 1

A flow diagram shows the Moving Forward Through Consensus Orthopaedic Oncology Research Program planning initiative.

Full size image
Table 1 Phase II criteria definitions
Full size table

Before conducting this consensus-based process, ethics approval was received from the Hamilton Integrated Research Ethics Board (Project No. 1765). For the qualitative assessment and rating evaluation, written informed consent was not obtained from participants because the completion and submission of questionnaires were considered implied consent. For the in-person consensus meeting participants, written informed consent was obtained before the commencement of the meeting; however, for remote participants, written informed consent was not obtained because access to the remote response system and submission of responses were again considered implied consent. All suggested research questions, scores, and rankings were kept strictly confidential and all identifiers were removed before any analysis. Data with direct identifiers were available to only one member of the planning committee (PJS).

An initial invitation to gauge interest in participating in this research program planning initiative was distributed to 351 individuals by email on May 24, 2016. The invitation was sent to all clinicians on the Prophylactic Antibiotic Regimens in Tumor Surgery (PARITY) network distribution list, which includes both individuals who have either expressed an interest or are actively participating in the PARITY trial (NCT No. 01479283) [6] as well as members of the Musculoskeletal Tumor Society (MSTS). Of the 351 individuals, 117 responded to the email (33%). Almost all of these 117 respondents (114 individuals) indicated that they would be interested in participating in all phases of this initiative with only two individuals indicating that they did not wish to participate and one individual wanting to participate only in the consensus meeting.

Phase I: Qualitative Assessment–Soliciting Research Questions of Interest

Participants

To be eligible for the qualitative assessment, participants had to be a clinician who was either: (1) interested or actively participating in the PARITY trial; or (2) an active or candidate MSTS member. An Orthopaedic Research and Education Foundation (OREF) representative and a clinical research manager of an orthopaedic oncology unit at a major clinical site (with extensive practical experience in orthopaedic oncology clinical research) were also included. A total of 114 individuals were invited to participate. Sixty-one individuals (54% response rate) provided complete responses to the Phase I questionnaire (Fig. 1). Of the participants, all but one were orthopaedic surgeons (98%), and almost all (90%) were men. Of the orthopaedic surgeons who participated, 88% had completed a fellowship in orthopaedic oncology. Participants represented orthopaedic oncology practices from 16 countries (Table 2).

Table 2 Participant demographics
Full size table

Interventions

The invitation to complete a web-based, open-ended questionnaire (with unrestricted answers) was distributed by email on June 14, 2016. The questionnaire was not accompanied by any additional facilitators or literature reviews. Rather, participants were asked to review the literature and consult with colleagues as they saw fit before proposing their ideas. The questionnaire also requested some demographic data and any potential financial or intellectual conflicts of interest. The questionnaire remained online for 4 weeks, and reminder emails were sent approximately every 7 days after the initial invitation to those who had not yet completed the questionnaire.

Phase II: Rating Evaluation–Rating of Research Questions

Participants

The same eligibility criteria for qualitative assessment were maintained for the rating evaluation. In total, 113 individuals (one individual withdrew) were invited to participate in this part of the study. Sixty individuals (59 complete, one incomplete; 53% response rate) responded to the rating evaluation questionnaire (Fig. 1). Of the participants, all but one were orthopaedic surgeons (97%); however, one of these orthopaedic surgeons specialized in veterinary medicine. Over two-thirds of the participants (85%) were men. Of the orthopaedic surgeons who participated, 88% had completed a fellowship in orthopaedic oncology. Participants represented orthopaedic oncology practices from 16 countries (Table 2).

Interventions

The second phase of the modified Delphi process used a web-based questionnaire that asked participants to rate each candidate research question individually on a 5-point Likert scale for five criteria: scientific merit, clinical significance, innovation, relevance, and feasibility (Table 3). Participants were also provided with the opportunity to further clarify or add to research questions, if necessary, and recommend additional research questions. Participants were invited to complete this questionnaire by email on August 10, 2016. The questionnaire remained online for 4 weeks, and reminder emails were sent approximately 7 days after the initial invitation to individuals who had not yet completed the questionnaire.

Table 3 Phase II 5-point Likert scale
Full size table

Phase III: Consensus Meeting–Vetting and Ranking of Research Priorities

Participants

Rather than utilize a random sample that is representative of the target population, the Delphi method uses a consensus panel of those invested in the process and its outcomes. There is little agreement on the definition of an expert [3]. Therefore, for the purposes of the current study, all participants from the qualitative assessment and rating evaluation were invited to participate as well as representatives from patient advocacy groups, the MSTS, and OREF. One hundred one individuals were invited to participate. In response to numerous requests, mostly from international clinicians, we opted to also provide a web-based response system and conferencing capabilities, which allowed us to ensure that the consensus panel membership was geographically diverse. The consensus panel included a total of 44 individuals from 12 countries (Fig. 1). The panel was comprised of one patient representative, one clinical research manager, and 42 were orthopaedic surgeons. The OREF selected a representative to attend the meeting, who is also an orthopaedic oncologist. Of the orthopaedic surgeon members, half (50%) were between the ages of 30 and 40 years, and almost two-thirds (57%) had been in practice for no more than 10 years (Table 2).

Interventions

The consensus meeting was held on October 7, 2016. The meeting was facilitated by an experienced and independent meeting facilitator with extensive prior involvement in strategic planning initiatives. At this meeting, the members of the consensus panel were given the opportunity to discuss the eligible candidate research questions as a group. A semistructured agenda that had flexible time parameters to allow for discussion and questions was followed, as previously published [29]. The consensus meeting was recorded, and all consensus panel members were informed that it was being recorded before the start of the meeting.

Absolute Scoring Stage

Onsite participants were provided with an audience response system device to anonymously score research questions throughout the meeting. Remote participants were provided with access to anonymously score research questions through the Internet in real time throughout the meeting. All candidate research questions were individually discussed by members of the consensus panel, thereby providing an opportunity for members to reconsider their Phase II ratings in light of other members’ views. However, priority was given to the discussion of higher ranking tiers. For each question, the facilitator first asked the consensus panel for any remarks in support of the research question; then the facilitator asked for any comments against the research question before asking for any additional comments. When differences in the ratings from Phase II appeared to have resulted from ambiguity in the wording of a research question, the members also used this time to agree on revised wording. After each question’s discussion, the members anonymously assigned a score using the response system.

Final Ranking Stage

Utilizing a web-based questionnaire, the candidate research questions that met the predetermined criteria as outlined below were subsequently distributed to the consensus panel members who were asked to rank their top three research questions [29]. All individuals (100%) involved in the consensus meeting provided their final rankings.

Analysis

Phase I: Qualitative Assessment–Soliciting Research Questions of Interest

The questionnaire responses were compiled for review. Responses from the qualitative assessment were initially reviewed by an orthopaedic oncologist (MG) on the planning committee. Similar ideas were clustered together into emerging research questions, and duplicate responses were removed. This review was independently repeated by a health research methodology expert (NE) on the planning committee. The two reviewers then met to discuss any differences produced from these independent reviews and reach a consensus on the list of candidate research questions that would progress to the next phase.

Phase II: Rating Evaluation–Rating of Research Questions

The planning committee compiled the ratings for each research question as well as any additional research questions. The results of the questionnaire were reviewed to determine whether each candidate research question met the predetermined consensus thresholds (Table 4). Research questions that met either the inclusion or nonconsensus thresholds progressed to Phase III for review by the consensus panel. Candidate research questions that met the exclusion consensus threshold were not brought forward for review.

Table 4 Phase II consensus thresholds
Full size table

Phase III: Consensus Meeting–Vetting and Ranking of Research Priorities

Absolute Scoring Stage

Consensus panel members anonymously assigned a score on a 9-point Likert scale for each candidate research question (Table 5). A larger Likert scale was used during this stage because it provided a better opportunity for respondents to adequately report their opinions on the relative importance of each candidate research question. This was especially important during this stage considering the large number of candidate research questions that progressed for in-person vetting.

Table 5 Phase III scoring scale
Full size table

Once the scores were compiled, questions meeting one of the following predetermined criteria were brought forward for final ranking: 100% of respondents scored the candidate research question as a 7, 8, or 9 or at least 10% of respondents scored the research question as a 9. If none of the research questions met these criteria, it was also decided a priori that the top 10 scoring candidate research questions would be brought forward for final ranking [29].

Final Ranking Stage

Consensus panel members then ranked their top three research questions. The corresponding point system was used to determine the highest ranking candidate research questions (Table 6). We decided a posteriori that a multiple regression analysis of the consensus meeting data, as stated in the study’s protocol [29], was not appropriate given the relatively small final sample size and the large number of priority research questions brought forward by the participants.

Table 6 Phase III final ranking scoring system
Full size table

Results

Phase I: Qualitative Assessment–Soliciting Research Questions of Interest

In total, 175 candidate research questions were proposed by the respondents. After the qualitative assessment review, during which 106 duplicates were removed, it was determined that 69 candidate research questions would progress to the rating evaluation phase (Appendix 1 [Supplemental materials are available with the online version of CORR ®.]).

Phase II: Rating Evaluation–Rating of Research Questions

In total, 60 individuals (59 complete, one incomplete) participated in the rating evaluation, rating the 69 candidate research questions on five criteria. Four additional research questions were proposed (Note: after the consensus meeting, none of these four questions were ranked highly.). As per the consensus thresholds (Table 4), only one candidate research question met the inclusion threshold. Sixty-eight research questions met the nonconsensus threshold, and no research questions met the exclusion threshold. Therefore, 73 candidate research questions (69 candidate research questions plus four new suggestions) progressed to the final phase for consideration and assessment at the consensus meeting (Appendix 2 [Supplemental materials are available with the online version of CORR ®.]).

As a result of the large number of candidate research questions that met the conditions for advancement, we divided them into the following seven tiers in preparation for the consensus meeting: Tier 1–met inclusion threshold; Tier 2–met nonconsensus threshold (average consensus score ≥ 70%); Tier 3–met nonconsensus threshold (average consensus score = 65%–69%); Tier 4–met nonconsensus threshold (average consensus score = 60%–64%); Tier 5–met nonconsensus threshold (average consensus score = 50%–59%); Tier 6–met nonconsensus threshold (average consensus score < 50%); and Tier 7–new suggestions. This was done to minimize time constraints and prioritize discussion of candidate research questions that had higher degrees of consensus in the rating evaluation phase.

Phase III: Consensus Meeting–Vetting and Ranking of Research Priorities

Final Research Priorities

The final stage identified four research priorities with proposed study methodologies (Table 7): (1) Does less intensive surveillance of patients with sarcoma affect survival? (2) What are the survival outcomes over time for orthopaedic oncology implants? (3) Does resection versus stabilization improve oncologic and functional outcomes in oligometastatic bone disease? (4) What is the natural history of untreated fibromatosis?

Table 7 Priority research questions for future research in the field
Full size table

Absolute Scoring Stage

The final top research priorities were identified through the scoring of each research question. Two candidate research questions met the criteria for progression to the final ranking stage. This was considered a priori an insufficient number of candidate research questions to conduct the final ranking, so the mean score was calculated for each research question and the top 10 were brought forward as per protocol [29]. Because one research question that met the criteria for progression did not score in the top 10 by mean score, we decided that 11 candidate research questions would advance to the final stage of this phase (Table 8).

Table 8 Phase III: priority research questions advancing to final ranking stage
Full size table

Final Ranking Stage

After each consensus meeting participant provided their final rankings, the top priority research questions were identified as outlined previously (Table 7). The research question with the fourth highest score was also included in this list because the total number of panel members that ranked this question in their top three was higher than that of both the second and third ranking questions with a final score only marginally lower than that of the third ranking question.

Discussion

Previous work has demonstrated a lack of high-quality evidence to guide clinical decisions in orthopaedic oncology [5]. As a result of the rarity of bone and soft tissue tumors, multicenter prospective collaboration is imperative for broadly meaningful research and evidence-based advances in patient care [18]. However, although agreement exists pertaining to the importance of collaborative research and the need for higher quality research in orthopaedic oncology, research priorities remain unclarified. Consensus methods are being increasingly used to develop research agendas in various medical and surgical specialties [17, 20, 23]. Research agendas can assist professional groups in allocating finite research resources to clinical investigations likely to provide the greatest value. They can also provide individual researchers with guidance to help prioritize their own endeavors. To establish a research agenda specific to orthopaedic oncology, we brought together international stakeholders and conducted a modified Delphi process, which identified the following four priorities: (1) Does less intensive surveillance of patients with sarcoma affect survival? (2) What are the survival outcomes over time for orthopaedic oncology implants? (3) Does resection versus stabilization improve oncologic and functional outcomes in oligometastatic bone disease? (4) What is the natural history of untreated fibromatosis?

Our study has some limitations that should be recognized. Although diverse, our sample’s composition was still dominated by individuals from North America, so our results may reflect a North American perspective that is not applicable to all regions as a result of differences in access to health care, referral patterns, availability of medical interventions, and cultural acceptability of treatment. However, these imbalances in perspective may have been resolved through discussions with international colleagues during the consensus meeting. Furthermore, underrepresentation of distinct groups may have resulted from our sampling method. We invited all interested or active PARITY investigators and MSTS members to participate in this initiative. However, a substantial number of orthopaedic oncologists are not affiliated with either the PARITY trial or MSTS; therefore, they may not be adequately represented in our study. Although we made numerous attempts to contact orthopaedic oncology patient advocacy groups for their input, we were only able to connect with one patient advocate, who was contacted directly and who was interested in supporting orthopaedic oncology research. It was, therefore, fruitful to directly contact patients rather than patient advocacy groups. Nevertheless, future opportunities exist to include a greater number of patient advocates and underrepresented stakeholders such as during the development process of specific research protocols to address the identified priority research questions. Finally, our response rates (33% [117 of 351] for the initial invitation and 54% [61 of 114] and 53% [60 of 113] for the qualitative assessment and rating evaluation questionnaires, respectively) were somewhat lower than those achieved in similar studies in other medical specialties [20, 23]. However, our initial invitation allowed us to specifically select a population that is interested in collaborative research in the field. Considering that our objective was to determine the priority research questions in orthopaedic oncology to drive future collaborative prospective research, this bias may have been favorable and resulted in inherently collaborative participants providing many insightful ideas. Individuals who did not respond to the initial invitation may not have been interested in participating in any collaborative research, may not have expected this initiative to be fruitful, or did not prioritize this initiative above other academic interests. However, response bias was minimized by allowing individuals to participate in any phase of the study. Future studies that are aimed at more rigorously evaluating potential sampling biases should include European and Asian orthopaedic oncology societies.

Despite these limitations, the design of the current initiative was structured, thorough, transparent, and aimed to include all invited individuals who expressed an interest. Furthermore, the use of a modified Delphi process for this initiative maximized the benefits of two common consensus approaches—the classic Delphi method and the Nominal Group Technique. Throughout the initiative, anonymity in scoring was maintained, even while the moderated discussion took place; this provided the opportunity for the causes of disagreement to be explored without particularly vocal participants dominating the discussion and overpowering the opinions of others [8, 15]. Therefore, it is unlikely that consensus was forced on participants nor that the priority research questions identified reflect the perspective of any one participant. Finally, the composition of the participant group is also thought to influence the outcome of the consensus process. Other studies have previously demonstrated that homogenous samples select different options than heterogeneous ones considering the same choices [31]. To maximize the feasibility and generalizability of the identified research priorities, we assembled a group of participants from a broad range of healthcare systems, geographic regions, career focus and stage, and prior research experience.

The natural succession of the identification of research priorities is the development of an action plan to answer the priority research questions and to identify viable funding mechanisms to support these research efforts. Surveillance after sarcoma treatment is a subject that spans all disciplines in cancer care. The balance between the cost of intense surveillance with respect to resources and patient quality of life and the potential benefit to identify relapsed disease in a curable stage must take into account many important factors [7, 10, 19, 22, 35]. This is, therefore, a complex question that will require coordinated protocol development among a long list of stakeholders, the most important being the patients themselves. Puri et al. [26] have published the only relevant randomized controlled trial (RCT) to date in the sarcoma field. This single-center study concluded that overall 3-year survival and disease-free survival were no different between patients with sarcoma who had more intensive surveillance (CT scans) and those with less (chest radiographs) [26]. However, as a result of the sample size, this trial could not conclusively demonstrate noninferiority in overall survival for a 6-monthly followup visit interval against a 3-monthly interval [26]. In addition, because this was a single-center study, generalizability of the results to other centers and countries is limited. A large international collaborative network will be required to implement a RCT protocol that addresses both the implications for healthcare systems and the preferences of patients with sarcoma. Government-level funding and large-scale grants from cancer research funding agencies would presumably be required to ensure the success of this effort.

Patients with sarcoma are often teenagers and young adults [24]. Therefore, those who survive after treatment may live for many decades. The implants used to reconstruct the extremities after tumor excision are prone to failure for a variety of reasons including infection, fracture, and aseptic loosening [14, 30]. Identifying emerging trends in the characteristics of those requiring revision surgery specific to the available implants may help identify risk factors so that the appropriate resources can be allocated to mitigate those risks. The success of the National Joint Registry (NJR) for England, Wales, Northern Ireland, and the Isle of Man in identifying implant designs that were failing at a proportionately higher rate illustrates the benefits of long-term observational data collection in other orthopaedic specialties [33]. An industry- or government-funded prospective implant registry specific to orthopaedic oncology would likely be the ideal approach to address this research priority.

With newer targeted systemic therapy, many patients with metastatic bone disease from primary carcinomas such as breast and renal cancers are living months and years longer with their disease [4, 32]. Therefore, a more aggressive surgical approach to resect entire bone metastases en bloc, as opposed to stabilization without tumor excision in its entirety, may be warranted to improve survival and possibly quality of life. This research priority could be answered with a multicenter RCT with a focus not only on survival rates, but also patient-reported outcomes specific to quality of life. However, such an endeavor would likely require generous government-level and nonprofit support. Similarly, there has been a paradigm shift in the treatment of fibromatosis (desmoid tumors), a benign diagnosis with aggressive local behavior. Once thought to be a surgical disease, it is now understood through retrospective data that with unacceptably high recurrence rates, “watchful waiting” may be the most appropriate management strategy for most patients [9, 12]. However, patients must be convinced that, if left untreated, the natural history of this disease is benign. A prospective multicenter cohort study that follows patients for symptoms and progression would assist in directing the care of patients in future generations. The Desmoid Tumor Foundation would be an ideal funding mechanism for this important study in orthopaedic oncology.

We have identified research priorities for international prospective research in orthopaedic oncology by conducting a three-step modified Delphi process. Top research priorities in orthopaedic oncology include evaluating different postoperative surveillance regimens in patients with extremity sarcoma, understanding the survival outcomes of orthopaedic oncology implants over time, evaluating whether resection versus stabilization improves outcomes in patients with oligometastatic bone disease, and understanding the natural history of untreated fibromatosis. These priority research questions highlight areas where international stakeholders have agreed by consensus that further knowledge would have a significant impact on the clinical care of orthopaedic oncology patients. Therefore, the MSTS and other professional orthopaedic oncology societies whose missions are to promote the advancement of orthopaedic oncology science and patient care may enable and support research efforts that address these priorities. These societies could do so by helping to facilitate the establishment of Working Groups and coordinating Working Group meetings, preferably by designating specific meeting times at annual conferences, to explore the next steps and develop action plans. The limited research funds of these professional societies could also be preferentially allocated to studies concentrating on one of the identified priorities. This research agenda could also be used by researchers to focus their research efforts and provide a rationale in competitive grant applications when applying for the financial resources to support endeavors directed at answering these priority research questions in orthopaedic oncology.