figure a

New methods for ACL reconstruction have been developed to improve the historical results of surgical treatment and it is important to critically evaluate these improvements in surgery. In addition, the value of non-operative treatment needs to be carefully investigated to determine the merits over surgical treatment. In the field of orthopaedic sports medicine, there are relatively few Level I randomized clinical trials that have investigated the methods for and benefits of reconstructing the anterior cruciate ligament (ACL) or to compare non-operative treatment to surgical reconstruction.

A well-done Level I randomized clinical trial requires substantial planning, collaboration, commitment and personnel and financial resources to successfully complete the study. In planning a clinical trial, selection of a patient-centred outcome measure is an important consideration. Patient-centred outcomes are those outcomes that are important to patients. Based on a Delphi study involving clinical experts conducted by the American Academy of Orthopaedic Surgeons (AAOS) Management of Anterior Cruciate Ligament Injuries Evidence-Based Clinical Practice Guidelines Work Group [3], critical and important outcomes for ACL injuries were found to include patient satisfaction, pain, function, ACL clinical failure, quality of life and return to play. Often these outcomes are assessed with the use of patient-reported outcome (PRO) measures, which are designed to assess the patient’s perception of their symptoms (pain, instability, etc.), activity limitations (difficulty walking, running, cutting, pivoting, etc.) and participation restrictions (restrictions participating in sports, recreation, work, etc.).

The most commonly used PRO measures for assessing the outcome of injury and treatment of ACL injuries are the International Knee Documentation Committee Subjective Knee Form (IKDC-SKF) and the Knee injury and Osteoarthritis Outcome Score (KOOS). In a systematic review of the 50 most frequently sited Level I clinical trials investigating the treatment of ACL injuries, the IKDC-SKF was used as an outcome measure in 28 studies, while the KOOS was used in 6 studies [1]. However, use of the KOOS (and pivot shift test) was associated with higher quality studies.

Because Level I studies constitute the highest level of evidence, it follows that the end-points used by these studies must also be selected with care and scrutiny based on their ability to provide relevant, reliable, valid and responsive measures of knee function in persons following ACL-R. More recently, to ensure that a PRO measure is truly patient-centred, there has been a greater emphasis placed on content of the PRO measure in the generation and selection of items. In considering content of a PRO, it is important to clearly specify the construct measured by the PRO measure. Once the construct is specified, it necessary to determine if the construct measured by the PRO is adequately represented by the items included on the PRO. This includes determination that collectively the items represent all important aspects of the construct (i.e. there is no under-representation of the construct) and that there are no items that are unrelated to construct (i.e. there no items that are irrelevant to the construct). Further, the comprehensibility and understandability of the items, including the stem (or question) and response options should also be evaluated. If a PRO measure lacks important content or includes content that is irrelevant to the construct, the resulting score may still demonstrate acceptable psychometric properties including reliability, validity and responsiveness; however, the meaning of the score may be uncertain. Further, under-representation of the construct or inclusion of irrelevant items may result in floor or ceiling effects and lack of responsiveness of the measure.

Historically, content of a PRO measure has been evaluated psychometrically by determining the presence of floor and ceiling effects and internal structure of the item responses using factor analytic methods and coefficient alpha to determine internal consistency. Today, cognitive interviews and focus groups with representative patient-stakeholders are recommended in the process to generate, review and select items during the development phase of the PRO measure. For legacy PRO measures, including the KOOS and IKDC-SKF, which were developed prior to the emphasis on the use of stakeholder engagement to establish content of the PRO measure, it is recommended that the items be reviewed by patient-stakeholders to determine if there is under-representation of the construct, inclusion of irrelevant items or difficulties with comprehensibility. If this process results in the creation of new items, removal of existing items, changes in the wording of the items, including the response options or recall period, then further evaluation of the content and psychometric properties of the revised PRO measure would be required before its adoption to replace the legacy measure.

Given the importance of the content of PRO measures, Hansen et al. [13] used the COSMIN Risk of Bias Checklist [20, 22, 27] to evaluate evidence for the content of five commonly used PRO measures by the orthopaedic sports medicine community, including the KOOS and IKDC-SKF. The authors concluded that development of the KOOS and IKDC-SKF was inadequate and both measures possess insufficient content for their target populations.

The KOOS is a comprehensive knee-specific PRO measure that was developed based on the Western and McMaster Universities Osteoarthritis Index (WOMAC), which was originally intended to evaluate symptoms and function of patients between the ages of 16 and 65 years with a focus on hip and knee osteoarthritis [23]. As evidenced by inclusion of the KOOS in the Scandinavian knee ligament registries, adoption of the KOOS has been widespread, in part due to support from influential orthopaedic surgeons and physical therapists despite limited evidence to support the content and psychometric properties of the KOOS.

Recent studies investigating the content validity of the KOOS to assess outcomes for individuals with an ACL injury have noted the issue of inclusion of items that are irrelevant to individuals with an ACL injury [5, 7, 13]. Further, review of the KOOS reveals under-representation of content important for individuals with an ACL injury, including the lack of items that represent the presence of instability or weakness and difficulty with activities such as starting and stopping quickly, sprinting, landing from a jump and changing directions quickly. Using a PRO measure with limited relevance and inadequate coverage of items specific to ACL injury, researchers risk the inability to detect important treatment effects, leading to false-negative results [5, 24, 25].

Of the five KOOS subscales, the sports/recreation and quality of life (QoL) subscales are the most relevant to individuals with ACL injury and are also the most important for achieving a patient acceptable symptom state (PASS) in terms of the KOOS [4, 21]. In contrast, the Pain and Activities of Daily Living (ADL) subscales have been found to be irrelevant for individuals with an ACL injury and demonstrate low levels of responsiveness for this patient population. [13, 21, 28]. Moreover, the Pain and ADL KOOS subscales demonstrate ceiling effects for patients with ACL injury, which leads to low discriminatory power in knee function following ACL-R in patients who attain scores at the higher end of the measurement spectrum [17]. Importantly, the Pain and ADL domains of the KOOS lack the ability to detect change over time in knee function 1 year after surgery or injury and fail to address factors that determine return to sport (RTS) after ACL injury [19, 24, 28].

Recognizing some of the shortcomings of the KOOS, efforts have been made to develop the 11-item KOOSglobal [17] and the composite KOOS4 [9]. While eliminating items and subscales that are irrelevant for individuals with an ACL injury may help to eliminate ceiling effects and the lack of responsiveness, these revisions have not been based on the input from individuals with an ACL injury. As such, these revised scales likely still suffer from under-representation of the construct and inclusion of irrelevant items. Further, the psychometric properties of these revised KOOS scores have not been validated for patients with ACL injuries and may, therefore, be unreliable study endpoints [18]. It is possible that compensating for the domains of the KOOS that are less than optimal for the assessment knee function following ACL injury may be a step in the wrong direction.

It is now clear that the PRO measures used in ACL clinical practice and research must be carefully considered to avoid inaccurate results and inadequate conclusions regarding the treatment of individuals with an ACL tear. While the ultimate goal should be the development and validation of PRO measures with strong evidence for content and psychometric properties specific to the ACL-injured patient, current alternatives to the KOOS should not be forgotten.

The International Knee Documentation Committee-Subjective Knee Form (IKDC-SKF) is the most common PRO measure used in studies investigating surgical reconstruction and/or non-operative treatment for ACL injuries [1, 14, 16]. The IKDC-SKF is a knee-specific PRO measure that was designed as an evaluative measure to detect improvement or deterioration in symptoms, function and sports activity for individuals with a variety of knee conditions, including ligament and meniscus injuries, articular cartilage lesions and patellofemoral pain [14].

Recently, development of the IKDC-SKF has been deemed to be inadequate with insufficient evidence for content validity, largely due to the lack of patient input in the generation of the initial pool of items [13]. However, item content for the IKDC-SKF was developed through review of existing knee-related PRO measures and with input from the International Knee Documentation Committee that consisted of highly recognized orthopaedic sports medicine surgeons from North America, Europe and the Pacific Rim. The initial pool of items consisted of 27 items related to symptoms, 8 items related to function during activities of daily living, 4 questions related to function during sports activities, 3 questions related to current function of the knee and 5 questions related to participation in sports and/or work. The initial pool of items was pilot tested (n = 144 patients) and field tested (n = 222 patients). By considering the content and statistical properties of the items, the data from the pilot and field testing were used to select the final set of 18 items for the IKDC-SKF.

Extensive evidence for internal consistency, test–retest reliability, construct validity, and responsiveness [8, 10, 14, 15] as well as normative data [2] and the Patient Acceptable Symptom Score [21] has been provided to support interpretation of the IKDC-SKF Scores. Further, evidence for content of the IKDC-SKF has been provided by ratings of the frequency of occurrence and importance of the item content by patients with ACL insufficiency or meniscus tears [26] and following ACL reconstruction [11, 28] or articular cartilage repair [12]. As a result of its widespread use and psychometric data supporting interpretation of the IKDC-SKF score, in 2019 the Panther ACL Consensus Conference endorsed the IKDC-SKF as the recommended PRO measure to assess outcome for individuals with an ACL tear [25].

To address the concerns related to the lack of evidence for content of the IKDC-SKF, consistent with the recommendations by Hansen et al., the American Orthopaedic Society for Sports Medicine and the American Board of Orthopaedic Surgery has funded a study that will conduct cognitive interviews with persons with a variety of knee complaints to determine item relevance and clarity and to identify gaps in item content coverage to measure the construct of symptoms, function and sports activity for individuals with a variety of knee conditions, including ligament and meniscus injuries, articular cartilage lesions and patellofemoral pain. Additionally, fit of an item response theory (IRT) model and a simulated computer adaptive test (CAT) will be performed to determine if use of IRT and CAT can enhance the efficiency of administering the IKDC-SKF without adversely affecting the resulting IKDC-SKF scores. If this work leads to the creation of new items, removal of existing items, or changes in the wording of the items, then further evaluation of the psychometric properties of the revised IKDC-SKF will be undertaken.

Recent reports have drawn attention to a relatively newly developed PRO measure, the Knee Numeric-Entity Evaluation Score ACL (KNEES-ACL), created for the specific assessment of ACL injury [6, 7, 13]. In patients before and up to 12 months after ACL-R, subscales of the KNEES-ACL have demonstrated superior responsiveness compared to the KOOS subscales and IKDC-SKF [7]. While use of the KNEES-ACL is currently limited, it is an ACL-specific, patient-generated and psychometrically validated PRO measure warranting further consideration in future clinical ACL studies.

Further improvements in the KOOS and other existing PRO measures that are used to measure ACL outcomes are necessary to enhance the evidence base for surgical and non-surgical treatment of ACL injuries. Despite its widespread use, there are some limitations related to the content of the KOOS as a measure of patient-reported outcome following surgical or non-surgical treatment of ACL injuries. The issues of inclusion of irrelevant content and omission of important content leads to problems with floor and ceiling effects and lack of responsiveness when using the KOOS to detect the effects of treatment of ACL injuries. Ultimately, identification and universal endorsement of an accepted PRO measure that has strong evidence for content as well as for reliability, validity and responsiveness needs to be adopted to serve as the primary outcome for all ACL clinical trials and knee ligament registries.