Reliability, validity and responsiveness of the Dutch version of the AOSpine PROST (Patient Reported Outcome Spine Trauma)

To validate the Dutch version of AOSpine PROST (Patient Reported Outcome Spine Trauma). Patients were recruited from two level-1 trauma centers from the Netherlands. Next to the AOSpine PROST, patients also filled out SF-36 for concurrent validity. Descriptive statistics were used to analyze the characteristics. Content validity was assessed by evaluating the number of inapplicable or missing questions. Also floor and ceiling effects were analyzed. Internal consistency was assessed by calculating Cronbach’s α and item-total correlation coefficients (itcc). Spearman correlation tests were performed within AOSpine PROST items and in correlation with SF-36. Test–retest reliability was analyzed using Intraclass Correlation Coefficients (ICC). Responsiveness was assessed by calculating effect sizes (ES) and standardized response mean (SRM). Factor analysis was performed to explore any dimensions within AOSpine PROST. Out of 179 enrolled patients, 163 (91.1%) were included. Good results were obtained for content validity. No floor or ceiling effects were seen. Internal consistency was excellent (Cronbach’s α = 0.96, itcc 0.50–0.86), with also good Spearman correlations (0.25–0.79). Compared to SF-36, the strongest correlation was seen for physical functioning (0.79; p < .001). Also test–retest reliability was excellent (ICC = 0.92). Concerning responsiveness analysis, very good results were seen with ES = 1.81 and SRM = 2.03 (p < 0.001). Factor analysis revealed two possible dimensions (Eigenvalues > 1), explaining 65.4% of variance. Very satisfactory results were obtained for reliability, validity and responsiveness of the Dutch version of AOSpine PROST. Treating surgeons are encouraged to use this novel and validated tool in clinical setting and research to contribute to evidence-based and patient-centered care.


Introduction
Measurement of the results of interventions on the individuals' health-related quality of life is not only relevant for optimal treatment strategies but also from the standpoint of cost-effectiveness. However, the outcomes of spine trauma patients have traditionally been limited to reporting of mortality and neurologic deficits or expressed with instruments designed for chronic conditions [1][2][3]. The use of different outcome measures, which were not designed for spine trauma, contributes to the ongoing controversies on the optimal treatment of this specific patient population [4,5].
To address this void, the AOSpine Knowledge Forum Trauma initiated a project to develop and validate a disease-specific outcome measure for spine trauma patients: the Patient Reported Outcome Spine Trauma (AOSpine PROST). The systematic approach and Core Set development methodology of the International Classification of Functioning, Disability, and Health (ICF) of the World Health Organization (WHO) was used as the basis for the development of the tool [6,7]. In a preparatory phase, three studies aimed to identify ICF categories relevant to measure outcomes of traumatic spinal column injuries from different perspectives: research, experts and patients. A fourth study investigated various question and response formats for use in AOSpine PROST. In the next phase, a formal consensus process integrated evidence from the preparatory studies and expert opinion and let to the selection of 25 ICF categories as 'core categories' and the appropriate response scale. Subsequently, a draft Dutch version of the tool was developed by clustering the 25 core ICF categories into 19 items and implementing those into the selected 0-100 Numeric Rating Scale (NRS-101). After pilot testing, a definitive Dutch version to be validated was developed [8] In the developmental process and initial validation, we sought to focus on patients sustaining injuries to their spinal column and excluded completely paralyzed and polytrauma patients, to identify specific problems related to spine trauma. This study aimed to validate the Dutch version of the AOSpine PROST among traumatic spinal column injury patients. More specifically, the psychometric properties were investigated to assess its reliability, validity and responsiveness.

Target population
Adult (≥ 18 years) traumatic spinal column injury patients who were capable of understanding and adequately filling out the questionnaires were included. Polytrauma patients (Injury Severity Score (ISS) > 15) and patients with complete paralysis (American Spinal Injury Association (ASIA) impairment grade A or B at discharge or transfer from hospital) were excluded.

Instruments
For the purpose of concurrent validity, the AOSpine PROST should be compared to a validated outcome instrument designed for patients with traumatic spinal column injuries. However, such instrument is not available. Therefore, a generic health-related quality-of-life (HRQoL) outcome instrument, the Medical Outcomes Study 36-item Short-Form Health Survey (SF-36), was also administered to patients as reference standard.
The AOSpine PROST consists of 19 questions on broad aspects of functioning. (Appendix 1 shows the translated and cross-cultural adapted English version.) Each item has a 0-100 numeric rating scale, with 0 indicating no function at all and 100 the functional level before trauma. The scale is supported by smileys at both ends of the ruler. The SF-36 includes 36 items measuring 8 health subscales, and is widely used to measure general health status of patients with different diseases [9]. The two summary measures, the physical component summary (PCS) and mental component summary (MCS), are calculated from the 8 health subscales. Scores range from 0 to 100, with higher scores indicating better health status. The Dutch version of SF-36 has shown good validity results [10,11]. These questionnaires (AOSpine PROST and SF-36) along with a limited number of additional questions were administered to the patients as one questionnaire via an online system. The additional questions aimed to explore the presence of irrelevant question in AOSpine PROST, the absence of relevant questions and patients' self-reported degree of recovery.
The health professionals participating in the study were asked to complete background data, consisting of sociodemographic characteristics and trauma-related variables and to make an assessment of patient's degree of recovery based on clinical and radiological assessments (not recovered at all, somewhat-, mainly-, and completely recovered).

Study procedures
Patients were recruited from two level-1 trauma centers in the Netherlands: University Medical Center, Utrecht (UMCU), and Radboud University Medical Center, Nijmegen (RUMC). The study consisted of two arms: test-retest and responsiveness. For the test-retest part, eligible patients who were seen at the outpatient clinic within 13 months post-trauma were invited to participate while in the responsiveness arm patients were recruited shortly

3
before discharge from hospital. After informed consent, patients received an email with a link to the questionnaire or postal mail with a login code. For the purpose of test-retest, one week after completion patients were asked to fill out the same questionnaire once more. In the responsiveness arm, the questionnaire was administered three times: at 2-week, 6-week and 3-month post-trauma. If it was not completed within 3 days, patients received a reminder via email or telephone.

Statistical analysis
Patient characteristics were analyzed using descriptive statistics and frequency analysis. Content validity was assessed by evaluating the number of inapplicable questions and the responses to the open question if any question was missing in AOSpine PROST. Also floor and ceiling effects were analyzed, which could occur if > 15% of the patients achieve the lowest or highest possible score, respectively. The mean total scores in correspondence to the degree of recovery, both as reported by patients and assessed by the clinicians, were analyzed using Welch's ANOVA.
Concurrent validity between AOSpine PROST and SF-36 was analyzed using Spearman correlation coefficient (r s ). The r s can take values from + 1 to − 1, with + 1 indicating a perfect association, 0 no association and − 1 a perfect negative association of ranks [12]. Concurrent validity is supported if the coefficient is at least 0.70 [13].
The internal consistency was assessed by calculating Cronbach's α and item-total correlation coefficients. It is suggested that the value of α should be > 0.70 for acceptance as satisfactory internal consistency [13,14]. Also pairwise Spearman correlation was performed to investigate the correlation between AOSpine PROST items.
Test-retest reliability was assessed using Intraclass Correlation Coefficients (ICCs), with good and excellent reliability indicated by values of 0.70 to 0.85 and > 0.85, respectively [13].
Responsiveness was analyzed using effect size (ES) and standardized response mean (SRM). ES was reflected as the change in score divided by the standard deviation (SD) at 2 weeks. In general, ES > 0.8 is regarded large based on Cohen's criteria [15]. The SRM is the change score divided by the SD of the change score.
Finally, exploratory factor analysis was performed to identify the dimensionality of the AOSpine PROST. Factors with an Eigenvalue greater than 1 were selected, and selection was confirmed by visual inspection of the scree plot. The factor loading of each item after varimax rotation was examined.
Statistical analyses were conducted using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA).

Patient characteristics
Out of 179 patients, a total of 163 (91.1%) were enrolled. Five patients from RUMC and 11 from UMCU were excluded as they did not complete any questionnaire. Out of the included patients, 14 (8.6%) were from RUMC and

Content validity
The mean registered time by the online system to complete AOSpine PROST was 7.0 ± 4.3 (range 1-30) minutes. No item in the questionnaire was indicated as inapplicable or irrelevant. Eight (4.9%) patients reported to have difficulties in filling out AOSpine PROST questions. However, in their clarification patients described in more detail their limitations relevant to a specific question rather than to have a practical difficulty or misunderstanding of the questions. Twentythree (14.1%) patients responded positive that a relevant question was missing. A detailed analyses of those responses revealed that no specific question was missing. However, patients did indicate that it was somewhat unclear whether the questions should be solely answered for the spine fracture or also to other sustained fractures.
As no patient had the minimum and only one patient (0.6%) the maximum total score, no floor and ceiling effects were observed. AOSpine PROST scores relative to the degree of recovery were more strongly related to the indication by patients (p < 0.001) compared to the clinicians' assessments ( Table 2).

Internal consistency
The internal consistency of AOSpine PROST total score was excellent (Cronbach's alpha = 0.960). With a range of 0.50 to 0.86 item-total correlation showed good results ( Table 4). The lowest values were seen for 'Urinating' (0.50) and 'Bowel movement' (0.58). Cronbach's alpha did not improve (0.95-0.96) when an item was removed. As shown in Appendix 2, Spearman correlation between AOSpine PROST items showed also good results (0.25-0.79).

Test-retest reliability
A total of 64 patients were included in the test-retest arm ( Table 4). The mean time after trauma was 5.6 ± 4.1 (range 0-13) months when completing the first questionnaire. The time between the first and second administrations was 9 ± 2.3 (range 4-14) days. Excellent test-retest reliability was seen for the total score (ICC = 0.92). When looking into detail per item, all had acceptable to excellent reliability results expect for 'Back and/or neck pain' item (ICC = 0.55).

Responsiveness
Out of initially enrolled 96 patients in the responsiveness arm, 59 (61.5%) had completed the questionnaires at all time points and could be included in the responsiveness analysis. AOSpine PROST mean and median scores for the different time points showed gradual increasing over time (Table 5), as would be expected with gradual recovery over time. The change in scores from 2 weeks to 3 months are shown in Table 6. The AOSpine PROST scores showed significant (p < 0.001) larger changes compared to SF-36. Also, the largest ES and SRM were seen for AOSpine PROST (ES = 1.81 and SRM = 2.03). Table 7 shows the changes in AOSpine PROST scores compared to patient-reported degree of recovery; a higher degree of recovery is reflected by a higher change in score with larger ES and SRM.

Factor analysis
Factor analysis revealed that two factors had an Eigenvalue > 1, i.e., possible identification of two dimensions across AOSpine PROST items (Factor 1 and Factor 2). These factors had an Eigenvalue of 11.0 and 1.4, and explaining 58.1% and 7.3% of the variance, respectively. Rotated factor analysis for the items among those two factors showed that most items load high on Factor 1 and low on Factor 2, indicating that those items considerably contribute to the dimension represented by Factor 1 (see Table 8 and Fig. 1). Contrarily, the items 'Urinating' and 'Bowel movement' load high on Factor 2 and low on Factor 1. No item loaded low on both factors which indicates that no possible third factor is expected.

Discussion
This study investigated the psychometric properties of the Dutch language version of the AOSpine PROST (Patient Reported Outcome Spine Trauma), a novel patient-reported outcome measure specifically designed for spine trauma patients. Although a number of outcome instruments have either been developed and validated, or used in, individuals with traumatic spinal cord injury, these tend to focus on the impact of paralysis, e.g., Spinal Cord Independence Measure (SCIM) and Functional Independence Measure (FIM) [3,16]. A unique approach in AOSpine PROST is asking patients to recall their pre-injury level of health, more specifically to compare their current function (0) with their pre-trauma level of function (100). This feature might have contributed to the good responsiveness of the tool. Comparing the health and function of spinal trauma patients with normative standardized data is not straightforward because the characteristics of spine trauma patients may very well deviate from those of the general population [17,18]. This explains our findings of in general weak correlation between AOSpine PROST and SF-36. Also, various patient characteristics may influence their outcomes, e.g., cause of trauma and comorbidities. Excellent results were obtained for internal consistency and test-retest reliability. Very high Cronbach's alpha values were obtained for all items with exception of 'Urinating' and 'Bowel movement.' This is in some contrast with findings from the factor analysis. When applying this to spine trauma patient population, these functions are likely to be adversely affected in spinal cord injured patients [19,20]. It is hypothesized that this bidimensional model will no longer be applicable when the tool is tested among ASIA A and B patients. As this will be performed in the next phase, it was decided not to make any changes to the current version of AOSpine PROST. Also redundancy of certain items will be investigated in future studies, as well as a detailed analysis per item including larger patients samples. Another interesting finding was the discrepancies in AOSpine PROST scores when related to patient-reported degree of recovery compared to the assessments of the clinicians. This supports the authors' aim to also develop an outcome instrument from the perspective of the treating surgeons: AOSpine CROST (Clinician Reported Outcome Spine Trauma) using the most relevant clinical and radiological parameters [21].
In a preparatory phase of the AOSpine PROST project, a systematic literature review found SF-36 to be the most frequently used generic instrument in studies including spine trauma patients [1]. This finding is supported by several other studies [22][23][24]. We found good concurrent validity for AOSpine PROST when compared to physical component of SF-36 scores and satisfactory but lower concurrent validity for mental SF-36 scores. This indicates that AOSpine PROST total scores reflect more the patient's experienced physical than the mental health. As was hypothesized further, the responsiveness analysis yielded much better results for AOSpine PROST with the highest ES and SR.
This study has several limitations. The ability to detect minimal clinically important differences has not been investigated completely as a larger patient sample would be required. We certainly aim to investigate this specific aspect in future studies. Second, the contribution of included patients was not equal from the two centers. This was due to the combination of different amounts of spine trauma exposure and practical difficulties in the enrollment process. Finally, somewhat heterogeneous patient population was seen with a relative high mean age and percentage of males. This aging spine trauma patient population is in general seen in the clinics nowadays and also underpinned by various publications [22,25]. Also, slight differences were seen between the recruited patients from the two centers. Further investigation of subgroups such as age, specific injuries and severity of spinal cord injury would, however, still be very interesting and will be performed in future studies. In conclusion, this study aimed to analyze the psychometric properties of the Dutch version of the AOSpine PROST and showed very satisfactory results for reliability, validity and responsiveness. In future studies, the applicability of the tool to complete paralyzed patients will also be investigated. The aim is to translate and cross-cultural adapt the AOSpine PROST into many languages in order to make it available for spine trauma patients around the world. Treating surgeons are encouraged to use this novel and validated tool in clinical setting and research to contribute to further evidence-based and patient-centered spine trauma care.
Funding This study was organized and funded by AOSpine through the AOSpine Knowledge Forum Trauma, a focused group of international Trauma experts. AOSpine is a clinical division of the AO Foundation which is an independent medically-guided not-for-profit organization. Study support was provided directly through the AOSpine Research Department.

Compliance with Ethical Statement
Conflict of interest The authors declare that they have no conflict of interest.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat iveco mmons .org/licen ses/by/4.0/.