Background

Restless Legs Syndrome (RLS) is a sensorimotor disorder that can substantially disrupt sleep. It affects 5-10% of the general population [1], predominantly women [1, 2], and has been characterized as chronic in individuals with moderate-to-severe symptoms requiring treatment [3]. The criteria that define RLS include an urge to move the legs due to unpleasant or uncomfortable sensations during periods of rest, the relief of symptoms through leg movements, and symptoms that follow a distinct circadian pattern [3]. The onset of RLS symptoms typically begin in the late evening and may persist through the night time hours. Thus, the primary morbidity in RLS is sleep disruption, the major reason patients cite for consulting their physicians [2, 3].

Sleep-related problems, such as trouble initiating or maintaining sleep and experiencing disturbed, non-restful or non-refreshing sleep, are more prevalent in patients with RLS than in those without RLS [4]. Nearly 90% of patients with RLS experience at least 1 sleep-associated symptom and over 40% of patients identified sleep-associated issues as the most bothersome RLS-related symptom [5]. Nearly 70% of patients with twice weekly RLS symptoms needed more than 30 minutes to fall asleep and 60% had more than 3 awakenings per night. On average, patients with severe RLS symptoms sleep only 4 to 5 hours per night [5].

Because of the subjective nature of RLS symptoms and the impact of these symptoms on sleep, patient-reported outcomes (PROs) play a prominent role as study endpoints in clinical trials investigating RLS treatments. However, for PROs to be credible, they must be subjected to a rigorous inspection of their measurement properties using measurement theory and standard psychometric assessments. While there were several pre-existing sleep instruments to choose from (i.e., MOS Sleep Questionnaire) most were limited in their use in RLS research because they were not disease specific, or were more difficult to administer within a clinical trial setting (i.e., clinician-administered). The hope was to be more responsive to the Food and Drug Administration's (FDA) recently emphasized expectations around patient reported outcome measures and the need to demonstrate the validity and reliability of such scales when used to measure endpoints in clinical trials [6].

The Post Sleep Questionnaire (PSQ) is a, 5-item PRO instrument designed to evaluate sleep disturbance in subjects with RLS. This study was a post-hoc evaluation of the validity of the PSQ, Version 2, using data from 2 well-controlled clinical trials in subjects with moderate-to-severe RLS [7, 8].

Methods

Data source

This study used pooled data from two 12-week, multicenter, randomized, double-blind, placebo-controlled clinical trials (XenoPort, Inc. protocols XP052 [ClinicalTrials.gov Identifier NCT00298623] and XP053 [ClinicalTrials.gov Identifier NCT00365352]) that evaluated the efficacy and tolerability of gabapentin enacarbil (GEn) for the treatment of moderate-to-severe primary RLS [7, 8]. The co-primary endpoints for the 1200 mg and placebo groups in both studies were the reduction in total IRLS score from baseline to Week 12 on the International Restless Legs Scale (IRLS) [9] and the proportion of responders rated by investigators as "very much improved" or "much improved" on the Clinical Global Impression-Improvement [CGI-I] scale) [10]. The secondary endpoints evaluated subject-rated improvements in sleep, mood, and quality of life.

The studies were of similar design. Subjects were randomly assigned for 12 weeks of treatment to GEn 1200 mg or matching placebo (XP052, XP053), with study XP053 also examining a third treatment group, GEn 600 mg, as a secondary endpoint compared with placebo. These studies demonstrated statistically significant efficacy with both GEn doses compared with placebo for reduction of RLS symptoms and improvements in sleep, mood, and quality of life [7, 8].

Study population

Study methodologies have been published elsewhere [7, 8]. Briefly, eligible subjects were ≥18 years of age, diagnosed with primary RLS, having RLS symptoms ≥15 nights of the month prior to study enrollement and for ≥4 of 7 consecutive nights in the week prior to their baseline assessment, and had a total RLS severity score of ≥15 (i.e., moderate-to-severe severity). If subjects were receiving RLS treatment at screening, then they must have had a symptom frequency of ≥15 nights per month prior to treatment initiation. A 2-week washout period prior to the baseline assessments was required for dopamine agonists, gabapentin, opioids, and benzodiazepines.

Subjects with non-RLS-related sleep disorders (e.g., sleep apnea), a history of RLS symptom augmentation or early-morning rebound with previous dopamine-agonist treatment, neurological disease or movement disorders other than RLS (e.g., diabetic neuropathy, Parkinson's disease, multiple sclerosis, dyskinesias, and dystonias), or other medical conditions that could confound study results were excluded.

PSQ

Figure 1 displays the 5 items comprising the PSQ, the sleep domains corresponding to each question, the associated response categories, and the scoring for each. The PSQ was developed to evaluate sleep disturbance in subjects with primary RLS over the past weeks using 4 Likert-type scale questions and 1 quantitative, open-ended question. The PSQ assesses several single-item sleep domains, including overall sleep quality, overall daytime functioning, frequency of night time RLS symptoms, and RLS-related sleep disturbances and latency. Higher PSQ scores indicate worse sleep.

Figure 1
figure 1

The 5-item PSQ with assessed domains, possible responses, and item scores. Higher scores indicate greater sleep difficulties associated with RLS.

Full size image

Validated sleep instruments

Seven PRO instruments were completed as part of the clinical trials assessments. Five of the instruments were previously validated and were used in the present study to validate the PSQ: 1) IRLS Rating Scale comprises 10 questions that assess symptom severity and frequency on a 40-point scale (0 = no symptom; 40 = very severe symptoms); subjects were also classified into RLS symptom severity categories using the IRLS total score (mild = 0-10, moderate = 11-20, severe = 21-30, and very severe ≥31) [11]; 2) investigator- and subject-rated CGI-I, evaluates improvements in any disease on a 7-point scale (1 = "very much improved"; 7 = "very much worse") [10]; 3) Profile of Mood States (POMS) is a 30-item scale that assesses mood in 6 domains and 1 total overall mood score, with higher scores indicating a more negative mood state [12]; 4) Medical Outcomes Study (MOS) Scale-Sleep assesses sleep constructs, including sleep initiation, maintenance, perceived adequacy, somnolence, respiratory impairments, and regularity in subjects with health comorbidities, such as RLS [13–16]; and 5) RLS Quality of Life Questionnaire (RLSQoL), a 100-point scale that assesses the direct impact of RLS on the daily life, emotional well-being, social life, and work-life of the subject, with lower scores indicating better quality of life [17, 18]. The remaining 2 instruments were diaries that provided important subject response information: 1) the 24-hour RLS Record assesses the frequency of RLS symptoms in half-hour increments during a 24-hour period and 2) the 7-Day RLS Symptom Diary assesses the frequency of nightly symptoms over a 1-week period.

Analyses

Analyses were conducted on pooled data irrespective of treatment assignment, as baseline assessments were used to generate the psychometric indices. Data from 540 subjects were included in these analyses: 220 from XP052 and 320 from XP053. As shown in Table 1, there were no major differences between the 2 study populations at baseline in terms of age, gender, ethnicity, or disease severity.

Table 1 Baseline demographic and characteristics of enrolled subjects from two clinical trials conducted in subjects with primary RLS
Full size table

The following psychometric properties were assessed.

Convergent validity

Convergent validity, the extent of agreement between 2 scales that measure similar domains or criteria that are linked with the domain of interest, was evaluated using the nonparametric Spearman-rank correlation coefficient. Convergent validity is accepted when the correlation estimates are in the correct theoretical direction [19] and are statistically significant. If correlation coefficients are statistically significant, Cohen [20] describes correlations of 0.1 as small, 0.3 as medium, and ≥0.5 as large convergence. Convergent validity for the PSQ was determined by correlating the PSQ item scores with the total scores from the IRLS, RLSQoL, and POMS, and the 4 individual domain scores of the MOS-Sleep scale.

Individual PSQ item scores were also correlated with similar items from daily diaries that captured RLS-related symptoms and sleep: PSQ Item 3 (nights with RLS symptoms) score and nights with RLS symptoms score from the 7-Day RLS Symptom Diary, PSQ Item 4 (RLS-related sleep disturbance) with number of RLS-related awakenings score from the 24-Hour RLS Symptom Diary, and PSQ Item 5 (RLS-related sleep latency) with the RLS-related sleep latency score from the 24-Hour RLS Record.

Divergent validity

An instrument demonstrates divergent validity when it shows a lack of statistical association with other instruments or related variables (e.g., demographic characteristics) unrelated to the focus instrument. This study was limited in the choice of instruments to assess divergent validity because it utilized only the instruments included in the clinical trials [7, 8] that were used to measure the effects of RLS treatment. To demonstrate divergent validity of the PSQ, we assessed correlations between the baseline PSQ item scores and three demographic variables: ethnicity, race, and gender. Because there was no theoretical basis to expect a relationship between sleep problems and these demographic variables, a lack of correlation would demonstrate that subjects with different demographic characteristics do not interpret the PSQ items differently. Cramer's V statistic was used to assess their associations [21], because this involved correlating a nominally scaled variable (i.e., 0 = Male, 1 = Female) with continuous variables. Cramer's V estimates range from 0 to 1, with low scores indicating poor association and demonstrating divergent validity.

Known-group validity

Known-group validity, or predictive validity, is the ability of a scale or scale items to statistically discriminate respondents in ways expected (predicted) as a result of independent assessments. The known groups in the current study were based on RLS symptom severity, as determined by the IRLS total score. Sleep qualities measured by the PSQ are expected to be most pronounced for persons with severe symptoms. Baseline PSQ scores were compared across severity groups using the Kruskal-Wallis test [22].

Responsiveness

Responsiveness means that the PSQ item scores will change in direction and magnitude with changes experienced by the responders. In this case, responsiveness was demonstrated if relationships between changes in the investigator- and subject-rated CGI-I assessments and changes in PSQ scores over time were in the same direction and statistically significant [15]. Change from baseline in individual PSQ items were compared with subject symptom changes as measured by the CGI-I using the Kruskal-Wallis test. Among all subjects in the clinical trials [7, 8], only a few experienced worsening of symptoms over time; as a result, the responsiveness analysis was conducted by re-categorizing subjects reporting 'minimally worse', 'much worse' and 'very much worse' into 1 category of 'worse'. Guyatt's statistic, a measure of test-retest reliability or responsiveness to change, which is the ratio of the mean change for each category divided by the standard deviation for the no change group, was estimated. An instrument with an absolute value of ≥0.20 for Guyatt's statistic has acceptable responsiveness and a value of ≥1.00 is considered highly responsive to change [23, 24].

Results

Subject characteristics

Of 544 total subjects, 540 and 451 completed the PSQ at Baseline and Week 12, respectively, and were included in the analysis. Table 1 shows that the baseline mean age of subjects was approximately 50 years, most subjects were non-Hispanic, white, and female. About 92% had moderate or severe RLS symptoms and 8% had very severe symptoms.

PSQ results

The mean scores for PSQ items indicate that most subjects improved over the 12-week study, and most of the improvements occurred by Week 4 (Table 2). For sleep quality, the percentage of subjects reporting their sleep quality as "excellent" increased by approximately 20% by Week 12. Daytime functioning showed similar positive gains, with 4.5 times as many subjects reporting their daytime functioning as "excellent" at Week 12 compared with baseline. Similar improvements in frequency of nighttime RLS symptoms, RLS-related sleep disturbances and sleep latency over the 12 weeks were observed.

Table 2 Subject responses and mean scores for PSQ items
Full size table

Convergent validity

Table 3 shows the Spearman rank correlation coefficients and the probability of each PSQ item based on the total scores of the IRLS, RLSQoL, and POMS. All correlations between the IRLS total score and each PSQ item were statistically significant, ranging from 0.25 to 0.49 (p > 0.0001, each). The strength of the associations showed moderate to moderately high convergence. The overall POMS total score had statistically significant correlations with the PSQ items, except for the item "nights with RLS symptoms". The correlations ranged from 0.04 to 0.48. All but 1 of the correlations were small in size; daytime functioning was moderately high. For the individual POMS items, 22 of 35 correlations were statistically significant (p < 0.05 each; data not shown). The correlations between the RLSQoL total score and the PSQ items were also statistically significant. The correlation coefficients ranged from -0.12 to -0.57 (p < 0.01 each). The convergence was large for daytime functioning, moderate for sleep quality, and small for all others.

Table 3 Convergent validity: PSQ item correlations at baseline with IRLS, POMS, and RLSQoL total scores
Full size table

The correlations between the MOS-Sleep scale domains and the PSQ items are provided in Table 4. The correlation coefficients ranged from -0.46 to 0.49; 18 of 20 correlations were statistically significant (p < 0.02). Of the 20 correlations, 9 had moderate to moderately large coefficients. Moderately high convergence was seen between the PSQ sleep quality item and the MOS sleep disturbance domain (higher sleep disturbance, lower sleep quality) and sleep adequacy (higher sleep quality, higher sleep adequacy), between PSQ daytime functioning and MOS daytime somnolence domain (worse functioning, worse somnolence), between PSQ sleep latency and MOS sleep disturbance domain (higher sleep latency, higher sleep disturbance). The correlations of the PSQ item nights with RLS symptoms were statistically significant with MOS sleep domains, but showed small convergence with MOS sleep disturbance, adequacy and quantity domains.

Table 4 Convergent validity: PSQ items and MOS Sleep domains at baseline
Full size table

The 7-day RLS Symptom Diary significantly correlated with the PSQ item #3 nights with RLS symptoms (r = 0.47; p < 0.0001; data not shown). The correlation between the 24-hour RLS Symptom Record Diary and the PSQ items RLS-related sleep disturbance and RLS-related sleep latency were 0.17 and 0.27, respectively (p < 0.0001, both).

Divergent validity

None of the correlation estimates from Cramer's V for any PSQ item baseline score across the 3 demographic variables (i.e., race, gender and ethnicity) were larger than 0.13 (data not shown) and none approached statistical significance, thereby demonstrating divergent validity. For race, the correlation estimates ranged from 0.05 for sleep latency to 0.13 for sleep disturbance. The respective estimates for ethnicity ranged from 0.02 to 0.07, and for gender, the range was 0.05 to 0.12.

Known-group validity

Positive and statistically significant differences (p < 0.0001, each) were found between baseline PSQ scores and the RLS severity groups moderate, severe, and very severe (Table 5). The estimates were linear across the severity groups with very severe RLS subjects having worse PSQ scores than subjects with severe RLS and subjects with severe RLS having worse PSQ scores than subjects with moderate RLS.

Table 5 Known-group validity PSQ items and RLS disease severity as assessed by IRLS total score at baseline
Full size table

Responsiveness

The mean change in PSQ scores from baseline to Week 12 indicate that improvements in sleep were achieved and scores were significantly improved across each of the investigator- and subject-rated CGI-I categories (Tables 6 and 7, respectively). The largest change was among those subjects judged by the clinician as being "very much improved" followed by "much improved", "minimally improved", "no change", and finally "worse." The magnitude of the effect size for change in PSQ scores from baseline also followed the above linear order (data not shown). The magnitude of Guyatt's statistic was moderate to large, ranging from 0.00 to 3.46 among improved subjects. Of the 50 coefficients, 7 did not meet the acceptable cutpoint of 0.20 or greater and all but 1 of the 7 was in the "worse" category. Additionally, 11 of the 50 coefficients were >1, indicating a highly responsive measure.

Table 6 Responsiveness: mean change from baseline in PSQ item scores and Guyatt's statistic by investigator-rated CGI-I at Week 12
Full size table
Table 7 Responsiveness: mean change from baseline in PSQ item scores, effect size, and Guyatt's statistic by subject-rated CGI-I at Week 12
Full size table

Discussion

This post hoc psychometric evaluation of an investigator-developed PRO instrument used data from 2 randomized, double-blind, placebo-controlled trials conducted in subjects with moderate-to-severe primary RLS to examine the validity of the PSQ in assessing RLS-related sleep disturbance and treatment improvements. The parent studies indicated that the PSQ scores showed improved sleep in subjects with RLS symptoms who received active RLS treatment compared with placebo [7, 8].

To verify the validity of the PSQ, several measurement properties were evaluated. The PSQ's convergent validity was estimated using correlations between the overall RLS symptom impact score, as assessed with the IRLS, and the individual PSQ items. These correlation estimates were all statistically significant and positive (i.e., worse symptom impact with worse sleep status). All PSQ items scores had moderate convergence with overall RLS symptom impact, except for the PSQ item "nights with RLS symptoms", which had a lower convergence.

Convergent validity was also assessed using the MOS Sleep Scale. Nearly half of the correlations with the MOS-Sleep Scale sleep domains showed moderate to moderately high convergence. Two PSQ items, nights with RLS symptoms and RLS-related sleep latency, showed little convergence with the MOS daytime somnolence domain. The PSQ sleep quality domain had sufficient convergence with the MOS sleep disturbance, sleep adequacy and sleep quantity domains. The PSQ daytime functioning item had sufficient convergence with all the MOS domains. The PSQ item, RLS-related sleep latency, had sufficient convergence with MOS sleep domains, except daytime somnolence. Taken together, these data suggest that the PSQ items had adequate convergence with the most pertinent MOS sleep domains.

Validity can also be shown via divergence with scales intended to measure other concepts. We found no strong associations across the 3 demographic variables of race, ethnicity and gender with the PSQ items. While demographics are not traditionally used for assessing divergent validity, we were limited to using the data collected in the trials. Given that, these findings clearly and robustly demonstrate that the PSQ results do not include systematic measurement errors that could be associated with personal attributes.

The PSQ results showed a consistent and robust ability to discriminate between RLS severity groups in a known-group validity analysis. All estimates were linear across the severity groups with increasingly worse sleep outcomes found in progressively worse symptom groups.

The PSQ was found to provide reliable results; for example, the PSQ was responsive to treatment differences over 12 weeks, and the findings were consistent regardless of whether the investigator or the subject rated the improvement, as indicated by the clinician and the patient rated CGI-Is. Further, the changes were seen regardless of treatment assignment, indicating that the PSQ is sensitive to small clinical changes over the course of treatment. Guyatt's statistic, an indication of responsiveness to change, was acceptably strong in all responder categories that indicated change and comparatively weaker in groups that indicated no change or worse change.

The post hoc nature of this study posed some challenges. Analyses were confined to the instruments collected as part of the clinical trials and were not included for the purposes of conducting a validation study. Thus, some aspects of the instruments used to validate the PSQ may not have been ideal. For example, items on some instruments are not directly related to sleep or are confounded by sleep. Finally, the recall periods varied among the instruments and some instruments were not specifically designed for RLS.

Conclusions

The PSQ demonstrated acceptable measurement properties of convergent validity, divergent validity, and known-group validity. PSQ scores were also reliable in terms of being responsive to change in symptoms. The PSQ is, therefore, a psychometrically valid instrument for assessing sleep among RLS subjects in clinical trial settings.