Introduction

Rhinitis and asthma are highly prevalent diseases that are closely associated. A few questionnaires have been developed and validated to assess the control of rhinitis[1, 2] and asthma[3, 4]. For over a decade, the Allergic Rhinitis and its Impact on Asthma (ARIA) initiative has recommended the simultaneous assessment and management of these diseases[5, 6]. Recent observational studies reinforced the association of rhinitis and asthma. In a cross-sectional study of asthma patients from 85 primary care practices in the United Kingdom, self-reported rhinitis was identified as a major predictor of poor asthma control[7]. Also, data from the West Sweden Asthma Study linked the degree of rhinitis with the risk of having multi-symptom asthma[8].

We have previously developed the Control of Allergic Rhinitis and Asthma Test – a 17-item version[9] – employing a formal methodological approach to ensure its quality and content validity[10]. Subsequently, we conducted a cross-sectional study, in which we performed factor analysis to reduce the questionnaire and assess its unidimensionality, resulting in a simple 10-item version (CARAT10)[11]. In that study, we described how 2 independent factors were extracted from the reduced version, matching the initial theoretical subdomains of asthma and allergic rhinitis. Nevertheless, we demonstrated that CARAT10 has good internal consistency, similar to that of widely used asthma control questionnaires, thus supporting the concept of assessing allergic rhinitis and asthma simultaneously (construct/structural validity). Finally, we also showed that CARAT10 has good discriminative properties and concurrent validity[11].

CARAT10 has been proposed as the first tool to implement ARIA guidelines in clinical practice[12]. To serve this purpose, such tool must have adequate properties to assess asthma and allergic rhinitis control of an individual over time.

Thus, this study aimed to prospectively assess the test-retest reliability, responsiveness and longitudinal validity of CARAT10.

Methods

Study design and setting

This prospective observational study comprised two visits, 4 to 6 weeks apart, and was conducted in the first semester of 2009.

Patients were enrolled in 4 allergy outpatient clinics of central hospitals in three Portuguese regions – north (Porto and Gaia), south (Lisbon) and the Azores islands (Ponta Delgada).

The study was approved by the Hospital S. João review board (Comissão de Ética para a Saúde) and was given the project nº 120/08. Each patient gave his/her written informed consent.

Participants

All patients between 18 and 70 years of age, with a medical diagnosis of asthma and allergic rhinitis and at least 6 months of follow-up at the clinic were eligible. Only patients unable to fill the questionnaire were excluded.

Data collection

At each visit, patients were asked to fill CARAT10 and additional tools: the Asthma Control Questionnaire (ACQ5)[3] and three visual analogue scales (VAS) concerning all airways symptoms, bronchial/pulmonary symptoms and nasal symptoms as measures of self-perceived control. They also underwent lung function tests followed by a medical evaluation. The study design is described in the Additional file1: Figure S1.

Lung function and exhaled nitric oxide (Fe NO50) were measured according to the 2005 ATS/ERS position statements[13, 14]. Lung function variables included the forced vital capacity (FVC), the forced expiratory volume in 1 second (FEV1) and the peak expiratory flow (PEF). Fe NO50 was evaluated only in two of the four centres and expressed as parts per billion (ppb).

The medical evaluation was carried out by the attending allergy specialist, who was blinded to the questionnaires’ answers. The physician classified the patient’s rhinitis and asthma severity and rated the rhinitis and asthma control in two 10 cm VAS, according to his/her judgment and bearing in mind the classifications of ARIA[5] and Global Initiative for Asthma (GINA)[15]. The treatment decisions were made by the attending physician, based on the clinical history, physical examination and lung function following ARIA and GINA guidelines. In addition, known allergies and current medication, as well as the decision regarding treatment plan (increase, maintain or decrease treatment) were registered. In the second visit, the same procedures were repeated and, additionally, the physician filled out two 10 cm VAS regarding the variation in asthma control and in rhinitis control (from 0 - greatest worsening, to 10 - greatest improvement).

Statistical analysis

The statistical analysis was carried out using SPSS 19.0 (SPSS Inc., Chicago, IL, USA). The level of significance was set at p < 0.05. The population was described using standard descriptive statistical techniques.

We derived dichotomous variables, classifying the patient as having controlled or not controlled asthma and controlled or not controlled rhinitis, from the physician’s control VAS. Patients were categorized as controlled if the physician filled out the VAS in the ]6,10]cm interval.

Patients who were graded between 5.0 and 6.0 in the second visit’s control variation VAS were considered to be clinically stable[3].

We classified the patients’ rhinitis severity according to the ARIA recommendations and the asthma control according to the GINA guidelines. The results from both visits were pooled and were plotted against the scores in CARAT10 and its factors.

Properties of CARAT10

The CARAT10 internal consistency analysis comprised the assessment of its internal consistency with Cronbach’s α.

The test-retest reliability was assessed using intra-class correlation coefficient (ICC) in clinically stable patients between visits[16].

The concurrent validity was studied using Spearman’s correlation coefficients between CARAT10/CARAT factors and 1) control assessment instruments or 2) physician’s assessment.

The responsiveness of CARAT10 was evaluated in the group of clinically unstable patients – those with either unstable asthma, unstable rhinitis, or both. We used a paired T-test to assess the within-patient change in the CARAT10 score. Moreover, the Guyatt’s responsiveness index (GRI) was calculated as[17, 18]:

G R I mean change of CARAT 10 in the unstable group SD of change CARAT 10 in the stable group
(1)

The longitudinal validity was assessed by Spearman’s correlation coefficients calculated between the variation of the CARAT10 score and the variation of the other measures.

For hypotheses testing, a priori predictions for the correlation coefficients were, based on previous studies as follows: (i) 0.6–0.8 with ACQ5; (ii) 0.6–0.8 with the symptoms VAS; (iii) 0.4–0.6 with the physician’s assessment[3, 10].

Results

We included 62 patients; one patient did not attend the 2nd visit. Seven patients in the first visit and 3 in the second incompletely filled out CARAT10 (frequencies of unanswered questions in the Additional file1: Table S1); 51 patients completely filled out the CARAT10 questionnaire in both visits. Table 1 summarizes the sample’s characteristics, physician assessment, lung function test results and scorings for the several questionnaires. Table 2 shows the CARAT10 scores according to the patients’ self-perceived control. Scores and subscores of CARAT10 in both visits, in patients with controlled and uncontrolled rhinitis and asthma, are presented in the Additional file1: Table S2.

Table 1 – Sample’s characteristics and results of physician assessment, control questionnaires and lung function tests
Full size table
Table 2 – Comparison of CARAT10 scores according to the patients’ self-perceived control
Full size table

Seventeen patients (28%) were classified as having clinically stable asthma and rhinitis. The unstable group included 30 patients with both unstable asthma and rhinitis, 9 only with unstable asthma and 5 only with unstable rhinitis.

The plots of CARAT10 scores according to the guidelines’ classifications of rhinitis severity and asthma control are shown in Figure 1.

Figure 1
figure 1

CARAT10 and its subscores plotted against the classifications of rhinitis severity and asthma control, according to ARIA recommendations and GINA guidelines.

Full size image

Properties of CARAT10

Regarding internal consistency, the Cronbach’s α was 0.84 for CARAT10, 0.76 for the rhinitis subscore and 0.80 for the asthma subscore. As for the test-retest reliability, an ICC of 0.82 was found in the stable group. Regarding concurrent validity, all the correlation coefficients between CARAT10/CARAT factors and 1) control assessment instruments or 2) physician’s assessment met the a priori predictions and were statistically significant with p < 0.001 (Table 3).

Table 3 – Spearman’s correlations of CARAT10 and its factors with external measures of rhinitis and asthma control in the first visit
Full size table

Concerning responsiveness, we observed a significant within-patient change of CARAT10 scoring in clinically unstable patients (95% confidence interval [-5.08; -1.31], p = 0.002). The Guyatt’s responsiveness index was 1.54.

The correlation coefficients of the measurements variations between visits ranged from 0.49 to 0.65 for ACQ5 and the symptoms VAS, while for the physician assessment of control the coefficients ranged from 0.31 to 0.41. Several correlation coefficients of external measures and CARAT subscores were lower than the a priori predictions, but only one was below 0.4[10]. These results are summarized in Table 4.

Table 4 – Longitudinal validity – comparison of the variation of CARAT10 with the variation of external measures of control
Full size table

Receiver operating characteristic (ROC) curves were plotted, comparing CARAT10’ scores (Figure 2) and subscores (Figure 3) with asthma and rhinitis assessment tools. The AUC for CARAT were between 0.941 and 0.948 for CARAT (Figure 2). Higher AUC were observed for the asthma factor (0.926-0.942) than for the rhinitis factor (0.682-0.893) (Figure 3).

Figure 2
figure 2

Receiver operating characteristic curves and diagnostic test properties of CARAT10’ with composite score of (A) ACQ score and Nasal VAS, (B), GINA and ARIA criteria and (C) ACQ, GINA, Nasal VAS and ARIA.

Full size image
Figure 3
figure 3

Receiver operating characteristic curves and diagnostic test properties of CARAT10’ asthma factor with (A) ACQ score, (B) GINE criteria, and (C) a composite score of GINA and ACQ; and CARAT10’ rhinitis factor with (D) ARIA classification, (E) the nasal symptoms VAS and (F) a composite score of ARIA and the symptoms VAS.

Full size image

Discussion

The longitudinal assessment of the Control of Allergic Rhinitis and Asthma Test (CARAT10) showed that it has adequate properties to assess individual patients over time.

This is the third phase of a project that aimed to develop and validate a tool to simultaneously assess the control of rhinitis and asthma. We have previously shown that CARAT10 has good internal consistency and internal validity, supporting the concept of a single questionnaire assessing allergic rhinitis and asthma simultaneously, and showing it may be used to compare groups[11]. We had observed good correlations between CARAT10 and ACQ5, symptoms VAS and the physician’s assessment of control, similar to that of other widely used control questionnaires for asthma. The new data confirmed our previous results. As for test-retest reliability, the ICC for CARAT10 scores between visits in stable patients was 0.82, better than the ICC previously reported for Asthma Control Test (ACT) (0.77)[19]. Also, the Guyatt’s responsiveness index observed is somewhat higher than that of ACQ7 (1.35)[3]. Regarding longitudinal validity, like ACT, CARAT10 correlates better with changes in ACQ than with the physician’s assessment; the correlation coefficient of CARAT10 with symptoms VAS was similar to the previously reported correlation of ACQ7 with asthma symptoms[3, 19]. The correlation coefficient for changes between visits of CARAT10 with the all symptoms VAS met the a priori predictions (Table 4). The correlation coefficients of CARAT10 subscores with external measures of rhinitis and asthma status were lower than predicted. These subscores were defined using standard procedures of exploratory factor analysis in a previous study[11]. However, questions on waking up at night or increasing medication may be related to asthma and/or rhinitis in different patients. Therefore, a reassessment of CARAT10 structure in different populations seems to be necessary.

In this multicenter longitudinal study, it was necessary to categorize patients as clinically stable or unstable between visits. The criterion was the physician’s rating of control variation between visits, since there is no gold standard for the measurement of control of allergic rhinitis and asthma. The intrinsic insufficiencies of physician rating[20] may undermine the results observed. However, the direction of this bias worsens the questionnaire’s properties. Therefore, the estimates for the evaluative and discriminative properties of CARAT10 are conservatively reported.

CARAT10 has been thoroughly studied in cross-sectional and prospective studies and now meets almost all requirements of COSMIN - COnsensus-based Standards for the selection of health Measurement Instruments[21]. Further studies with larger datasets are needed to firmly establish the cut values for the CARAT10 global score and the rhinitis and asthma subscores[22]. The existing data seems to suggest that a CARAT10 score over 24 identifies controlled patients, with a rhinitis subscore over 8 and an asthma subscore equal or greater than 16.

The availability of CARAT10 for clinical practice in multiple countries requires cross-cultural validation in other languages. A website to support an adequate process of translation was developed (http://www.caratnetwork.org) and researchers are invited to participate. Cross-cultural adaptation of CARAT has started in over 10 different countries[23]. It was organized in 3 phases, following the Global Allergy and Asthma European Network (GA2LEN) network recommendations[24]: forward translation, backward translation and patient testing; clinical validation studies for these languages should follow.

In summary, this study showed that CARAT10 has adequate test-retest reliability, responsiveness and longitudinal validity while confirming its high internal consistency and concurrent validity. Therefore, CARAT10 can be used both in clinical studies and in clinical practice, to compare groups and to evaluate individual patients over time.