Measuring Anxiety as a Treatment Endpoint in Youth with Autism Spectrum Disorder

Abstract

Despite the high rate of anxiety in individuals with autism spectrum disorder (ASD), measuring anxiety in ASD is fraught with uncertainty. This is due, in part, to incomplete consensus on the manifestations of anxiety in this population. Autism Speaks assembled a panel of experts to conduct a systematic review of available measures for anxiety in youth with ASD. To complete the review, the panel held monthly conference calls and two face-to-face meetings over a fourteen-month period. Thirty eight published studies were reviewed and ten assessment measures were examined: four were deemed appropriate for use in clinical trials, although with conditions; three were judged to be potentially appropriate, while three were considered not useful for clinical trials assessing anxiety. Despite recent advances, additional relevant, reliable and valid outcome measures are needed to evaluate treatments for anxiety in ASD.

Appendix: Descriptive and Evaluative Criteria for Measures Reviewed (for details on rating of all instruments reviewed contact the authors)

• 1. Respondent type indicates who reports on behavior, including self, caregiver, clinician, or other person (sibling, classmate).

• 2. Age and requirement of verbal skills reflects the degree to which the measure requires verbal skills by the person with ASD and the extent to which it can be used across a range of developmental level and chronological age. To be reliable and valid, some measures may rely less on language and more on other communication skills. For example, individuals may have language but may not have the pragmatic skills to meet the demands of some tests. Other measures may require the individual to respond within a specific time period or require a motoric response. These tests may require adequate receptive language skills. Abilities and knowledge in these areas may vary for children with ASD. The notation in this category indicates whether verbal skills are required by the instrument to assess the domain of interest.

• 3. Comment on the use in ASD, DD population or both follows from the unique challenges of assessing behavior in youth with ASD. A “yes” in this column indicates that the instrument has been studied in one or more samples of individuals with ASD (e.g., community sample, clinical sample, an intervention trial).

• 4. Clinical relevance reflects the degree to which the measure captures dimensions of anxiety that are applicable to youth with ASD.

• 5. Reliability reflects the extent to which the measure assesses anxiety consistently. Measures of reliability include internal consistency (items or dimensions contribute evenly to the total score), test–retest reliability (correlation of score over brief periods of time), inter-rater reliability (level of agreement across raters). Inter-rater reliability is particularly important in clinician-rated instruments (e.g., Pediatric Anxiety Rating Scale). The demand for inter-rater reliability may be lower for informant-based measures (e.g., parents and teachers may not be in agreement about a child’s behavior given the differences in setting). The following rating system was adapted from the Centre for Childhood Disability Research (2004) for this review.

  • Excellent (e.g., publication of more than 3 studies with reliability statistics in the excellent range plus use of the measure in at least two clinical trials with adequate statistical power.

  • Good (e.g. 2 reliability studies completed with adequate to excellent reliability values plus use of at least one clinical trial with adequate statistical power.

  • Adequate (e.g. 1 reliability study completed with adequate to excellent reliability values)

  • Poor (e.g., reliability studies poorly completed, or reliability studies showing inadequate levels of reliability)

  • No evidence available

• 6. Validity reflects the degree to which the instrument measures what it purports to measure. The review considered (a) content validity, i.e., The degree to which the measure adequately reflects anxiety; (b) construct validity, i.e., the degree to which the measure positively correlates with another measure of anxiety (convergent validity) and the degree to which it does not correlate with a measure presumed to measure a construct separate from anxiety (divergent validity). The following ratings were used:

  Content Validity

  • Excellent: (e.g., expert judgment or statistical method (e.g., factor analysis) was used to determine that the measure adequately covers the domain of interest without inclusion of unrelated material;

  • Adequate: Has reported content validity but no specific method was used

  • Poor: Instrument is not comprehensive

  • No evidence available

    Construct Validity

  • Excellent: (e.g., more than 2 independent, experimental studies have shown adequate agreement with a gold standard measure, stronger for overall evaluation if determination of both convergent and divergent validity reported)

  • Adequate: (e.g. 1–2 studies demonstrate confirmation of theoretical formulation; determination of either convergent or divergent validity reported)

  • Poor: (e.g., construct validation poorly documented or construct validity not supported by the study)

  • No evidence available

• 7. Sensitivity to change. Measures with sufficient assay sensitivity are typically used as the primary endpoint in pivotal clinical trials in support of regulatory approval for a new drug or device, or new indications for existing drug or device. Validity and reliability are pre-requisites for assay sensitivity, but high levels of validity and reliability do not guarantee sensitivity to change. Measures of assay sensitivity include effect sizes and receiver operating characteristic curves (Diggle et al. 2002). The assessment of assay sensitivity may include review of design elements (e.g., entry criteria, length of trial, and selection of control condition) and properties of the measure itself (scoring range, number of items). Although sensitivity to change is essential for a measure to be useful as a primary endpoint, failure of a given measure to demonstrate sensitivity to change may reflect misapplication of the measure, design problems in the trial or lack of efficacy for the intervention. Therefore, in this review, failure of a measure to demonstrate sensitivity to change would not result in discounting the potential for the measure as a primary endpoint. Ratings of demonstrated sensitivity to change were defined as follows:

  • Excellent: More than 2 experimental trials have demonstrated that measure captures changes in outcome (improvement or decline), in a clear and consistent manner

  • Adequate: 1–2 studies (experimental or quasi-experimental) have demonstrated that the measure captures changes in outcome, but evidence is incomplete or inconclusive. For example, evidence is limited to only one or two studies; evidence is available for more than two studies, but findings across studies are not consistent. An assessment of adequate could also be given if available evidence is inconclusive due to study design limitations making it difficult for the instrument to detect change due to variability. For example, a relatively small sample with unclear entry criteria resulting in large variability on the primary outcome.

  • Poor: Available studies have failed to show the capacity of the measure to differentiate active from a control condition.

  • No evidence available or limited evidence due to type of study (open label, small pilot trial, randomized trial with design limitations, single dose study only).

• 8. Burden reflects the time, demand structure or difficulty associated with collection of the measure. The review considered: (1) the individual with autism, (2) other persons who would complete the measure, such as parents, teachers, and clinicians, and (3) the investigator. For the individual with ASD, considerations included whether the test involves an unusually high level of demand, such as sitting motionless for an extended period of time, or an expectation that a person with ASD might find uncomfortable or particularly challenging, such as interacting with groups of people, being touched, or required to comprehend complex, verbal instructions. The length of a measure and frequency of measurement could also affect burden. For informants such as a parent, clinician, or teacher, the review also included length and frequency of the test, level of literacy required, or features of the test that might make the informant uncomfortable, such as inquiring about highly personal matters. For clinician-rated interviews, considerations include the level of training or specific qualifications required to conduct the measure, need for detailed supervision following collection of the measure, whether specialized and/or costly equipment is needed, high cost of administration (e.g., need for complex coding of behavior before analyzing data) and time required to complete the rating. Ratings of burden were classified as follows:

  • Low or none: The participant is observed in a natural environment for 15 min or less or takes 15–30 min to complete the measure by a trained observer. The measure is relatively brief (<20 min), includes current observable behaviors (e.g., past month or less) and requires simple scale (e.g., 0–3). There is little or no distressing content. Measure is easily downloadable, free of charge, and simple to score and interpret. Test is available and has been validated in more than one other language other than English.

  • Medium: The participant is observed or engaged with an unfamiliar interviewer/reporter, for 15–30 min or informant-based rating that takes >20 min to complete. An informant rating includes a long list of questions (e.g., >60) or assessment may have <60 questions, but requires complex directions or responses. There is low to moderate levels of verbal or pragmatic comprehension required. There may be some distressing content. It uses a Likert or Likert type scale, and the rater is required to interact and observe with participant while simultaneously recording responses. The rating may require a manual; however it does not require extensive training. Measure is not free of charge. Test is offered in at least one other language than English, but may not be validated for use in non-English languages. Need for conversion of raw to standardized scores to be calculated by an additional program at a cost.

  • High: The participant is required to spend 30 min or more performing tasks that require sustained attention or the assessment requires extensive preparation (e.g., fMRI) or participant training or high levels of verbal/pragmatic comprehension that may involve complex or distressing responses. The rater is required to interact and observe with participant while simultaneously recording responses using extended semi-structured or non-standardized measures. The measure requires extensive training, must be administered by clinician with at least master’s level and the measure requires >45 min to administer. Measure may not require extended time to administer, but it may involve distressing content or evoke distressing responses. Use of the tool is costly and administration would require investigator to provide additional training to personnel. Test is only available in English and requires use of translator.

• 9. Overall rating involved the inclusion of each category Most weight was placed on clinical relevance, psychometric soundness, and use in ASD population. However, sensitivity to change and burden were also considered when determining rating as follows:

  • Appropriate: Measure achieved high ratings on clinical relevance; good to excellent reliability and validity with information available on all pertinent indices (e.g., test–retest, internal consistency and construct validity) across a range of youth with ASD or developmentally disabled population. On clinician-rated measures, high priority was given to inter-rater reliability (consistency across informant ratings such as parents and teachers was less important). If sensitivity to change has been demonstrated in an ASD or developmentally disabled population, this enhanced the assessment of the rating. The instrument has low to medium burden to individual, parent, and clinician.

  • Appropriate with Conditions: Measure was considered appropriate with one or more qualifications. The psychometric properties have been evaluated in an ASD or developmentally disabled population, but available information is restricted to a specific age range or a specific subgroup such as high functioning ASD. Measure was considered relevant with good to excellent reliability and validity with information on several, but not all, pertinent indices (e.g., limited information on test–retest reliability or convergent validity). On clinician-rated measures, high priority was given to inter-rater reliability (consistency across informant ratings such as parents and teachers was less important). However, it may not be available in multiple languages and translation would be needed for a large-scale international study. Measure has medium or high level of burden and some data on sensitivity to change.

  • Potentially Appropriate: The instrument measures a clinically relevant outcome; however, there is emerging or inconsistent reliability and validity in ASD or developmentally disabled populations. The measure may have demonstrated validity for screening or diagnosis, but no evidence supporting its use as an outcome measure. Measure may be appropriate for a subgroup in the ASD population, but more study is needed to establish the reliability, validity and usefulness as an outcome measure.

  • Unproven: The instrument measures a clinically relevant outcome. However, reliability and validity have been inadequately evaluated or not evaluated. Measure may have evidence for use in other pediatric populations, but has little or no evidence supporting its use in an ASD or developmentally disabled populations.

  • Not Appropriate: Measure is not relevant to the outcome of interest or was developed for screening purposes rather than as a change measure. Available information indicates poor reliability, validity or both.