Introduction

CDKL5 Deficiency Disorder (CDD) is classified as a developmental and epileptic encephalopathy (DEE), characterized by severe developmental impairments, epileptic seizures from infancy and other comorbidities including corticovisual impairment (CVI), poor sleep and gastrointestinal problems (Leonard et al., 2022). Currently, there are clinical trials evaluating therapeutics for CDD and capability for disease modifying therapies, such as gene therapy, is imminent (Leonard et al., 2022). A critical barrier to strong clinical trial protocols in CDD is the lack of validated outcome measures that can demonstrate meaningful interventional changes.

At a Federal Drug Administration patient-focused drug development meeting for CDD, 79% (n = 40) of caregivers identified that global developmental delay, particularly communication impairment, was one of the most burdensome symptoms for both the child and caregiver (Mingorance et al., 2020). In children with CDD aged over 1.5 years, only 16% (17/105) had attained the ability to use single words (Fehr et al., 2015) although higher proportions of children used nonverbal modes of communication (Fehr et al., 2016). Younger and older children had used similar methods of communication (Fehr et al., 2016) suggesting that developmental delays became persistent impairments.

The Communication and Symbolic Behavior Scales–Developmental Profile (CSBS–DP) is a set of assessments originally designed for screening and evaluation of communication disorders in children aged between 6 and 24 months (Wetherby & Prizant, 1993, 2002). They comprise two parent report measures, the Infant–Toddler Checklist (CSBS-DP ITC) and a Caregiver Questionnaire, and a direct observation measure, the Behavior Sample (CSBS-DP BS; Wetherby et al., 2002). The CSBS-DP ITC is a parent-report screening tool.

The CSBS–DP ITC has recently been used for children with rare genetically caused developmental encephalopathies (DEs) and DEEs. It was a secondary outcome measure in a randomized crossover trial testing mecasermin (rhIGF-1) (O'Leary et al., 2018) and is in the protocol for a phase 3 trial of trofinitide (Neul et al., 2022) in Rett syndrome. Social subscale scores were significantly higher for the treatment group in the mecasermin. For other neurogenetic syndromes (Angelman, Prader-Willi and Williams syndromes), total and subscale scores were lower than for typically developing children with the lowest scores for Angelman syndrome (Hamrick & Tonnsen, 2019) but no other evidence for its validation. In five DEEs (Dravet syndrome n = 40, KCNB1 n = 14, KCNQ2 n = 21, Lennox-Gastaut syndrome n = 36, SCN2A n = 9), raw scores correlated strongly with raw Adaptive Behavior Assessment System (ABAS) communication scale scores (Berg et al., 2022). There are limited validation data for the CSBS-DP IT in neurogenetic syndromes including DEEs.

Assessing the psychometric properties of the CSBS-DP ITC for specific populations such as CDD is necessary prior to its application in clinical trials. The aim of this study was to inform whether the CSBS-DP ITC is fit for purpose for CDD by describing the distribution of scores, evaluating the factor structure and analyzing test–retest reliability for the CSBS-DP ITC, and comparing CSBS-DP ITC scores with categorical descriptors of communication and other functional abilities.

Methods

Data Source

Caregivers who had completed the initial baseline questionnaire with the International CDKL5 Disorder Database (ICDD) (Fehr et al., 2015) were invited to complete an online follow up questionnaire between April 2018 and August 2019 (Leonard et al., 2021). Data were collected from 150 caregivers of children and young people with genetically confirmed CDD. Additionally, the CSBS-DP ITC was subsequently administered twice with a four-week interval to 73 caregivers. This included 30 children not in the validity dataset who had not provided follow up data in 2018 or were newly recruited.

Measures of Functioning

Communication Measures

The CSBS-DP ITC comprises 24 items encompassing seven clusters which yield three composite scores: Social (comprised of the Emotion and Eye Gaze, Communication and Gestures clusters), Speech (comprised of the Sounds and Words clusters) and Symbolic (comprised of the Understanding and Object Use clusters), and a Total score (Wetherby & Prizant, 2002). Using other ICDD data collected at the same time, current communication skills were categorized as: words, gestures/signs or vocalisations, and unable.

Other Variables

Biological sex and genotype were described. The mutation groups were categorized according to the predicted structural/functional consequence as previously described (Fehr et al., 2015): mutations causing no functional protein, missense/in-frame mutations, truncating mutations located from aa 172 to aa 781 inclusive and truncating mutations occurring after aa 781. In those aged 12 months or older, hand function was categorized as the ability to: grasp small and large objects, pick up or hold a large but not a small object and no hand function for grasping objects. For children aged 18 months or older, mobility was categorized as: no walking, assisted and independent. Gastrostomy was categorized as: oral only, oral/gastrostomy, gastrostomy only.

Characteristics of the children and young people in the validity and reliability datasets are presented in Table 1. For the validity sample, age ranged from 1 to 29 years and 82.0% (n = 123) were female. Nearly two-thirds (n = 96) took all food orally and close to one-fifth (n = 27) communicated using some words. Approximately one third (n = 57, 38%) could grasp and pick up a selection of large and small objects and 33 (22.3%) could walk independently. Each of the four mutation groups (Fehr et al., 2015) were represented. The reliability sample had similar distributions for biological sex, genotype, walking and communication, but higher proportions took all food orally (74%) and could grasp and pick up a selection of large and small objects (51%) (Table 1).

Table 1 Characteristics of individuals with CDD in the CFA Analysis and Reliability Analysis, and CSBS-DP ITC total score data for questionnaires with no missing data
Full size table

Statistical Analysis

Descriptive statistics were used to summarize the characteristics of the individuals with CDD as well as their CSBS-DP ITC item and total scores. Confirmatory factor analysis was performed to verify the factor structure of the composite (social, speech and symbolic) scores. The three factors were allowed to be correlated with each other, but no correlated errors were permitted. All items were treated as ordinal variables and hence the weighted least square mean and variance adjusted (WLSMV) estimator was used (Lei & Shiverdecker, 2020). Standardized factor loadings were reported. Goodness of fit of the 3-factor model was assessed using the following statistics: CMIN/df value, root mean square error approximation (RMSEA), the Comparative Fit Index (CFI) and the Tucker–Lewis Index (TLI). Cronbach’s alpha, the composite reliability and average variance extracted (AVE) statistics were calculated for each factor to assess convergent validity. The maximum correlation squared (MCS) value was calculated for each factor and compared with AVE to assess divergent validity. As the divergent validity of the 3-factor model was found to be sub-optimal, confirmatory factor analysis of 1-factor model was performed and associated goodness of fit statistics were obtained. The relationships between the CSBS-DP ITC total score as continuous variable and functional abilities (walking, communication, feeding method, hand function) were investigated using simple linear regression. Multivariable regression was also carried out to examine the associations adjusted for age and genotype. Point estimates and their 95% confidence intervals were reported. To avoid unnecessary burden on the families we recruited a smaller number for the test–retest reliability analysis whilst still retaining good precision for the reliability estimate. With 73 families, we calculated that the 95% confidence interval around our anticipated reliability value of ICC = 0.9 (good agreement) would lie between 0.85 and 0.95. Test–retest reliability was assessed using intra-class correlations (ICCs). ICCs were interpreted as ≤ 0.40 slight agreement, 0.41–0.60 fair agreement, 0.61–0.80 moderate agreement, and 0.81–1.00 substantial agreement (Landis & Koch, 1977). Statistical analysis of data was performed using Stata 15.1 (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC.), and, for confirmatory factor analysis, Mplus version 8.5 (Muthén & Muthén, 1998–2017).

Results

The mean (SD) CSBS-DP total score was 14.6 (12.7) and the median (IQR) was 12 (5–18) out of a total possible score of 57, based on a complete dataset for 131 individuals (Table 1). The percent of items with a score of 0 is shown in Table 1. For all participants, over half (57.4%) of items received a score of 0. There was a greater percentage of items that scored 0 (65.4%) for males compared to females (55.7%). Individuals with lower abilities in walking, communication, and hand function had in a greater percentage of items scored 0. Also, individuals who are fed through GI tube only had a higher percent of 0 scores (75.2%) compared to children who engaged in oral feeding only (5.02%).

For each item, each response category was scored by at least one individual (Table 2) although the total score was highly skewed (1.6; Fig. 1), indicating a floor effect. The majority scored zero for most items, but seven items contributed greater variability to the scores: item 1 (Do you know when your child is happy and when your child is upset?), item 3 (Does your child smile or laugh while looking at you?), item 6 (When you are not paying attention to your child, does he/she try to get your attention?), item 14 (Does your child use sounds or words to get attention or help?), item 16 (About how many of the following consonant sounds does your child use: ma, na, ba, da, ga, wa, la, ya, sa, sha?), item 19 (When you call your child’s name, does he/she respond by looking or turning toward you?) and item 20 (About how many different words or phrases does your child under- stand without gestures?) (Table 2).

Table 2 Descriptive statistics of CSBS DP Infant–Toddler checklist item scores, including composite (e.g., SOCIAL), cluster (e.g., Emotion and Eye Gaze), and total scores, in 150 individuals with CDD
Full size table
Fig. 1
figure 1

Frequency distribution of CSBS DP Infant–Toddler checklist total score

Full size image

Confirmatory Factor Analysis

Confirmatory factor analysis was conducted for 150 individuals.

Three-Factor Analysis

Factor loadings for each of the items in the three-factor model were all greater than 0.6 (Table 3). The three-factor model showed satisfactory indices of relative fit using CMIN/df (1.387), RMSEA (0.051), CFI (0.991) and TLI (0.990; Table 4). Cronbach’s alpha values for the three-factor model ranged from 0.850 for Symbolic to 0.924 for Social, composite reliability values ranged from 0.934 for Symbolic to 0.975 for Social, and average variance extracted ranged from 0.708 for Symbolic to 0.820 for Speech, indicating satisfactory convergent reliability (Table 4). However, for each domain, the AVE values were smaller than the MCS values, therefore giving clear evidence of a lack of divergent validity of these domains (Hu & Bentler, 1999).

Table 3 Standardized factor loading (95% confidence interval) values from 3-factor and 1-factor confirmatory factor analysis for 150 individuals with CDD
Full size table
Table 4 Goodness of fit statistics, internal consistency and discriminant validity statistics from the 3-factor and 1-factor confirmatory factor analysis
Full size table

One-Factor Analysis

Factor loadings for each of the items in the single factor model were all greater than 0.6 (Table 3). This model showed satisfactory indices of relative fit using CMIN/df (1.542), RMSEA (0.060), CFI (0.988) and TLI (0.987; Table 4). For the one-factor model, internal consistency values were extremely high (Cronbach’s alpha 0.959, composite reliability 0.985) and convergent validity was satisfactory (AVE 0.733) (Table 4).

Comparison of Known Groups

Comparisons of the CSBS-DP ITC one-factor score for the functional group categories using data from 131 individuals is shown in Table 5. This analysis was limited to individuals aged 18 months or older at the time of questionnaire completion to avoid evaluating skills, such as walking, in children who would not yet be expected to. Children who required assistance (coefficient − 16.9, 95% CI − 22.9, − 10.8) or were unable to walk (coefficient − 20.5, 95% CI − 24.3, − 16.6) had significantly lower scores than those who walked independently. Children who had oral and enteral feeding (coefficient − 6.9, 95% CI − 12.7, − 1.2) or enteral feeding only (coefficient − 10.9, 95% CI − 16.2, − 5.6) had significantly lower scores than oral feeding only, also adjusting for age-group and genotype. Children able to grasp large objects only (coefficient − 12.4, 95% CI − 17.2, − 7.7) or had no grasping ability (coefficient − 16.7, 95% CI − 20.7, − 12.7) had significantly lower scores than children able to grasp small and large objects. For all functional abilities, those in the lower performing group scored lower on the CSBS-DP ITC.

Table 5 Univariable linear regression analysis of the associations between the CSBS-DP ITC total score and covariates (n = 131)
Full size table

Test–retest Reliability

Test–retest reliability had an extremely high level of agreement (ICC = 0.96, 95% CI 0.94–0.98). The mean interval between test and retest was 36.6 days.

Discussion

This is the first study to examine the distribution of scores, validity and reliability of the CSBS-DP ITC in a large sample of children and young adults with CDD, an essential task as no communication measures have been validated for CDD. This work is critical if clinical trials with non-seizure endpoints are to successfully proceed to a reliable outcome. This validation study provided mixed evidence of the suitability of the CSBS-DP ITC for the CDD population. Although the three-factor model did not show satisfactory divergent validity, the one-factor model showed satisfactory convergent and known groups validity. However, the distribution of scores showed a floor effect, and the internal consistency and test–retest reliability values were unusually high.

There is increasing evidence describing the psychometric properties of the CSBS-DP ITC in neurogenetic syndromes including DEEs, beyond its original evaluations as an early childhood screening tool for autism (Wetherby et al., 2004). When used as an outcome measure in clinical trials for RTT, significant differences were observed for the social composite score (O'Leary et al., 2018). This prompted us to test the CSBS-DP ITC with CDD. More recent evidence suggests that CSBS-DP ITC scores correlate with other communication measures for a group of DEEs (Berg et al., 2022) and scores differ by diagnostic group as would be expected. Consistently in our study, scores differed by broad functional ability groups. We used CFA to investigate psychometric properties further and reject the three-factor model for CDD because the factors were too highly correlated to justify separation of the items into the three domains. The one-factor model showed more satisfactory psychometric properties.

However, the distribution of the CSBS-DP ITC total score was very right skewed, indicating a floor effect, meaning the CSBS-DP ITC may be problematic for use in the CDD population and is dependent on the meaningfulness of change in a clinical trial setting for which data are not yet available. CDD is associated with severe impairments and there is high risk of a floor effect when measuring functional abilities with scales that were designed to evaluate a larger range of skills and pay limited attention to small incremental skills at the lower range of functioning (Berg et al., 2020). For example, approximately 75% of items were scored as 0 for groups with the most severe impairments. We note that the floor effect appears to have been less marked for other DEEs (Berg et al., 2022). We also observed an extremely high level of agreement with test–retest reliability, with minimal within subject noise. This could be related to the floor effect and the CSBS having poor granularity in the subtle communication skills associated with severe disability.

Other aspects of the CSBS-DP ITC could be advantageous. It includes questions on social communication skills, as does the Communication Matrix (Rowland & Fried-Oken, 2010), consistent with the notion that social communication is an important part of quality of life (Tangarorang et al., 2019). In addition, rather than scoring as “Never” or “0”, the CSBS-DP ITC uses the wording “not yet”. Parents participating in our Consumer Reference Group meetings have described how they value the scoring category "not yet", especially when their child achieves few skills. Parents with a child with CDD live with day-to-day challenges to their mental health (Mori et al., 2017) and measurement should be efficient and strengths-based to reduce their burden. On the other hand, it is also important to note the limitations of the CSBS-DP ITC. For instance, item 1 asks if the child indicates being happy or upset combining two concepts within the one item. Potential use of an Augmentative and Alternative Communication (AAC) device is not addressed and it can be difficult for caregivers to know exactly how many words or phrases some children can understand, if the child has extremely limited expressive communication and movement skills, and is without access to an AAC device (Mingorance et al., 2020). Supplementing the CSBS-DP ITC with additional items could reduce the floor effect and enable greater capacity for the measure to identify responsiveness to change if impairments are severe. Qualitative studies to understand communication in children with severe impairments and mapping of items across available communication measures would inform potential modifications.

The ICDD was used for recruitment, resulting in a sample size well powered for this analysis. With 24 items in the checklist, the sample was larger than the generally recommended sample size of five participants per item for factor analysis (Streiner, 1994). Parent reported questionnaires are commonly used in pediatric and intellectual disability fields and there is evidence for their suitability in young children and those with intellectual disabilities (Fisher et al., 2014; Wetherby et al., 2002), suggesting suitability for CDD. Our findings could be relevant to other DEE conditions with severe communication impairments although additional validation studies are needed. We acknowledge that further validation studies of other available communication outcome measures could yield suitable measures of communication for DEE conditions.

Conclusions

This study is the first to generate validation data for a communication measure for CDD. Our mixed findings suggest caution in using the CSBS-DP ITC in clinical trials. We found evidence that using the total score is preferable to the three composite scores but that there could be value in exploring whether additional items could buttress its capacity to describe communication more incrementally at the most impaired levels. There is an urgent need for validated outcome measures for CDD and other DEEs to support upcoming clinical trials of novel treatments, especially with targeted gene therapies on the horizon. Our study represents affirmative action in this arena.