Introduction

Chiari II malformation is a complex developmental malformation of the central nervous system. It is characterized by a small posterior fossa and downward displacement of the cerebellum and brainstem through an enlarged foramen magnum (hindbrain herniation) [1]. Chiari II malformation is almost uniquely associated with open spinal dysraphism [2]. McLone and Knepper [3] hypothesized that leakage of cerebrospinal fluid through the spinal anomaly reduces the distension of the embryonic ventricular system. The decreased inductive pressure on the surrounding mesenchyme results in an abnormally small posterior fossa. Approximately one third of the patients with Chiari II malformation develop signs and symptoms of brainstem compression [4]. The mortality in this symptomatic group is 15 to 35 % [5, 6].

Usually, Chiari II malformation is clinically diagnosed with the help of MR imaging. On MR images, the malformation is characterized by a constellation of morphological features (Table 1). Most of these features were originally derived from post-mortem examinations [7–10] and computed tomography studies [11–14]. With the introduction of MR imaging, most features were simply adopted to evaluate MR images [15–19]. However, the interpretation of features as seen on MR images may not always be straightforward. First, the malformation is heterogeneous in itself and in its relation with spinal dysraphism. Second, an abundance of features exist, which may obscure unambiguous assessment of Chiari II malformation. Third, the definitions of some features are equivocal and reviewers may interpret features differently. Although most features are typical for Chiari II malformation, knowledge about the reliability of rating these features on MR images is lacking.

Table 1 Definitions of features of Chiari II malformation
Full size table

Still, brain MR imaging plays a substantial role in clinical decision making regarding the management of children with spina bifida [18, 20]. On the one hand, the discussion on selective treatment of severely affected newborn infants is still ongoing [21]. On the other hand, fetal imaging and prenatal surgery are becoming more important every day. Recently, a randomized control trial showed important improvement of hindbrain herniation following prenatal surgery for spina bifida [22]. However, the assessment of Chiari II malformation may be even more complicated in prenatal MR imaging. A discrepancy of 41 % was seen in judgment of the degree of cerebellar herniation in prenatal MR imaging studies [23]. When choices have to be made about pre- and postnatal treatment options, it is important to have consensus about the morphological features that unambiguously characterize Chiari II malformation. As a proper reference standard is not available, however, testing the validity of different features is unattainable. The next best method to appraise these features is to evaluate interobserver reliability.

Therefore, we initiated a study to investigate the interobserver reliability of morphological features of Chiari II malformation on MR images. The purpose of this study was to select those features among the abundance of features that are essential for the diagnosis of the malformation, hypothesizing that several features would be too unreliable to adequately characterize Chiari II malformation.

Material and methods

Patients

Brain MR images of 79 children [mean age 10.6 (SD 3.2; range, 6-16) years] were evaluated. Of these children, 26 had open spinal dysraphism, while 17 children had closed spinal dysraphism (13 with lipomyelomeningocele and four children with other types of closed spinal dysraphism). The children with open spinal dysraphism were presumed to have Chiari II malformation [2], while children with closed spinal dysraphism might have some features of hindbrain herniation according to the literature [24, 25]. The latter group was included to reduce context bias [26]. The majority of these children with spinal dysraphism (n = 36) were recruited at the outpatient clinics of Pediatric Neurology of the Radboud University Nijmegen Medical Centre (RUNMC) as part of a prospective research program dedicated to outcome and prognosis of spina bifida. MR images of the remaining seven children were obtained retrospectively from the archives of the Department of Radiology of the RUNMC, from which we also obtained MR images of 36 children without spinal dysraphism, who were presumed to have no cerebral pathology. Although MR imaging in these 36 children was performed with suspicion of or to rule out cerebral pathology, the images had been assessed as normal by an independent radiologist in a clinical setting before the start of the study.

MR imaging

All MR images were acquired using a 1.5-T MR imaging unit (Siemens Avanto; Siemens Medical Solutions, Erlangen, Germany) with a standard head coil. MR imaging in the 36 children who were part of the prospective research program consisted of T1-weigthed images in the sagittal plane and T2-weigthed images in the axial and coronal plane. The retrospectively obtained MR images were acquired using comparable sequences. For different reasons, MR images were not acquired in three planes for all 79 children. Images in the sagittal plane were available for 69 children (41 with spinal dysraphism), images in the axial plane for 58 children (32 with spinal dysraphism), and images in the coronal plane for 51 children (37 with spinal dysraphism).

The Regional Committee on Research involving Human Subjects approved the study protocol. Prior to inclusion in the study, written informed consent was obtained from the parents of all 36 children and all children above 12 years of age taking part in the prospective research program.

Image analysis

All MR images were blinded for demographic and diagnostic information. The MR images were mixed and arranged by plane into three data sets: a sagittal set, an axial set, and a coronal set. These three data sets were reviewed consecutively and independently by three observers: a junior pediatric neurologist (N.G.) with 6 years of experience in reviewing pediatric brain MR images, a senior pediatric neurologist (R.A.M.), and a senior neuroradiologist (T.V.), both with more than 20 years of experience in reviewing pediatric brain MR images. A few weeks separated the reviews of the three datasets to prevent bias by recognition of images from a former set as much as possible. The images were available on compact disks and were reviewed on an Agfa workstation or on a personal computer using Agfa software (Impax Client, release 4.5).

The morphological features of Chiari II malformation to be assessed were selected from the literature and incorporated in a review protocol (Table 1). First, the feasibility of the protocol was evaluated in a pilot study (n = 10), resulting in a final set of study features with their definitions. The observers rated all features as being present, absent, or indefinable.

Statistical analysis

For each feature, the ‘present’, ‘absent’, and ‘indefinable’ ratings were tallied up per observer. First, the ‘indefinable’ ratings were evaluated to assess the applicability of each feature. If two or three observers rated a feature as indefinable in more than 5 % of the MR images, it was qualified as non-applicable and subsequently excluded from the further analyses.

Interobserver agreement analyses were performed for the applicable features using only the ‘present’ and ‘absent’ ratings. The percentages of agreement were obtained from contingency tables. Based on these tables, κ values for multiple observers were calculated to measure the extent of agreement among the three observers [27]. To comprehend possible sources of disagreement, κ values were also calculated for pairs of observers. We considered a feature reliable when the κ value was above 0.8, which denotes almost perfect agreement [28]. The analyses were performed using SAS software version 8.2 (SAS Institute).

Results

For each feature, the percentages of ‘present’ and ‘indefinable’ ratings are summarized per observer in Table 2. All observers rated most features in the sagittal plane as present in 20–35 % of the MR images, whereas the percentages of ‘present’ ratings in the axial and coronal planes varied substantially among features and among observers. In general, observer C rated features as ‘present’ less often than the other two observers did, whereas observer B rated features as ‘indefinable’ more often than the other two observers did. In the sagittal plane, all but one feature (Stenogyria) turned out to be applicable. In contrast, in the axial and coronal plane more than half of the features turned out to be non-applicable (Table 2). One observer rated Enlarged massa intermedia in the axial plane as indefinable in all but one MR image. The ratings of features in children with open or closed spinal dysraphism or without spinal dysraphism are presented in Table 3. With a few exceptions, features were quite common in children with open spinal dysraphism and hardly seen in the other children.

Table 2 Proportions of ‘present’ and ‘indefinable’ ratings per observer for each feature of Chiari II malformation
Full size table
Table 3 Features of Chiari II malformation present on MR images in children with open or closed spinal dysraphism or without spinal dysraphism
Full size table

The interobserver agreement of the applicable features is presented in Table 4. The right panel of the table shows the percentages of agreement and disagreement, while the left panel shows the κ values. The interobserver agreement among all three observers was almost perfect (κ value > 0.8) for the following features in the sagittal plane: Downward herniation cerebellum, Downward herniation tonsil, Downward displacement medulla, Downward displacement fourth ventricle, Medullary kinking, Abnormal width fourth ventricle, Hypoplastic tentorium, and Beaked tectum (Fig. 1). Only one feature in the axial plane (Small fourth ventricle) showed almost perfect agreement, while none of the features in the coronal plane did. The overall κ values for the remaining features ranged from 0.50 (Cerebellum wrapped around brainstem) to 0.75 (Downward displacement pons), except for a very low κ value for Enlarged massa intermedia (0.10). Table 4 also lists the κ values for pairs of observers. For seven features, the κ values differed substantially among pairs of observers: Downward herniation vermis, Upward herniation cerebellum, Downward displacement pons, and Abnormal course straight sinus in the sagittal plane; Cerebellum wrapped around brainstem in the axial plane; and Indentation and Gyral interdigitation in the coronal plane. In general, the agreement between observers A and B was stronger than the agreement of each of them with observer C.

Table 4 Overall and pairwise interobserver agreement of the applicable features of Chiari II malformation
Full size table
Fig. 1
figure 1

a Sagittal T1-weighted brain MR image in 16-year-old child with open spinal dysraphism. The image shows herniation of the vermis (large white arrow), herniation of the tonsil (large white open arrow), and medullary kinking (small white arrow); b sagittal T1-weighted brain MR image in 12-year-old child with open spinal dysraphism. The image shows herniation of the cerebellum (large white arrow). The vermis and tonsil cannot be demarcated from each other. Note the beaked tectum (small white arrow) and the hypoplastic tentorium. Also, note the downward displacement of the medulla and pons and the small fourth ventricle in both images

Full size image

Discussion

On brain MR images, Chiari II malformation is generally assessed based on a constellation of morphological features. The current study reports on the reliability of these features leading to the identification of essential features that may improve consensus on the diagnosis of Chiari II malformation.

In this study, reliable features were distinguished from unreliable features, with reliable features predominantly being found in the sagittal plane. This in itself is not surprising, as most of the morphological abnormalities are best shown in the midsagittal plane, which is usually used to assess Chiari II malformation. Still, a substantial number of features in the sagittal plane (six out of 14) showed less than perfect or poor reliability and most features in the axial and coronal plane were non-applicable. These results support our assumption that the MR interpretation of Chiari II malformation is not always straightforward. The unreliability of features may be explained by their qualitative nature and the fact that the distinction between normal and abnormal brain development is not defined by an unambiguous cutoff point. Judgment of the features is further complicated by the morphological diversity of the malformation and the fact that MR images capture features to various degrees. These general explanations mainly apply to features with random disagreement, that is to say, when the overall κ value and all pairwise κ values are low (e.g., Upward herniation cerebellum, Flattened pons, and Gyral interdigitation; Table 4).

On the other hand, the results for pairwise agreement showed systematic disagreement for some features; i.e., stronger agreement between observers A and B than the agreement for each of them with observer C. Perhaps, reappraisal of some definitions may further improve reliability, for instance, for Cerebellum wrapped around brainstem and Indentation (Figs. 2 and 3).

Fig. 2
figure 2

a Axial T2-weighted brain MR image in 16-year-old child with open spinal dysraphism. The image clearly shows that the cerebellar hemispheres are wrapped around the brainstem (small white arrows); b axial T2-weighted brain MR image in 12-year-old child with open spinal dysraphism. In this image, it is questionable whether the cerebellar hemispheres are wrapped around the brainstem (small white arrows). Also note the small fourth ventricle (large white arrow)

Full size image
Fig. 3
figure 3

a Coronal T2-weighted brain MR image in 9-year-old child with open spinal dysraphism. The image clearly shows that the tentorium indents the cerebellar hemispheres (white arrows); b coronal T2-weighted brain MR image in 12-year-old child with open spinal dysraphism. In this image, it is questionable whether the tentorium indents the cerebellar hemispheres (white arrows)

Full size image

The systematic disagreement for Downward herniation vermis is of special interest. Blurred cerebellar contours in a crowded posterior fossa and partial volume effects may hamper precise demarcation of the vermis and may make it difficult to distinguish the vermis from the tonsil and from medullary kinking (Fig. 1). This is in agreement with previous studies that reported that the vermis could not be clearly delineated in about 50 % of children with Chiari II malformation [15, 16]. On the other hand, systematic disagreement may have resulted from different concepts about the morphology of Chiari II malformation. Observer C, in contrast to the other two observers, considered Downward herniation vermis to be present more often than Downward herniation tonsil (Table 2). Yet, from post-mortem studies, it is known that herniation of the vermis without herniation of the tonsils does not occur [9]. Therefore, we recommend to assess downward herniation of the cerebellum irrespective of this being herniation of the vermis or herniation of the tonsils.

One of the limitations of this study was the possibility of context bias, i.e., knowledge from other sources that exaggerated interobserver agreement [26]. To deal with this phenomenon, we mixed the images expected to show Chiari II malformation with images expected to be without abnormalities and with images in which some features of hindbrain herniation could be present. However, observers may have tended to rate a feature according to the general appearance of the cerebellum, as complete blinding of each solitary feature was impossible. Another potential source of bias was the ratio between present and absent ratings as excess of one of the two affects the κ value [29]. In the current study, the proportion of present ratings per feature generally ranged from 25 to 35 % (Table 2). Within this small range, κ values can be safely compared among features. Yet, a few features were rated as present in considerably lower proportions. As the κ value will underestimate agreement in case of low proportions [29], reliability of the features in question may be better than expected from the actual κ values. Furthermore, response bias may have decreased κ values [29, 30]. This is particularly relevant when a rating is ambiguous. Although the observers had the opportunity to rate ambiguous features as indefinable, response bias was not completely avoided, since observers A and B generally rated features more often as present than observer C. As this was clearly the case for Downward displacement pons and the κ value was just below the cutoff point of 0.8, underestimation of agreement may be relevant for this feature. Potential institutional bias may be another limitation of the study. All observers worked at the same academic hospital, which might have increased agreement. However, the observers differed in terms of experience and educational and professional background. These differences might have reduced the interobserver agreement. On the other hand, the participation of senior and junior specialists with different backgrounds implies that the results are particularly useful for radiologist and other specialists who might be less familiar with reviewing brain MR images.

Nevertheless, this study showed that among all features that are evaluated while diagnosing Chiari II malformation, only a subset seems to be reliable. Although the Chiari II malformation seems to be a clear entity, clinicians and researchers should be aware of the different interpretations of its features among observers. The use of reliable features may facilitate plain communication about Chiari II malformation in clinical and research settings. In the management of individual patients, decisions about treatment options should be based on clinical signs and symptoms in combination with reliable MR findings. Although Chiari II malformation is almost uniquely associated with open spinal dysraphism, there might be exceptions. In such cases, the reliable features presented might be useful. In discussions on prenatal surgery and postnatal selective treatment of spina bifida, this study provides clinicians and researchers with features that unambiguously describe the Chiari II malformation.

In addition to the qualitative method, a morphometric approach quantifying the morphological distortions may be helpful to overcome the problems of unreliable features. Morphometric measures are less subjective and may be less liable to interobserver variability. They may also provide cutoff points that distinguish between normal and abnormal brain development. The reliability and diagnostic performance of morphometric measures is subject of the second part of our study on MR assessment of Chiari II malformation.

In conclusion, the following morphological features can reliably be used to assess Chiari II malformation on MR images: downward herniation of the cerebellum, downward displacement of the medulla, pons, and fourth ventricle, medullary kinking, abnormally shaped fourth ventricle, hypoplastic tentorium, and beaked mesencephalic tectum. The use of these essential features may improve MR assessment of Chiari II malformation by providing a solid basis for consensus on the diagnosis.