Introduction

Different methods of measuring the Cobb angle include manual measurement and computer-assisted semi-automated measurements. The most widely used manual method of measurement requires extensive technical experience and is prone to intra- and inter-observer errors of 2.8–8° [1, 2]. Although semi-automated measurement methods can improve work efficiency, clinicians still need to select the end vertebrae and determine the positions of endplates. Therefore, it is difficult for inexperienced observers to measure Cobb angles accurately. Besides, inevitable manual operations increase the work burden [3].

Deep learning technology can be used to optimize diagnosis, prognosis, and outcome prediction in spinal diseases [4,5,6,7,8]. However, the use of deep learning in the field of scoliosis measurement is still immature. Previous studies have performed deep learning measurements based on mild scoliosis cases or chest X-ray images, or non-deep learning methods with redundant pixel-level segmentation of images, requiring excessive manual assistance [9,10,11]. Therefore, the measurement methods in these studies may not apply to severe scoliosis. Most studies have only measured the main scoliosis curve, but the upper and lower curves adjacent to the main curve are also crucial for making treatment decisions.

To address these problems and compare manual and automated measurement of Cobb angle in idiopathic scoliosis, we chose anterior–posterior (AP) spine X-ray images as the research object and proposed an automated measurement system based on deep learning object detection and keypoint detection technology. The system consists of three parts: isolated vertebra detection, vertebral corner localization, and Cobb angle calculation. The Cobb angle measurement based on the four corners of the vertebra can avoid the influence of irregular or unclear contours during measurement.

Materials and methods

Experimental materials

The inclusion criteria for images were as follows: patients with idiopathic scoliosis (IS) or normal adults without scoliosis, no other previous spinal disease or surgery, and a full-length AP spinal X-ray image should include the entire spine without interference blocking the vertebrae. 181 AP spinal X-ray images were included in this study, including 165 IS cases (Cobb angle range: 10–93°) and 16 normal adults without scoliosis (Cobb angle range 0–10°). We used the image annotation tool ‘Labelme’ to label the four corners of each vertebra from C7 to L5. We used 145 X-ray images (2610 vertebrae) for training and 36 X-ray images (648 vertebrae) for testing.

Proposed methods

Our automated measurement method consisted of three steps (Fig. 1). First, the C7-L5 vertebrae were detected and cropped from the X-ray images. The resolution of the X-ray images was zoomed out from 1750 × 3064 to 768 × 1152 pixels to locate each vertebra efficiently with global information. The vertebral regions were cropped from the original X-ray images to facilitate the predictions made by the deep learning detection model. Second, the four vertebral corners were located in the cropped regions. The regions were zoomed in to 512 × 512 pixels before being fed into the deep learning corner detection model to precisely locate the four corners. Third, the Cobb angles were measured automatically using the connecting line between the upper or lower corners, which, in clinical practice, is equivalent to the upper or lower endplate.

Fig. 1
figure 1

Cobb angle measurement flow

Full size image

C7-L5 vertebrae localization

To locate the region of interest of the spine, the zoomed-out X-ray images were fed into the CenterNet-based [12] model named CenterNet1 to approximately locate the C7-L5 vertebral regions. CenterNet is a state-of-the-art deep learning object detection technology that uses keypoint estimation to find center points and regresses by measuring the size of the borders. After the vertebrae were located, their sequence was determined by the longitudinal height. The lowest detection result was L5 in each X-ray image, and the other vertebrae included L4-C7 from bottom to top. To ensure that four corners were included in the vertebral region, the widths and heights of the bounding boxes were enlarged 1.5-times. Subsequently, each vertebral region was cropped from the high-resolution X-ray images and zoomed in to 512 × 512 pixels. (Fig. 2).

Fig. 2
figure 2

Image processing flow for vertebrae localization

Full size image

Vertebral corner localization

Next, we used keypoint detection technology to locate the four corners in each vertebral region. The proposed vertebral corner detection model was another CenterNet-based model named CenterNet2 consisting of two parts: the backbone and the output heads (Fig. 3). The backbone used the pertaining DLA-34 [13]. The vertebral corner detection model included three output heads, which were used to estimate the center heatmap, corner locations, and corner offset. Corner locations were the relative locations of the corners compared to the center. The corner offset was the discretization error of the corners.

Fig. 3
figure 3

The architecture of the vertebral corner detection model

Full size image

The input of the vertebral corner detection model was the cropped vertebrae images described above. During the deep learning system training, supervision consisted of center heatmap loss, corner location loss, and corner offset loss. The center heatmap loss used the focal loss, whereas the other losses used the L1-norm loss. The left-top, right-top, left-bottom, and right-bottom corners of each vertebra were determined according to the vertical and horizontal coordinates after the vertebral corner detection.

Cobb angle measurement

The Cobb angle is defined as the angle between the upper endplate of the upper end vertebra and the lower endplate of the lower end vertebra. This study used the line connecting the left-top corner and the right-top corner as the upper endplate, and the line connecting the left-bottom corner and the right-bottom corner as the lower endplate for each vertebra (Fig. 4).

Fig. 4
figure 4

Illustration of the vertebral endplates

Full size image

To obtain the Cobb angle, each angle composed by different vertebrae is calculated recursively by changing the lower and upper end vertebra from top to bottom (Fig. 5a). After the Cobb angle of the main curve is detected, Cobb angles in its upper and lower curves will also be measured in a similar way. A spinal X-ray is given as an example to show the detection result (Fig. 5b).

Fig. 5
figure 5

a The traversal computation for the Cobb angle. b An illustration of the Cobb angle measurement results with Cobb angles of 11°, 29° and 13° corresponding to the upper, main and lower curves

Full size image

Finally, descriptive statistics were calculated for all patient characteristics as means and standard deviations for each group (expert annotated and deep learning located). To measure the reliability of the Cobb angles calculated by the proposed deep learning method, the intra-class correlation coefficient (ICC) was utilized, whereas the Pearson correlation coefficient, the mean absolute error and Spearman rank-order correlation were used to explore the relationships between deep learning results and the experts’ annotation as appropriate. The level of significance was set at p < 0.05. The data were analyzed using Python (rpy2 3.3.6) and calling R language.

Results

The GPU-accelerated deep learning system measured 36 test set images simultaneously, and the average time required for each image was 4.45 s. The automated system was set up to mark the name of each vertebra and its four corners, then to measure the Cobb angle of the main curve and display any adjacent curve > 10° above and below it (Table 1). We selected four test images for demonstration, the most severe one had a Cobb angle of 88° for the main curve, and its upper and lower curves were 30 and 49°, respectively (Fig. 6).

Table 1 36 cases of idiopathic scoliosis X-ray orthopedic expert measurement results and deep learning system measurement results
Full size table
Fig. 6
figure 6

The Cobb angles measured by the deep learning system. a A patient with severe idiopathic scoliosis (IS). Deep learning measurements were 88° for the main thoracic (MT) curve, 30° for the proximal thoracic (PT) curve, and 49° for the lumbar curve. b A patient with IS. Deep learning measurements were 62° for the MT curve, 44° for the PT curve, and 28° for the lumbar curve. c A patient with Lenke 5C IS. Deep learning measurements were 42° for the main lumbar curve, and there is a compensatory non-structural curve of 18° at T2–T12. d A patient with mild IS. Deep learning measurements were 20° for the thoracolumbar curve and 16° for the thoracic curve, which is a compensatory non-structural curve

Full size image

The mean Cobb angles of the two methods and correlation coefficients are given in Table 2. The statistical analysis indicated that the automated measurements agree with the experts’ annotation.

Table 2 Comparison of the Cobb angle measurements between expert annotation and automated location
Full size table

The Spearman rank-order correlation of 0.984 (p < 0.001) indicated that the Cobb angle results obtained using the deep learning-located corners and those obtained using the expert-annotated corners highly correlated.

Besides, since deep learning models always give the identical output on the identical input, intra-rater variability of the models involved in this paper always remains zero.

Discussion

Cobb angle is the most common way to quantify the severity of scoliosis. Currently, manual measurement is widely used. However, inexperienced observers often lead to improper localization of end vertebrae and endplates, resulting in inevitable random errors. The larger the curve, the more variable the measurement accuracy [14, 15]. In addition, manual measurement is time-consuming [15]. To address these problems, some researchers have explored methods of automatically or semi-automatically measuring the Cobb angle [9,10,11, 16,17,18,19]. Compared to manual measurement and semi-automated measurement, automated measurement can reduce differences in the selection of end vertebrae and errors caused by manually drawing endplate lines, thereby achieving high-precision in the measurements.

To reduce artificial errors, many previous studies have proposed various semi-automated methods of measuring the Cobb angle, which could minimize manual participation in the measurement process [11, 16,17,18]. In 2002, Chockalingam et al. [16] developed a semi-automated computerized technique to measure the Cobb angle. Subjectivity arises when the observer determines the upper and lower end vertebra of the curve on the spinal image, and when the medial and lateral edges of the vertebra are defined. The mean analysis time for each radiograph was 2.3 min. Although research on semi-automated measurement has deepened, and measurement time and accuracy are improving, the process of manual participation cannot be avoided. For example, a system developed by Zhang et al. [11] could achieve an accuracy similar to that of experienced physicians, and the average computing time was less than half a minute, but it still required manual selection and labeling of the end vertebrae.

The use of deep learning technology in identifying and measuring spinal X-ray images is still in its early stages, and there were significant limitations in previous studies [9, 10, 19]. The automated systems developed by Pan et al. [9] measured Cobb angles in 248 chest X-rays with mild scoliosis (6.6–48.3°; average 14.87 ± 5.57°), whereas the ICC between the system and the physician’s manual measurement was 0.854 and the mean absolute difference was 3.32°. In other words, the sensitivity (89.59%) and specificity (70.37%) of this system for identifying scoliosis were unsatisfactory. Horng et al. [10] only tested their automated measurement system in a Cobb angle range of 0–20.1°. The ICC and Pearson correlation coefficient were within the range of 0.936–0.971, and the Spearman rank-order correlation between the expert manual measurements and the system measurements was 0.891. This system exhibited high accuracy, but it was not verified with a Cobb angle > 20.1°. Zhang et al. [19] also used the vertebral corner keypoint detection method to achieve automated measurement of the Cobb angle. However, they did not use statistical methods to compare the automated method’s consistency against the manual method. Furthermore, the range of Cobb angles that can be measured effectively was not mentioned.

To solve these problems, we proposed a new scoliosis measurement method based on deep learning. Our system can achieve automated measurement of Cobb angles through the vertebral positioning based on deep learning object detection technology and vertebral corner positioning based on deep learning keypoint detection technology. Our study included various Cobb angles in the training set (145 X-ray images), ranging from 0 to 93°. We used the trained system to automatically identify and measure Cobb angles on 36 AP spine X-ray images that were not included in the training set. There were no significant differences in the measurements of 36 scoliosis cases with different severity by both the system and orthopedic experts. The trained deep learning system could satisfactorily recognize curves up to almost 90°. Compared to previous studies, our system has obvious advantages in its scope of application and degree of automation. The vertebral positioning based on deep learning object detection technology and vertebral corner positioning based on deep learning keypoint detection technology allowed faster measurement of Cobb angles by our system, which is more robust and finer-grained (Fig. 7). On the one hand, our automated system can identify each vertebra and obtain Cobb angles rapidly (average 4.45 s per image), avoiding human assistance at each step. On the other hand, this system can accurately measure severe Cobb angles up to almost 90° and locate multiple curves in the same scoliosis case simultaneously.

Fig. 7
figure 7

In each case, the Cobb angle of scoliosis with the biggest difference between expert annotation and automated positioning was selected as the maximum error of this case, and the number of cases with the maximum error from 0 to 9° in the test set of 36 cases was shown in the form of histogram. The median of maximum Cobb angle error was 3.0°, the mean value of the maximum Cobb angle error is 3.2°

Full size image

Our study has some limitations. First, our experiments found that the measurements made by our system greatly deviated from manual measurements when the curve was greater than 90°. The reason may be that, in severe scoliosis, the vertebrae are tilted and rotated severely, and the vertebrae may overlap in X-ray images, especially in the thoracic spine, where the ribs interfere with the image. Our system cannot accurately segment the vertebrae and identify the four corners of the vertebrae in these severe cases. Second, our system has difficulty identifying the four corners of the L5 vertebra. Compared to other lumbar vertebrae, L5 is usually structurally different and more inclined in the sagittal plane. Even an experienced orthopedist may not be able to identify the four corners of L5 precisely. We measured the angles between the upper and lower endplates of L1-L5 and the horizontal line on sagittal X-ray images from 36 test sets (Fig. 8). The results show that L5 is the most inclined vertebra in the sagittal plane of the lumbar spine, which leads to projection of the upper and lower endplates of the L5 vertebra closer to a “plane” rather than a “line” on the coronal plane. The “planes” of the upper and lower endplates are closer to each other, or even overlap, which makes it difficult to identify the four corners of the L5 vertebra on the coronal plane. Hopefully, future studies can overcome these problems and achieve more accurate measurements.

Fig. 8
figure 8

The Angle of sagittal inclination of the upper and lower endplates of L1-L5 in 36 test cases

Full size image

Conclusion

We developed an automated measurement method for Cobb angle in idiopathic scoliosis based on deep learning keypoint detection and compared it to manual measurement. The automated measurement results agreed with the experts’ annotation and had a high degree of reliability when the Cobb angle did not exceed 90°. The technique could also locate multiple curves in the same scoliosis case simultaneously in a short period of time. However, the number of images in our test set was small, the results of this experimental study need to be verified in more cases in the future.