Comparison of manual versus automated measurement of Cobb angle in idiopathic scoliosis based on a deep learning keypoint detection technology

The present study compared manual and automated measurement of Cobb angle in idiopathic scoliosis based on deep learning keypoint detection technology. A total of 181 anterior–posterior spinal X-rays were included in this study, including 165 cases of idiopathic scoliosis and 16 normal adult cases without scoliosis. We labeled all images and randomly chose 145 as the training set and 36 as the test set. Two state-of-the-art deep learning object detection models based on convolutional neural networks were used in sequence to segment each vertebra and locate the vertebral corners. Cobb angles measured from the output of the models were compared to manual measurements performed by orthopedic experts. The mean Cobb angle in test cases was 27.4° ± 19.2° (range 0.00–91.00°) with manual measurements and 26.4° ± 18.9° (range 0.00–88.00°) with automated measurements. The automated method needed 4.45 s on average to measure each radiograph. The intra-class correlation coefficient (ICC) for the reliability of the automated measurement of the Cobb angle was 0.994. The Pearson correlation coefficient and mean absolute error between automated positioning and expert annotation were 0.990 and 2.2° ± 2.0°, respectively. The analytical result for the Spearman rank-order correlation was 0.984 (p < 0.001). The automated measurement results agreed with the experts’ annotation and had a high degree of reliability when the Cobb angle did not exceed 90° and could locate multiple curves in the same scoliosis case simultaneously in a short period of time. Our results need to be verified in more cases in the future.


Introduction
Different methods of measuring the Cobb angle include manual measurement and computer-assisted semi-automated measurements. The most widely used manual method of measurement requires extensive technical experience and is prone to intra-and inter-observer errors of 2.8-8° [1,2]. Although semi-automated measurement methods can improve work efficiency, clinicians still need to select the end vertebrae and determine the positions of endplates. Therefore, it is difficult for inexperienced observers to measure Cobb angles accurately. Besides, inevitable manual operations increase the work burden [3].
Deep learning technology can be used to optimize diagnosis, prognosis, and outcome prediction in spinal diseases [4][5][6][7][8]. However, the use of deep learning in the field of scoliosis measurement is still immature. Previous studies have performed deep learning measurements based on mild scoliosis cases or chest X-ray images, or non-deep learning methods with redundant pixel-level segmentation of images, requiring excessive manual assistance [9][10][11]. Therefore, the measurement methods in these studies may not apply Yu Sun, Yaozhong Xing, Zian Zhao, Xianglong Meng have contributed equally to this study, they are co-first authors of this article. to severe scoliosis. Most studies have only measured the main scoliosis curve, but the upper and lower curves adjacent to the main curve are also crucial for making treatment decisions.
To address these problems and compare manual and automated measurement of Cobb angle in idiopathic scoliosis, we chose anterior-posterior (AP) spine X-ray images as the research object and proposed an automated measurement system based on deep learning object detection and keypoint detection technology. The system consists of three parts: isolated vertebra detection, vertebral corner localization, and Cobb angle calculation. The Cobb angle measurement based on the four corners of the vertebra can avoid the influence of irregular or unclear contours during measurement.

Experimental materials
The inclusion criteria for images were as follows: patients with idiopathic scoliosis (IS) or normal adults without scoliosis, no other previous spinal disease or surgery, and a full-length AP spinal X-ray image should include the entire spine without interference blocking the vertebrae. 181 AP spinal X-ray images were included in this study, including 165 IS cases (Cobb angle range: 10-93°) and 16 normal adults without scoliosis (Cobb angle range 0-10°). We used the image annotation tool 'Labelme' to label the four corners of each vertebra from C7 to L5. We used 145 X-ray images (2610 vertebrae) for training and 36 X-ray images (648 vertebrae) for testing.

Proposed methods
Our automated measurement method consisted of three steps (Fig. 1). First, the C7-L5 vertebrae were detected and cropped from the X-ray images. The resolution of the X-ray images was zoomed out from 1750 × 3064 to 768 × 1152 pixels to locate each vertebra efficiently with global information. The vertebral regions were cropped from the original X-ray images to facilitate the predictions made by the deep learning detection model. Second, the four vertebral corners were located in the cropped regions. The regions were zoomed in to 512 × 512 pixels before being fed into the deep learning corner detection model to precisely locate the four corners. Third, the Cobb angles were measured automatically using the connecting line between the upper or lower corners, which, in clinical practice, is equivalent to the upper or lower endplate.

C7-L5 vertebrae localization
To locate the region of interest of the spine, the zoomed-out X-ray images were fed into the CenterNet-based [12] model named CenterNet1 to approximately locate the C7-L5 vertebral regions. CenterNet is a state-of-the-art deep learning object detection technology that uses keypoint estimation to find center points and regresses by measuring the size of the borders. After the vertebrae were located, their sequence was determined by the longitudinal height. The lowest detection result was L5 in each X-ray image, and the other vertebrae included L4-C7 from bottom to top. To ensure that four corners were included in the vertebral region, the widths and heights of the bounding boxes were enlarged 1.5-times. Subsequently, each vertebral region was cropped from the high-resolution X-ray images and zoomed in to 512 × 512 pixels. (Fig. 2).

Vertebral corner localization
Next, we used keypoint detection technology to locate the four corners in each vertebral region. The proposed vertebral corner detection model was another CenterNet-based model named CenterNet2 consisting of two parts: the backbone and the output heads (Fig. 3). The backbone used the pertaining DLA-34 [13]. The vertebral corner detection model included three output heads, which were used to estimate the center heatmap, corner locations, and corner offset. Corner locations were the relative locations of the corners compared to the center. The corner offset was the discretization error of the corners.
The input of the vertebral corner detection model was the cropped vertebrae images described above. During the deep learning system training, supervision consisted of center heatmap loss, corner location loss, and corner offset loss. The center heatmap loss used the focal loss, whereas the other losses used the L1-norm loss. The left-top, right-top, left-bottom, and right-bottom corners of each vertebra were determined according to the vertical and horizontal coordinates after the vertebral corner detection.

Cobb angle measurement
The Cobb angle is defined as the angle between the upper endplate of the upper end vertebra and the lower endplate of the lower end vertebra. This study used the line connecting the left-top corner and the right-top corner as the upper endplate, and the line connecting the left-bottom corner and the right-bottom corner as the lower endplate for each vertebra (Fig. 4).
To obtain the Cobb angle, each angle composed by different vertebrae is calculated recursively by changing the lower and upper end vertebra from top to bottom (Fig. 5a). After the Cobb angle of the main curve is detected, Cobb angles in its upper and lower curves will also be measured in a similar way. A spinal X-ray is given as an example to show the detection result ( Fig. 5b). Finally, descriptive statistics were calculated for all patient characteristics as means and standard deviations for each group (expert annotated and deep learning located). To measure the reliability of the Cobb angles calculated by the proposed deep learning method, the intraclass correlation coefficient (ICC) was utilized, whereas the Pearson correlation coefficient, the mean absolute error and Spearman rank-order correlation were used to explore the relationships between deep learning results and the experts' annotation as appropriate. The level of significance was set at p < 0.05. The data were analyzed using Python (rpy2 3.3.6) and calling R language.

Results
The GPU-accelerated deep learning system measured 36 test set images simultaneously, and the average time required for each image was 4.45 s. The automated system was set up to mark the name of each vertebra and its four corners, then to measure the Cobb angle of the main curve and display any adjacent curve > 10° above and below it ( Table 1). We selected four test images for demonstration, the most severe one had a Cobb angle of 88° for the main curve, and its upper and lower curves were 30 and 49°, respectively (Fig. 6).
The mean Cobb angles of the two methods and correlation coefficients are given in Table 2. The statistical analysis indicated that the automated measurements agree with the experts' annotation.
The Spearman rank-order correlation of 0.984 (p < 0.001) indicated that the Cobb angle results obtained using the deep learning-located corners and those obtained using the expert-annotated corners highly correlated.
Besides, since deep learning models always give the identical output on the identical input, intra-rater variability of the models involved in this paper always remains zero.

Discussion
Cobb angle is the most common way to quantify the severity of scoliosis. Currently, manual measurement is widely used. However, inexperienced observers often lead to improper localization of end vertebrae and endplates, resulting in inevitable random errors. The larger the curve, the more variable the measurement accuracy [14,15]. In addition, manual measurement is time-consuming [15]. To address these problems, some researchers have explored methods of automatically or semi-automatically measuring the Cobb angle [9][10][11][16][17][18][19]. Compared to manual measurement and semi-automated measurement, automated measurement can reduce differences in the selection of end vertebrae and errors caused by manually drawing endplate lines, thereby achieving high-precision in the measurements.
To reduce artificial errors, many previous studies have proposed various semi-automated methods of measuring the Cobb angle, which could minimize manual participation in the measurement process [11,[16][17][18]. In 2002, Chockalingam et al. [16] developed a semi-automated computerized technique to measure the Cobb angle. Subjectivity arises when the observer determines the upper and lower end vertebra of the curve on the spinal image,  Although research on semi-automated measurement has deepened, and measurement time and accuracy are improving, the process of manual participation cannot be avoided. For example, a system developed by Zhang et al. [11] could achieve an accuracy similar to that of experienced physicians, and the average computing time was less than half a minute, but it still required manual selection and labeling of the end vertebrae.
The use of deep learning technology in identifying and measuring spinal X-ray images is still in its early stages, and there were significant limitations in previous studies [9,10,19]. The automated systems developed by Pan et al. [9] measured Cobb angles in 248 chest X-rays with mild scoliosis (6.6-48.3°; average 14.87 ± 5.57°), whereas the ICC between the system and the physician's manual measurement was 0.854 and the mean absolute difference was 3.32°. In other words, the sensitivity (89.59%) and specificity (70.37%) of this system for identifying scoliosis were unsatisfactory. Horng et al. [10] only tested their automated measurement system in a Cobb angle range of 0-20.1°. The ICC and Pearson correlation coefficient were within the range of 0.936-0.971, and the Spearman rank-order correlation between the expert manual measurements and the system measurements was 0.891. This system exhibited high accuracy, but it was not verified with a Cobb angle > 20.1°. Zhang et al. [19] also used the vertebral corner keypoint detection method to achieve automated measurement of the Cobb angle. However, they did not use statistical methods to compare the automated method's consistency against the manual method. Furthermore, the range of Cobb angles that can be measured effectively was not mentioned.

3
leads to projection of the upper and lower endplates of the L5 vertebra closer to a "plane" rather than a "line" on the coronal plane. The "planes" of the upper and lower endplates are closer to each other, or even overlap, which makes it difficult to identify the four corners of the L5 vertebra on the coronal plane. Hopefully, future studies can overcome these problems and achieve more accurate measurements.

Conclusion
We developed an automated measurement method for Cobb angle in idiopathic scoliosis based on deep learning keypoint detection and compared it to manual measurement. The automated measurement results agreed with the experts' annotation and had a high degree of reliability when the Cobb angle did not exceed 90°. The technique could also locate multiple curves in the same scoliosis case simultaneously in a short period of time. However, the number of images in our test set was small, the results of this experimental study need to be verified in more cases in the future. Fig. 6 The Cobb angles measured by the deep learning system. a A patient with severe idiopathic scoliosis (IS). Deep learning measurements were 88° for the main thoracic (MT) curve, 30° for the proximal thoracic (PT) curve, and 49° for the lumbar curve. b A patient with IS. Deep learning measurements were 62° for the MT curve, 44° for the PT curve, and 28° for the lumbar curve. c A patient with Lenke 5C IS. Deep learning measurements were 42° for the main lumbar curve, and there is a compensatory non-structural curve of 18° at T2-T12. d A patient with mild IS. Deep learning measurements were 20° for the thoracolumbar curve and 16° for the thoracic curve, which is a compensatory non-structural curve  Number of cases Fig. 7 In each case, the Cobb angle of scoliosis with the biggest difference between expert annotation and automated positioning was selected as the maximum error of this case, and the number of cases with the maximum error from 0 to 9° in the test set of 36 cases was shown in the form of histogram. The median of maximum Cobb angle error was 3.0°, the mean value of the maximum Cobb angle error is 3.2°