ROC operating point selection for classification of imbalanced data with application to computer-aided polyp detection in CT colonography
Computer-aided detection and diagnosis (CAD) of colonic polyps always faces the challenge of classifying imbalanced data. In this paper, three new operating point selection strategies based on receiver operating characteristic curve are proposed to address the problem.
Classification on imbalanced data performs inferiorly because of a major reason that the best differentiation threshold shifts due to the degree of data imbalance. To address this decision threshold shifting issue, three operating point selection strategies, i.e., shortest distance, harmonic mean and anti-harmonic mean, are proposed and their performances are investigated.
Experiments were conducted on a class-imbalanced database, which contains 64 polyps in 786 polyp candidates. Support vector machine (SVM) and random forests (RFs) were employed as basic classifiers. Two imbalanced data correcting techniques, i.e., cost-sensitive learning and training data down sampling, were applied to SVM and RFs, and their performances were compared with the proposed strategies. Comparing to the original thresholding method, i.e., 0.488 sensitivity and 0.986 specificity for RFs and 0.526 sensitivity and 0.977 specificity for SVM, our strategies achieved more balanced results, which are around 0.89 sensitivity and 0.92 specificity for RFs and 0.88 sensitivity and 0.90 specificity for SVM. Meanwhile, their performance remained at the same level regardless of whether other correcting methods are used.
Based on the above experiments, the gain of our proposed strategies is noticeable: the sensitivity improved from 0.5 to around 0.88 for RFs and 0.89 for SVM while remaining a relatively high level of specificity, i.e., 0.92 for RFs and 0.90 for SVM. The performance of our proposed strategies was adaptive and robust with different levels of imbalanced data. This indicates a feasible solution to the shifting problem for favorable sensitivity and specificity in CAD of polyps from imbalanced data.
KeywordsComputer-aided detection and diagnosis (CAD) Computed tomography colonography (CTC) Random forests Harmonic mean Support vector machine (SVM) Receiver operating characteristic (ROC)
This work was supported in part by the NIH/NCI under Grants #CA082402 and #CA143111.
Conflict of Interest
Bowen Song, Guopeng Zhang, Wei Zhu and Zhengrong Liang declare that they have no conflict of interest.
- 1.American Cancer Society (2012) Cancer facts & figures 2012. American Cancer Society, AtlantaGoogle Scholar
- 3.Gluecker T, Johnson C, Harmsen W, Offord K, Harris A, Wilson L, Ahlquist D (2003) Colorectal cancer screening with CT colonography, colonoscopy, and double-contrast barium enema examination: prospective assessment of patient perceptions and preferences. Radiology 227(2):378–384PubMedCrossRefGoogle Scholar
- 8.Hossain M, Hassan M, Kirley M, Bailey J (2008) ROC-tree: a novel decision tree induction algorithm based on receiver operating characteristics to classify gene expression data. In: Proceedings of the 2008 SIAM international conference on data mining (SDM), pp 455–465Google Scholar
- 10.Rakotomamonjy A (2004) Optimizing area under ROC curve with SVMs. ROC Analysis in Artificial Intelligence, pp 71–80Google Scholar
- 11.Zhao P, Hoi SCH, Jin R, Yang T (2011) Online AUC maximization. In: Proceeding of international conference of machine learningGoogle Scholar
- 15.Zhu H, Duan C, Pickhardt P, Wang S, Liang Z (2009) CAD of colonic polyps with level set-based adaptive convolution in volumetric mucosa to advance CT colonography toward a screening modality. J Cancer Manag Res DOVE Med Press 1:1–13Google Scholar
- 16.Marelo F, Musé P, Aguirre S, Sapiro G (2010) Automatic colon polyp flagging via geometric and texture features. Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE, pp 3170–3173Google Scholar
- 18.American College of Radiology (2005) ACR practice guideline for the performance of computed tomography (CT) colonography in adults. ACR Pract Guidel 29:295–298Google Scholar
- 19.Breiman L (1996) Bagging predictors. Mach Learn 24:123–140Google Scholar
- 20.Breiman L (2001) Random forests. Mach Learn 45(1):5–32Google Scholar
- 21.Vapnik V (1998) Statistical learning theory. Wiley, New YorkGoogle Scholar
- 22.Morik K, Brokhausen P, Joachims T (1999) Combining statistical learning with a knowledge-based approach—a case study in intensive care monitoring. In: Proceedings 16th international conference on machine learningGoogle Scholar
- 23.Chang C, Lin C (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2:27:1–27:27. Software available at http://www.csie.ntu.edu.tw/~jlin/libsvm
- 24.Osuna E, Freund R, Girosi F (1997) Training support vector machines: an application to face detection. In: Proceedings computer vision and pattern recognition pp 130–136Google Scholar
- 25.Pontil M, Verri A (1998) Object recognition with support vector machines. IEEE Trans Pattern Anal Mach Intell 20:637–646 Google Scholar
- 26.Diaz-Uriarte R, Alvarez de Andres S (2006) Gene selection and classification of microarray data using random forest. BMC Bioinformatics. doi: 10.1186/1471-2105-7-3
- 27.Alexandre LA, Casteleiro J, Nobreinst N (2007) Polyp detection in endoscopic video using SVMs. Lect Notes Comput Sci 4702:358–365Google Scholar
- 29.Liu M, Lu L, Bi J, Raykar V, Wolf M, Salganicoff M (2011) Robust large scale prone-supine polyp matching using local features: a metric learning approach. Med Image Comput Assist Interv 14(3):75–82Google Scholar
- 30.Liu M, Lu L, Ye X, Yu J, Salganicoff M (2011) Sparse classification for computer aided diagnosis using learned dictionaries. In: Proceedings of the 14th international conference on medical image computing and computer assisted intervention (MICCAI), September 18–22, 2011, Toronto, CanadaGoogle Scholar
- 32.Chen C, Liaw A, Breiman L (2004) Using random forest to learn Imbalanced data. Technical Report of Dept. of Stat., UC, BerkeleyGoogle Scholar
- 35.Maloof M (2003) Learning when data sets are imbalanced and when cost are unequal and unknown. In: Proceedings ICML workshop learn imbalanced data sets, pp 73–80Google Scholar