Healthcare Text Classification System and its Performance Evaluation: A Source of Better Intelligence by Characterizing Healthcare Text

  • Saurabh Kumar Srivastava
  • Sandeep Kumar Singh
  • Jasjit S. Suri
Education & Training
  • 25 Downloads
Part of the following topical collections:
  1. Education & Training

Abstract

A machine learning (ML)-based text classification system has several classifiers. The performance evaluation (PE) of the ML system is typically driven by the training data size and the partition protocols used. Such systems lead to low accuracy because the text classification systems lack the ability to model the input text data in terms of noise characteristics. This research study proposes a concept of misrepresentation ratio (MRR) on input healthcare text data and models the PE criteria for validating the hypothesis. Further, such a novel system provides a platform to amalgamate several attributes of the ML system such as: data size, classifier type, partitioning protocol and percentage MRR. Our comprehensive data analysis consisted of five types of text data sets (TwitterA, WebKB4, Disease, Reuters (R8), and SMS); five kinds of classifiers (support vector machine with linear kernel (SVM-L), MLP-based neural network, AdaBoost, stochastic gradient descent and decision tree); and five types of training protocols (K2, K4, K5, K10 and JK). Using the decreasing order of MRR, our ML system demonstrates the mean classification accuracies as: 70.13 ± 0.15%, 87.34 ± 0.06%, 93.73 ± 0.03%, 94.45 ± 0.03% and 97.83 ± 0.01%, respectively, using all the classifiers and protocols. The corresponding AUC is 0.98 for SMS data using Multi-Layer Perceptron (MLP) based neural network. All the classifiers, the best accuracy of 91.84 ± 0.04% is shown to be of MLP-based neural network and this is 6% better over previously published. Further we observed that as MRR decreases, the system robustness increases and validated by standard deviations. The overall text system accuracy using all data types, classifiers, protocols is 89%, thereby showing the entire ML system to be novel, robust and unique. The system is also tested for stability and reliability.

Keywords

Healthcare text classification Machine learning Classifiers Misrepresentation ratio Reliability Stability 

Notes

Compliance with Ethical Standards

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

  1. 1.
    Rico, T. M., dos Santos Machado, K., Fernandes, V. P., Madruga, S. W., Noguez, P. T., Barcelos, C. R. G., and Dumith, S. C., Text messaging (SMS) helping cancer care in patients undergoing chemotherapy treatment: a Pilot study. J. Med. Syst. 41(11):181, 2017.CrossRefPubMedGoogle Scholar
  2. 2.
    Lee, K., Agrawal, A., and Choudhary, A., Real-time disease surveillance using twitter data: demonstration on flu and cancer. In Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining, 1474-1477, 2013.Google Scholar
  3. 3.
    Rios-Alvarado, A. B., Lopez-Arevalo, I., Tello-Leal, E., and Sosa-Sosa, V. J., An approach for learning expressive ontologies in medical domain. J. Med. Syst. 39(8):75, 2015.CrossRefPubMedGoogle Scholar
  4. 4.
    Li, G. Z., Yang, J., Liu, G. P., and Xue, L., Feature selection for multi-class problems using support vector machines. In PRICAI, 292-300, 2004.Google Scholar
  5. 5.
    Vahdat, S., Hamzehgardeshi, L., Hessam, S., and Hamzehgardeshi, Z., Patient involvement in health care decision making: a review. Iran Red Crescent Med. J. 16(1):1–7, 2014.CrossRefGoogle Scholar
  6. 6.
    Acharya, U. R., Faust, O., Sree, S. V., Molinari, F., Saba, L., Nicolaides, A., and Suri, J. S., An accurate and generalized approach to plaque characterization in 346 carotid ultrasound scans. IEEE Trans. Instrum. Measure. 61(4):1045–1053, 2012.CrossRefGoogle Scholar
  7. 7.
    Acharya, U. R., Sree, S. V., Saba, L., Molinari, F., Guerriero, S., and Suri, J. S., Ovarian tumor characterization and classification using ultrasound—a new online paradigm. J. Digit. Imaging 26(3):544–553, 2013.CrossRefPubMedGoogle Scholar
  8. 8.
    Kautz, T., Eskofier, B. M., and Pasluosta, C. F., Generic performance measure for multiclass-classifiers. Pattern Recogn. 68:111–125, 2017.CrossRefGoogle Scholar
  9. 9.
    Japkowicz, N., and Shah, M., Evaluating learning algorithms: a classification perspective. Cambridge University Press. 2011.Google Scholar
  10. 10.
    Sokolova, M., and Lapalme, G., A systematic analysis of performance measures for classification tasks. Inform. Process. Manag. 45(4):427–437, 2009.CrossRefGoogle Scholar
  11. 11.
    Huang, J., and Ling, C. X., Constructing new and better evaluation measures for machine learning. In IJCAI, 859-864, 2007.Google Scholar
  12. 12.
    Wong, A. K., Lee, J. W., and Yeung, D. S., Improving text classifier performance based on AUC. In Pattern Recognition, 2006. ICPR 2006. 18 th , 1-4, 2006. Google Scholar
  13. 13.
    Iwata, T., Tanaka, T., Yamada, T., and Ueda, N., Improving classifier performance using data with different taxonomies. IEEE Trans. Knowledge Data Eng. 23(11):1668–1677, 2011.CrossRefGoogle Scholar
  14. 14.
    Sriram, B., Fuhry, D., Demir, E., Ferhatosmanoglu, H., and Demirbas, M., Short text classification in twitter to improve information filtering. In Proceedings of the33rd international ACM SIGIR conference on Research and development in information retrieval, 841-842, 2010.Google Scholar
  15. 15.
    Caragea, C., Wu, J., Gollapalli, S. D., and Giles, C. L., Document Type Classification in Online Digital Libraries. AAAI, 3997-4002, 2016.Google Scholar
  16. 16.
    Shrivastava, V. K., Londhe, N. D., Sonawane, R. S., and Suri, J. S., Reliable and accurate psoriasis disease classification in dermatology images using comprehensive feature space in machine learning paradigm. Expert Syst. Appl. 42(15):6184–6195, 2015.CrossRefGoogle Scholar
  17. 17.
    Shrivastava, V. K., Londhe, N. D., Sonawane, R. S., and Suri, J. S., Computer- aided diagnosis of psoriasis skin images with HOS, texture and color features: a first comparative study of its kind. Comput. Methods Prog. Biomed. 126:98–109, 2016.CrossRefGoogle Scholar
  18. 18.
    Cormack, G. V., Gómez Hidalgo, J. M., and Sánz, E. P., Spam filtering for short messages. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, 313-320, 2007.Google Scholar
  19. 19.
    Liang, J. G., Zhou, X. F., Liu, P., Guo, L., and Bai, S., An EMM-based Approach for Text Classification. Proc. Comput. Sci. 17:506–513, 2013.CrossRefGoogle Scholar
  20. 20.
    Lu, C., Zhang, X., Park, J. R., Hu, X., & He, T., Web clustering based on the information of sibling pages. In Granular Computing, 2008. GrC 2008. IEEE International Conference, 480–485, 2008.Google Scholar
  21. 21.
    Tuarob, S., Tucker, C. S., Salathe, M., and Ram, N., An ensemble heterogeneous classification methodology for discovering health-related knowledge in social media messages. J. Biomed. Inform. 49:255–268, 2014.CrossRefPubMedGoogle Scholar
  22. 22.
    Roesslein, J. (2009). tweepy documentation. Online http://tweepy.readthedocs.io/en/v3, 5.
  23. 23.
    Velardi, P., Stilo, G., Tozzi, A. E., and Gesualdo, F., Twitter mining for fine- grained syndromic surveillance. Artif. Intell. Med. 61(3):153–163, 2014.CrossRefPubMedGoogle Scholar
  24. 24.
    Srivastava, S. K., and Singh, S. K., Multi-Parameter Based Performance Evaluation Of Classification Algorithms. Int. J. Comput Sci. Inform. Technol. (IJCSIT) 7:115–125, 2015.CrossRefGoogle Scholar
  25. 25.
    Acharya, U. R., Mookiah, M. R. K., Sree, S. V., Afonso, D., Sanches, J., Shafique, S., and Suri, J. S., Atherosclerotic plaque tissue characterization in 2D ultrasound longitudinal carotid scans for automated classification: a paradigm for stroke risk assessment. Med Biol. Eng. Comput. 51(5):513–523, 2013.CrossRefPubMedGoogle Scholar
  26. 26.
    Sanchez, A., and V. D., Advanced support vector machines and kernel methods. Neurocomputing 55(1–2):5–20, 2003.CrossRefGoogle Scholar
  27. 27.
    Muller, K. R., Mika, S., Ratsch, G., Tsuda, K., and Scholkopf, B., An introduction to kernel-based learning algorithms. IEEE Trans. Neural Netw. 12(2):181–201, 2001.CrossRefPubMedGoogle Scholar
  28. 28.
    Acharya, R. U., Faust, O., Alvin, A. P. C., Sree, S. V., Molinari, F., Saba, L., and Suri, J. S., Symptomatic vs. asymptomatic plaque classification in carotid ultrasound. J. Med. Syst. 36(3):1861–1871, 2012.CrossRefPubMedGoogle Scholar
  29. 29.
    Chakravarty, S. (2011). Stochastic Gradient Descent Methods for large scale pattern classification.Google Scholar
  30. 30.
    Martineau, J., and Finin, T., Delta TFIDF: an improved feature space for sentiment analysis. Icwsm 9:106, 2009.Google Scholar
  31. 31.
    Robert, M. H., & Linda, G. S., Computer and robot vision. Vol. I, Addison-Wesley, 28–48, 1992.Google Scholar
  32. 32.
    Suri, J. S., Haralick, R. M., and Sheehan, F.H., Left ventricle longitudinal axis fitting and its apex estimation using a robust algorithm and its performance: a parametric apex model. In Image Processing, 1997. Proceedings., International Conference on (Vol. 3, pp. 118-121). IEEE, 1997.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Saurabh Kumar Srivastava
    • 1
  • Sandeep Kumar Singh
    • 1
  • Jasjit S. Suri
    • 2
  1. 1.Department of Computer Science & EngineeringJIITNoidaIndia
  2. 2.Advanced Knowledge Engineering CenterGlobal Biomedical Technologies, Inc.RosevilleUSA

Personalised recommendations