Skip to main content
SpringerLink
Log in

Healthcare Text Classification System and its Performance Evaluation: A Source of Better Intelligence by Characterizing Healthcare Text

  • Education & Training
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

A machine learning (ML)-based text classification system has several classifiers. The performance evaluation (PE) of the ML system is typically driven by the training data size and the partition protocols used. Such systems lead to low accuracy because the text classification systems lack the ability to model the input text data in terms of noise characteristics. This research study proposes a concept of misrepresentation ratio (MRR) on input healthcare text data and models the PE criteria for validating the hypothesis. Further, such a novel system provides a platform to amalgamate several attributes of the ML system such as: data size, classifier type, partitioning protocol and percentage MRR. Our comprehensive data analysis consisted of five types of text data sets (TwitterA, WebKB4, Disease, Reuters (R8), and SMS); five kinds of classifiers (support vector machine with linear kernel (SVM-L), MLP-based neural network, AdaBoost, stochastic gradient descent and decision tree); and five types of training protocols (K2, K4, K5, K10 and JK). Using the decreasing order of MRR, our ML system demonstrates the mean classification accuracies as: 70.13 ± 0.15%, 87.34 ± 0.06%, 93.73 ± 0.03%, 94.45 ± 0.03% and 97.83 ± 0.01%, respectively, using all the classifiers and protocols. The corresponding AUC is 0.98 for SMS data using Multi-Layer Perceptron (MLP) based neural network. All the classifiers, the best accuracy of 91.84 ± 0.04% is shown to be of MLP-based neural network and this is 6% better over previously published. Further we observed that as MRR decreases, the system robustness increases and validated by standard deviations. The overall text system accuracy using all data types, classifiers, protocols is 89%, thereby showing the entire ML system to be novel, robust and unique. The system is also tested for stability and reliability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Rico, T. M., dos Santos Machado, K., Fernandes, V. P., Madruga, S. W., Noguez, P. T., Barcelos, C. R. G., and Dumith, S. C., Text messaging (SMS) helping cancer care in patients undergoing chemotherapy treatment: a Pilot study. J. Med. Syst. 41(11):181, 2017.

    Article  PubMed  Google Scholar 

  2. Lee, K., Agrawal, A., and Choudhary, A., Real-time disease surveillance using twitter data: demonstration on flu and cancer. In Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining, 1474-1477, 2013.

  3. Rios-Alvarado, A. B., Lopez-Arevalo, I., Tello-Leal, E., and Sosa-Sosa, V. J., An approach for learning expressive ontologies in medical domain. J. Med. Syst. 39(8):75, 2015.

    Article  PubMed  Google Scholar 

  4. Li, G. Z., Yang, J., Liu, G. P., and Xue, L., Feature selection for multi-class problems using support vector machines. In PRICAI, 292-300, 2004.

  5. Vahdat, S., Hamzehgardeshi, L., Hessam, S., and Hamzehgardeshi, Z., Patient involvement in health care decision making: a review. Iran Red Crescent Med. J. 16(1):1–7, 2014.

    Article  Google Scholar 

  6. Acharya, U. R., Faust, O., Sree, S. V., Molinari, F., Saba, L., Nicolaides, A., and Suri, J. S., An accurate and generalized approach to plaque characterization in 346 carotid ultrasound scans. IEEE Trans. Instrum. Measure. 61(4):1045–1053, 2012.

    Article  Google Scholar 

  7. Acharya, U. R., Sree, S. V., Saba, L., Molinari, F., Guerriero, S., and Suri, J. S., Ovarian tumor characterization and classification using ultrasound—a new online paradigm. J. Digit. Imaging 26(3):544–553, 2013.

    Article  PubMed  Google Scholar 

  8. Kautz, T., Eskofier, B. M., and Pasluosta, C. F., Generic performance measure for multiclass-classifiers. Pattern Recogn. 68:111–125, 2017.

    Article  Google Scholar 

  9. Japkowicz, N., and Shah, M., Evaluating learning algorithms: a classification perspective. Cambridge University Press. 2011.

  10. Sokolova, M., and Lapalme, G., A systematic analysis of performance measures for classification tasks. Inform. Process. Manag. 45(4):427–437, 2009.

    Article  Google Scholar 

  11. Huang, J., and Ling, C. X., Constructing new and better evaluation measures for machine learning. In IJCAI, 859-864, 2007.

  12. Wong, A. K., Lee, J. W., and Yeung, D. S., Improving text classifier performance based on AUC. In Pattern Recognition, 2006. ICPR 2006. 18 th , 1-4, 2006.

  13. Iwata, T., Tanaka, T., Yamada, T., and Ueda, N., Improving classifier performance using data with different taxonomies. IEEE Trans. Knowledge Data Eng. 23(11):1668–1677, 2011.

    Article  Google Scholar 

  14. Sriram, B., Fuhry, D., Demir, E., Ferhatosmanoglu, H., and Demirbas, M., Short text classification in twitter to improve information filtering. In Proceedings of the33rd international ACM SIGIR conference on Research and development in information retrieval, 841-842, 2010.

  15. Caragea, C., Wu, J., Gollapalli, S. D., and Giles, C. L., Document Type Classification in Online Digital Libraries. AAAI, 3997-4002, 2016.

  16. Shrivastava, V. K., Londhe, N. D., Sonawane, R. S., and Suri, J. S., Reliable and accurate psoriasis disease classification in dermatology images using comprehensive feature space in machine learning paradigm. Expert Syst. Appl. 42(15):6184–6195, 2015.

    Article  Google Scholar 

  17. Shrivastava, V. K., Londhe, N. D., Sonawane, R. S., and Suri, J. S., Computer- aided diagnosis of psoriasis skin images with HOS, texture and color features: a first comparative study of its kind. Comput. Methods Prog. Biomed. 126:98–109, 2016.

    Article  Google Scholar 

  18. Cormack, G. V., Gómez Hidalgo, J. M., and Sánz, E. P., Spam filtering for short messages. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, 313-320, 2007.

  19. Liang, J. G., Zhou, X. F., Liu, P., Guo, L., and Bai, S., An EMM-based Approach for Text Classification. Proc. Comput. Sci. 17:506–513, 2013.

    Article  Google Scholar 

  20. Lu, C., Zhang, X., Park, J. R., Hu, X., & He, T., Web clustering based on the information of sibling pages. In Granular Computing, 2008. GrC 2008. IEEE International Conference, 480–485, 2008.

  21. Tuarob, S., Tucker, C. S., Salathe, M., and Ram, N., An ensemble heterogeneous classification methodology for discovering health-related knowledge in social media messages. J. Biomed. Inform. 49:255–268, 2014.

    Article  PubMed  Google Scholar 

  22. Roesslein, J. (2009). tweepy documentation. Online http://tweepy.readthedocs.io/en/v3 , 5.

  23. Velardi, P., Stilo, G., Tozzi, A. E., and Gesualdo, F., Twitter mining for fine- grained syndromic surveillance. Artif. Intell. Med. 61(3):153–163, 2014.

    Article  PubMed  Google Scholar 

  24. Srivastava, S. K., and Singh, S. K., Multi-Parameter Based Performance Evaluation Of Classification Algorithms. Int. J. Comput Sci. Inform. Technol. (IJCSIT) 7:115–125, 2015.

    Article  Google Scholar 

  25. Acharya, U. R., Mookiah, M. R. K., Sree, S. V., Afonso, D., Sanches, J., Shafique, S., and Suri, J. S., Atherosclerotic plaque tissue characterization in 2D ultrasound longitudinal carotid scans for automated classification: a paradigm for stroke risk assessment. Med Biol. Eng. Comput. 51(5):513–523, 2013.

    Article  PubMed  Google Scholar 

  26. Sanchez, A., and V. D., Advanced support vector machines and kernel methods. Neurocomputing 55(1–2):5–20, 2003.

    Article  Google Scholar 

  27. Muller, K. R., Mika, S., Ratsch, G., Tsuda, K., and Scholkopf, B., An introduction to kernel-based learning algorithms. IEEE Trans. Neural Netw. 12(2):181–201, 2001.

    Article  PubMed  CAS  Google Scholar 

  28. Acharya, R. U., Faust, O., Alvin, A. P. C., Sree, S. V., Molinari, F., Saba, L., and Suri, J. S., Symptomatic vs. asymptomatic plaque classification in carotid ultrasound. J. Med. Syst. 36(3):1861–1871, 2012.

    Article  PubMed  Google Scholar 

  29. Chakravarty, S. (2011). Stochastic Gradient Descent Methods for large scale pattern classification.

    Google Scholar 

  30. Martineau, J., and Finin, T., Delta TFIDF: an improved feature space for sentiment analysis. Icwsm 9:106, 2009.

    Google Scholar 

  31. Robert, M. H., & Linda, G. S., Computer and robot vision. Vol. I, Addison-Wesley, 28–48, 1992.

  32. Suri, J. S., Haralick, R. M., and Sheehan, F.H., Left ventricle longitudinal axis fitting and its apex estimation using a robust algorithm and its performance: a parametric apex model. In Image Processing, 1997. Proceedings., International Conference on (Vol. 3, pp. 118-121). IEEE, 1997.

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, JIIT, Noida, India

    Saurabh Kumar Srivastava & Sandeep Kumar Singh

  2. Advanced Knowledge Engineering Center, Global Biomedical Technologies, Inc., Roseville, CA, USA

    Jasjit S. Suri

Authors
  1. Saurabh Kumar Srivastava
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sandeep Kumar Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jasjit S. Suri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jasjit S. Suri.

Ethics declarations

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Additional information

This article is part of the Topical Collection on Education & Training

Appendices

Appendix A: Types of Dataset used in the study

A.1. TwitterA Dataset

A.2. WebKB4 Dataset

A.3. Disease Dataset

A.4. Reuters (R8) Dataset

A.5. SMS Dataset

Appendix B: Labels used in different text data types

Appendix C: ROC Curves

C1: ROC curves for K2 protocol using five classifier

C2: ROC curves for K4 protocol using five classifier

C3: ROC curves for K5 protocol using five classifier

C4: ROC curves for K10 protocol using five classifier

C5: ROC curves for JK protocol using five classifier

Appendix D: AUC Tables

Appendix E: Postive Predictive Value Tables

Appendix F: Sensitivity Tables

Appendix G: Specificity Tables

Appendix H: List of Abbreviations/Symbols

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Srivastava, S.K., Singh, S.K. & Suri, J.S. Healthcare Text Classification System and its Performance Evaluation: A Source of Better Intelligence by Characterizing Healthcare Text. J Med Syst 42, 97 (2018). https://doi.org/10.1007/s10916-018-0941-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-018-0941-6

Keywords

Search

Navigation