Skip to main content

Advertisement

SpringerLink
Log in

R-JaunLab: Automatic Multi-Class Recognition of Jaundice on Photos of Subjects with Region Annotation Networks

  • Original Paper
  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

Jaundice occurs as a symptom of various diseases, such as hepatitis, the liver cancer, gallbladder or pancreas. Therefore, clinical measurement with special equipment is a common method that is used to identify the total serum bilirubin level in patients. Fully automated multi-class recognition of jaundice combines two key issues: (1) the critical difficulties in multi-class recognition of jaundice approaches contrasting with the binary class and (2) the subtle difficulties in multi-class recognition of jaundice represent extensive individuals variability of high-resolution photos of subjects, huge coherency between healthy controls and occult jaundice, as well as broadly inhomogeneous color distribution. We introduce a novel approach for multi-class recognition of jaundice to detect occult jaundice, obvious jaundice and healthy controls. First, region annotation network is developed and trained to propose eye candidates. Subsequently, an efficient jaundice recognizer is proposed to learn similarities, context, localization features and globalization characteristics on photos of subjects. Finally, both networks are unified by using shared convolutional layer. Evaluation of the structured model in a comparative study resulted in a significant performance boost (categorical accuracy for mean 91.38%) over the independent human observer. Our work was exceeded against the state-of-the-art convolutional neural network (96.85% and 90.06% for training and validation subset, respectively) and showed a remarkable categorical result for mean 95.33% on testing subset. The proposed network makes a performance better than physicians. This work demonstrates the strength of our proposal to help bringing an efficient tool for multi-class recognition of jaundice into clinical practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Chambers CV, Irwin CE: Intense jaundice in an adolescent. an unusual presentation of infectious mononucleosis. J. Adolesc. Health Care, 7(3):195–197, 1986

  2. Anand AC, Puri P: Jaundice in malaria. J Gastroenterol Hepatol, 20(9):1322–1332, 2010

    Article  Google Scholar 

  3. Chan LY, Tsang WC, Hui Y, Leung WY, Chan KL, Sung JY: The role of lamivudine and predictors of mortality in severe flare-up of chronic hepatitis b with jaundice. J Viral Hepat, 9(6):424–428, 2010

    Article  Google Scholar 

  4. Howard R, Watson CJ: Antecedent jaundice in cirrhosis of the liver. Arch Intern Med, 80(1):1–10, 1947

    Article  CAS  Google Scholar 

  5. Hawkins WG, Dematteo RP, Jarnagin WR, Ben-Porat L, Fong Y: Jaundice predicts advanced disease and early mortality in patients with gallbladder cancer. Ann Surg Oncol, 11(3):310–315, 2004

    Article  PubMed  Google Scholar 

  6. Brandabur JJ, Kozarek RA, Ball TJ, Hofer BO, Jr RJ, Traverso LW, Freeny PC, Lewis GP: Nonoperative versus operative treatment of obstructive jaundice in pancreatic cancer: cost and survival analysis. Am J Gastroenterol, 83(10):1132, 1988

  7. Larry C: Oxford textbook of primary medical care. J R Soc Med, 97(6):304, 2004

    Google Scholar 

  8. Maisels MJ: Managing the jaundiced newborn: a persistent challenge, CMAJ

  9. Roche SP, Kobos R: Jaundice in the adult patient. Am Fam Physician, 69(2):299–304, 2004

    PubMed  Google Scholar 

  10. Labori KJ, Raeder MG: Diagnostic approach to the patient with jaundice following trauma. Scandinavian Journal of Surgery Sjs Official Organ for the Finnish Surgical Society & the Scandinavian Surgical Society, 93(3):176, 2004

    CAS  Google Scholar 

  11. Winger J, Michelfelder A: Diagnostic approach to the patient with jaundice. Prim Care, 38(3):469–482, 2011

    Article  PubMed  Google Scholar 

  12. Aydım M, Hardala FC, Ural B, Karap S: Neonatal jaundice detection system, J Med Syst. 40(7):166, 2016

  13. Halder A, Banerjee M, Singh S, Adhikari A, Sarkar PK, Bhattacharya AM, Chakrabarti P, Bhattacharyya D, Mallick AK, Pal SK: A novel whole spectrum-based non-invasive screening device for neonatal hyperbilirubinemia. IEEE J Biomed Health Inform, PP(99):1.

  14. Mannino RG, Myers DR, Tyburski EA, Caruso C, Boudreaux J, Leong T, Clifford GD, Lam WA: Smartphone app for non-invasive detection of anemia using only patient-sourced photos. Nat Commun, 9(1), 2018

  15. Padidar P, Shaker M, Amoozgar H, Khorraminejad-Shirazi M, Hemmati F, Najib KS, Pourarian S: Detection of neonatal jaundice by using an android os-based smartphone application. Iran J Pediatr, 29(2):e84397, 2019

    Google Scholar 

  16. Thompson BL, Wyckoff SL, Haverstick DM, Landers JP: Simple, reagentless quantification of total bilirubin in blood via microfluidic phototreatment and image analysis. Anal Chem, 89(5):3228–3234, 2017.

    Article  CAS  PubMed  Google Scholar 

  17. Saha S, Saha S, Bhattacharyya PP: Classifier fusion for liver function test based indian jaundice classification. In International Conference on Man & Machine Interfacing, 2016.

  18. Saini N, Kumar A: Comparison of non-invasive bilirubin detection techniques for jaundice prediction, 2016.

  19. Wang X, Zhang A, Han Y, Wang P, Sun H, Song G, Dong T, Yuan Y, Yuan X, Zhang M. Urine metabolomics analysis for biomarker discovery and detection of jaundice syndrome in patients with liver disease. Mol Cell Proteomics Mcp, 11(8):370, 2012

    Article  Google Scholar 

  20. Zulkarnay Z, Jurimah AJ, Ibrahim B, Shazwani S, Nasir MAKA: An overview on jaundice measurement and application in biomedical: The potential of non-invasive method. In International Conference on Biomedical Engineering, 2015.

  21. Knill-Jones RP, Stern RB, Girmes DH, Maxwell JD, Thompson RP, Williams R: Use of sequential bayesian model in diagnosis of jaundice by computer. Br Med J, 1(5852):530–533, 1973

  22. Laddi A, Kumar S, Sharma S, Kumar A: Non-invasive jaundice detection using machine vision. IETE J Res, 59(5):591–596, 2013

    Article  Google Scholar 

  23. Krizhevsky A, Sutskever I, Hinton G: Imagenet classification with deep convolutional neural networks. In International Conference on Neural Information Processing Systems, 2012

  24. Kumar M, Dargon S: A survey of deep learning and its applications: A new paradigm to machine learning. Arch Comput Meth Eng, 2019

  25. LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD: Backpropagation applied to handwritten zip code recognition. Neural Comput, 1(4):541–551, 2014

    Article  Google Scholar 

  26. Payal C, Kumar GN, Munish K. Content-based image retrieval system using orb and sift features. Neural Comput Applic, 32:2725–2733, 2020

    Article  Google Scholar 

  27. He K, Zhang X, Ren S, Sun J: Deep residual learning for image recognition. In 2016 IEEE Conf Comp Vis Pattern Recognit (CVPR), pages 770–778, 2016

  28. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ: Densely connected convolutional networks. In 2017 IEEE Conf Comput Visi Pattern Recognit (CVPR), pages 2261–2269, 2017

  29. Simonyan K, Zisserman A: Very deep convolutional networks for large-scale image recognition. Computer Science, 2014

  30. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z: Rethinking the inception architecture for computer vision. 2015

  31. Girshick R: Fast r-cnn. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1440–1448, 2015

  32. Girshick R, Donahue J, Darrelland T, Malik J: Rich feature hierarchies for object detection and semantic segmentation. In IEEE Conference on Computer Vision & Pattern Recognition, 2014

  33. He K, Georgia G, Piotr D, Ross G: Mask r-cnn. IEEE Trans Pattern Anal Mach Intell, PP(99):1, 2017

  34. He K, Zhang X, Ren S, Sun J: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell, 37(9):1904–16, 2014

    Article  Google Scholar 

  35. Jung C, Sun T, Jiao L: Eye detection under varying illumination using the retinex theory. Neurocomputing, 113(596):130–137, 2013

    Article  Google Scholar 

  36. Ren S, He K, Girshick R, Sun J. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Trans Patt Anal Mach Intell, 39(6):1137–1149, 2017

    Article  PubMed  Google Scholar 

  37. Soetedjo A: Eye detection based-on color and shape features. Int J Adv Comput Sci Appl, 3(5), 2012

  38. Kumar M, Bansal M, Kumar M. 2d object recognition techniques: State-of-the-art work. Archives of Computational Methods in Engineering, 02 2020

  39. Kumar M, Dargan S: A comprehensive survey on the biometric recognition systems based on physiological and behavioral modalities. Expert Systems with Applications, pages 1–27, 11 2019

  40. Kumar M, Gupta S, Thakur K: 2d-human face recognition using sift and surf descriptors of face’s feature regions. Vis Comput, 01 2020

  41. Kumar M, Kumar R, Kaur P: A healthcare monitoring system using random forest and internet of things (iot). Multimed Tools Appl, 02 2019

  42. Bland JM, Altman D: Measuring agreement in method comparison studies. Stat Methods Med Res, 8:135–60, 1999

    Article  CAS  PubMed  Google Scholar 

  43. PS Myles, Cui JI: using the bland altman method to measure agreement with repeated measures. Br J Anaesth, 99(3):309–311, 2007

  44. Ronneberger O, Fischer P, Brox T: U-net: Convolutional networks for biomedical image segmentation. 2015

  45. Nair V, Hinton GE: Rectified Linear Units Improve Restricted Boltzmann Machines. In International Conference on International Conference on Machine Learning, 2010

  46. Goroshin R, Mathieu M, LeCun Y: Learning to Linearize Under Uncertainty. CoRR, abs/1506.03011, 2015

  47. Creswell A, Arulkumaran K, Bharath AA: On denoising autoencoders trained to minimise binary cross-entropy. 2017

  48. Deng J, Dong W, Socher R, Li LJ, Li FF: Imagenet: a large-scale hierarchical image database. In IEEE Conference on Computer Vision & Pattern Recognition, 2009

  49. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M: Imagenet large scale visual recognition challenge. Int J Comput Vis, 115(3):211–252, 2015

    Article  Google Scholar 

  50. Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou S: Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging, 35(5):1207–1216, 2016

    Article  PubMed  Google Scholar 

  51. Cho J, Lee K, Shin E, Choy G, Do S: How much data is needed to train a medical image deep learning system to achieve necessary high accuracy? Computer Science, 2015

  52. Greenspan H, van Ginneken B, Summers RM: Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Trans Med Imaging, 35(5):1153–1159, 2016

    Article  Google Scholar 

  53. Rajkomar A, Lingam S, Taylor AG, Blum M, Mongan J: High-throughput classification of radiographs using deep convolutional neural networks. J Dig Imaging, 30(1):95–101, 2017

    Article  Google Scholar 

  54. Shin HC, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM: Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging, 35(5):1285–1298, 2016

    Article  PubMed  Google Scholar 

  55. Tajbakhsh N, Shin JY, Gurudu SR, Hurst RT, Kendall CB, Gotway MB, Liang J. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Trans Med Imaging, 35(5):1299–1312, 2016

    Article  PubMed  Google Scholar 

  56. Wang Z, Meng Y, Weng F, Chen Y, Lu F, Liu X, Hou M, Zhang J: An effective cnn method for fully automated segmenting subcutaneous and visceral adipose tissue on ct scans. Ann Biomed Eng, pages 1–17, 2019

  57. KA: Cs231n course notes: Transfer learning [online]. Accessed: 19-May-2016 http://cs231n.github.io/transfer-learning.

  58. Razavian AS, Azizpour H, Sullivan J, Carlsson S: Cnn features off-the-shelf: An astounding baseline for recognition. pages 512–519, 2014

  59. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A: Learning deep features for discriminative localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2921–2929, 2016

  60. Zeiler MD, Fergus R: Visualizing understanding convolutional networks. 2013

  61. Dosovitskiy A, Springenberg JT, Riedmiller M, Brox T: Discriminative unsupervised feature learning with convolutional neural networks. 2014

  62. Wong SC, Gatt A, Stamatescu V, Mcdonnell MD: Understanding data augmentation for classification: When to warp? 2016

  63. Hinton G, Tieleman T: Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, (4):26–30, 2012

  64. Bengio Y: Speeding up stochastic gradient descent. 2007

  65. Kaleem, Rashid, Sreepathi, Pingali, Keshav. Stochastic gradient descent on gpus. 2015

  66. Dutta P, Saha S, Gulati S: Graph-based hub gene selection technique using protein interaction information: Application to sample classification. IEEE J Biomed Health Inform, PP(99):1, 2019

  67. Ji J, Zhang A, Liu C, Quan X, Liu Z: Survey: Functional module detection from protein-protein interaction networks. IEEE Trans Knowl Data Eng, 26(2):261–277, 2013

    Article  Google Scholar 

  68. Jr MA, Niswender GD, Rebar RW. Principles for the assessment of the reliability of radioimmunoassay methods (precision, accuracy, sensitivity, specificity). Acta Endocrinologica Supplementum, 142(1 Suppl):163, 1969

  69. Kessler RC, Abelson JM, Demler O, Escobar JI, Zheng H: Clinical calibration of dsm-iv diagnoses in the world mental health (wmh) version of the world health organization (who) composite international diagnostic interview (cidi). 13(2):122–139, 2004

  70. Spanhol FA, Oliveira LS, Petitjean C, Heutte L: Breast cancer histopathological image classification using convolutional neural networks. In International Joint Conference on Neural Networks, 2016

  71. Chollet F: Xception: Deep learning with depthwise separable convolutions. In 2017 IEEE Conf Comput Vis Pattern Recognit (CVPR), pages 1800–1807, 2017

  72. Kouvaris K, Clune J, Kounios L, Brede M, Watson RA: How evolution learns to generalise: Principles of under-fitting, over-fitting and induction in the evolution of developmental organisation. Journal of the Society of English & American Literature Kansei Gakuin University, 52(4):93–107, 2015

    Google Scholar 

  73. Cordero C, Schieve LA, Croen LA, Engel SM, Maria ASR, Herring AH, Vladutiu CJ, Seashore CJ, Daniels JL: Neonatal jaundice in association with autism spectrum disorder and developmental disorder. Journal of perinatology: official journal of the California Perinatal Association, 2019

  74. Redfern V, Mortimore G: Right hypochondrial pain leading to diagnosis of cholestatic jaundice and cholecystitis: a review and case study. Gastrointestinal Nursing

  75. Xu X, Zhang X: The application of intravoxel incoherent motion diffusion-weighted imaging in the diagnosis of hilar obstructive jaundice. J Comput Assist Tomogr, 43(2):1, 2019

    Article  CAS  Google Scholar 

  76. Tabatabaee RS, Golmohammadi H, Ahmadi SH: Easy diagnosis of jaundice: A smartphone-based nanosensor bioplatform using photoluminescent bacterial nanopaper for point-of-care diagnosis of hyperbilirubinemia. ACS sensors, 4(4):1063–1071, 2019

    Article  CAS  PubMed  Google Scholar 

  77. Tibana TK, Grubert RM, Fornazari VAV, Barbosa FCP, Bacelar B, Oliveira AF, Marchiori E, Nunes TF: The role of percutaneous transhepatic biliary biopsy in the diagnosis of patients with obstructive jaundice: an initial experience. Radiologia Brasileira, (AHEAD), 2019

  78. Sunwoo MH, Lee JW, Kim JH: Method and apparatus for jaundice diagnosis based on an image, Apr. 18 2019. US Patent App. 16/115,821

  79. Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. Computer Science, 14(7):38–39, 2015

    Google Scholar 

  80. Litjens G, Sanchez CI, Timofeeva N, Hermsen M, Nagtegaal I, Kovacs I, Hulsbergenvan DKC, Bult P, Van GB, Van DLJ: Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Sci Rep, 6(1):26286, 2016

  81. Wang D, Khosla A, Gargeya R, Irshad H, Beck AH: Deep learning for identifying metastatic breast cancer. 2016

  82. Han Z, Wei B, Zheng Y, Yin Y, Li K, Li S. Breast cancer multi-classification from histopathological images with structured deep learning model. Scientific Reports, 7(1):4172, 2017

    Article  PubMed  PubMed Central  Google Scholar 

  83. Takiyama H, Ozawa T, Ishihara S, Fujishiro M, Shichijo S, Nomura S, Miura M, Tada T: Automatic anatomical classification of esophagogastroduodenoscopy images using deep convolutional neural networks. Sci Rep, 8(7497):7497, 2018

    Article  PubMed  PubMed Central  Google Scholar 

  84. Causey JL, Zhang J, Ma S, Jiang B, Qualls JA, Politte DG, Prior F, Zhang S, Huang X: Highly accurate model for prediction of lung nodule malignancy with ct scans. Sci Rep, 8(1):9286, 2018

    Article  PubMed  PubMed Central  Google Scholar 

  85. Deniz CM, Hallyburton S, Welbeck A, Honig S, Cho K, Chang G: Segmentation of the proximal femur from mr images using deep convolutional neural networks. Sci Rep, 8(1), 2018

  86. Ghafoorian M, Karssemeijer N, Heskes T, Uden IWM, Sanchez CI, Litjens G,  Leeuw FE, Ginneken B, Marchiori E, Platel B: Location sensitive deep convolutional neural networks for segmentation of white matter hyperintensities. Sci Rep, 7(1):5110, 2017

    Article  PubMed  PubMed Central  Google Scholar 

  87. Hao C, Qi D, Xi W, Jing Q, Heng PA: Mitosis detection in breast cancer histology images via deep cascaded networks. In Thirtieth Aaai Conference on Artificial Intelligence, 2016

  88. Suk HI, Lee SW, Shen D: Hierarchical feature representation and multimodal fusion with deep learning for ad/mci diagnosis. Neuroimage, 101:569–582, 2014

    Article  PubMed  Google Scholar 

  89. Wang S, Kim M, Wu G, Shen D: Chapter11c scalable high performance image registration framework by unsupervised deep feature representations learning. IEEE Trans Biomed Eng, 63(7):1505–1516, 2016

  90. Hinton GE, Salakhutdinov RR: Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006

    Article  CAS  Google Scholar 

  91. Lecun Y, Bengio Y, Hinton G: Deep learning. Nature, 521(7553):436, 2015

    Article  CAS  Google Scholar 

  92. Shen D, Wu G, Suk HI: Deep learning in medical image analysis. Annu Rev Biomed Eng, 19(1):221–248, 2017

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  93. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A: Going deeper with convolutions. In 2015 IEEE Conf Comput Vis Pattern Recognit (CVPR), pages 1–9, 2015

Download references

Acknowledgements

This work was supported by the Projects of the National Social Science Foundation of China under Grant 19BTJ011 and also funded by the Graduate Student Innovation Foundation of Central South University (2019zzts213).

Author information

Author notes

  1. Zheng Wang and Ying Xiao contributed equally to this work.

Authors and Affiliations

  1. School of Mathematics and Statistics, Central South University, Changsha, Hunan, 410083, China

    Zheng Wang, Futian Weng & Muzhou Hou

  2. Department of Gastroenterology and Hepatology, Shenzhen University General Hospital, Shenzhen, 518055, China

    Yu Meng

  3. Science and Engineering School, Hunan First Normal University, Changsha, 410205, China

    Zheng Wang

  4. Gastroenterology Department of Xiangya Hospital, Central South University, Changsha, 410083, China

    Ying Xiao, Xiaojun Li & Xiaowei Liu

  5. The Second Xiangya Hospital, Central South University, 410083, Changsha, China

    Fanggen Lu

  6. Department of Gastroenterology, Hunan Provincial People’s Hospital, Changsha, 410002, China

    Danhua Zhu

Authors
  1. Zheng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Futian Weng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaojun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Danhua Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Fanggen Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xiaowei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Muzhou Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yu Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Muzhou Hou or Yu Meng.

Ethics declarations

Conflicts of Interest

The authors declare no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Z., Xiao, Y., Weng, F. et al. R-JaunLab: Automatic Multi-Class Recognition of Jaundice on Photos of Subjects with Region Annotation Networks. J Digit Imaging 34, 337–350 (2021). https://doi.org/10.1007/s10278-021-00432-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-021-00432-7

Keywords

Search

Navigation