Skip to main content
SpringerLink
Log in

Convolutional herbal prescription building method from multi-scale facial features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In Traditional Chinese Medicine (TCM), facial features are important basis for diagnosis and treatment. A doctor of TCM can prescribe according to a patient’s physical indicators such as face, tongue, voice, symptoms, pulse. Previous works analyze and generate prescription according to symptoms. However, research work to mine the association between facial features and prescriptions has not been found for the time being. In this work, we try to use deep learning methods to mine the relationship between the patient’s face and herbal prescriptions (TCM prescriptions), and propose to construct convolutional neural networks that generate TCM prescriptions according to the patient’s face image. It is a novel and challenging job. In order to mine features from different granularities of faces, we design a multi-scale convolutional neural network based on three-grained face, which mines the patient’s face information from the organs, local regions, and the entire face. Our experiments show that convolutional neural networks can learn relevant information from face to prescribe, and the multi-scale convolutional neural networks based on three-grained face perform better.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou S (2016) Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging 35(5):1207–1216

    Article  Google Scholar 

  2. Bayramoglu N, Kannala J, Heikkilä J (2016) Deep learning for magnification independent breast cancer histopathology image classification. In: 2016 23rd International conference on pattern recognition (ICPR), pp 2440–2445

  3. Bottou L (2012) Stochastic gradient descent tricks. In: Neural networks: tricks of the trade. 9780201398298. Springer, Berlin, pp 421–436

    Google Scholar 

  4. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  Google Scholar 

  5. Chaabouni S, Benois-pineau J, Tison F, Ben Amar C, Zemmari A (2017) Prediction of visual attention with deep cnn on artificially degraded videos for studies of attention of patients with dementia. Multimed Tools Appl 76(21):22527–22546

    Article  Google Scholar 

  6. Chen M, Shi X, Zhang Y, Wu D, Guizani M (2017) Deep features learning for medical image analysis with convolutional autoencoder neural network. IEEE Trans Big Data, 1

  7. Cheung F (2011) TCM: made in China. Nature 480:S82

    Article  Google Scholar 

  8. Chougrad H, Zouaki H, Alheyane O (2018) Deep convolutional neural networks for breast cancer screening. Comput Methods Programs Biomed 157:19–30

    Article  Google Scholar 

  9. Dehan L, Jia W, Yimin C, Hamid G (2014) Classification of Chinese herbal medicines based on SVM. In: 2014 International conference on information science, electronics and electrical engineering, vol 1, pp 453–456

  10. Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: AISTATS ’11: Proceedings of the 14th international conference on artificial intelligence and statistics, pp 315–323, 1502.03167

  11. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  12. Hon M, Khan NM (2017) Towards Alzheimer’s disease classification through transfer learning. In: 2017 IEEE International conference on bioinformatics and biomedicine (BIBM), pp 1166–1169

  13. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: The IEEE Conference on computer vision and pattern recognition (CVPR)

  14. Huang G, Liu Z, v d Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 2261–2269

  15. Jain V, Learned-Miller E (2010) Fddb: a benchmark for face detection in unconstrained settings. Tech. Rep UM-CS-2010-009. University of Massachusetts, Amherst

    Google Scholar 

  16. Jones AL (2018) The influence of shape and colour cue classes on facial health perception. Evol Hum Behav 39(1):19–29

    Article  Google Scholar 

  17. Kassim YM, Prasath VBS, Glinskii OV, Glinsky VV, Huxley VH, Palaniappan K (2017) Microvasculature segmentation of arterioles using deep CNN. In: 2017 IEEE International conference on image processing (ICIP), pp 580–584

  18. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758

    Google Scholar 

  19. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Advances In Neural Information Processing Systems, pp 1097–1105, 1102.0183

  20. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez C I (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  21. Liu B, Zhou X, Wang Y, Hu J, He L, Zhang R, Chen S, Guo Y (2012) Data processing and analysis in real-world traditional Chinese medicine clinical data: challenges and approaches. Stat Med 31(7): 653–660

    Article  MathSciNet  Google Scholar 

  22. Peng H (1996) Dictionary of traditional Chinese medicine prescriptions. People Health Press, Beijing

    Google Scholar 

  23. Peng Y, Wang N, Wang Y, Wang M (2019) Segmentation of dermoscopy image using adversarial networks. Multimed Tools Appl 78(8):10965–10981

    Article  Google Scholar 

  24. Qiu J (2007) Traditional medicine: a culture in the balance. Nature 448:126

    Article  Google Scholar 

  25. Sekaran K, Chandana P, Krishna NM, Kadry S (2019) Deep learning convolutional neural network (cnn) with gaussian mixture model for predicting pancreatic cancer. Multimedia Tools and Applications

  26. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. Int Conf Learn Represent (ICRL) 1409:1556

    Google Scholar 

  27. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958. 1102.4807

    MathSciNet  MATH  Google Scholar 

  28. Stanitsas P, Cherian A, Truskinovsky A, Morellas V, Papanikolopoulos N (2017) Active convolutional neural networks for cancerous tissue recognition. In: 2017 IEEE International conference on image processing (ICIP), pp 1367–1371

  29. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: 2015 IEEE Conference on computer vision and pattern recognition (CVPR), pp 1–9

  30. Taigman Y, Yang M, Ranzato M, Wolf L (2014) DeepFace: closing the gap to human-level performance in face verification. In: 2014 IEEE Conference on computer vision and pattern recognition, pp 1701–1708

  31. Wang J, Ding H, Bidgoli FA, Zhou B, Iribarren C, Molloi S, Baldi P (2017) Detecting cardiovascular disease from Mammograms with deep learning. IEEE Trans Med Imaging 36(5):1172–1181

    Article  Google Scholar 

  32. Weng H, Liu Z, Yan S, Fan M, Ou A, Chen D, Hao T (2017) A framework for automated knowledge graph construction towards traditional Chinese medicine. In: Siuly S, Huang Z, Aickelin U, Zhou R, Wang H, Zhang Y, Klimenko S (eds) Health information science. Springer International Publishing, Cham, pp 170–181

    Chapter  Google Scholar 

  33. Weng JC, Hu MC, Lan KC (2017) Recognition of easily-confused TCM herbs using deep learning. In: Proceedings of the 8th ACM on multimedia systems conference, MMSys’17. ACM, New York, pp 233–234

  34. Xie D, Pei W, Zhu W, Li X (2017) Traditional Chinese medicine prescription mining based on abstract text. In: 2017 IEEE 19th International conference on e-health networking, applications and services (Healthcom), pp 1–5

  35. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 5987–5995

  36. Xu Z, Liu X, Cheng XE, Song JL, Zhang JQ (2017) Diagnosis of cirrhosis stage via deep neural network. In: 2017 IEEE International conference on bioinformatics and biomedicine (BIBM), pp 745–749

  37. Yao L, Zhang Y, Wei B, Wang W, Zhang Y, Ren X, Bian Y (2015) Discovering treatment pattern in traditional Chinese Medicine clinical cases by exploiting supervised topic model and domain knowledge. J Biomed Inform 58:260–267

    Article  Google Scholar 

  38. Yao L, Zhang Y, Wei B, Zhang W, Jin Z (2018) A topic modeling approach for traditional Chinese medicine prescriptions. IEEE Trans Knowl Data Eng 30(6):1007–1021

    Article  Google Scholar 

  39. Yiqin W (2012) Objective application of TCM inspection of face and tongue. Chin Arch Tradit Chin Med 30(2):349–352

    Google Scholar 

  40. Yu L, Chen H, Dou Q, Qin J, Heng PA (2017) Automated melanoma recognition in dermoscopy images via very deep residual networks. IEEE Trans Med Imaging 36(4):994–1004

    Article  Google Scholar 

  41. Yu T, Li J, Yu Q, Tian Y, Shun X, Xu L, Zhu L, Gao H (2017) Knowledge graph for TCM health preservation: design, construction, and applications. Artif Intell Med 77:48–52

    Article  Google Scholar 

  42. Yuan Y, Chao M, Lo YC (2017) Automatic skin lesion segmentation using deep fully convolutional networks with Jaccard distance. IEEE Trans Med Imaging 36 (9):1876–1886

    Article  Google Scholar 

  43. Zagoruyko S, Komodakis N (2016) Wide residual networks. In: Wilson RCERH, Smith WAP (eds) Proceedings of the British machine vision conference (BMVC). BMVA Press, pp 87.1–87.12

  44. Zhang ML, Zhou ZH (2014) A review on multi-label learning algorithms. IEEE Trans Knowl Data Eng 26(8):1819–1837

    Article  Google Scholar 

  45. Zhang NL, Zhang R, Chen T (2012) Discovery of regularities in the use of herbs in traditional Chinese medicine prescriptions. In: Cao L, Huang JZ, Bailey J, Koh Y S, Luo J (eds) New frontiers in applied data mining. Springer, Berlin, pp 353–360

    Chapter  Google Scholar 

  46. Zhao Y, Dong Q, Chen H, Iraji A, Li Y, Makkie M, Kou Z, Liu T (2017) Constructing fine-granularity functional brain network atlases via deep convolutional autoencoder. Med Image Anal 42:200–211

    Article  Google Scholar 

  47. Zhao X, Wu Y, Song G, Li Z, Zhang Y, Fan Y (2018) A deep learning model integrating FCNNs and CRFs for brain tumor segmentation. Med Image Anal 43:98–111

    Article  Google Scholar 

  48. Zheng G, Jiang M, Lu C, Lu A (2014) Prescription analysis and mining. Springer International Publishing, Cham, pp 97–109

    Google Scholar 

  49. Zhu X, Liu Y, Li Q, Zhang Y, Wen C (2019) Mining patterns of chinese medicinal prescription for diabetes mellitus based on therapeutic effect. Multimedia Tools and Applications

Download references

Acknowledgements

This study was supported by the China National Science Foundation (60973083, 61273363), Science and Technology Planning Project of Guangdong Province (2014A010103009, 2015A020217002), and Guangzhou Science and Technology Planning Project (201504291154480, 2016040- 20179, 201803010088).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering in South China University of Technology, Guangzhou, China

    Huiqiang Liao, Guihua Wen & Yang Hu

  2. Department of Traditional Chinese Medicine in Guangdong General Hospital, Guangzhou, China

    ChangJun Wang

Authors
  1. Huiqiang Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guihua Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. ChangJun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guihua Wen.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liao, H., Wen, G., Hu, Y. et al. Convolutional herbal prescription building method from multi-scale facial features. Multimed Tools Appl 78, 35665–35688 (2019). https://doi.org/10.1007/s11042-019-08118-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08118-7

Keywords

Search

Navigation