Multimedia Tools and Applications

, Volume 78, Issue 24, pp 35665–35688 | Cite as

Convolutional herbal prescription building method from multi-scale facial features

  • Huiqiang Liao
  • Guihua WenEmail author
  • Yang Hu
  • ChangJun Wang


In Traditional Chinese Medicine (TCM), facial features are important basis for diagnosis and treatment. A doctor of TCM can prescribe according to a patient’s physical indicators such as face, tongue, voice, symptoms, pulse. Previous works analyze and generate prescription according to symptoms. However, research work to mine the association between facial features and prescriptions has not been found for the time being. In this work, we try to use deep learning methods to mine the relationship between the patient’s face and herbal prescriptions (TCM prescriptions), and propose to construct convolutional neural networks that generate TCM prescriptions according to the patient’s face image. It is a novel and challenging job. In order to mine features from different granularities of faces, we design a multi-scale convolutional neural network based on three-grained face, which mines the patient’s face information from the organs, local regions, and the entire face. Our experiments show that convolutional neural networks can learn relevant information from face to prescribe, and the multi-scale convolutional neural networks based on three-grained face perform better.


Convolutional neural networks Face Prescription Traditional Chinese medicine 



This study was supported by the China National Science Foundation (60973083, 61273363), Science and Technology Planning Project of Guangdong Province (2014A010103009, 2015A020217002), and Guangzhou Science and Technology Planning Project (201504291154480, 2016040- 20179, 201803010088).


  1. 1.
    Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou S (2016) Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging 35(5):1207–1216CrossRefGoogle Scholar
  2. 2.
    Bayramoglu N, Kannala J, Heikkilä J (2016) Deep learning for magnification independent breast cancer histopathology image classification. In: 2016 23rd International conference on pattern recognition (ICPR), pp 2440–2445Google Scholar
  3. 3.
    Bottou L (2012) Stochastic gradient descent tricks. In: Neural networks: tricks of the trade. 9780201398298. Springer, Berlin, pp 421–436Google Scholar
  4. 4.
    Breiman L (2001) Random forests. Mach Learn 45(1):5–32CrossRefGoogle Scholar
  5. 5.
    Chaabouni S, Benois-pineau J, Tison F, Ben Amar C, Zemmari A (2017) Prediction of visual attention with deep cnn on artificially degraded videos for studies of attention of patients with dementia. Multimed Tools Appl 76(21):22527–22546CrossRefGoogle Scholar
  6. 6.
    Chen M, Shi X, Zhang Y, Wu D, Guizani M (2017) Deep features learning for medical image analysis with convolutional autoencoder neural network. IEEE Trans Big Data, 1Google Scholar
  7. 7.
    Cheung F (2011) TCM: made in China. Nature 480:S82CrossRefGoogle Scholar
  8. 8.
    Chougrad H, Zouaki H, Alheyane O (2018) Deep convolutional neural networks for breast cancer screening. Comput Methods Programs Biomed 157:19–30CrossRefGoogle Scholar
  9. 9.
    Dehan L, Jia W, Yimin C, Hamid G (2014) Classification of Chinese herbal medicines based on SVM. In: 2014 International conference on information science, electronics and electrical engineering, vol 1, pp 453–456Google Scholar
  10. 10.
    Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: AISTATS ’11: Proceedings of the 14th international conference on artificial intelligence and statistics, pp 315–323, 1502.03167Google Scholar
  11. 11.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778Google Scholar
  12. 12.
    Hon M, Khan NM (2017) Towards Alzheimer’s disease classification through transfer learning. In: 2017 IEEE International conference on bioinformatics and biomedicine (BIBM), pp 1166–1169Google Scholar
  13. 13.
    Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: The IEEE Conference on computer vision and pattern recognition (CVPR)Google Scholar
  14. 14.
    Huang G, Liu Z, v d Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 2261–2269Google Scholar
  15. 15.
    Jain V, Learned-Miller E (2010) Fddb: a benchmark for face detection in unconstrained settings. Tech. Rep UM-CS-2010-009. University of Massachusetts, AmherstGoogle Scholar
  16. 16.
    Jones AL (2018) The influence of shape and colour cue classes on facial health perception. Evol Hum Behav 39(1):19–29CrossRefGoogle Scholar
  17. 17.
    Kassim YM, Prasath VBS, Glinskii OV, Glinsky VV, Huxley VH, Palaniappan K (2017) Microvasculature segmentation of arterioles using deep CNN. In: 2017 IEEE International conference on image processing (ICIP), pp 580–584Google Scholar
  18. 18.
    King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758Google Scholar
  19. 19.
    Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Advances In Neural Information Processing Systems, pp 1097–1105, 1102.0183Google Scholar
  20. 20.
    Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez C I (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88CrossRefGoogle Scholar
  21. 21.
    Liu B, Zhou X, Wang Y, Hu J, He L, Zhang R, Chen S, Guo Y (2012) Data processing and analysis in real-world traditional Chinese medicine clinical data: challenges and approaches. Stat Med 31(7): 653–660MathSciNetCrossRefGoogle Scholar
  22. 22.
    Peng H (1996) Dictionary of traditional Chinese medicine prescriptions. People Health Press, BeijingGoogle Scholar
  23. 23.
    Peng Y, Wang N, Wang Y, Wang M (2019) Segmentation of dermoscopy image using adversarial networks. Multimed Tools Appl 78(8):10965–10981CrossRefGoogle Scholar
  24. 24.
    Qiu J (2007) Traditional medicine: a culture in the balance. Nature 448:126CrossRefGoogle Scholar
  25. 25.
    Sekaran K, Chandana P, Krishna NM, Kadry S (2019) Deep learning convolutional neural network (cnn) with gaussian mixture model for predicting pancreatic cancer. Multimedia Tools and ApplicationsGoogle Scholar
  26. 26.
    Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. Int Conf Learn Represent (ICRL) 1409:1556Google Scholar
  27. 27.
    Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958. 1102.4807MathSciNetzbMATHGoogle Scholar
  28. 28.
    Stanitsas P, Cherian A, Truskinovsky A, Morellas V, Papanikolopoulos N (2017) Active convolutional neural networks for cancerous tissue recognition. In: 2017 IEEE International conference on image processing (ICIP), pp 1367–1371Google Scholar
  29. 29.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: 2015 IEEE Conference on computer vision and pattern recognition (CVPR), pp 1–9Google Scholar
  30. 30.
    Taigman Y, Yang M, Ranzato M, Wolf L (2014) DeepFace: closing the gap to human-level performance in face verification. In: 2014 IEEE Conference on computer vision and pattern recognition, pp 1701–1708Google Scholar
  31. 31.
    Wang J, Ding H, Bidgoli FA, Zhou B, Iribarren C, Molloi S, Baldi P (2017) Detecting cardiovascular disease from Mammograms with deep learning. IEEE Trans Med Imaging 36(5):1172–1181CrossRefGoogle Scholar
  32. 32.
    Weng H, Liu Z, Yan S, Fan M, Ou A, Chen D, Hao T (2017) A framework for automated knowledge graph construction towards traditional Chinese medicine. In: Siuly S, Huang Z, Aickelin U, Zhou R, Wang H, Zhang Y, Klimenko S (eds) Health information science. Springer International Publishing, Cham, pp 170–181CrossRefGoogle Scholar
  33. 33.
    Weng JC, Hu MC, Lan KC (2017) Recognition of easily-confused TCM herbs using deep learning. In: Proceedings of the 8th ACM on multimedia systems conference, MMSys’17. ACM, New York, pp 233–234Google Scholar
  34. 34.
    Xie D, Pei W, Zhu W, Li X (2017) Traditional Chinese medicine prescription mining based on abstract text. In: 2017 IEEE 19th International conference on e-health networking, applications and services (Healthcom), pp 1–5Google Scholar
  35. 35.
    Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 5987–5995Google Scholar
  36. 36.
    Xu Z, Liu X, Cheng XE, Song JL, Zhang JQ (2017) Diagnosis of cirrhosis stage via deep neural network. In: 2017 IEEE International conference on bioinformatics and biomedicine (BIBM), pp 745–749Google Scholar
  37. 37.
    Yao L, Zhang Y, Wei B, Wang W, Zhang Y, Ren X, Bian Y (2015) Discovering treatment pattern in traditional Chinese Medicine clinical cases by exploiting supervised topic model and domain knowledge. J Biomed Inform 58:260–267CrossRefGoogle Scholar
  38. 38.
    Yao L, Zhang Y, Wei B, Zhang W, Jin Z (2018) A topic modeling approach for traditional Chinese medicine prescriptions. IEEE Trans Knowl Data Eng 30(6):1007–1021CrossRefGoogle Scholar
  39. 39.
    Yiqin W (2012) Objective application of TCM inspection of face and tongue. Chin Arch Tradit Chin Med 30(2):349–352Google Scholar
  40. 40.
    Yu L, Chen H, Dou Q, Qin J, Heng PA (2017) Automated melanoma recognition in dermoscopy images via very deep residual networks. IEEE Trans Med Imaging 36(4):994–1004CrossRefGoogle Scholar
  41. 41.
    Yu T, Li J, Yu Q, Tian Y, Shun X, Xu L, Zhu L, Gao H (2017) Knowledge graph for TCM health preservation: design, construction, and applications. Artif Intell Med 77:48–52CrossRefGoogle Scholar
  42. 42.
    Yuan Y, Chao M, Lo YC (2017) Automatic skin lesion segmentation using deep fully convolutional networks with Jaccard distance. IEEE Trans Med Imaging 36 (9):1876–1886CrossRefGoogle Scholar
  43. 43.
    Zagoruyko S, Komodakis N (2016) Wide residual networks. In: Wilson RCERH, Smith WAP (eds) Proceedings of the British machine vision conference (BMVC). BMVA Press, pp 87.1–87.12Google Scholar
  44. 44.
    Zhang ML, Zhou ZH (2014) A review on multi-label learning algorithms. IEEE Trans Knowl Data Eng 26(8):1819–1837CrossRefGoogle Scholar
  45. 45.
    Zhang NL, Zhang R, Chen T (2012) Discovery of regularities in the use of herbs in traditional Chinese medicine prescriptions. In: Cao L, Huang JZ, Bailey J, Koh Y S, Luo J (eds) New frontiers in applied data mining. Springer, Berlin, pp 353–360CrossRefGoogle Scholar
  46. 46.
    Zhao Y, Dong Q, Chen H, Iraji A, Li Y, Makkie M, Kou Z, Liu T (2017) Constructing fine-granularity functional brain network atlases via deep convolutional autoencoder. Med Image Anal 42:200–211CrossRefGoogle Scholar
  47. 47.
    Zhao X, Wu Y, Song G, Li Z, Zhang Y, Fan Y (2018) A deep learning model integrating FCNNs and CRFs for brain tumor segmentation. Med Image Anal 43:98–111CrossRefGoogle Scholar
  48. 48.
    Zheng G, Jiang M, Lu C, Lu A (2014) Prescription analysis and mining. Springer International Publishing, Cham, pp 97–109Google Scholar
  49. 49.
    Zhu X, Liu Y, Li Q, Zhang Y, Wen C (2019) Mining patterns of chinese medicinal prescription for diabetes mellitus based on therapeutic effect. Multimedia Tools and ApplicationsGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Computer Science and Engineering in South China University of TechnologyGuangzhouChina
  2. 2.Department of Traditional Chinese Medicine in Guangdong General HospitalGuangzhouChina

Personalised recommendations