Skip to main content

Advertisement

SpringerLink
Log in

A multi-modal approach to predict the strength of doctor–patient relationships

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Advances in healthcare social media and information about the doctor-patient (D–P) communication regarding the prior patients’ treatment experience, can positively influence the D–P relationship. In pace with prior patients’ photo-sharing on healthcare social media websites from personal computers and smartphones regarding their treatment experience, the amount of multi-modal content has been growing exponentially. Therefore, there is an increasing need for coping with such information to mine useful knowledge about the D–P communication. Scraping 68,610 reviews, including 4618 photos from a popular physician-rating site, Yelp.com, this study proposes a novel, real-time, multi-modal classification framework, which uses textual and visual modalities as a source of information. Furthermore, this work suggests a social media image filtering mechanism that filters duplicate and irrelevant information from the data. Results show that the data filtering enhances the information reliability, whereas the addition of novel text and visual feature sets improves the classification accuracy up to 16.94%. In addition, fusing textual and visual features enhance the performance of the classifier by 18.24%, which produces better results than considering them separately. The findings also revealed that deep learning algorithms outperformed the classical machine learning algorithms across the entire novel features model, indicating the usefulness and suitability of the proposed methodology. Lastly, the findings from extensive experiments on the physicians’ reviews dataset will guide the doctors to demonstrate the implication of the proposed system for improving the D–P relationship.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://nlp.stanford.edu/software/tokenizer.html

  2. http://www.nltk.org/api/nltk.stem.html?highlight=lemmatizer

  3. https://nlp.stanford.edu/software/tagger.html

  4. http://image-net.org/challenges/LSVRC/2014/results#clsloc

  5. https://deeplearning4j.org/.

  6. http://www.phash.org/.

  7. http://crowdflower.com/

  8. https://scikit-learn.org/stable/

  9. https://keras.io/.

  10. http://radimrehurek.com/gensim/.

  11. http://www.tensorflow.org/.

References

  1. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M (2016) Tensorflow: large-scale machine learning on heterogeneous distributed systems. http://arxiv.org/abs/1603.04467

  2. Alam F, Ofli F, Imran M (2018) Processing social media images by combining human and machine computing during crises. Int J Hum-Comput Int 34(4):311–327. https://doi.org/10.1080/10447318.2018.1427831

    Article  Google Scholar 

  3. Alemi F, Torii M, Clementz L, Aron DC (2012) Feasibility of real-time satisfaction surveys through automated analysis of patients' unstructured comments and sentiments. Qual Manag Health Care 21(1):9–19. https://doi.org/10.1097/qmh.0b013e3182417fc4

    Article  Google Scholar 

  4. Amplayo RK, Song M (2017) An adaptable fine-grained sentiment analysis for summarization of multiple short online reviews. Data Knowl Eng 110:54–67. https://doi.org/10.1016/j.datak.2017.03.009

    Article  Google Scholar 

  5. Analytics H (2020) Using Visual Analytics, Big Data Dashboards for Healthcare Insights. https://healthitanalytics.com/features/using-visual-analytics-big-data-dashboards-for-healthcare-insights. Accessed 15 April 2020

  6. Audrain-Pontevia A-F, Menvielle L (2018) Do online health communities enhance patient–physician relationship? An assessment of the impact of social support and patient empowerment. Health Serv Manag Res 31(3):154–162. https://doi.org/10.1177/0951484817748462

    Article  Google Scholar 

  7. Basole RC, Park H, Gupta M, Braunstein ML, Chau DH, Thompson M (2015) A visual analytics approach to understanding care process variation and conformance. In: Proceedings of the 2015 Workshop on Visual Analytics in Healthcare, Chicago, Illinois, ACM, pp 1-8. https://doi.org/10.1145/2836034.2836040

  8. Batarseh FA, Latif EA (2016) Assessing the quality of service using big data Analytics: with application to healthcare. Big Data Res 4:13–24. https://doi.org/10.1016/j.bdr.2015.10.001

    Article  Google Scholar 

  9. Bennett JK, Fuertes JN, Keitel M, Phillips R (2011) The role of patient attachment and working alliance on patient adherence, satisfaction, and health-related quality of life in lupus treatment. Patient Educ Couns 85(1):53–59. https://doi.org/10.1016/j.pec.2010.08.005

    Article  Google Scholar 

  10. Bertaglia TFC, Nunes MDGV (2016) Exploring Word Embeddings for Unsupervised Textual User-Generated Content Normalization. In: Proceedings of the 26th International Conference on Computational Linguistics (COLING 16), Osaka, Japan, The COLING 2016, pp 112–120. http://arxiv.org/abs/1603.04467

  11. Bishop CM (2011) Pattern recognition and machine learning. Information Science and Statistics. Springer-Verlag, Berlin, Heidelberg

  12. Boquiren VM, Hack TF, Beaver K, Williamson S (2015) What do measures of patient satisfaction with the doctor tell us? Patient Educ Couns 98(12):1465–1473. https://doi.org/10.1016/j.pec.2015.05.020

    Article  Google Scholar 

  13. Cambria E, Hussain A, Durrani T, Havasi C, Eckl C, Munro J (2010) Sentic computing for patient centered applications. In: Proceedings of the IEEE 10th International Conference on Signal Processing Beijing, China, IEEE, pp. 1279–1282. https://doi.org/10.1109/ICOSP.2010.5657072

  14. Cambria E, Howard N, Hsu J, Hussain A (2013) Sentic blending: scalable multimodal fusion for the continuous interpretation of semantics and sentics. In: Proceedings of the 2013 IEEE Symposium on Computational Intelligence for Human-like Intelligence (CIHLI), Singapore, IEEE, pp 108-117. https://doi.org/10.1109/CIHLI.2013.6613272

  15. Campbell TA, Auerbach SM, Kiesler DJ (2007) Relationship of interpersonal behaviors and health-related control appraisals to patient satisfaction and compliance in a university health center. J Am Coll Heal 55(6):333–340. https://doi.org/10.3200/JACH.55.6.333-340

    Article  Google Scholar 

  16. Campos V, Jou B, Giró-i-Nieto X (2017) From pixels to sentiment: fine-tuning CNNs for visual sentiment prediction. Image Vis Comput 65:15–22. https://doi.org/10.1016/j.imavis.2017.01.011

    Article  Google Scholar 

  17. Caropreso MF, Matwin S, Sebastiani F (2001) A learner-independent evaluation of the usefulness of statistical phrases for automated text categorization. In: Chin AG (ed) Text databases & document management. IGI Global, Hershey, PA, USA, pp 78-102. http://dl.acm.org/citation.cfm?id=374247.374254

  18. CDC (2016) Deaths and Mortality. https://www.cdc.gov/nchs/data/hus/2017/019.pdf. Accessed 2 May 2018

  19. Chang MK, Cheung W, Tang M (2013) Building trust online: interactions among trust building mechanisms. Inf Manag 50(7):439–445. https://doi.org/10.1016/j.im.2013.06.003

    Article  Google Scholar 

  20. Chen X, Wang Y, Liu Q (2017) Visual and textual sentiment analysis using deep fusion convolutional neural networks. In: Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China, IEEE, pp 1557-1561. https://doi.org/10.1109/ICIP.2017.8296543

  21. Cherry MG, Fletcher I, Berridge D, O’Sullivan H (2018) Do doctors’ attachment styles and emotional intelligence influence patients’ emotional expressions in primary care consultations? An exploratory study using multilevel analysis. Patient Educ Couns 101(4):659–664. https://doi.org/10.1016/j.pec.2017.10.017

    Article  Google Scholar 

  22. Deng S, Yang N, Li S, Wang W, Yan H, Li H (2018) Doctors’ job satisfaction and its relationships with doctor-patient relationship and work-family conflict in China: a structural equation modeling. INQUIRY: J Health Car 55:1–11. https://doi.org/10.1177/0046958018790831

    Article  Google Scholar 

  23. Detz A, Lopez A, Sarkar U (2013) Long-term doctor-patient relationships: patient perspective from online reviews. J Med Internet Res 15(7):e131. https://doi.org/10.2196/jmir.2552

    Article  Google Scholar 

  24. Dhankhar P (2019) ResNet-50 and VGG-16 for recognizing facial emotions. Int J Innov Eng Technol (IJIET) 13(4):126–130

    Google Scholar 

  25. Fernandez JM, Cenador MBG, Manuel López Millan J, Méndez JAJ, Ledesma MJS (2017) Use of Information and Communication Technologies in Clinical Practice Related to the Treatment of Pain. Influence on the Professional Activity and the Doctor-Patient Relationship. J Med Syst 41(5):77. https://doi.org/10.1007/s10916-017-0724-5

    Article  Google Scholar 

  26. Galavotti L, Sebastiani F, Simi M (2000) Experiments on the use of feature selection and negative evidence in automated text categorization. In: Borbinha J, Baker T (eds) Research and advanced Technology for Digital Libraries. Springer, Berlin, pp 59–68. https://doi.org/10.1007/3-540-45268-0_6

    Chapter  Google Scholar 

  27. Goldberg Y, Levy O (2014) word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling Word-Embedding Method. http://arxiv.org/abs/1402.3722

  28. Goodfellow I, Bengio Y, Courville A (2016) Deep Learning. The MIT Press, Cambridge

    MATH  Google Scholar 

  29. Gotz DH, Sun J, Cao N (2012) Multifaceted visual analytics for healthcare applications. IBM J Res Dev 56(5):6:1-6:12. https://doi.org/10.1147/JRD.2012.2199170

    Article  Google Scholar 

  30. Greaves F, Ramirez-Cano D, Millett C, Darzi A, Donaldson L (2013) Use of sentiment analysis for capturing patient experience from free-text comments posted online. J Med Internet Res 15(11):e239. https://doi.org/10.2196/jmir.2721

    Article  Google Scholar 

  31. Grünloh C, Myreteg G, Cajander Å, Rexhepi H (2018) “Why do they need to check me?” patient participation through eHealth and the doctor-patient relationship: qualitative study. J Med Internet Res 20(1):e11. https://doi.org/10.2196/jmir.8444

    Article  Google Scholar 

  32. Guo S, Guo X, Zhang X, Vogel D (2018) Doctor–patient relationship strength’s impact in an online healthcare community. Inf Technol Dev 24(2):279–300. https://doi.org/10.1080/02681102.2017.1283287

    Article  Google Scholar 

  33. Haluza D, Naszay M, Stockinger A, Jungwirth D (2017) Digital natives versus digital immigrants: influence of online health information seeking on the doctor–patient relationship. Health Commun 32(11):1342–1349. https://doi.org/10.1080/10410236.2016.1220044

    Article  Google Scholar 

  34. Hamming R (1950) Error detecting and error correcting codes. Bell System Technical Journal 29. https://doi.org/10.1002/j.1538-7305.1950.tb00463.x

  35. Hao H, Zhang K (2016) The voice of Chinese health consumers: a text mining approach to web-based physician reviews. J Med Internet Res 18(5):e108. https://doi.org/10.2196/jmir.4430

    Article  Google Scholar 

  36. Hao H, Zhang K, Wang W, Gao G (2017) A tale of two countries: international comparison of online doctor reviews between China and the United States. Int J Med Inform 99:37–44. https://doi.org/10.1016/j.ijmedinf.2016.12.007

    Article  Google Scholar 

  37. Haskard Zolnierek KB, DiMatteo MR (2009) Physician communication and patient adherence to treatment: a meta-analysis. Med Care 47(8):826–834. https://doi.org/10.1097/MLR.0b013e31819a5acc

    Article  Google Scholar 

  38. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  39. How convolutional neural networks see the world - an exploration of convnet filters with keras.

  40. James TL, Villacis Calderon ED, Cook DF (2017) Exploring patient perceptions of healthcare service quality through analysis of unstructured feedback. Expert Syst Appl 71:479–492. https://doi.org/10.1016/j.eswa.2016.11.004

    Article  Google Scholar 

  41. Jiménez-Zafra SM, Martín-Valdivia MT, Molina-González MD, Ureña-López LA (2019) How do we talk about doctors and drugs? Sentiment analysis in forums expressing opinions for medical domain. Artif Intell Med 93:50–57. https://doi.org/10.1016/j.artmed.2018.03.007

    Article  Google Scholar 

  42. Kamal N, Wiebe S, Engbers J, Hill M (2014) Big data and visual Analytics in health and medicine: from pipe dream to reality. J Health Med Inform 5:e125. https://doi.org/10.4172/2157-7420.1000e125

    Article  Google Scholar 

  43. Kaymak S, Helwan A, Uzun D (2017) Breast cancer image classification using artificial neural networks. In: Proceedings of the 9th International Conference on Theory and Application of Soft Computing, Computing with Words and Perception, Budapest, Hungary, Elsevier, pp 126-131. https://doi.org/10.1016/j.procs.2017.11.219

  44. Kee JWY, Khoo HS, Lim I, Koh MYH (2018) Communication skills in patient-doctor interactions: learning from patient complaints. Health Prof Educ 4(2):97–106. https://doi.org/10.1016/j.hpe.2017.03.006

    Article  Google Scholar 

  45. Keras visualization toolkit https://raghakot.github.io/keras-vis/

  46. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. http://arxiv.org/abs/1412.6980

  47. Kuang H, Che C, Zhang Q, Wei X (2017) LSTM based classification model and its application for doctor-patient relationship evaluation. Paper presented at the 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom), Dalian, China, pp 1-5. https://doi.org/10.1109/HealthCom.2017.8210781

  48. Laugesen J, Hassanein K, Yuan Y (2015) The impact of internet health information on patient compliance: a research model and an empirical study. J Med Internet Res 17(6):e143. https://doi.org/10.2196/jmir.4333

    Article  Google Scholar 

  49. Li W, Chen H (2014) Identifying top sellers in underground economy using deep learning-based sentiment analysis. In: Proceedings of the 2014 IEEE Joint Intelligence and Security Informatics Conference, The Hague, Netherlands, IEEE, pp 64-67. https://doi.org/10.1109/JISIC.2014.19

  50. Liu X, Liu QB, Guo X (2016) Patients’ Use of Social Media Improves Doctor-patient Relationship and Patient Wellbeing: Evidence from a Natural Experiment in China. In: Proceedings of the 37th International Conference on Information Systems (ICIS 2016), Dublin, Ireland, AIS, pp. 1–14

  51. Lu X, Zhang R (2019) Impact of physician-patient communication in online health communities on patient compliance: cross-sectional questionnaire study. J Med Internet Res 21(5):e12891. https://doi.org/10.2196/12891

    Article  Google Scholar 

  52. Majumder N, Hazarika D, Gelbukh A, Cambria E, Poria S (2018) Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowl-Based Syst 161:124–133. https://doi.org/10.1016/j.knosys.2018.07.041

    Article  Google Scholar 

  53. Martinez D, Ananda-Rajah MR, Suominen H, Slavin MA, Thursky KA, Cavedon L (2015) Automatic detection of patients with invasive fungal disease from free-text computed tomography (CT) scans. J Biomed Inform 53:251–260. https://doi.org/10.1016/j.jbi.2014.11.009

    Article  Google Scholar 

  54. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. http://arxiv.org/abs/1301.3781

  55. Morency L-P, Mihalcea R, Doshi P (2011) Towards multimodal sentiment analysis: harvesting opinions from the web. In: Proceedings of the 13th international conference on multimodal interfaces, Alicante, Spain, ACM, pp 169-176. https://doi.org/10.1145/2070481.2070509

  56. Niaz U, Merialdo B (2013) Fusion methods for multi-modal indexing of web data. In: Proceedings of the 14th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Paris, France, IEEE, pp 1-4. https://doi.org/10.1109/WIAMIS.2013.6616129

  57. Pang B, Lee L, Vaithyanathan S (2002) Thumbs up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL-02 conference on Empirical methods in natural language processing - Volume 10, Association for Computational Linguistics, Stroudsburg, PA, USA, pp 79-86. https://doi.org/10.3115/1118693.1118704

  58. Pang L, Zhu S, Ngo C (2015) Deep multimodal learning for affective analysis and retrieval. IEEE T Multimedia 17(11):2008–2020. https://doi.org/10.1109/TMM.2015.2482228

    Article  Google Scholar 

  59. Paolanti M, Kaiser C, Schallner R, Frontoni E, Zingaretti P (2017) Visual and textual sentiment analysis of brand-related social media pictures using deep convolutional neural networks. In: Battiato S, Gallo G, Schettini R, Stanco F (eds) Image Analysis and Processing - ICIAP 2017. Springer International Publishing, Cham, pp 402–413. https://doi.org/10.1007/978-3-319-68560-1_36

    Chapter  Google Scholar 

  60. Poria S, Chaturvedi I, Cambria E, Hussain A (2016) Convolutional MKL based multimodal emotion recognition and sentiment analysis. In: Proceedings of the IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain, IEEE, pp 439-448. https://doi.org/10.1109/ICDM.2016.0055

  61. Poria S, Cambria E, Howard N, Huang G-B, Hussain A (2016) Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174:50–59. https://doi.org/10.1016/j.neucom.2015.01.095

    Article  Google Scholar 

  62. Poria S, Cambria E, Bajpai R, Hussain A (2017) A review of affective computing: from unimodal analysis to multimodal fusion. Inform Fusion 37:98–125. https://doi.org/10.1016/j.inffus.2017.02.003

    Article  Google Scholar 

  63. Poria S, Peng H, Hussain A, Howard N, Cambria E (2017) Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing 261:217–230. https://doi.org/10.1016/j.neucom.2016.09.117

    Article  Google Scholar 

  64. Preim B, Lawonn K (2020) A survey of visual Analytics for public health. Comput Graph Forum 39(1):543–580. https://doi.org/10.1111/cgf.13891

    Article  Google Scholar 

  65. Qian X, Li M, Ren Y, Jiang S (2019) Social media based event summarization by user–text–image co-clustering. Knowl-Based Syst 164:107–121. https://doi.org/10.1016/j.knosys.2018.10.028

    Article  Google Scholar 

  66. Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, Liu PJ, Liu X, Marcus J, Sun M (2018) Scalable and accurate deep learning with electronic health records. NPJ Digit Med 1(1):18. https://doi.org/10.1038/s41746-018-0029-1

    Article  Google Scholar 

  67. Reddy BK, Delen D (2018) Predicting hospital readmission for lupus patients: An RNN-LSTM-based deep-learning methodology. Comput Biol Med 101:199–209. https://doi.org/10.1016/j.compbiomed.2018.08.029

    Article  Google Scholar 

  68. Roberts L, Cornell C, Bostrom M, Goldsmith S, Ologhobo T, Roberts T, Robbins L (2018) Communication skills training for surgical residents: learning to relate to the needs of older adults. J Surg Educ 75(5):1180–1187. https://doi.org/10.1016/j.jsurg.2018.02.005

    Article  Google Scholar 

  69. Roettl J, Bidmon S, Terlutter R (2016) What predicts patients’ willingness to undergo online treatment and pay for online treatment? Results from a web-based survey to investigate the changing patient-physician relationship. J Med Internet Res 18(2):e32. https://doi.org/10.2196/jmir.5244

    Article  Google Scholar 

  70. Rothenfluh F, Schulz PJ (2017) Physician rating websites: what aspects are important to identify a good doctor, and are patients capable of assessing them? A mixed-methods approach including physicians’ and health care consumers’ perspectives. J Med Internet Res 19(5):e127. https://doi.org/10.2196/jmir.6875

    Article  Google Scholar 

  71. Sacha D, Senaratne H, Kwon BC, Ellis G, Keim DA (2016) The role of uncertainty, awareness, and Trust in Visual Analytics. IEEE T Vis Comput Gr 22(1):240–249. https://doi.org/10.1109/TVCG.2015.2467591

    Article  Google Scholar 

  72. Sacha D, Sedlmair M, Zhang L, Lee JA, Peltonen J, Weiskopf D, North SC, Keim DA (2017) What you see is what you can change: human-centered machine learning by interactive visualization. Neurocomputing 268:164–175. https://doi.org/10.1016/j.neucom.2017.01.105

    Article  Google Scholar 

  73. Saha M, Chakraborty C, Racoceanu D (2018) Efficient deep learning model for mitosis detection using breast histopathology images. Comput Med Imaging Graph 64:29–40. https://doi.org/10.1016/j.compmedimag.2017.12.001

    Article  Google Scholar 

  74. Sarker A, Gonzalez G (2015) Portable automatic text classification for adverse drug reaction detection via multi-corpus training. J Biomed Inform 53:196–207. https://doi.org/10.1016/j.jbi.2014.11.002

    Article  Google Scholar 

  75. Segal J, Sacopulos M, Sheets V, Thurston I, Brooks K, Puccia R (2012) Online doctor reviews: do they track surgeon volume, a proxy for quality of care? J Med Internet Res 14(2):e50. https://doi.org/10.2196/jmir.2005

    Article  Google Scholar 

  76. Shah AM, Yan X, Shah SAA, Mamirkulova G (2019) Mining patient opinion to evaluate the service quality in healthcare: a deep-learning approach. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-019-01434-8

  77. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Paper presented at the ICLR 2015: International Conference on Learning Representation 2015, San Diego, CA, USA, 09/04. http://arxiv.org/abs/1409.1556

  78. Simpao AF, Ahumada LM, Rehman MA (2015) Big data and visual analytics in anaesthesia and health care. Br J Anaesth 115(3):350–356. https://doi.org/10.1093/bja/aeu552

    Article  Google Scholar 

  79. Singh N, Singh S (2017) Object classification to analyze medical imaging data using deep learning. In: proceedings of the 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), Coimbatore, India, IEEE, pp 1-4. https://doi.org/10.1109/ICIIECS.2017.8276099

  80. Stewart Loane S, D'Alessandro S (2014) Empowered and knowledgeable health consumers: the impact of online support groups on the doctor–patient relationship. Australas Mark J AMJ 22(3):238–245. https://doi.org/10.1016/j.ausmj.2014.08.007

    Article  Google Scholar 

  81. Strang KD, Sun Z (2019) Hidden big data analytics issues in the healthcare industry. Health Inform J 0 (0):1460458219854603. https://doi.org/10.1177/1460458219854603, 26, 1460458219854998

  82. Sudha G, Suguna S (2018) A survey on contribution of visual Analytics in health care domain. Paper presented at the 2018 IADS International Conference on Computing, Communications & Data Engineering (CCODE), India, pp 1-6. https://doi.org/10.2139/ssrn.3165309

  83. Tan SS-L, Goonawardene N (2017) Internet health information seeking and the patient-physician relationship: a systematic review. J Med Internet Res 19(1):e9. https://doi.org/10.2196/jmir.5729

    Article  Google Scholar 

  84. Team TTD (2016) Theano: a Python framework for fast computation of mathematical expressions. https://arxiv.org/abs/1605.02688.pdf

  85. Thomas J, Cook K (2005) Illuminating the path: Research and Development agenda for visual Analytics. IEEE Comput. Graph. Appl. IEEE Computer Society Press, Los Alamitos, CA, USA. https://doi.org/10.1109/MCG.2006.5

  86. Truong Q-T, Lauw HW (2017) Visual sentiment analysis for review images with item-oriented and user-oriented CNN. In: Proceeding of the 25th ACM international conference on Multimedia, Mountain View, California, USA, ACM, 3123374, pp 1274-1282. https://doi.org/10.1145/3123266.3123374

  87. Tucker JD, Cheng Y, Wong B, Gong N, Nie J-B, Zhu W, McLaughlin MM, Xie R, Deng Y, Huang M, Wong WCW, Lan P, Liu H, Miao W, Kleinman A (2015) Patient–physician mistrust and violence against physicians in Guangdong Province, China: a qualitative study. BMJ Open 5(10):e008221. https://doi.org/10.1136/bmjopen-2015-008221

    Article  Google Scholar 

  88. Umar N, Litaker D, Schaarschmidt M-L, Peitsch WK, Schmieder A, Terris DD (2012) Outcomes associated with matching patients' treatment preferences to physicians' recommendations: study methodology. BMC Health Serv Res 12(1):1. https://doi.org/10.1186/1472-6963-12-1

    Article  Google Scholar 

  89. Ureña R, Kou G, Dong Y, Chiclana F, Herrera-Viedma E (2019) A review on trust propagation and opinion dynamics in social networks and group decision making frameworks. Inf Sci 478:461–475. https://doi.org/10.1016/j.ins.2018.11.037

    Article  Google Scholar 

  90. Vaitsis C, Nilsson G, Zary N (2014) Visual analytics in healthcare education: exploring novel ways to analyze and represent big data in undergraduate medical education. PeerJ 2:e683. https://doi.org/10.7717/peerj.683

    Article  Google Scholar 

  91. Vellappally S, Al Kheraif AA, Anil S, Assery MK, Kumar KA, Divakar DD (2018) Analyzing relationship between patient and doctor in public dental health using particle Memetic multivariable logistic regression analysis approach (MLRA2). J Med Syst 42(10):183. https://doi.org/10.1007/s10916-018-1037-z

    Article  Google Scholar 

  92. Wang J, Li S, An Z, Jiang X, Qian W, Ji S (2018) Batch-normalized deep neural networks for achieving fast intelligent fault diagnosis of machines. Neurocomputing 329:53–65. https://doi.org/10.1016/j.neucom.2018.10.049

    Article  Google Scholar 

  93. Wu B (2018) Patient continued use of online health care communities: web Mining of Patient-Doctor Communication. J Med Internet Res 20(4):e126. https://doi.org/10.2196/jmir.9127

    Article  Google Scholar 

  94. Wu H, Lu N (2017) Online written consultation, telephone consultation and offline appointment: An examination of the channel effect in online health communities. Int J Med Inform 107:107–119. https://doi.org/10.1016/j.ijmedinf.2017.08.009

    Article  Google Scholar 

  95. Xiao Y, Wu J, Lin Z, Zhao X (2018) A deep learning-based multi-model ensemble method for cancer prediction. Comput Methods Prog Biomed 153:1–9. https://doi.org/10.1016/j.cmpb.2017.09.005

    Article  Google Scholar 

  96. Yellowlees P, Richard Chan S, Burke Parish M (2015) The hybrid doctor–patient relationship in the age of technology – Telepsychiatry consultations and the use of virtual space. Int Rev Psychiatry 27(6):476–489. https://doi.org/10.3109/09540261.2015.1082987

    Article  Google Scholar 

  97. You Q, Luo J, Jin H, Yang J (2015) Joint visual-textual sentiment analysis with deep neural networks. In: proceedings of the 23rd ACM international conference on multimedia, Brisbane, Australia, ACM, pp 1071-1074. https://doi.org/10.1145/2733373.2806284

  98. You Q, Jin H, Luo J (2017) Visual sentiment analysis by attending on local image regions. In: proceedings of the thirty-first AAAI conference on artificial intelligence, San Francisco, California, USA, AAAI press, pp 231-237. http://dl.acm.org/citation.cfm?id=3298239.3298274

  99. Young T, Hazarika D, Poria S, Cambria E (2018) Recent trends in deep learning based natural language processing [Review Article]. IEEE Comput Intell M 13(3):55–75. https://doi.org/10.1109/MCI.2018.2840738

    Article  Google Scholar 

  100. Yu Y, Lin H, Meng J, Zhao Z (2016) Visual and textual sentiment analysis of a microblog using deep convolutional neural networks. Algorithms 9(2):41. https://doi.org/10.3390/a9020041

    Article  MathSciNet  Google Scholar 

  101. Zanini C, Sarzi-Puttini P, Atzeni F, Di Franco M, Rubinelli S (2014) Doctors insights into the patient perspective: a qualitative study in the field of chronic pain. Biomed Res Int 2014:6. https://doi.org/10.1155/2014/514230

    Article  Google Scholar 

  102. Zhai C, Massung S (2016) Text data management and analysis: a practical introduction to information retrieval and text mining. Association for Computing Machinery and Morgan & Claypool, New York, NY, USA. https://doi.org/10.1145/2915031

  103. Zhang Y, Sun Y, Kim Y (2017) The influence of individual differences on consumer's selection of online sources for health information. Comput Hum Behav 67:303–312. https://doi.org/10.1016/j.chb.2016.11.008

    Article  Google Scholar 

Download references

Acknowledgments

This research is supported by the National Natural Science Foundation, People’s Republic of China (No.71531013, 71729001).

Author information

Authors and Affiliations

  1. Department of Management Science and Engineering, School of Management, Harbin Institute of Technology, Harbin, 150001, People’s Republic of China

    Adnan Muhammad Shah, Xiangbin Yan, Salim Khan & Waqas Khurrum

  2. Department of Finance, Beijing Technology and Business University, Beijing, 100048, People’s Republic of China

    Qasim Raza Khan

Authors
  1. Adnan Muhammad Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiangbin Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Salim Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Waqas Khurrum
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qasim Raza Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adnan Muhammad Shah.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shah, A.M., Yan, X., Khan, S. et al. A multi-modal approach to predict the strength of doctor–patient relationships. Multimed Tools Appl 80, 23207–23240 (2021). https://doi.org/10.1007/s11042-020-09596-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09596-w

Keywords

Search

Navigation