Abstract
The emergence of social media has created several opportunities for patients to evaluate the quality of healthcare services by posting online reviews. The rich text and photographic information in the online reviews results insights into how patients’ experience with the doctor and their satisfaction with healthcare service delivery. Various studies have performed patients’ opinion analysis using textual contents. This study presents a novel multimodal approach to analyze the patients’ sentiment regarding the quality of healthcare service delivery (high vs. low). In comparison with existing studies, we consider not only the unique textual contents but photographic contents as well from the Yelp.com platform, which is more challenging due to feature extraction. We evaluate the performance of the baseline, and deep learning algorithms across textual cues, visual cues, and the fusion of both text and visual cues. Experimental results indicated that the addition of novel features increases the models’ accuracy to 15.0% across text and visual cues. Furthermore, fusing text and visual contents improve the classification accuracy by at least 12.64% instead of considering the contents alone. Preliminary comparative experiments result show that the deep learning model outperformed all other algorithms. In comparison with several other state-of-the-art methods in the biomedical domain, the proposed model could significantly enhance the performance of the classifier indicating the effectiveness and suitability of the methodology.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akter S, D’Ambra J, Ray P (2013) Development and validation of an instrument to measure user perceived service quality of mHealth. Inf Manag 50:181–195. https://doi.org/10.1016/j.im.2013.03.001
Alemi F, Torii M, Clementz L, Aron DC (2012) Feasibility of real-time satisfaction surveys through automated analysis of patients’ unstructured comments and sentiments. Qual Manag Healthc 21:9–19. https://doi.org/10.1097/QMH.0b013e3182417fc4
Alexa T (2018) Find website traffic, statistics, and analytics. https://www.alexa.com. Accessed 3 May 2018
Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5:157–166. https://doi.org/10.1109/72.279181
Bridle JS (1990) Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In: Soulié FF, Hérault J (eds) Proceedings of the neurocomputing. Springer, Berlin, pp 227–236
Burkle CM, Keegan MT (2015) Popularity of internet physician rating sites and their apparent influence on patients’ choices of physicians. BMC Health Serv Res 15:416. https://doi.org/10.1186/s12913-015-1099-2
Cambria E, Hussain A, Durrani T, Havasi C, Eckl C, Munro J (2010) Sentic computing for patient centered applications. In: Proceedings of IEEE 10th international conference on signal processing. IEEE, Beijing, pp 1279–1282. http://dx.doi.org/10.1109/ICOSP.2010.5657072
Campos V, Jou B, Giró-i-Nieto X (2017) From pixels to sentiment: fine-tuning CNNs for visual sentiment prediction. Image Vis Comput 65:15–22. https://doi.org/10.1016/j.imavis.2017.01.011
Cao X, Liu Y, Zhu Z, Hu J, Chen X (2017) Online selection of a physician by patients: empirical study from elaboration likelihood perspective. Comput Hum Behav 73:403–412. https://doi.org/10.1016/j.chb.2017.03.060
CDC (2016) Deaths and mortality. https://www.cdc.gov/nchs/data/hus/2017/019.pdf. Accessed 2 May 2018
Chen X, Wang Y, Liu Q (2017) Visual and textual sentiment analysis using deep fusion convolutional neural networks. In: Proceedings of the 2017 IEEE international conference on image processing (ICIP). IEEE, Beijing, pp 1557–1561. https://doi.org/10.1109/ICIP.2017.8296543
Csáji BC (2001) Approximation with artificial neural networks, vol 24. Faculty of Sciences, Etvs Lornd University, Budapest, p 48
Czech D, Mishra A, Inggs M (2018) A CNN and LSTM-based approach to classifying transient radio frequency interference. Astron Comput 25:52–57. https://doi.org/10.1016/j.ascom.2018.07.002
Danielson B et al (2011) Development of indicators of the quality of radiotherapy for localized prostate cancer. Radiother Oncol 99:29–36. https://doi.org/10.1016/j.radonc.2011.02.013
Emmert M, Meier F (2013) An analysis of online evaluations on a physician rating website: evidence from a German public reporting instrument. J Med Internet Res 15:e157. https://doi.org/10.2196/jmir.2655
Emmert M, Sander U, Pisch F (2013) Eight questions about physician-rating websites: a systematic review. J Med Internet Res 15:e24. https://doi.org/10.2196/jmir.2360
Emmert M, Meszmer N, Sander U (2016) Do health care providers use online patient ratings to improve the quality of care? Results from an online-based cross-sectional study. J Med Internet Res 18:e254. https://doi.org/10.2196/jmir.5889
Gao GG, McCullough JS, Agarwal R, Jha AK (2012) A changing landscape of physician quality reporting: analysis of patients’ online ratings of their physicians over a 5-year period. J Med Internet Res 14:e38. https://doi.org/10.2196/jmir.2003
Garrod B (2008) Exploring place perception a photo-based analysis. Ann Tour Res 35:381–401. https://doi.org/10.1016/j.annals.2007.09.004
Goodfellow I, Bengio Y, Courville A (2016) Deep learning. The MIT Press, Cambridge
Grabner-Kräuter S, Waiguny MK (2015) Insights into the impact of online physician reviews on patients’ decision making: randomized experiment. J Med Internet Res 17:e93. https://doi.org/10.2196/jmir.3991
Greaves F, Pape UJ, Lee H, Smith DM, Darzi A, Majeed A, Millett C (2012) Patients’ ratings of family physician practices on the internet: usage and associations with conventional measures of quality in the English National Health Service. J Med Internet Res 14:e146. https://doi.org/10.2196/jmir.2280
Greaves F, Ramirez-Cano D, Millett C, Darzi A, Donaldson L (2013) Use of sentiment analysis for capturing patient experience from free-text comments posted online. J Med Internet Res 15:e239. https://doi.org/10.2196/jmir.2721
Gu J et al (2018) Recent advances in convolutional neural networks. Pattern Recognit 77:354–377. https://doi.org/10.1016/j.patcog.2017.10.013
Hao H (2015) The development of online doctor reviews in China: an analysis of the largest online doctor review website in China. J Med Internet Res 17:e134. https://doi.org/10.2196/jmir.4365
Hao H, Zhang K (2016) The voice of chinese health consumers: a text mining approach to web-based physician reviews. J Med Internet Res 18:e108. https://doi.org/10.2196/jmir.4430
Hao H, Zhang K, Wang W, Gao G (2017) A tale of two countries: international comparison of online doctor reviews between China and the United States. Int J Med Inform 99:37–44. https://doi.org/10.1016/j.ijmedinf.2016.12.007
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Las Vegas, pp 770–778. https://doi.org/10.1109/CVPR.2016.90
Hochreiter S, Bengio Y, Frasconi P, Schmidhuber J (2001) Gradient flow in recurrent nets: the difficulty of learning long term dependencies. In: John FK, Stefan CK (eds) A field guide to dynamical recurrent networks. 1st edn. Wiley-IEEE, pp 237–243
Hu Z, Zhang Z, Yang H, Chen Q, Zuo D (2017) A deep learning approach for predicting the quality of online health expert question-answering services. J Biomed Inform 71:241–253. https://doi.org/10.1016/j.jbi.2017.06.012
Hu Z, Zhang Z, Yang H, Chen Q, Zhu R, Zuo D (2018) Predicting the quality of online health expert question-answering services with temporal features in a deep learning framework. Neurocomputing 275:2769–2782. https://doi.org/10.1016/j.neucom.2017.11.039
James TL, Villacis Calderon ED, Cook DF (2017) Exploring patient perceptions of healthcare service quality through analysis of unstructured feedback. Expert Syst Appl 71:479–492. https://doi.org/10.1016/j.eswa.2016.11.004
Jiménez-Zafra SM, Martín-Valdivia MT, Molina-González MD, Ureña-López LA (2019) How do we talk about doctors and drugs? Sentiment analysis in forums expressing opinions for medical domain. Artif Intell Med 93:50–57. https://doi.org/10.1016/j.artmed.2018.03.007
Jung Y, Hur C, Jung D, Kim M (2015) Identifying key hospital service quality factors in online health communities. J Med Internet Res 17:e90. https://doi.org/10.2196/jmir.3646
Kadry B, Chu LF, Kadry B, Gammas D, Macario A (2011) Analysis of 4999 online physician ratings indicates that most patients give physicians a favorable rating. J Med Internet Res 13:e95. https://doi.org/10.2196/jmir.1960
Kasiri LA, Cheng KTG, Sambasivan M, Sidin SM (2017) Integration of standardization and customization: impact on service quality, customer satisfaction, and loyalty. J Retail Consum Serv 35:91–97. https://doi.org/10.1016/j.jretconser.2016.11.007
Kaymak S, Helwan A, Uzun D (2017) Breast cancer image classification using artificial neural networks. In: Proceedings of the 9th international conference on theory and application of soft computing, computing with words and perception. Elsevier, Budapest, pp 126–131
Khan I, Dongping H, Wahab A (2016) Does culture matter in effectiveness of social media marketing strategy? An investigation of brand fan pages. Aslib J Inf Manag 68:694–715. https://doi.org/10.1108/AJIM-03-2016-0035
Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint. arXiv:14126980
Kuang H, Che C, Zhang Q, Wei X (2017) LSTM based classification model and its application for doctor-patient relationship evaluation. In: Proceedings of the 2017 IEEE 19th international conference on e-health networking, applications and services (Healthcom). IEEE, Dalian, pp 1–5. http://doi.org/10.1109/HealthCom.2017.8210781
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436. https://doi.org/10.1038/nature14539
Li Y, Cao H (2017) Prediction for tourism flow based on LSTM neural network. In: Proceedings of the 2017 international conference on identification, information and knowledge in the Internet of Things. Elsevier, pp 277–283. https://doi.org/10.1016/j.procs.2018.03.076
Li W, Chen H (2014) Identifying top sellers in underground economy using deep learning-based sentiment analysis. In: Proceedings of the 2014 IEEE joint intelligence and security informatics conference, The Hague, Netherlands, pp 64–67. http://dx.doi.org/10.1109/JISIC.2014.19
Li J, Liu M, Li X, Liu X, Liu J (2018) Developing embedded taxonomy and mining patients’ interests from web-based physician reviews: mixed-methods approach. J Med Internet Res 20:e254. https://doi.org/10.2196/jmir.8868
Li J, Tang J, Jiang L, Yen DC, Liu X (2019) Economic success of physicians in the online consultation market: a signaling theory perspective. Int J Electron Commer 23:244–271. https://doi.org/10.1080/10864415.2018.1564552
Liu Z, Yang M, Wang X, Chen Q, Tang B, Wang Z, Xu H (2017) Entity recognition from clinical texts via recurrent neural network. BMC Med Inform Decis Mak 17:67. https://doi.org/10.1186/s12911-017-0468-7
Lu N, Wu H (2016) Exploring the impact of word-of-mouth about Physicians’ service quality on patient choice based on online health communities. BMC Med Inform Decis Mak 16:151. https://doi.org/10.1186/s12911-016-0386-0
Luarn P, Lin Y-F, Chiu Y-P (2015) Influence of Facebook brand-page posts on online engagement. Online Inf Rev 39:505–519. https://doi.org/10.1108/OIR-01-2015-0029
Ma Y, Xiang Z, Du Q, Fan W (2018) Effects of user-provided photos on hotel review helpfulness: an analytical approach with deep leaning. Int J Hosp Manag 71:120–131. https://doi.org/10.1016/j.ijhm.2017.12.008
Majumder N, Hazarika D, Gelbukh A, Cambria E, Poria S (2018) Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowl Based Syst 161:124–133. https://doi.org/10.1016/j.knosys.2018.07.041
Martinez D, Ananda-Rajah MR, Suominen H, Slavin MA, Thursky KA, Cavedon L (2015) Automatic detection of patients with invasive fungal disease from free-text computed tomography (CT) scans. J Biomed Inform 53:251–260. https://doi.org/10.1016/j.jbi.2014.11.009
McAllister P, Zheng H, Bond R, Moorhead A (2018) Combining deep residual neural network features with supervised machine learning algorithms to classify diverse food image datasets. Comput Biol Med 95:217–233. https://doi.org/10.1016/j.compbiomed.2018.02.008
Meesala A, Paul J (2018) Service quality, consumer satisfaction and loyalty in hospitals: thinking for the future. J Retail Consum Serv 40:261–269. https://doi.org/10.1016/j.jretconser.2016.10.011
Mitropoulos P, Vasileiou K, Mitropoulos I (2018) Understanding quality and satisfaction in public hospital services: a nationwide inpatient survey in Greece. J Retail Consum Serv 40:270–275. https://doi.org/10.1016/j.jretconser.2017.03.004
Okike K, Peter-Bibb TK, Xie KC, Okike ON (2016) Association between physician online rating and quality of care. J Med Internet Res 18:e324. https://doi.org/10.2196/jmir.6612
Orlando JI, Prokofyeva E, del Fresno M, Blaschko MB (2018) An ensemble deep learning based approach for red lesion detection in fundus images. Comput Methods Progr Biomed 153:115–127. https://doi.org/10.1016/j.cmpb.2017.10.017
Paolanti M, Kaiser C, Schallner R, Frontoni E, Zingaretti P (2017) Visual and textual sentiment analysis of brand-related social media pictures using deep convolutional neural networks. In: Proceedings of the 2017 international conference on image analysis and processing, springer, image analysis and processing—ICIAP 2017, vol 10484, pp 402–413. https://doi.org/10.1007/978-3-030-03649-2_19
Papacharissi Z (2019) The virtual sphere. The Internet as a public sphere. In: Stempfhuber M, Wagner E (eds) Praktiken der Überwachten. Springer, Berlin, pp 43–60
Pascanu R, Mikolov T, Bengio Y (2012) Understanding the exploding gradient problem. CoRR. arXiv:abs/121150632
Poria S, Cambria E, Howard N, Huang G-B, Hussain A (2016) Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174:50–59. https://doi.org/10.1016/j.neucom.2015.01.095
Poria S, Peng H, Hussain A, Howard N, Cambria E (2017) Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing 261:217–230. https://doi.org/10.1016/j.neucom.2016.09.117
Portal (2018) Leading 10 US states based on the total number of active physicians as of 2018. https://www.cdc.gov/nchs/data/hus/2017/019.pdf. Accessed 5 May 2018
Qian X, Li M, Ren Y, Jiang S (2019) Social media based event summarization by user–text–image co-clustering. Knowl Based Syst 164:107–121. https://doi.org/10.1016/j.knosys.2018.10.028
Rajkomar A et al (2018) Scalable and accurate deep learning with electronic health records NPJ. Digit Med 1:18. https://doi.org/10.1038/s41746-018-0029-1
Reddy BK, Delen D (2018) Predicting hospital readmission for lupus patients: an RNN-LSTM-based deep-learning methodology. Comput Biol Med 101:199–209. https://doi.org/10.1016/j.compbiomed.2018.08.029
Rothenfluh F, Germeni E, Schulz PJ (2016) Consumer decision-making based on review websites: are there differences between choosing a hotel and choosing a physician? J Med Internet Res 18:e129. https://doi.org/10.2196/jmir.5580
Saha M, Chakraborty C, Racoceanu D (2018) Efficient deep learning model for mitosis detection using breast histopathology images. Comput Med Imaging Graph 64:29–40. https://doi.org/10.1016/j.compmedimag.2017.12.001
Sahu SK, Anand A (2018) Drug-drug interaction extraction from biomedical texts using long short-term memory network. J Biomed Inform 86:15–24. https://doi.org/10.1016/j.jbi.2018.08.005
Sajid M, Baig M (2007) Quality of health care: an absolute necessity for public satisfaction. Int J Health Care Qual Assur 20:545–548. https://doi.org/10.1108/09526860710819477
Sirinukunwattana K, Raza SEA, Tsang Y, Snead DRJ, Cree IA, Rajpoot NM (2016) Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imaging 35:1196–1206. https://doi.org/10.1109/TMI.2016.2525803
Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958
Terlutter R, Bidmon S, Röttl J (2014) Who uses physician-rating websites? Differences in sociodemographic variables, psychographic variables, and health status of users and nonusers of physician-rating websites. J Med Internet Res 16:e97. https://doi.org/10.2196/jmir.3145
Tutubalina E, Nikolenko S (2018) Exploring convolutional neural networks and topic models for user profiling from drug reviews. Multimed Tools Appl 77:4791–4809. https://doi.org/10.1007/s11042-017-5336-z
Vedaldi A, Lenc K (2015) MatConvNet: convolutional neural networks for MATLAB. In: Paper presented at the Proceedings of the 23rd ACM international conference on Multimedia, Brisbane, Australia
Wallace BC, Paul MJ, Sarkar U, Trikalinos TA, Dredze M (2014) A large-scale quantitative analysis of latent factors and sentiment in online doctor reviews. J Am Med Inform Assoc 21:1098–1103. https://doi.org/10.1136/amiajnl-2014-002711
Wang J, Peng B, Zhang X (2018) Using a stacked residual LSTM model for sentiment intensity prediction. Neurocomputing 322:93–101. https://doi.org/10.1016/j.neucom.2018.09.049
Xia C, Zhao D, Wang J, Liu J, Ma J (2018) ICSH 2018: LSTM based sentiment analysis for patient experience narratives in E-survey tools. In: Proceedings of the 6th international conference on smart health. Springer, Wuhan, pp 231–239. https://doi.org/10.1007/978-3-030-03649-2_23
Xiao Y, Wu J, Lin Z, Zhao X (2018) A deep learning-based multi-model ensemble method for cancer prediction. Comput Methods Progr Biomed 153:1–9. https://doi.org/10.1016/j.cmpb.2017.09.005
Xie J, Zhang B, Zeng D (2018) Readmission prediction using trajectory-based deep learning approach. In: Proceedings of the 6th international conference on smart health. Springer, Wuhan, pp 224–230. http://dx.doi.org/10.1007/978-3-030-03649-2_22
Yang H, Guo X, Wu T (2015a) Exploring the influence of the online physician service delivery process on patient satisfaction. Decis Support Syst 78:113–121. https://doi.org/10.1016/j.dss.2015.05.006
Yang H, Guo X, Wu T, Ju X (2015b) Exploring the effects of patient-generated and system-generated information on patients’ online search, evaluation and decision. Electron Commer Res Appl 14:192–203. https://doi.org/10.1016/j.elerap.2015.04.001
Yu Y, Lin H, Meng J, Zhao Z (2016) Visual and textual sentiment analysis of a microblog using deep convolutional neural networks. Algorithms 9:41. https://doi.org/10.3390/a9020041
Zhou F, Hang R, Liu Q, Yuan X (2018) Hyperspectral image classification using spectral-spatial LSTMs. Neurocomputing. https://doi.org/10.1016/j.neucom.2018.02.105
Acknowledgements
This research is supported by the National Natural Science Foundation, People’s Republic of China (nos. 71531013, 71401047, 71729001).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Shah, A.M., Yan, X., Shah, S.A.A. et al. Mining patient opinion to evaluate the service quality in healthcare: a deep-learning approach. J Ambient Intell Human Comput 11, 2925–2942 (2020). https://doi.org/10.1007/s12652-019-01434-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-019-01434-8