Abstract
In many healthcare applications, identifying pills given their captured images under various conditions and backgrounds has been becoming more and more essential. Several efforts have been devoted to utilizing the deep learning-based approach to tackle the pill recognition problem in the literature. However, due to the high similarity between pills’ appearance, misrecognition often occurs, leaving pill recognition a challenge. To this end, in this paper, we introduce a novel approach named PIKA that leverages external knowledge to enhance pill recognition accuracy. Specifically, we address a practical scenario (which we call contextual pill recognition), aiming to identify pills in a picture of a patient’s pill intake. Firstly, we propose a novel method for modeling the implicit association between pills in the presence of an external data source, in this case, prescriptions. Secondly, we present a walk-based graph embedding model that transforms from the graph space to vector space and extracts condensed relational features of the pills. Thirdly, a final framework is provided that leverages both image-based visual and graph-based relational features to accomplish the pill identification task. Within this framework, the visual representation of each pill is mapped to the graph embedding space, which is then used to execute attention over the graph representation, resulting in a semantically-rich context vector that aids in the final classification. To our knowledge, this is the first study to use external prescription data to establish associations between medicines and to classify them using this aiding information. The architecture of PIKA is lightweight and has the flexibility to incorporate into any recognition backbones. The experimental results show that by leveraging the external knowledge graph, PIKA can improve the recognition accuracy from \(4.8\%\) to \(34.1\%\) in terms of F1-score, compared to baselines.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
World patient safety day (2022). https://www.who.int/news-room/events/detail/2022/09/17/default-calendar/world-patient-safety-day-2022. Accessed 14 Apr 2022
Chang, et al.: A deep learning-based intelligent medicine recognition system for chronic patients. IEEE Access 7, 44441–44458 (2019). https://doi.org/10.1109/ACCESS.2019.2908843
Chang, et al.: Medglasses: a wearable smart-glasses-based drug pill recognition system using deep learning for visually impaired chronic patients. IEEE Access 8, 17013–17024 (2020). https://doi.org/10.1109/ACCESS.2020.2967400
Chollet, et al.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer vision and Pattern Recognition, pp. 1251–1258 (2017)
Hang, J.Y., Zhang, M.L.: Collaborative learning of label semantics and deep label-specific features for multi-label classification. IEEE Trans. Pattern Anal. Mach. Intell. 44, 9860–9871 (2021). https://doi.org/10.1109/TPAMI.2021.3136592
He, et al.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Hinton, et al.: Stochastic neighbor embedding. In: Becker, S., Thrun, S., Obermayer, K. (eds.) Advances in Neural Information Processing Systems. vol. 15. MIT Press (2002). https://proceedings.neurips.cc/paper/2002/file/6150ccc6069bea6b5716254057a194ef-Paper.pdf
Li, Q., Qiao, M., Bian, W., Tao, D.: Conditional graphical lasso for multi-label image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
Ling, et al.: Few-shot pill recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020
Loshchilov, I., et al.: Decoupled weight decay regularization. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019. OpenReview.net (2019), https://openreview.net/forum?id=Bkg6RiCqY7
van der Maaten, L., et al.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(86), 2579–2605 (2008). http://jmlr.org/papers/v9/vandermaaten08a.html
Proma, et al.: Medicine recognition from colors and text. In: Proceedings of the 2019 3rd International Conference on Graphics and Signal Processing. ICGSP 2019, pp. 39–43., Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3338472.3338484, https://doi.org/10.1145/3338472.3338484
Scott, et al.: Multivariate Density Estimation: Theory, Practice, and Visualization, 2nd edn., March 2015. https://doi.org/10.1002/9781118575574
Simonyan, K., et al.: Very deep convolutional networks for large-scale image recognition. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 March 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1409.1556
Ting, H.W., et al.: A drug identification model developed using deep learning technologies: experience of a medical center in Taiwan. BMC Health Serv. Res. 20 (2020). https://doi.org/10.1186/s12913-020-05166-w, https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-020-05166-w#citeas
Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., Xu, W.: CNN-RNN: a unified framework for multi-label image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016
Wang, Y., et al.: Multi-label classification with label graph superimposing. Proc. AAAI Conf. Artif. Intell. 34(07), 12265–12272 (2020). https://doi.org/10.1609/aaai.v34i07.6909, https://ojs.aaai.org/index.php/AAAI/article/view/6909
Wang, Y., Xie, Y., Liu, Y., Zhou, K., Li, X.: Fast graph convolution network based multi-label image recognition via cross-modal fusion. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM 2020, pp. 1575–1584. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3340531.3411880
Wong, Y.F., et al.: Development of fine-grained pill identification algorithm using deep convolutional network. J. Biomed. Inform. 74, 130–136 (2017). https://doi.org/10.1016/j.jbi.2017.09.005, https://www.sciencedirect.com/science/article/pii/S1532046417302022
Yaniv, et al.: The national library of medicine pill image recognition challenge: an initial report. In: 2016 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), pp. 1–9 (2016). https://doi.org/10.1109/AIPR.2016.8010584
Acknowledgement
This work was funded by Vingroup Joint Stock Company (Vingroup JSC), Vingroup, and supported by Vingroup Innovation Foundation (VINIF) under project code VINIF.2021.DA00128.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Nguyen, A.D., Nguyen, T.D., Pham, H.H., Nguyen, T.H., Nguyen, P.L. (2022). Image-Based Contextual Pill Recognition with Medical Knowledge Graph Assistance. In: Szczerbicki, E., Wojtkiewicz, K., Nguyen, S.V., Pietranik, M., Krótkiewicz, M. (eds) Recent Challenges in Intelligent Information and Database Systems. ACIIDS 2022. Communications in Computer and Information Science, vol 1716. Springer, Singapore. https://doi.org/10.1007/978-981-19-8234-7_28
Download citation
DOI: https://doi.org/10.1007/978-981-19-8234-7_28
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-8233-0
Online ISBN: 978-981-19-8234-7
eBook Packages: Computer ScienceComputer Science (R0)