Automated Anatomic Labeling Architecture for Content Discovery in Medical Imaging Repositories
The combination of textual data with visual features is known to enhance medical image search capabilities. However, the most advanced imaging archives today only index the studies’ available meta-data, often containing limited amounts of clinically useful information. This work proposes an anatomic labeling architecture, integrated with an open source archive software, for improved multimodal content discovery in real-world medical imaging repositories. The proposed solution includes a technical specification for classifiers in an extensible medical imaging archive, a classification database for querying over the extracted information, and a set of proof-of-concept convolutional neural network classifiers for identifying the presence of organs in computed tomography scans. The system automatically extracts the anatomic region features, which are saved in the proposed database for later consumption by multimodal querying mechanisms. The classifiers were evaluated with cross-validation, yielding a best F1-score of 96% and an average accuracy of 97%. We expect these capabilities to become common-place in production environments in the future, as automated detection solutions improve in terms of accuracy, computational performance, and interoperability.
KeywordsMedical imaging informatics Picture archiving and communication systems Content-based image retrieval Open source software
This work is financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme, and by National Funds through the FCT – Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and Technology) within project PTDC/EEI-ESS/6815/2014; POCI-01-0145-FEDER-016694. Eduardo Pinho was funded by FCT under grant agreement PD/BD/105806/2014.
Compliance with ethical standards
Conflict of interest
The authors Eduardo Pinho and Carlos Costa declare that they have no conflicts of interest.
This article does not contain any studies with human participants or animals performed by any of the authors. All medical imaging data used in this work were anonymized before provision.
- 1.National Electrical Manufacturers Association (NEMA), Digital imaging and communications in medicine (DICOM) standard. Rosslyn: National Electrical Association, 2018.Google Scholar
- 2.Guld, M. O. et al. (2002) Quality of DICOM header information for image categorization, in SPIE, vol. 4685, pp. 280–287.Google Scholar
- 3.Valente, F., Costa, C., Silva, A. (2013) Content based retrieval Systems in a Clinical Context, Medical Imaging in Clinical Practice, pp. 3–22.Google Scholar
- 11.Zhou, X et al. (2004) Automated recognition of human strucure from torso CT images, in International conference on visualization, imaging, and image processing, pp. 584–589.Google Scholar
- 14.H. R. Roth et al., (2015) Anatomy-specific classification of medical images using deep convolutional nets, in IEEE 12th international symposium on biomedical imaging, pp. 101–104.Google Scholar
- 15.Kumar, A., Kim, J., Lyndon, D., Fulham, M., and Feng, D., An Ensemble of Fine-Tuned Convolutional Neural Networks for medical image classification. IEEE Journal of Biomedical and Health Informatics PP(99):1, 2016.Google Scholar
- 17.Ronneberger, O., Fischer, P., Brox, T. (2015) U-net: Convolutional networks for biomedical image segmentation, in International conference on medical image computing and computer-assisted intervention, pp. 234–241.Google Scholar
- 18.Wang, Y., Qiu, Y., Thai, T., Moore, K., Liu, H., and Zheng, B., A two-step convolutional neural network based computer-aided detection scheme for automatically segmenting adipose tissue volume depicting on CT images. Computer Methods and Programs in Biomedicine 144:97–104, 2017.CrossRefPubMedPubMedCentralGoogle Scholar
- 19.Lehmann, T. M., Wein, B. E., Keysers, D., Kohnen, M., Schubert, H. (2002) A monohierarchical multiaxial classification code for medical images in content-based retrieval, in IEEE international symposium on biomedical imaging, pp. 313–316.Google Scholar
- 20.Markonis, D. et al. (2014) A Visual Information Retrieval System for Radiology Reports and the Medical Literature, in Multimedia modeling conference.Google Scholar
- 26.Pinho, E., Godinho, T., Valente, F., Costa, C., A multimodal search engine for medical imaging studies, Journal of Digital Imaging 30(1):39–48, 2017.Google Scholar
- 27.Pinho, E, Costa, C. (2016) Extensible Architecture for Multimodal Information Retrieval in Medical Imaging Archives, in Signal image technology & internet based systems, pp. 316–322.Google Scholar
- 28.Krizhevsky, A., Sutskever, I., Hinton, G. E. (2012) ImageNet classification with deep convolutional neural networks, in Advances in neural information processing systems, pp. 1097–1105.Google Scholar
- 29.Ioffe, S., Szegedy, C. (2015) Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, arXiv preprint arXiv:1502.03167, pp. 1–11.Google Scholar
- 30.He, K., Zhang, X., Ren, S., Sun, J. (2015) Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, in IEEE international conference on computer vision, pp. 1026–1034.Google Scholar
- 31.Kingma, D. P., Ba, J. L. (2015) Adam: A Method for Stochastic Optimization, in International conference on learning representations.Google Scholar
- 32.National Electrical Manufacturers Association (NEMA) (2018) Digital imaging and communications in medicine (DICOM) standard, PS3.3 – information object definitions. National Electrical Association, Rosslyn, VA, USA.Google Scholar
- 33.Silva, L. A. B., Costa, C., Oliveira, J. L. (2014) Semantic search over DICOM repositories, in IEEE international conference on healthcare informatics, pp. 238–246.Google Scholar