Journal of Medical Systems

, 42:145 | Cite as

Automated Anatomic Labeling Architecture for Content Discovery in Medical Imaging Repositories

  • Eduardo PinhoEmail author
  • Carlos Costa
Systems-Level Quality Improvement
Part of the following topical collections:
  1. Systems-Level Quality Improvement


The combination of textual data with visual features is known to enhance medical image search capabilities. However, the most advanced imaging archives today only index the studies’ available meta-data, often containing limited amounts of clinically useful information. This work proposes an anatomic labeling architecture, integrated with an open source archive software, for improved multimodal content discovery in real-world medical imaging repositories. The proposed solution includes a technical specification for classifiers in an extensible medical imaging archive, a classification database for querying over the extracted information, and a set of proof-of-concept convolutional neural network classifiers for identifying the presence of organs in computed tomography scans. The system automatically extracts the anatomic region features, which are saved in the proposed database for later consumption by multimodal querying mechanisms. The classifiers were evaluated with cross-validation, yielding a best F1-score of 96% and an average accuracy of 97%. We expect these capabilities to become common-place in production environments in the future, as automated detection solutions improve in terms of accuracy, computational performance, and interoperability.


Medical imaging informatics Picture archiving and communication systems Content-based image retrieval Open source software 



This work is financed by the ERDF – European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme, and by National Funds through the FCT – Fundação para a Ciência e a Tecnologia (Portuguese Foundation for Science and Technology) within project PTDC/EEI-ESS/6815/2014; POCI-01-0145-FEDER-016694. Eduardo Pinho was funded by FCT under grant agreement PD/BD/105806/2014.

Compliance with ethical standards

Conflict of interest

The authors Eduardo Pinho and Carlos Costa declare that they have no conflicts of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors. All medical imaging data used in this work were anonymized before provision.


  1. 1.
    National Electrical Manufacturers Association (NEMA), Digital imaging and communications in medicine (DICOM) standard. Rosslyn: National Electrical Association, 2018.Google Scholar
  2. 2.
    Guld, M. O. et al. (2002) Quality of DICOM header information for image categorization, in SPIE, vol. 4685, pp. 280–287.Google Scholar
  3. 3.
    Valente, F., Costa, C., Silva, A. (2013) Content based retrieval Systems in a Clinical Context, Medical Imaging in Clinical Practice, pp. 3–22.Google Scholar
  4. 4.
    Akgül, C. B., Rubin, D. L., Napel, S., Beaulieu, C. F., Greenspan, H., and Acar, B., Content-based image retrieval in radiology: Current status and future directions. Journal of Digital Imaging 24(2):208–222, 2011.CrossRefPubMedGoogle Scholar
  5. 5.
    Faruque, J., Beaulieu, C. F., Rosenberg, J., Rubin, D. L., Yao, D., and Napel, S., Content-based image retrieval in radiology: Analysis of variability in human perception of similarity. Journal of Medical Imaging 2(2):25501, 2015.CrossRefGoogle Scholar
  6. 6.
    Kumar, A., Kim, J., Cai, W., Fulham, M., and Feng, D., Content-based medical image retrieval: A survey of applications to multidimensional and multimodality data. Journal of Digital Imaging 26(6):1025–1039, 2013.CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Valente, F., Silva, L. A. B., Godinho, T. M., and Costa, C., Anatomy of an extensible open source PACS. Journal of Digital Imaging 29(3):284–296, Jun. 2016.CrossRefPubMedGoogle Scholar
  8. 8.
    Doi, K., Computer-aided diagnosis in medical imaging: Historical review, current status and future potential. Computerized Medical Imaging and Graphics 31(4):198–211, 2007.CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Le, A. H. T., Liu, B., and Huang, H. K., Integration of computer-aided diagnosis/detection (CAD) results in a PACS environment using CAD–PACS toolkit and DICOM SR. International Journal of Computer Assisted Radiology and Surgery 4(4):317–329, 2009.CrossRefPubMedGoogle Scholar
  10. 10.
    Dicken, V., Lindow, B., Bornemann, L., Drexl, J., Nikoubashman, A., and Peitgen, H.-O., Rapid image recognition of body parts scanned in computed tomography datasets. International Journal of Computer Assisted Radiology and Surgery 5(5):527–535, 2010.CrossRefPubMedGoogle Scholar
  11. 11.
    Zhou, X et al. (2004) Automated recognition of human strucure from torso CT images, in International conference on visualization, imaging, and image processing, pp. 584–589.Google Scholar
  12. 12.
    Desikan, R. S. et al., An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. Neuroimage 31(3):968–980, 2006.CrossRefPubMedGoogle Scholar
  13. 13.
    Litjens, G. et al., A survey on deep learning in medical image analysis. Medical Image Analysis 42:60–88, 2017.CrossRefPubMedGoogle Scholar
  14. 14.
    H. R. Roth et al., (2015) Anatomy-specific classification of medical images using deep convolutional nets, in IEEE 12th international symposium on biomedical imaging, pp. 101–104.Google Scholar
  15. 15.
    Kumar, A., Kim, J., Lyndon, D., Fulham, M., and Feng, D., An Ensemble of Fine-Tuned Convolutional Neural Networks for medical image classification. IEEE Journal of Biomedical and Health Informatics PP(99):1, 2016.Google Scholar
  16. 16.
    Rajkomar, A., Lingam, S., Taylor, A. G., Blum, M., and Mongan, J., High-throughput classification of radiographs using deep convolutional neural networks. Journal of Digital Imaging 30(1):95–101, 2017.CrossRefPubMedGoogle Scholar
  17. 17.
    Ronneberger, O., Fischer, P., Brox, T. (2015) U-net: Convolutional networks for biomedical image segmentation, in International conference on medical image computing and computer-assisted intervention, pp. 234–241.Google Scholar
  18. 18.
    Wang, Y., Qiu, Y., Thai, T., Moore, K., Liu, H., and Zheng, B., A two-step convolutional neural network based computer-aided detection scheme for automatically segmenting adipose tissue volume depicting on CT images. Computer Methods and Programs in Biomedicine 144:97–104, 2017.CrossRefPubMedPubMedCentralGoogle Scholar
  19. 19.
    Lehmann, T. M., Wein, B. E., Keysers, D., Kohnen, M., Schubert, H. (2002) A monohierarchical multiaxial classification code for medical images in content-based retrieval, in IEEE international symposium on biomedical imaging, pp. 313–316.Google Scholar
  20. 20.
    Markonis, D. et al. (2014) A Visual Information Retrieval System for Radiology Reports and the Medical Literature, in Multimedia modeling conference.Google Scholar
  21. 21.
    Riegler, M. et al., From annotation to computer-aided diagnosis. ACM Transactions on Multimedia Computing, Communications, and Applications 13(3):1–26, 2017.CrossRefGoogle Scholar
  22. 22.
    Kundu, M. K., Chowdhury, M., and Das, S., Interactive radiographic image retrieval system. Computer Methods and Programs in Biomedicine 139:209–220, 2017.CrossRefPubMedGoogle Scholar
  23. 23.
    Valente, F., Viana-Ferreira, C., Costa, C., and Oliveira, J. L., A RESTful image gateway for multiple medical image repositories. IEEE Transactions on Information Technology in Biomedicine 16(3):356–364, 2012.CrossRefPubMedGoogle Scholar
  24. 24.
    Costa, C., Freitas, F., Pereira, M., Silva, A., and Oliveira, J. L., Indexing and retrieving DICOM data in disperse and unstructured archives. International Journal of Computer Assisted Radiology and Surgery 4(1):71–77, 2009.CrossRefPubMedGoogle Scholar
  25. 25.
    Valente, F., Costa, C., and Silva, A., Dicoogle, a PACS featuring profiled content based image retrieval. Public Library of Science One 8(5):e61888, 2013.PubMedGoogle Scholar
  26. 26.
    Pinho, E., Godinho, T., Valente, F., Costa, C., A multimodal search engine for medical imaging studies, Journal of Digital Imaging 30(1):39–48, 2017.Google Scholar
  27. 27.
    Pinho, E, Costa, C. (2016) Extensible Architecture for Multimodal Information Retrieval in Medical Imaging Archives, in Signal image technology & internet based systems, pp. 316–322.Google Scholar
  28. 28.
    Krizhevsky, A., Sutskever, I., Hinton, G. E. (2012) ImageNet classification with deep convolutional neural networks, in Advances in neural information processing systems, pp. 1097–1105.Google Scholar
  29. 29.
    Ioffe, S., Szegedy, C. (2015) Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, arXiv preprint arXiv:1502.03167, pp. 1–11.Google Scholar
  30. 30.
    He, K., Zhang, X., Ren, S., Sun, J. (2015) Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, in IEEE international conference on computer vision, pp. 1026–1034.Google Scholar
  31. 31.
    Kingma, D. P., Ba, J. L. (2015) Adam: A Method for Stochastic Optimization, in International conference on learning representations.Google Scholar
  32. 32.
    National Electrical Manufacturers Association (NEMA) (2018) Digital imaging and communications in medicine (DICOM) standard, PS3.3 – information object definitions. National Electrical Association, Rosslyn, VA, USA.Google Scholar
  33. 33.
    Silva, L. A. B., Costa, C., Oliveira, J. L. (2014) Semantic search over DICOM repositories, in IEEE international conference on healthcare informatics, pp. 238–246.Google Scholar
  34. 34.
    Kurtz, C., Depeursinge, A., Napel, S., Beaulieu, C. F., and Rubin, D. L., On combining image-based and ontological semantic dissimilarities for medical image retrieval applications. Medical Image Analysis 18(7):1082–1100, 2014.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Instituto de Engenharia Electrónica e Informática de Aveiro, DETI / IEETA - University of AveiroCampus Universitário de SantiagoAveiroPortugal

Personalised recommendations