Journal of Digital Imaging

, Volume 30, Issue 1, pp 39–48 | Cite as

A Multimodal Search Engine for Medical Imaging Studies

  • Eduardo PinhoEmail author
  • Tiago Godinho
  • Frederico Valente
  • Carlos Costa


The use of digital medical imaging systems in healthcare institutions has increased significantly, and the large amounts of data in these systems have led to the conception of powerful support tools: recent studies on content-based image retrieval (CBIR) and multimodal information retrieval in the field hold great potential in decision support, as well as for addressing multiple challenges in healthcare systems, such as computer-aided diagnosis (CAD). However, the subject is still under heavy research, and very few solutions have become part of Picture Archiving and Communication Systems (PACS) in hospitals and clinics. This paper proposes an extensible platform for multimodal medical image retrieval, integrated in an open-source PACS software with profile-based CBIR capabilities. In this article, we detail a technical approach to the problem by describing its main architecture and each sub-component, as well as the available web interfaces and the multimodal query techniques applied. Finally, we assess our implementation of the engine with computational performance benchmarks.


Content-based image retrieval Computer systems Graphical user interface (GUI) Information storage and retrieval PACS Reproducibility of results Software design Multimodal information retrieval Query fusion Web services 


  1. 1.
    Myers B: U.S. medical imaging informatics industry reconnects with growth in the enterprise image archiving market. 2012. [Online]. Available: [Accessed: 08-Feb-2016]
  2. 2.
    National Electrical Manufacturers Association (NEMA): Digital Imaging and Communications in Medicine (DICOM) standard. Rosslyn, VA, USAGoogle Scholar
  3. 3.
    Valente F, Viana-Ferreira C, Costa C, Oliveira JL: A RESTful image gateway for multiple medical image repositories. IEEE Trans Inf Technol Biomed 16(3):356–364, 2012CrossRefPubMedGoogle Scholar
  4. 4.
    Akgül CB, Rubin DL, Napel S, Beaulieu CF, Greenspan H, Acar B: Content-based image retrieval in radiology: current status and future directions. J Digit Imaging 24(2):208–222, 2011CrossRefPubMedGoogle Scholar
  5. 5.
    Müller H, Despont C: Health care professionals’ image use and search behaviour. Proc Med Inform Eur. pp 24–32, 2006Google Scholar
  6. 6.
    Hanjalic A, Lienhart R, Ma W-Y, Smith JR: The holy grail of multimedia information retrieval: So close or yet so far away? Proc IEEE 4(96):541–547, 2008CrossRefGoogle Scholar
  7. 7.
    Atrey PK, Hossain MA, El Saddik A, Kankanhalli MS: Multimodal fusion for multimedia analysis: a survey. Multimedia Systems 16(6):345–379, 2010CrossRefGoogle Scholar
  8. 8.
    Valente F, Costa C, Silva A: Dicoogle, a PACS featuring profiled content based image retrieval. PLoS One 8(5):e61888, 2013CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Müller H, Geissbuhler A: Medical multimedia retrieval 2.0. Yearb Med Inform 47(1):55–63, 2008Google Scholar
  10. 10.
    Cao Y, Steffey S, Jianbiao H, Xiao D, Tao C, Chen P, Müller H: Medical image retrieval: a multimodal approach. Cancer Inform, 2015Google Scholar
  11. 11.
    Mourão A, Flávio M: NovaMedSearch: A multimodal search engine for medical case-based retrieval. In Proceedings of the 10th Conference on Open Research Areas in Information Retrieval. Le Centre de Hautes Etudes Internationales D’Informatique Documentaire, 2013, pp 223–224Google Scholar
  12. 12.
    Hanbury A, Boyer C, Gschwandtner M, Müller H: KHRESMOI: towards a multi-lingual search and access system for biomedical information. Med-e-Tel, Luxembourg, 2011, pp 412–416Google Scholar
  13. 13.
    Schaer R, Markonis D, Müller H: Architecture and applications of the parallel distributed image search engine (ParaDISE). FoRESEE, Stuttgart, 2014Google Scholar
  14. 14.
    Widmer A, Schaer R, Markonis D, Müller H: Gesture interaction for content–based medical image retrieval. In Proceedings of international conference on multimedia retrieval, 2014, p 503Google Scholar
  15. 15.
    Markonis D, Donner R, Holzer M, Schlegl T, Dungs S, Kriewel S, Langs G, Müller H: A visual information retrieval system for radiology reports and the medical literature. In Multimedia modeling conference, 2014Google Scholar
  16. 16.
    Rahman MM, You D, Simpson MS, Antani SK, Demner-Fushman D, Thoma GR: Multimodal biomedical image retrieval using hierarchical classification and modality fusion. Int J Multimed Inf Retr 2(3):159–173, 2013CrossRefGoogle Scholar
  17. 17.
    Valente F, Silva LB, Godinho TM, Costa C: Anatomy of an extensible open source PACS. J Digit Imaging 29(3):284–296, 2016CrossRefPubMedGoogle Scholar
  18. 18.
    Costa C, Freitas F, Pereira M, Silva A, Oliveira JL: Indexing and retrieving DICOM data in disperse and unstructured archives. Int J Comput Assist Radiol Surg 4(1):71–77, 2009CrossRefPubMedGoogle Scholar
  19. 19.
    Datta R, Joshi D, Li J, Wang JZ: Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv (CSUR) 40(2):5, 2008CrossRefGoogle Scholar
  20. 20.
    Mourão A, Martins F, Magalhães J: Multimodal medical information retrieval with unsupervised rank fusion. Comput Med Imaging Graph 39:35–45, 2015CrossRefPubMedGoogle Scholar
  21. 21.
    Fox EA, Shaw JA: Combination of multiple searches. NIST SPECIAL PUBLICATION SP, p 243, 1994Google Scholar
  22. 22.
    Cormack GV, Clarke CLA, Buettcher S: Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on research and development in information retrieval, 2009, pp 758–759Google Scholar
  23. 23.
    MontagueM, Aslam JA: Relevance score normalization for metasearch. In Proceedings of the tenth international conference on information and knowledge management, 2001, pp 427–433Google Scholar
  24. 24.
    Lee JH: Analyses of multiple evidence combination. In ACM SIGIR forum, 1997, vol 31, pp 267–276Google Scholar
  25. 25.
    Müller W, Müller H, Marchand-Maillet S, Pun T, Squire DM, Pecenovic Z, Giess C, De Vries AP: MRML: an extensible communication protocol for interoperability and benchmarking of multimedia information retrieval systems. In Information technologies 2000, 2000, pp. 124–133Google Scholar
  26. 26.
    Markonis D, Holzer M, Baroz F, De Castaneda RLR, Boyer C, Langs G, Müller H: User-oriented evaluation of a medical image retrieval system for radiologists. Int J Med Inform, 2015Google Scholar
  27. 27.
    Rocchio JJ: Relevance feedback in information retrieval. In The SMART retrieval system, experiments in automatic document processing, 1971, pp 313–323Google Scholar
  28. 28.
    Markonis D, Schaer R, Müller H: Evaluating multimodal relevance feedback techniques for medical image retrieval. Inf Retr J, pp 1–13, 2016Google Scholar
  29. 29.
    Faruque J, Beaulieu CF, Rosenberg J, Rubin DL, Yao D, Napel S: Content-based image retrieval in radiology: analysis of variability in human perception of similarity. J Med Imaging 2(2):25501, 2015CrossRefGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2016

Authors and Affiliations

  • Eduardo Pinho
    • 1
    Email author
  • Tiago Godinho
    • 1
  • Frederico Valente
    • 1
  • Carlos Costa
    • 1
  1. 1.DETI/IEETAUniversidade de AveiroAveiroPortugal

Personalised recommendations