Skip to main content
SpringerLink
Log in

A Multimodal Search Engine for Medical Imaging Studies

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

The use of digital medical imaging systems in healthcare institutions has increased significantly, and the large amounts of data in these systems have led to the conception of powerful support tools: recent studies on content-based image retrieval (CBIR) and multimodal information retrieval in the field hold great potential in decision support, as well as for addressing multiple challenges in healthcare systems, such as computer-aided diagnosis (CAD). However, the subject is still under heavy research, and very few solutions have become part of Picture Archiving and Communication Systems (PACS) in hospitals and clinics. This paper proposes an extensible platform for multimodal medical image retrieval, integrated in an open-source PACS software with profile-based CBIR capabilities. In this article, we detail a technical approach to the problem by describing its main architecture and each sub-component, as well as the available web interfaces and the multimodal query techniques applied. Finally, we assess our implementation of the engine with computational performance benchmarks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. www.mrml.net (as seen in August 3rd 2012)

References

  1. Myers B: U.S. medical imaging informatics industry reconnects with growth in the enterprise image archiving market. 2012. [Online]. Available: http://www.frost.com/prod/servlet/press-release.pag?docid=268728701. [Accessed: 08-Feb-2016]

  2. National Electrical Manufacturers Association (NEMA): Digital Imaging and Communications in Medicine (DICOM) standard. Rosslyn, VA, USA

  3. Valente F, Viana-Ferreira C, Costa C, Oliveira JL: A RESTful image gateway for multiple medical image repositories. IEEE Trans Inf Technol Biomed 16(3):356–364, 2012

    Article  PubMed  Google Scholar 

  4. Akgül CB, Rubin DL, Napel S, Beaulieu CF, Greenspan H, Acar B: Content-based image retrieval in radiology: current status and future directions. J Digit Imaging 24(2):208–222, 2011

    Article  PubMed  Google Scholar 

  5. Müller H, Despont C: Health care professionals’ image use and search behaviour. Proc Med Inform Eur. pp 24–32, 2006

  6. Hanjalic A, Lienhart R, Ma W-Y, Smith JR: The holy grail of multimedia information retrieval: So close or yet so far away? Proc IEEE 4(96):541–547, 2008

    Article  Google Scholar 

  7. Atrey PK, Hossain MA, El Saddik A, Kankanhalli MS: Multimodal fusion for multimedia analysis: a survey. Multimedia Systems 16(6):345–379, 2010

    Article  Google Scholar 

  8. Valente F, Costa C, Silva A: Dicoogle, a PACS featuring profiled content based image retrieval. PLoS One 8(5):e61888, 2013

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  9. Müller H, Geissbuhler A: Medical multimedia retrieval 2.0. Yearb Med Inform 47(1):55–63, 2008

    Google Scholar 

  10. Cao Y, Steffey S, Jianbiao H, Xiao D, Tao C, Chen P, Müller H: Medical image retrieval: a multimodal approach. Cancer Inform, 2015

  11. Mourão A, Flávio M: NovaMedSearch: A multimodal search engine for medical case-based retrieval. In Proceedings of the 10th Conference on Open Research Areas in Information Retrieval. Le Centre de Hautes Etudes Internationales D’Informatique Documentaire, 2013, pp 223–224

  12. Hanbury A, Boyer C, Gschwandtner M, Müller H: KHRESMOI: towards a multi-lingual search and access system for biomedical information. Med-e-Tel, Luxembourg, 2011, pp 412–416

  13. Schaer R, Markonis D, Müller H: Architecture and applications of the parallel distributed image search engine (ParaDISE). FoRESEE, Stuttgart, 2014

    Google Scholar 

  14. Widmer A, Schaer R, Markonis D, Müller H: Gesture interaction for content–based medical image retrieval. In Proceedings of international conference on multimedia retrieval, 2014, p 503

  15. Markonis D, Donner R, Holzer M, Schlegl T, Dungs S, Kriewel S, Langs G, Müller H: A visual information retrieval system for radiology reports and the medical literature. In Multimedia modeling conference, 2014

  16. Rahman MM, You D, Simpson MS, Antani SK, Demner-Fushman D, Thoma GR: Multimodal biomedical image retrieval using hierarchical classification and modality fusion. Int J Multimed Inf Retr 2(3):159–173, 2013

    Article  Google Scholar 

  17. Valente F, Silva LB, Godinho TM, Costa C: Anatomy of an extensible open source PACS. J Digit Imaging 29(3):284–296, 2016

    Article  PubMed  Google Scholar 

  18. Costa C, Freitas F, Pereira M, Silva A, Oliveira JL: Indexing and retrieving DICOM data in disperse and unstructured archives. Int J Comput Assist Radiol Surg 4(1):71–77, 2009

    Article  PubMed  Google Scholar 

  19. Datta R, Joshi D, Li J, Wang JZ: Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv (CSUR) 40(2):5, 2008

    Article  Google Scholar 

  20. Mourão A, Martins F, Magalhães J: Multimodal medical information retrieval with unsupervised rank fusion. Comput Med Imaging Graph 39:35–45, 2015

    Article  PubMed  Google Scholar 

  21. Fox EA, Shaw JA: Combination of multiple searches. NIST SPECIAL PUBLICATION SP, p 243, 1994

  22. Cormack GV, Clarke CLA, Buettcher S: Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on research and development in information retrieval, 2009, pp 758–759

  23. MontagueM, Aslam JA: Relevance score normalization for metasearch. In Proceedings of the tenth international conference on information and knowledge management, 2001, pp 427–433

  24. Lee JH: Analyses of multiple evidence combination. In ACM SIGIR forum, 1997, vol 31, pp 267–276

  25. Müller W, Müller H, Marchand-Maillet S, Pun T, Squire DM, Pecenovic Z, Giess C, De Vries AP: MRML: an extensible communication protocol for interoperability and benchmarking of multimedia information retrieval systems. In Information technologies 2000, 2000, pp. 124–133

  26. Markonis D, Holzer M, Baroz F, De Castaneda RLR, Boyer C, Langs G, Müller H: User-oriented evaluation of a medical image retrieval system for radiologists. Int J Med Inform, 2015

  27. Rocchio JJ: Relevance feedback in information retrieval. In The SMART retrieval system, experiments in automatic document processing, 1971, pp 313–323

  28. Markonis D, Schaer R, Müller H: Evaluating multimodal relevance feedback techniques for medical image retrieval. Inf Retr J, pp 1–13, 2016

  29. Faruque J, Beaulieu CF, Rosenberg J, Rubin DL, Yao D, Napel S: Content-based image retrieval in radiology: analysis of variability in human perception of similarity. J Med Imaging 2(2):25501, 2015

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. DETI/IEETA, Universidade de Aveiro, Campus Universitário de Santiago, 3810-193, Aveiro, Portugal

    Eduardo Pinho, Tiago Godinho, Frederico Valente & Carlos Costa

Authors
  1. Eduardo Pinho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tiago Godinho
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frederico Valente
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carlos Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eduardo Pinho.

Appendix: Multimodal Query Schema

Appendix: Multimodal Query Schema

figure a

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pinho, E., Godinho, T., Valente, F. et al. A Multimodal Search Engine for Medical Imaging Studies. J Digit Imaging 30, 39–48 (2017). https://doi.org/10.1007/s10278-016-9903-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-016-9903-z

Keywords

Search

Navigation