Journal of Digital Imaging

, Volume 21, Issue 3, pp 280–289 | Cite as

Extended Query Refinement for Medical Image Retrieval

  • Thomas M. Deserno
  • Mark O. Güld
  • Bartosz Plodowski
  • Klaus Spitzer
  • Berthold B. Wein
  • Henning Schubert
  • Hermann Ney
  • Thomas Seidl


The impact of image pattern recognition on accessing large databases of medical images has recently been explored, and content-based image retrieval (CBIR) in medical applications (IRMA) is researched. At the present, however, the impact of image retrieval on diagnosis is limited, and practical applications are scarce. One reason is the lack of suitable mechanisms for query refinement, in particular, the ability to (1) restore previous session states, (2) combine individual queries by Boolean operators, and (3) provide continuous-valued query refinement. This paper presents a powerful user interface for CBIR that provides all three mechanisms for extended query refinement. The various mechanisms of man–machine interaction during a retrieval session are grouped into four classes: (1) output modules, (2) parameter modules, (3) transaction modules, and (4) process modules, all of which are controlled by a detailed query logging. The query logging is linked to a relational database. Nested loops for interaction provide a maximum of flexibility within a minimum of complexity, as the entire data flow is still controlled within a single Web page. Our approach is implemented to support various modalities, orientations, and body regions using global features that model gray scale, texture, structure, and global shape characteristics. The resulting extended query refinement has a significant impact for medical CBIR applications.

Key words

Graphical user interface (GUI) web-based interface query refinement relevance feedback usability 


  1. 1.
    Tagare HD, Jaffe CC, Dungan J: Medical image databases—A content-based retrieval approach. J Am Med Inform Assoc 4:184–198, 1997PubMedGoogle Scholar
  2. 2.
    Müller H, Michoux N, Bandon D, Geissbuhler A: A review of content-based image retrieval systems in medical applications—Clinical benefits and future directions. Int J Med Inform 73(1):1–23, 2004PubMedCrossRefGoogle Scholar
  3. 3.
    Deselaers T, Müller H, Clough P, Ney H, Deserno TM: The CLEF 2005 automatic medical image annotation task. International Journal of Computer Vision 2007 (in press) DOI 10.1007/s11263-006-0007-y
  4. 4.
    Müller H, Deselaers T, Deserno TM, Clough P, Kim E, Hersh W: Overview of the ImageCLEFmed 2006 medical retrieval and annotation tasks. Lect Notes Comput Sci (in press)Google Scholar
  5. 5.
    Deserno TM, Güld MO, Deselaers T, Keysers D, Schubert H, Spitzer K, Ney H, Wein BB: Automatic categorization of medical images for content-based retrieval and data mining. Comput Med Imaging Graph 29(2):143–155, 2005CrossRefGoogle Scholar
  6. 6.
    Zhou W, Smalheiser NR, Yu C: A tutorial on information retrieval: Basic terms and concepts. J Biomed Discov Collab 1:2, 2006PubMedCrossRefGoogle Scholar
  7. 7.
    Celetyno A, Di Sciascio E: Feature integration and relevance feedback analysis in image similarity evaluation. J Electron Imaging 7(2):308–317, 1998CrossRefGoogle Scholar
  8. 8.
    Rui Y, Huang TS, Ortega M, Mehrotra S: Relecance feedback—A power tool for interactive content-based image retrieval. IEEE Trans Circuits Syst Video Technol 8(5):644–655, 1998CrossRefGoogle Scholar
  9. 9.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R: Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380, 2000CrossRefGoogle Scholar
  10. 10.
    Naster C, Mitschke M, Meilhac C: Efficient query refinement for image retrieval. Proceedings of the IEEE Computer Society Conference on Comput Vis Pattern Recognit 547–552, 1998Google Scholar
  11. 11.
    Müller H, Müller W, Squire DM, Marchand-Maillet S, Pun T: Strategies for positive and negative relevance feedback in image retrieval. Proceedings 15th International Conference on Pattern Recognition 1043–1046, 2000Google Scholar
  12. 12.
    Bartolini I, Ciaccia P, Waas F: FeedbackBypass—A new approach to interactive similarity query processing. Proceedings 27th International Conference on Very Large Data Bases 201–210, 2001Google Scholar
  13. 13.
    Laaksonen J, Koskela M, Laakso S, Oja E: Self-organizing maps as a relevance feedback technique in content-based image retrieval. Pattern Anal Appl 4(2–3):140–152, 2001CrossRefGoogle Scholar
  14. 14.
    Chen JY, Bouman CA, Dalton JC: Active browsing using similarity pyramids. Proceedings SPIE 3656:144–154, 1998CrossRefGoogle Scholar
  15. 15.
    He XF, King O, Ma WY, Li MJ, Zhang HJ: Learning a semantic space from relevance feedback for image retrieval. IEEE Trans Circuits Syst Video Technol 13(1):39–48, 2003CrossRefGoogle Scholar
  16. 16.
    Crestani F, De la Fuente P, Vegas J: Design of a graphical user interface for focussed retrieval of structured documents. Proceedings 8th Symposium on String Processing and Information Retrieval 246–249, 2001Google Scholar
  17. 17.
    Vegas J, De la Fuente P, Crestani F: A graphical user interface for structured document retrieval. Lect Notes Comput Sci 2291:268–283, 2002CrossRefGoogle Scholar
  18. 18.
    Fox KL, Frieder O, Knepper MM, Snowberg EJ: SENTINEL—A multiple engine information retrieval and visualization system. J Am Soc Inf Sci 50(7):616–625, 1999CrossRefGoogle Scholar
  19. 19.
    Meiers T, Sikora T, Keller I: Hierarchical image database browsing environment with embedded relevance feedback. Proceedings 2002 International Conference on Image Processing 2:593–596, 2002Google Scholar
  20. 20.
    Zhu XQ, Zhang HJ, Liu WY, Hu CH, Wu L: New query refinement and semantics integrated image retrieval system with semiautomatic annotation scheme. J Electron Imaging 10(4):850–860, 2001CrossRefGoogle Scholar
  21. 21.
    Deserno TM, Güld MO, Thies C, Fischer B, Spitzer K, Keysers D, Ney H, Kohnen M, Schubert H, Wein BB: Content-based image retrieval in medical applications. Methods Inf Med 43(4):354–361, 2004Google Scholar
  22. 22.
    Güld MO, Thies C, Fischer B, Deserno TM: A generic concept for the implementation of medical image retrieval systems. Int J Med Inf 76(2–3):252–259, 2007CrossRefGoogle Scholar
  23. 23.
    Deserno TM, Plodowski B, Spitzer K, Wein BB, Ney H, Seidl T: Extended query refinement for content-based access to large medical image databases. Proceedings SPIE 5371:90–98, 2004CrossRefGoogle Scholar
  24. 24.
    Berlage T; A selective undo mechanism for graphical user interfaces based on command objects. ACM Trans Comput Hum Interact 1(3):269–294, 1994CrossRefGoogle Scholar
  25. 25.
    Kreuseler M, Nocke T, Schumann H: A history mechanism for visual data mining. Proceedings IEEE symposium on information visualization (INFOVIS) 49–56, 2004Google Scholar
  26. 26.
    Rui Y, Huang TS, Ortega M, Mehrotra S: Relevance feedback: A power tool for interactive content-based image retrieval. IEEE Trans Circuits Syst Video Technol 8(5):644–655, 1998CrossRefGoogle Scholar
  27. 27.
    Nielsen J, Mack RL Eds.: Usability Inspections Methods. Wiley, Chichester, UK, 1994Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2007

Authors and Affiliations

  • Thomas M. Deserno
    • 1
  • Mark O. Güld
    • 1
  • Bartosz Plodowski
    • 1
  • Klaus Spitzer
    • 1
  • Berthold B. Wein
    • 2
  • Henning Schubert
    • 2
  • Hermann Ney
    • 3
  • Thomas Seidl
    • 4
  1. 1.Department of Medical InformaticsAachen University of Technology (RWTH)AachenGermany
  2. 2.Department of Diagnostic RadiologyAachen University of Technology (RWTH)AachenGermany
  3. 3.Department of Computer Science 6Aachen University of Technology (RWTH)AachenGermany
  4. 4.Department of Computer Science 9Aachen University of Technology (RWTH)AachenGermany

Personalised recommendations