Immediate Structured Visual Search for Medical Images

  • Karen Simonyan
  • Andrew Zisserman
  • Antonio Criminisi
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6893)


The objective of this work is a scalable, real-time visual search engine for medical images. In contrast to existing systems that retrieve images that are globally similar to a query image, we enable the user to select a query Region Of Interest (ROI) and automatically detect the corresponding regions within all returned images. This allows the returned images to be ranked on the content of the ROI, rather than the entire image. Our contribution is two-fold: (i) immediate retrieval – the data is appropriately pre-processed so that the search engine returns results in real-time for any query image and ROI; (ii) structured output – returning ROIs with a choice of ranking functions. The retrieval performance is assessed on a number of annotated queries for images from the IRMA X-ray dataset and compared to a baseline.


Image Retrieval Target Image Ranking Function Query Image Retrieval Performance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Chui, H., Rangarajan, A.: A new point matching algorithm for non-rigid registration. Comp. Vis. and Image Understanding 89(2-3), 114–141 (2003)CrossRefzbMATHGoogle Scholar
  2. 2.
    Chum, O., Matas, J., Obdržálek, Š.: Enhancing RANSAC by generalized model optimization. In: Proc. of Asian Conf. on Comp. Vis., vol. 2, pp. 812–817 (2004)Google Scholar
  3. 3.
  4. 4.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes (VOC) challenge. Int. J. Comp. Vis. 88(2), 303–338 (2010)CrossRefGoogle Scholar
  5. 5.
    Guld, M.O., Kohnen, M., Keysers, D., Schubert, H., Wein, B., Bredno, J., Lehmann, T.M.: Quality of DICOM header information for image categorization. In: Int. Symp. Med. Imag. SPIE, vol. 4685, pp. 280–287 (February 2002)Google Scholar
  6. 6.
    Isgum, I., Staring, M., Rutten, A., Prokop, M., Viergever, M., van Ginneken, B.: Multi-atlas-based segmentation with local decision fusion – application to cardiac and aortic segmentation in CT scans. IEEE Trans. Med. Imag. 28(7), 1000–1010 (2009)CrossRefGoogle Scholar
  7. 7.
    Klein, S., Staring, M., Murphy, K., Viergever, M.A., Pluim, J.P.W.: elastix: A toolbox for intensity-based medical image registration. IEEE Trans. Med. Imag. 29(1), 196–205 (2010)CrossRefGoogle Scholar
  8. 8.
    Lam, M., Disney, T., Pham, M., Raicu, D., Furst, J., Susomboon, R.: Contentbased image retrieval for pulmonary computed tomography nodule images. In: Med. Imag. 2007: PACS and Imag. Inform. SPIE, vol. 6516 (March 2007)Google Scholar
  9. 9.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comp. Vis. 60, 91–110 (2004)CrossRefGoogle Scholar
  10. 10.
    Müller, H., Michoux, N., Bandon, D., Geissbuhler, A.: A review of content-based image retrieval systems in medical applications – clinical benefits and future directions. Int. J. Med. Inform. 73(1), 1–23 (2004)CrossRefGoogle Scholar
  11. 11.
    Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: Proc. IEEE Conf. on Comp. Vis. and Patt. Recog. (2007)Google Scholar
  12. 12.
    Sivic, J., Zisserman, A.: Efficient visual search of videos cast as text retrieval. IEEE Trans. Patt. Anal. and Mach. Intell. 31(4), 591–606 (2009)CrossRefGoogle Scholar
  13. 13.
    Vedaldi, A., Gulshan, V., Varma, M., Zisserman, A.: Multiple kernels for object detection. In: Proc. Int. Conf. Comp. Vis., pp. 606–613 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Karen Simonyan
    • 1
  • Andrew Zisserman
    • 1
  • Antonio Criminisi
    • 2
  1. 1.University of OxfordUK
  2. 2.Microsoft ResearchCambridgeUK

Personalised recommendations