Skip to main content

Optimal Query-Based Relevance Feedback in Medical Image Retrieval Using Score Fusion-Based Classification

Journal of Digital Imaging volume 28, pages 160–178 (2015)Cite this article

Abstract

In this paper, a new content-based medical image retrieval (CBMIR) framework using an effective classification method and a novel relevance feedback (RF) approach are proposed. For a large-scale database with diverse collection of different modalities, query image classification is inevitable due to firstly, reducing the computational complexity and secondly, increasing influence of data fusion by removing unimportant data and focus on the more valuable information. Hence, we find probability distribution of classes in the database using Gaussian mixture model (GMM) for each feature descriptor and then using the fusion of obtained scores from the dependency probabilities, the most relevant clusters are identified for a given query. Afterwards, visual similarity of query image and images in relevant clusters are calculated. This method is performed separately on all feature descriptors, and then the results are fused together using feature similarity ranking level fusion algorithm. In the RF level, we propose a new approach to find the optimal queries based on relevant images. The main idea is based on density function estimation of positive images and strategy of moving toward the aggregation of estimated density function. The proposed framework has been evaluated on ImageCLEF 2005 database consisting of 10,000 medical X-ray images of 57 semantic classes. The experimental results show that compared with the existing CBMIR systems, our framework obtains the acceptable performance both in the image classification and in the image retrieval by RF.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

References

  1. Tamura H, Yokoya N: Image database systems: a survey. Pattern Recognit 17(1):29–43, 1984

    Article  Google Scholar 

  2. Chang SK, Hsu A: Image information systems: where do we go from here? IEEE Trans Knowl Data Eng 5(5):431–442, 1992

    Article  Google Scholar 

  3. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R: Content-based image retrieval at the end of the early years, IEEE Trans. Pattern Anal Mach Intell 22(12):1349–1380, 2000

    Article  Google Scholar 

  4. Rui Y, Huang TS, Chang S-F: Image retrieval: current techniques, promising directions, and open issues. J Vis Commun Image Represent 10(4):39–62, 1999

    Article  Google Scholar 

  5. Jafari Fesharaki N, Pourghassem H: Medical x-ray image hierarchical classification using a merging and splitting scheme in feature space. J Med Signals Sensors 3(3):150–163, 2013

    Google Scholar 

  6. Liu Y, Zhanga D, Lua G, Ma WY: A survey of content-based image retrieval with high-level semantics. Pattern Recognit 40(1):262–282, 2007

    Article  Google Scholar 

  7. Pourghassem H, Ghassemian H: Content-based medical image classification using a new hierarchical merging scheme. J Comput Med Imaging Graph 32(8):651–661, 2008

    Article  Google Scholar 

  8. N. Jafari Fesharaki, H. Pourghassem: Medical X-ray Images Classification Based on Shape Features and Bayesian Rule, Fourth International Conference on Computational Intelligence and Communication Networks (CICN2012): Mathura, Uttar Pradesh, India 3–5 Nov. 2012, pp 369-373

  9. Rahman MM, Desai BC, Bhattacharya P: Medical image retrieval with probabilistic multi-class support vector machine classifiers and adaptive similarity fusion. J Comput Med Imaging Graph 32(2):95–108, 2008

    Article  Google Scholar 

  10. Greenspan H, Pinhas AT: Medical image categorization and retrieval for PACS using the GMM-KL framework. IEEE Trans Inf Tech Bio 11(2):190–202, 2007

    Article  Google Scholar 

  11. Marakakis A, Siolas G, Galatsanos N, Likas A, Stafylopatis A: Relevance feedback approach for image retrieval combining support vector machines and adapted Gaussian mixture models. IET Image Process 5(6):531–540, 2011

    Article  Google Scholar 

  12. Xie Z, Wang S, Hu D: New insight at level set & Gaussian mixture model for natural image segmentation. SIViP 7(3):521–536, 2013

    Article  Google Scholar 

  13. Chatzichristofis SA, Zagoris K, Boutalis Y, Arampatzis A: A Fuzzy Rank-Based Late Fusion Method for Image Retrieval. 18th International Conference, MMM(2012). Springer Berlin Heidelberg, Klagenfurt, Austria, 2012, pp 463–472

    Google Scholar 

  14. M. Jović, Y. Hatakeyama, F. Dong, K. Hirota: Image Retrieval Based on Similarity Score Fusion from Feature Similarity Ranking Lists, Third International Conference, FSKD 2006, Xi’an: China, Sep. 2006, pp 461–470

  15. Su JH, Huang WH, Yu PS, Tseng VS: Efficient relevance feedback for content-based image retrieval by mining user navigation patterns. IEEE Trans Knowl Data Eng 23(3):360–372, 2011

    Article  Google Scholar 

  16. J.J. Rocchio: Relevance Feedback in Information Retrieval: The SMART Retrieval System—Experiments in Automatic Document Processing, 1971, pp 313–323

  17. Y. Rui, T. S. Huang, and S. Mehrotra: Content-Based Image Retrieval with Relevance Feedback in MARS: Proc. IEEE Int’l Conf. Image Processing, Oct. 1997, pp 815–818

  18. Rui Y, Huang TS, Ortega M, Mehrotra S: Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans Circ Syst Video Techno 8(5):644–655, 1998

    Article  Google Scholar 

  19. K. Porkaew, K. Chakrabarti, and S. Mehrotra: Query Refinement for Multimedia Similarity Retrieval in MARS: Proc. ACM Int’l Multimedia Conf. (ACMMM), 1999, pp 235–238

  20. Rahman MM, Bhattacharya P, Desai BC: A framework for medical image retrieval using machine learning and statistical similarity matching techniques with relevance feedback. IEEE Trans Inf Tech Bio 11(1):58–69, 2007

    Article  Google Scholar 

  21. Pourghassem H, Daneshvar S: A framework for medical image retrieval using merging-based classification with dependency probability-based relevance feedback. Turk J Electr Eng Comput Sci 21(3):882–896, 2013

    Google Scholar 

  22. Rahman MM, Antani SK, Thoma GR: A learning-based similarity fusion and filtering approach for biomedical image retrieval using SVM classification and relevance feedback. IEEE Trans Inf Technol Biomed 15(4):640–646, 2011

    Article  PubMed  Google Scholar 

  23. Yang L, Jin R, Mummert L, Sukthankar R, Goode A, Zheng B, Hoi SCH, Satyanarayanan M: A boosting framework for visuality-preserving distance metric learning and its application to medical image retrieval. IEEE Trans Pattern Anal Mach Intell 32(1):30–44, 2010

    Article  PubMed  Google Scholar 

  24. Duda RO, Hart PE, Stork DG: Pattern Classification, 2nd edition. John Wiley & Sons, New York, NY, 2001

    Google Scholar 

  25. Nurmohamadi M, Pourghassem H: Clavulanic acid production estimation based on color and structural features of streptomyces clavuligerus bacteria using self-organizing map and genetic algorithm. Comput Methods Programs Biomed 114(3):337–348, 2014

    Article  PubMed  Google Scholar 

  26. M. Behnam, H. Pourghassem: Feature Descriptor Optimization in Medical Image Retrieval Based on Genetic Algorithm: 20th Iranian Conference on Biomedical Engineering (ICBME2013), Tehran, Iran: 18–20 Dec, 2013, pp 280–285

  27. da Silva SF, Ribeiro MX, Neto JB, TrainaJr C, Traina AJM: Improving the ranking quality of medical image retrieval using a genetic feature selection method. Decis Support Syst 51(4):810–820, 2011

    Article  Google Scholar 

  28. Steji’c Z, Takama Y, Hirota K: Genetic algorithms for a family of image similarity models incorporated in the relevance feedback mechanism. Appl Soft Comput 2(4):306–327, 2003

    Article  Google Scholar 

  29. Lai CC, Chen YC: A user-oriented image retrieval system based on interactive genetic algorithm. IEEE Trans Instrum Meas 60(10):3318–3325, 2011

    Article  Google Scholar 

  30. Ojala T, Pietikäinen M, Mäenpää T: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987, 2002

    Article  Google Scholar 

  31. Zhao Y, Huang DS, Jia W: Completed local binary count for rotation invariant texture classification. IEEE Trans Image Process 21(10):4492–4497, 2012

    Article  PubMed  Google Scholar 

  32. Guo ZH, Zhang L, Zhang D: A completed modeling of local binary pattern operator for texture classification. IEEE Trans Image Process 19(6):1657–1663, 2010

    Article  PubMed  Google Scholar 

  33. Persoon E, Fu K: Shape discrimination using fourier descriptors. IEEE Trans Pattern Anal Mach Intell 8(3):388–397, 1977

    Google Scholar 

  34. Zahn CT, Roskies RZ: Fourier descriptors for plane closed curves. IEEE Trans Comput 21(3):269–281, 1972

    Article  Google Scholar 

  35. Tahmasebi A, Pourghassem H: A novel intra-class distance-based signature identification algorithm using weighted gabor features and dynamic characteristics. Arab J Sci Eng 38(11):3019–3029, 2013

    Article  Google Scholar 

  36. Zhang DS, Lu G: Generic fourier descriptor for shape-based image retrieval. IEEE Int Conf Multimed Expo 1:425–428, 2002

    Article  Google Scholar 

  37. D. S. Zhang and G. Lu: Enhanced generic Fourier descriptors for object-based image retrieval: IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP2002), 2002, pp 3668_3671

  38. Shahbeig S, Pourghassem H: A fast and automatic algorithm for optic disc extraction in retinal images using PCA-based pre-processing and curvelet transform. J Opt Soc Am A 30(1):13–21, 2013

    Article  Google Scholar 

  39. Pourghassem H: A novel material detection algorithm based on 2D GMM-based power density function and image detail addition scheme in dual energy x-ray images. J X-Ray Sci Technol (IOS press) 20(2):213–228, 2012

    Google Scholar 

  40. Pourghassem H: A relevance feedback approach based on modification of similarity measure using particle swarm optimization in a medical x-ray image retrieval system. Majlesi J Electrical Eng 4(2):9–17, 2010

    Google Scholar 

  41. P. Clough, H. Muller, T. Deselaers, M. Grubinger, T. M. Lehmann, J. Jensen, W. Hersh: The CLEF 2005 Cross–Language Image Retrieval Track: 6th Workshop of the Cross-Language Evaluation Forum, CLEF 2005, Vienna, Austria: 21–23, 2006, September, 2005, pp. 535–557

  42. Smith JR, Chang S: Tools and techniques for color image retrieval. Storage Retrieval Image Video Databases IV 2670:1–12, 1996

    Google Scholar 

  43. Deselaers T, Keysers D, Ney H: Classification error rate for quantitative evaluation of content-based image retrieval systems. Proc 17th International Conf Pattern Recognit 2:505–508, 2004

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the IRMA Group, Aachen, Germany, for making the database available for the experiments.

Author information

Affiliations

  1. Department of Electrical Engineering, Najafabad Branch, Islamic Azad University, 517, Najafabad, Isfahan, Iran

    Mohammad Behnam & Hossein Pourghassem

Authors
  1. Mohammad Behnam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hossein Pourghassem
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hossein Pourghassem.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Behnam, M., Pourghassem, H. Optimal Query-Based Relevance Feedback in Medical Image Retrieval Using Score Fusion-Based Classification. J Digit Imaging 28, 160–178 (2015). https://doi.org/10.1007/s10278-014-9730-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-014-9730-z

Keywords

  • Content-based medical image retrieval (CBMIR)
  • Visual features
  • Semantic concepts
  • Score fusion
  • Relevance feedback
  • Nonparametric estimation
  • Optimal queries