Case-based fracture image retrieval

  • Xin ZhouEmail author
  • Richard Stern
  • Henning Müller
Original Article



Case-based fracture image retrieval can assist surgeons in decisions regarding new cases by supplying visually similar past cases. This tool may guide fracture fixation and management through comparison of long-term outcomes in similar cases.


A fracture image database collected over 10 years at the orthopedic service of the University Hospitals of Geneva was used. This database contains 2,690 fracture cases associated with 43 classes (based on the AO/OTA classification). A case-based retrieval engine was developed and evaluated using retrieval precision as a performance metric. Only cases in the same class as the query case are considered as relevant. The scale-invariant feature transform (SIFT) is used for image analysis. Performance evaluation was computed in terms of mean average precision (MAP) and early precision (P10, P30). Retrieval results produced with the GNU image finding tool (GIFT) were used as a baseline.

Two sampling strategies were evaluated. One used a dense 40 × 40 pixel grid sampling, and the second one used the standard SIFT features. Based on dense pixel grid sampling, three unsupervised feature selection strategies were introduced to further improve retrieval performance. With dense pixel grid sampling, the image is divided into 1,600 (40 × 40) square blocks. The goal is to emphasize the salient regions (blocks) and ignore irrelevant regions. Regions are considered as important when a high variance of the visual features is found. The first strategy is to calculate the variance of all descriptors on the global database. The second strategy is to calculate the variance of all descriptors for each case. A third strategy is to perform a thumbnail image clustering in a first step and then to calculate the variance for each cluster. Finally, a fusion between a SIFT-based system and GIFT is performed.


A first comparison on the selection of sampling strategies using SIFT features shows that dense sampling using a pixel grid (MAP = 0.18) outperformed the SIFT detector-based sampling approach (MAP = 0.10). In a second step, three unsupervised feature selection strategies were evaluated. A grid parameter search is applied to optimize parameters for feature selection and clustering. Results show that using half of the regions (700 or 800) obtains the best performance for all three strategies. Increasing the number of clusters in clustering can also improve the retrieval performance. The SIFT descriptor variance in each case gave the best indication of saliency for the regions (MAP = 0.23), better than the other two strategies (MAP = 0.20 and 0.21). Combining GIFT (MAP = 0.23) and the best SIFT strategy (MAP = 0.23) produced significantly better results (MAP = 0.27) than each system alone.


A case-based fracture retrieval engine was developed and is available for online demonstration. SIFT is used to extract local features, and three feature selection strategies were introduced and evaluated. A baseline using the GIFT system was used to evaluate the salient point-based approaches. Without supervised learning, SIFT-based systems with optimized parameters slightly outperformed the GIFT system. A fusion of the two approaches shows that the information contained in the two approaches is complementary. Supervised learning on the feature space is foreseen as the next step of this study.


Content-based image retrieval Feature selection Fracture database Medical imaging Decision support system 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abdullah A, Veltkamp RC, Wiering MA (2010) Ensembles of novel visual keywords descriptors for image categorization. In: 11th International conference on control automation robotics vision (ICARCV), pp 1206–1211Google Scholar
  2. 2.
    Avni U, Greenspan H, Konen E, Sharon M, Goldberger J (2011) X-ray categorization and retrieval on the organ and pathology level, using patch-based visual words. IEEE Trans Med Imaging 30(3): 733–746PubMedCrossRefGoogle Scholar
  3. 3.
    Barnard K, Duygulu P, Forsyth D, de Freitas N, Blei DM, Jordan MI (2003) Matching words and pictures. J. Mach. Learn. Res. 3: 1107–1135Google Scholar
  4. 4.
    Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (surf). Comput Vis Image Underst 110(3): 346–359CrossRefGoogle Scholar
  5. 5.
    Clough P, Müller H, Sanderson M (2005) The CLEF cross-language image retrieval track (ImageCLEF) 2004. In: Peters C, Clough P, Gonzalo J, Jones GJF, Kluck M, Magnini B (eds) Multilingual information access for text, speech and images: result of the fifth CLEF evaluation campaign. Lecture notes in computer science (LNCS), vol 3491. Springer, Bath, UK, pp 597–613Google Scholar
  6. 6.
    Clough P, Sanderson M, Müller H (2004) The CLEF cross language image retrieval track (ImageCLEF) 2004. In: The challenge of image and video retrieval (CIVR 2004), Lecture notes in computer science, vol 3115, Springer, pp 243–251Google Scholar
  7. 7.
    Croft WB (2000) Combining approaches to information retrieval. In: Advances in information retrieval. Springer, US, pp 1–36Google Scholar
  8. 8.
    Csurka G, Dance CR, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In: In workshop on statistical learning in computer vision, ECCV, pp 1–22Google Scholar
  9. 9.
    Danis R (1949) Théorie et Pratique de l’Ostéosynthèse. Masson and Cie, Paris, FranceGoogle Scholar
  10. 10.
    Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2): 1–60CrossRefGoogle Scholar
  11. 11.
    Deselaers T, Keysers D, Ney H (2008) Features for image retrieval: an experimental comparison. Inf Retr 11: 77–107CrossRefGoogle Scholar
  12. 12.
    Deselaers T, Weyand T, Keysers D, Macherey W, Ney H (2005) FIRE in ImageCLEF 2005: combining content-based image retrieval with textual information retrieval. In: Working notes of the CLEF workshop. Vienna, AustriaGoogle Scholar
  13. 13.
    Donnelley M (2008) Computer aided long-bone segmentation and fracture detection. Ph.D. thesis, Flinders University of South Australia, Adelaide, South AustraliaGoogle Scholar
  14. 14.
    Donnelley M, Knowles G (2005) Computer aided long bone fracture detection. In: Proceedings of the 8th international symposium on signal processing and its applications (ISSPA 2005), vol 1. Sydney, Australia, pp 175–178Google Scholar
  15. 15.
    Donnelley M, Knowles G, Hearn T (2008) A cad system for long-bone segmentation and fracture detection. In: Proceedings of the 3rd international conference on image and signal processing (ICISP 2008). Lecture notes in computer science, vol 5099. Springer, pp 153–162Google Scholar
  16. 16.
    Fox EA, Shaw JA (1993) Combination of multiple searches. In: Text retrieval conference, pp 243–252Google Scholar
  17. 17.
    He JC, Leow WK, Howe TS (2007) Hierarchical classifiers for detection of fractures in X-ray images. In: Proceedings of the 12th international conference on computer analysis of images and patterns (CAIP 2007). Lecture notes in computer science, vol 4673. Springer, Vienna, Austria, pp 962–969Google Scholar
  18. 18.
    Hsu W, Antani S, Long LR, Neve L, Thoma GR (2009) Spirs: a web-based image retrieval system for large biomedical databases. Int J Med Inform 78(Supplement 1):S13–S24. MedInfo 2007Google Scholar
  19. 19.
    Jiang YG, Ngo CW, Yang J (2007) Towards optimal bag-of-features for object categorization and semantic video retrieval. In: CIVR ’07: proceedings of the 6th ACM international conference on image and video retrieval. ACM, New York, NY, USA, pp 494–501Google Scholar
  20. 20.
    Ke Y, Sukthankar R (2004) Pca-sift: a more distinctive representation for local image descriptors. In: IEEE computer society conference on computer vision and pattern recognition (CVPR 2004), vol 2. Washington, DC, USA, pp 506–513Google Scholar
  21. 21.
    Lehmann TM, Güld MO, Thies C, Fischer B, Spitzer K, Keysers D, Ney H, Kohnen M, Schubert H, Wein BB (2004) Content-based image retrieval in medical applications. Methods Inf Med 43: 354–361PubMedGoogle Scholar
  22. 22.
    Lim SE, Xing Y, Chen Y, Leow WK, Howe TS, Png MA (2004) Detection of femur and radius fractures in X-ray images. In: Proceedings of 2nd international conference on advances in medical signal and information processing. Sliema, Malta, pp 249–256Google Scholar
  23. 23.
    Liu C, Yuen J, Torralba A, Sivic J, Freeman WT (2008) SIFT flow: dense correspondence across different scenes. In: ECCV ’08: proceedings of the 10th European conference on computer vision. Springer, Berlin, Heidelberg. pp 28–42Google Scholar
  24. 24.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2): 91–110CrossRefGoogle Scholar
  25. 25.
    Lum VLF, Leow WK, Chen Y, Howe TS, Png MA (2005) Combining classifiers for bone fracture detection in X-ray images. In: IEEE international conference on image processing (ICIP’2005), vol 1. Genoa, Italy, pp 1149–1152Google Scholar
  26. 26.
    Marsh JL, Slongo TF, Agel J, Broderick JS, Creevey W, DeCoster TA, Prokuski L, Sirkin MS, Ziran B, Henley B, Audigé L (2007) Fracture and dislocation classification compendium—2007: Orthopaedic trauma association classification, database and outcomes committee. J Orthop Trauma 21(10 Suppl): S1–S133PubMedCrossRefGoogle Scholar
  27. 27.
    Matas J (2004) Robust wide-baseline stereo from maximally stable extremal regions. Image Vis Comput 22(10): 761–767CrossRefGoogle Scholar
  28. 28.
    Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Trans Pattern Anal Mach Intell 27(10): 1615–1630PubMedCrossRefGoogle Scholar
  29. 29.
    Müller H, Deselaers T, Kim E, Kalpathy-Cramer J, Deserno TM, Clough P, Hersh W (2008) Overview of the ImageCLEFmed 2007 medical retrieval and annotation tasks. In: CLEF 2007 proceedings. Lecture notes in computer science (LNCS), vol 5152. Springer, Budapest, Hungary, pp 473–491Google Scholar
  30. 30.
    Müller H, Deselaers T, Lehmann T, Clough P, Kim E, Hersh W (2007) Overview of the ImageCLEFmed 2006 medical retrieval and medical annotation tasks. In: CLEF 2006 proceedings. Lecture notes in computer science (LNCS), vol 4730. Springer, Alicante, Spain, pp 595–608Google Scholar
  31. 31.
    Müller H, Fabry P, Lovis C, Geissbuhler A (2003) MedGIFT—retrieving medical image by their visual content. In: World summit of the information society, forum science and society. Geneva, SwitzerlandGoogle Scholar
  32. 32.
    Müller H, Kalpathy-Cramer J, Kahn CE Jr, Hatt W, Bedrick S, Hersh W (2009) Overview of the ImageCLEFmed 2008 medical image retrieval task. In: Peters C, Giampiccolo D, Ferro N, Petras V, Gonzalo J, Peñas A, Deselaers T, Mandl T, Jones G, Kurimo M (eds) Evaluating systems for multilingual and multimodal information access—9th workshop of the cross-language evaluation forum. Lecture notes in computer science.,vol 5706. Aarhus, Denmark, pp 500–510Google Scholar
  33. 33.
    Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medicine-clinical benefits and future directions. Int J Med Inform 73(1): 1–23PubMedCrossRefGoogle Scholar
  34. 34.
    Muller H, Zhou X, Depeursinge A, Pitkanen M, Iavindrasana J, Geissbuhler A (2007) Medical visual information retrieval: state of the art and challenges ahead. In: Proceedings of the 2007 IEEE international conference on multimedia and expo, ICME’07. IEEE, pp 683–686Google Scholar
  35. 35.
    Nehemiah HK, Khanna A, Kumar DS (2006) Intelligent fractured image retrieval from medical image databases. Asian J Inf Technol 5: 448–453Google Scholar
  36. 36.
    Quellec G, Lamard M, Cazuguel G, Roux C, Cochener B (2011) Case retrieval in medical databases by fusing heterogeneous information. IEEE Trans Med Imaging 30(1): 108–118PubMedCrossRefGoogle Scholar
  37. 37.
    Rosset A, Müller H, Martins M, Dfouni N, Vallée JP, Ratib O (2004) Casimage project—a digital teaching files authoring environment. J Thorac Imaging 19(2): 1–6CrossRefGoogle Scholar
  38. 38.
    Savoy J (2002) Report on CLEF—2001 experiments. In: Report on the CLEF conference 2001 (Cross Language Evaluation Forum). LNCS 2406, Springer, Darmstadt, Germany, pp 27–43Google Scholar
  39. 39.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12): 1349–1380CrossRefGoogle Scholar
  40. 40.
    Squire DM, Müller W, Müller H, Pun T (2000) Content-based query of image databases: inspirations from text retrieval. In: Ersboll BK, Johansen P (eds) Pattern recognition letters (selected papers from the 11th scandinavian conference on image analysis SCIA ‘99), vol 21, pp 1193–1198Google Scholar
  41. 41.
    Stern R, Hoffmeyer P, Rosset A, Garcia J (2003) Fractures. University of Geneva, Geneva, SwitzerlandGoogle Scholar
  42. 42.
    Swain MJ, Ballard DH (1991) Color indexing. Int J Comput Vis 7(1): 11–32CrossRefGoogle Scholar
  43. 43.
    Tommasi T, Orabona F, Caputo B (2008) CLEF2008 image annotation task: an SVM confidence-based approach. In: Working notes of the 2008 CLEF workshop. Aarhus, DenmarkGoogle Scholar
  44. 44.
    Weber B (1966) Die verletzungen des oberen sprungge-lenkes. Aktuelle Probleme in der Chirurgie. Huber, StuttgartGoogle Scholar
  45. 45.
    Zhou X, Depeursinge A, Müller H (2010) Information fusion for combining visual and textual image retrieval. In: International conference on pattern recognition, ICPR’10. IEEE Computer Society, Los Alamitos, CA, USAGoogle Scholar
  46. 46.
    Zhou X, Eggel I, Müller H (2009) The MedGIFT group at ImageCLEF 2009. In: Working notes of CLEF 2009 (Cross Language Evaluation Forum). Corfu, GreeceGoogle Scholar
  47. 47.
    Zhou X, Gobeill J, Müller H (2009) The MedGIFT group at ImageCLEF 2008. In: CLEF 2008 proceedings. Lecture notes in computer science (LNCS), vol 5706. Springer, Aarhus, Denmark, pp 712–718Google Scholar
  48. 48.
    Zhou X, Gobeill J, Ruch P, Müller H (2008) University and hospitals of geneva participating at imageclef 2007. In: CLEF 2007 proceedings. Lecture notes in computer science (LNCS), vol 5152. Springer, Budapest, Hungary, pp 649–656Google Scholar
  49. 49.
    Zhou X, Krabbenhöft H, Niinimäki M, Depeursinge A, Möller S, Müller H (2009) An easy setup for parallel medical image processing: using Taverna and ARC. In: Proceedings of HealthGrid 2009. Berlin, GermanyGoogle Scholar

Copyright information

© CARS 2011

Authors and Affiliations

  1. 1.Geneva University Hospitals and University of Geneva (HUG)GenevaSwitzerland
  2. 2.Division of Orthopaedics and Trauma SurgeryGeneva University Hospitals and University of Geneva (HUG)GenevaSwitzerland
  3. 3.MedGIFT group, Business Information SystemsUniversity of Applied Sciences Western Switzerland (HES-SO)SierreSwitzerland

Personalised recommendations