Case-based fracture image retrieval



Case-based fracture image retrieval can assist surgeons in decisions regarding new cases by supplying visually similar past cases. This tool may guide fracture fixation and management through comparison of long-term outcomes in similar cases.


A fracture image database collected over 10 years at the orthopedic service of the University Hospitals of Geneva was used. This database contains 2,690 fracture cases associated with 43 classes (based on the AO/OTA classification). A case-based retrieval engine was developed and evaluated using retrieval precision as a performance metric. Only cases in the same class as the query case are considered as relevant. The scale-invariant feature transform (SIFT) is used for image analysis. Performance evaluation was computed in terms of mean average precision (MAP) and early precision (P10, P30). Retrieval results produced with the GNU image finding tool (GIFT) were used as a baseline.

Two sampling strategies were evaluated. One used a dense 40 × 40 pixel grid sampling, and the second one used the standard SIFT features. Based on dense pixel grid sampling, three unsupervised feature selection strategies were introduced to further improve retrieval performance. With dense pixel grid sampling, the image is divided into 1,600 (40 × 40) square blocks. The goal is to emphasize the salient regions (blocks) and ignore irrelevant regions. Regions are considered as important when a high variance of the visual features is found. The first strategy is to calculate the variance of all descriptors on the global database. The second strategy is to calculate the variance of all descriptors for each case. A third strategy is to perform a thumbnail image clustering in a first step and then to calculate the variance for each cluster. Finally, a fusion between a SIFT-based system and GIFT is performed.


A first comparison on the selection of sampling strategies using SIFT features shows that dense sampling using a pixel grid (MAP = 0.18) outperformed the SIFT detector-based sampling approach (MAP = 0.10). In a second step, three unsupervised feature selection strategies were evaluated. A grid parameter search is applied to optimize parameters for feature selection and clustering. Results show that using half of the regions (700 or 800) obtains the best performance for all three strategies. Increasing the number of clusters in clustering can also improve the retrieval performance. The SIFT descriptor variance in each case gave the best indication of saliency for the regions (MAP = 0.23), better than the other two strategies (MAP = 0.20 and 0.21). Combining GIFT (MAP = 0.23) and the best SIFT strategy (MAP = 0.23) produced significantly better results (MAP = 0.27) than each system alone.


A case-based fracture retrieval engine was developed and is available for online demonstration. SIFT is used to extract local features, and three feature selection strategies were introduced and evaluated. A baseline using the GIFT system was used to evaluate the salient point-based approaches. Without supervised learning, SIFT-based systems with optimized parameters slightly outperformed the GIFT system. A fusion of the two approaches shows that the information contained in the two approaches is complementary. Supervised learning on the feature space is foreseen as the next step of this study.

This is a preview of subscription content, access via your institution.


  1. 1

    Abdullah A, Veltkamp RC, Wiering MA (2010) Ensembles of novel visual keywords descriptors for image categorization. In: 11th International conference on control automation robotics vision (ICARCV), pp 1206–1211

  2. 2

    Avni U, Greenspan H, Konen E, Sharon M, Goldberger J (2011) X-ray categorization and retrieval on the organ and pathology level, using patch-based visual words. IEEE Trans Med Imaging 30(3): 733–746

    PubMed  Article  Google Scholar 

  3. 3

    Barnard K, Duygulu P, Forsyth D, de Freitas N, Blei DM, Jordan MI (2003) Matching words and pictures. J. Mach. Learn. Res. 3: 1107–1135

    Google Scholar 

  4. 4

    Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (surf). Comput Vis Image Underst 110(3): 346–359

    Article  Google Scholar 

  5. 5

    Clough P, Müller H, Sanderson M (2005) The CLEF cross-language image retrieval track (ImageCLEF) 2004. In: Peters C, Clough P, Gonzalo J, Jones GJF, Kluck M, Magnini B (eds) Multilingual information access for text, speech and images: result of the fifth CLEF evaluation campaign. Lecture notes in computer science (LNCS), vol 3491. Springer, Bath, UK, pp 597–613

  6. 6

    Clough P, Sanderson M, Müller H (2004) The CLEF cross language image retrieval track (ImageCLEF) 2004. In: The challenge of image and video retrieval (CIVR 2004), Lecture notes in computer science, vol 3115, Springer, pp 243–251

  7. 7

    Croft WB (2000) Combining approaches to information retrieval. In: Advances in information retrieval. Springer, US, pp 1–36

  8. 8

    Csurka G, Dance CR, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In: In workshop on statistical learning in computer vision, ECCV, pp 1–22

  9. 9

    Danis R (1949) Théorie et Pratique de l’Ostéosynthèse. Masson and Cie, Paris, France

    Google Scholar 

  10. 10

    Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2): 1–60

    Article  Google Scholar 

  11. 11

    Deselaers T, Keysers D, Ney H (2008) Features for image retrieval: an experimental comparison. Inf Retr 11: 77–107

    Article  Google Scholar 

  12. 12

    Deselaers T, Weyand T, Keysers D, Macherey W, Ney H (2005) FIRE in ImageCLEF 2005: combining content-based image retrieval with textual information retrieval. In: Working notes of the CLEF workshop. Vienna, Austria

  13. 13

    Donnelley M (2008) Computer aided long-bone segmentation and fracture detection. Ph.D. thesis, Flinders University of South Australia, Adelaide, South Australia

  14. 14

    Donnelley M, Knowles G (2005) Computer aided long bone fracture detection. In: Proceedings of the 8th international symposium on signal processing and its applications (ISSPA 2005), vol 1. Sydney, Australia, pp 175–178

  15. 15

    Donnelley M, Knowles G, Hearn T (2008) A cad system for long-bone segmentation and fracture detection. In: Proceedings of the 3rd international conference on image and signal processing (ICISP 2008). Lecture notes in computer science, vol 5099. Springer, pp 153–162

  16. 16

    Fox EA, Shaw JA (1993) Combination of multiple searches. In: Text retrieval conference, pp 243–252

  17. 17

    He JC, Leow WK, Howe TS (2007) Hierarchical classifiers for detection of fractures in X-ray images. In: Proceedings of the 12th international conference on computer analysis of images and patterns (CAIP 2007). Lecture notes in computer science, vol 4673. Springer, Vienna, Austria, pp 962–969

  18. 18

    Hsu W, Antani S, Long LR, Neve L, Thoma GR (2009) Spirs: a web-based image retrieval system for large biomedical databases. Int J Med Inform 78(Supplement 1):S13–S24. MedInfo 2007

    Google Scholar 

  19. 19

    Jiang YG, Ngo CW, Yang J (2007) Towards optimal bag-of-features for object categorization and semantic video retrieval. In: CIVR ’07: proceedings of the 6th ACM international conference on image and video retrieval. ACM, New York, NY, USA, pp 494–501

  20. 20

    Ke Y, Sukthankar R (2004) Pca-sift: a more distinctive representation for local image descriptors. In: IEEE computer society conference on computer vision and pattern recognition (CVPR 2004), vol 2. Washington, DC, USA, pp 506–513

  21. 21

    Lehmann TM, Güld MO, Thies C, Fischer B, Spitzer K, Keysers D, Ney H, Kohnen M, Schubert H, Wein BB (2004) Content-based image retrieval in medical applications. Methods Inf Med 43: 354–361

    PubMed  CAS  Google Scholar 

  22. 22

    Lim SE, Xing Y, Chen Y, Leow WK, Howe TS, Png MA (2004) Detection of femur and radius fractures in X-ray images. In: Proceedings of 2nd international conference on advances in medical signal and information processing. Sliema, Malta, pp 249–256

  23. 23

    Liu C, Yuen J, Torralba A, Sivic J, Freeman WT (2008) SIFT flow: dense correspondence across different scenes. In: ECCV ’08: proceedings of the 10th European conference on computer vision. Springer, Berlin, Heidelberg. pp 28–42

  24. 24

    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2): 91–110

    Article  Google Scholar 

  25. 25

    Lum VLF, Leow WK, Chen Y, Howe TS, Png MA (2005) Combining classifiers for bone fracture detection in X-ray images. In: IEEE international conference on image processing (ICIP’2005), vol 1. Genoa, Italy, pp 1149–1152

  26. 26

    Marsh JL, Slongo TF, Agel J, Broderick JS, Creevey W, DeCoster TA, Prokuski L, Sirkin MS, Ziran B, Henley B, Audigé L (2007) Fracture and dislocation classification compendium—2007: Orthopaedic trauma association classification, database and outcomes committee. J Orthop Trauma 21(10 Suppl): S1–S133

    PubMed  Article  CAS  Google Scholar 

  27. 27

    Matas J (2004) Robust wide-baseline stereo from maximally stable extremal regions. Image Vis Comput 22(10): 761–767

    Article  Google Scholar 

  28. 28

    Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Trans Pattern Anal Mach Intell 27(10): 1615–1630

    PubMed  Article  Google Scholar 

  29. 29

    Müller H, Deselaers T, Kim E, Kalpathy-Cramer J, Deserno TM, Clough P, Hersh W (2008) Overview of the ImageCLEFmed 2007 medical retrieval and annotation tasks. In: CLEF 2007 proceedings. Lecture notes in computer science (LNCS), vol 5152. Springer, Budapest, Hungary, pp 473–491

  30. 30

    Müller H, Deselaers T, Lehmann T, Clough P, Kim E, Hersh W (2007) Overview of the ImageCLEFmed 2006 medical retrieval and medical annotation tasks. In: CLEF 2006 proceedings. Lecture notes in computer science (LNCS), vol 4730. Springer, Alicante, Spain, pp 595–608

  31. 31

    Müller H, Fabry P, Lovis C, Geissbuhler A (2003) MedGIFT—retrieving medical image by their visual content. In: World summit of the information society, forum science and society. Geneva, Switzerland

  32. 32

    Müller H, Kalpathy-Cramer J, Kahn CE Jr, Hatt W, Bedrick S, Hersh W (2009) Overview of the ImageCLEFmed 2008 medical image retrieval task. In: Peters C, Giampiccolo D, Ferro N, Petras V, Gonzalo J, Peñas A, Deselaers T, Mandl T, Jones G, Kurimo M (eds) Evaluating systems for multilingual and multimodal information access—9th workshop of the cross-language evaluation forum. Lecture notes in computer science.,vol 5706. Aarhus, Denmark, pp 500–510

  33. 33

    Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medicine-clinical benefits and future directions. Int J Med Inform 73(1): 1–23

    PubMed  Article  Google Scholar 

  34. 34

    Muller H, Zhou X, Depeursinge A, Pitkanen M, Iavindrasana J, Geissbuhler A (2007) Medical visual information retrieval: state of the art and challenges ahead. In: Proceedings of the 2007 IEEE international conference on multimedia and expo, ICME’07. IEEE, pp 683–686

  35. 35

    Nehemiah HK, Khanna A, Kumar DS (2006) Intelligent fractured image retrieval from medical image databases. Asian J Inf Technol 5: 448–453

    Google Scholar 

  36. 36

    Quellec G, Lamard M, Cazuguel G, Roux C, Cochener B (2011) Case retrieval in medical databases by fusing heterogeneous information. IEEE Trans Med Imaging 30(1): 108–118

    PubMed  Article  Google Scholar 

  37. 37

    Rosset A, Müller H, Martins M, Dfouni N, Vallée JP, Ratib O (2004) Casimage project—a digital teaching files authoring environment. J Thorac Imaging 19(2): 1–6

    Article  Google Scholar 

  38. 38

    Savoy J (2002) Report on CLEF—2001 experiments. In: Report on the CLEF conference 2001 (Cross Language Evaluation Forum). LNCS 2406, Springer, Darmstadt, Germany, pp 27–43

  39. 39

    Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12): 1349–1380

    Article  Google Scholar 

  40. 40

    Squire DM, Müller W, Müller H, Pun T (2000) Content-based query of image databases: inspirations from text retrieval. In: Ersboll BK, Johansen P (eds) Pattern recognition letters (selected papers from the 11th scandinavian conference on image analysis SCIA ‘99), vol 21, pp 1193–1198

  41. 41

    Stern R, Hoffmeyer P, Rosset A, Garcia J (2003) Fractures. University of Geneva, Geneva, Switzerland

    Google Scholar 

  42. 42

    Swain MJ, Ballard DH (1991) Color indexing. Int J Comput Vis 7(1): 11–32

    Article  Google Scholar 

  43. 43

    Tommasi T, Orabona F, Caputo B (2008) CLEF2008 image annotation task: an SVM confidence-based approach. In: Working notes of the 2008 CLEF workshop. Aarhus, Denmark

  44. 44

    Weber B (1966) Die verletzungen des oberen sprungge-lenkes. Aktuelle Probleme in der Chirurgie. Huber, Stuttgart

    Google Scholar 

  45. 45

    Zhou X, Depeursinge A, Müller H (2010) Information fusion for combining visual and textual image retrieval. In: International conference on pattern recognition, ICPR’10. IEEE Computer Society, Los Alamitos, CA, USA

  46. 46

    Zhou X, Eggel I, Müller H (2009) The MedGIFT group at ImageCLEF 2009. In: Working notes of CLEF 2009 (Cross Language Evaluation Forum). Corfu, Greece

  47. 47

    Zhou X, Gobeill J, Müller H (2009) The MedGIFT group at ImageCLEF 2008. In: CLEF 2008 proceedings. Lecture notes in computer science (LNCS), vol 5706. Springer, Aarhus, Denmark, pp 712–718

  48. 48

    Zhou X, Gobeill J, Ruch P, Müller H (2008) University and hospitals of geneva participating at imageclef 2007. In: CLEF 2007 proceedings. Lecture notes in computer science (LNCS), vol 5152. Springer, Budapest, Hungary, pp 649–656

  49. 49

    Zhou X, Krabbenhöft H, Niinimäki M, Depeursinge A, Möller S, Müller H (2009) An easy setup for parallel medical image processing: using Taverna and ARC. In: Proceedings of HealthGrid 2009. Berlin, Germany

Download references

Author information



Corresponding author

Correspondence to Xin Zhou.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Zhou, X., Stern, R. & Müller, H. Case-based fracture image retrieval. Int J CARS 7, 401–411 (2012).

Download citation


  • Content-based image retrieval
  • Feature selection
  • Fracture database
  • Medical imaging
  • Decision support system