Case-based fracture image retrieval can assist surgeons in decisions regarding new cases by supplying visually similar past cases. This tool may guide fracture fixation and management through comparison of long-term outcomes in similar cases.
A fracture image database collected over 10 years at the orthopedic service of the University Hospitals of Geneva was used. This database contains 2,690 fracture cases associated with 43 classes (based on the AO/OTA classification). A case-based retrieval engine was developed and evaluated using retrieval precision as a performance metric. Only cases in the same class as the query case are considered as relevant. The scale-invariant feature transform (SIFT) is used for image analysis. Performance evaluation was computed in terms of mean average precision (MAP) and early precision (P10, P30). Retrieval results produced with the GNU image finding tool (GIFT) were used as a baseline.
Two sampling strategies were evaluated. One used a dense 40 × 40 pixel grid sampling, and the second one used the standard SIFT features. Based on dense pixel grid sampling, three unsupervised feature selection strategies were introduced to further improve retrieval performance. With dense pixel grid sampling, the image is divided into 1,600 (40 × 40) square blocks. The goal is to emphasize the salient regions (blocks) and ignore irrelevant regions. Regions are considered as important when a high variance of the visual features is found. The first strategy is to calculate the variance of all descriptors on the global database. The second strategy is to calculate the variance of all descriptors for each case. A third strategy is to perform a thumbnail image clustering in a first step and then to calculate the variance for each cluster. Finally, a fusion between a SIFT-based system and GIFT is performed.
A first comparison on the selection of sampling strategies using SIFT features shows that dense sampling using a pixel grid (MAP = 0.18) outperformed the SIFT detector-based sampling approach (MAP = 0.10). In a second step, three unsupervised feature selection strategies were evaluated. A grid parameter search is applied to optimize parameters for feature selection and clustering. Results show that using half of the regions (700 or 800) obtains the best performance for all three strategies. Increasing the number of clusters in clustering can also improve the retrieval performance. The SIFT descriptor variance in each case gave the best indication of saliency for the regions (MAP = 0.23), better than the other two strategies (MAP = 0.20 and 0.21). Combining GIFT (MAP = 0.23) and the best SIFT strategy (MAP = 0.23) produced significantly better results (MAP = 0.27) than each system alone.
A case-based fracture retrieval engine was developed and is available for online demonstration. SIFT is used to extract local features, and three feature selection strategies were introduced and evaluated. A baseline using the GIFT system was used to evaluate the salient point-based approaches. Without supervised learning, SIFT-based systems with optimized parameters slightly outperformed the GIFT system. A fusion of the two approaches shows that the information contained in the two approaches is complementary. Supervised learning on the feature space is foreseen as the next step of this study.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Abdullah A, Veltkamp RC, Wiering MA (2010) Ensembles of novel visual keywords descriptors for image categorization. In: 11th International conference on control automation robotics vision (ICARCV), pp 1206–1211
Avni U, Greenspan H, Konen E, Sharon M, Goldberger J (2011) X-ray categorization and retrieval on the organ and pathology level, using patch-based visual words. IEEE Trans Med Imaging 30(3): 733–746
Barnard K, Duygulu P, Forsyth D, de Freitas N, Blei DM, Jordan MI (2003) Matching words and pictures. J. Mach. Learn. Res. 3: 1107–1135
Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (surf). Comput Vis Image Underst 110(3): 346–359
Clough P, Müller H, Sanderson M (2005) The CLEF cross-language image retrieval track (ImageCLEF) 2004. In: Peters C, Clough P, Gonzalo J, Jones GJF, Kluck M, Magnini B (eds) Multilingual information access for text, speech and images: result of the fifth CLEF evaluation campaign. Lecture notes in computer science (LNCS), vol 3491. Springer, Bath, UK, pp 597–613
Clough P, Sanderson M, Müller H (2004) The CLEF cross language image retrieval track (ImageCLEF) 2004. In: The challenge of image and video retrieval (CIVR 2004), Lecture notes in computer science, vol 3115, Springer, pp 243–251
Croft WB (2000) Combining approaches to information retrieval. In: Advances in information retrieval. Springer, US, pp 1–36
Csurka G, Dance CR, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of keypoints. In: In workshop on statistical learning in computer vision, ECCV, pp 1–22
Danis R (1949) Théorie et Pratique de l’Ostéosynthèse. Masson and Cie, Paris, France
Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2): 1–60
Deselaers T, Keysers D, Ney H (2008) Features for image retrieval: an experimental comparison. Inf Retr 11: 77–107
Deselaers T, Weyand T, Keysers D, Macherey W, Ney H (2005) FIRE in ImageCLEF 2005: combining content-based image retrieval with textual information retrieval. In: Working notes of the CLEF workshop. Vienna, Austria
Donnelley M (2008) Computer aided long-bone segmentation and fracture detection. Ph.D. thesis, Flinders University of South Australia, Adelaide, South Australia
Donnelley M, Knowles G (2005) Computer aided long bone fracture detection. In: Proceedings of the 8th international symposium on signal processing and its applications (ISSPA 2005), vol 1. Sydney, Australia, pp 175–178
Donnelley M, Knowles G, Hearn T (2008) A cad system for long-bone segmentation and fracture detection. In: Proceedings of the 3rd international conference on image and signal processing (ICISP 2008). Lecture notes in computer science, vol 5099. Springer, pp 153–162
Fox EA, Shaw JA (1993) Combination of multiple searches. In: Text retrieval conference, pp 243–252
He JC, Leow WK, Howe TS (2007) Hierarchical classifiers for detection of fractures in X-ray images. In: Proceedings of the 12th international conference on computer analysis of images and patterns (CAIP 2007). Lecture notes in computer science, vol 4673. Springer, Vienna, Austria, pp 962–969
Hsu W, Antani S, Long LR, Neve L, Thoma GR (2009) Spirs: a web-based image retrieval system for large biomedical databases. Int J Med Inform 78(Supplement 1):S13–S24. MedInfo 2007
Jiang YG, Ngo CW, Yang J (2007) Towards optimal bag-of-features for object categorization and semantic video retrieval. In: CIVR ’07: proceedings of the 6th ACM international conference on image and video retrieval. ACM, New York, NY, USA, pp 494–501
Ke Y, Sukthankar R (2004) Pca-sift: a more distinctive representation for local image descriptors. In: IEEE computer society conference on computer vision and pattern recognition (CVPR 2004), vol 2. Washington, DC, USA, pp 506–513
Lehmann TM, Güld MO, Thies C, Fischer B, Spitzer K, Keysers D, Ney H, Kohnen M, Schubert H, Wein BB (2004) Content-based image retrieval in medical applications. Methods Inf Med 43: 354–361
Lim SE, Xing Y, Chen Y, Leow WK, Howe TS, Png MA (2004) Detection of femur and radius fractures in X-ray images. In: Proceedings of 2nd international conference on advances in medical signal and information processing. Sliema, Malta, pp 249–256
Liu C, Yuen J, Torralba A, Sivic J, Freeman WT (2008) SIFT flow: dense correspondence across different scenes. In: ECCV ’08: proceedings of the 10th European conference on computer vision. Springer, Berlin, Heidelberg. pp 28–42
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2): 91–110
Lum VLF, Leow WK, Chen Y, Howe TS, Png MA (2005) Combining classifiers for bone fracture detection in X-ray images. In: IEEE international conference on image processing (ICIP’2005), vol 1. Genoa, Italy, pp 1149–1152
Marsh JL, Slongo TF, Agel J, Broderick JS, Creevey W, DeCoster TA, Prokuski L, Sirkin MS, Ziran B, Henley B, Audigé L (2007) Fracture and dislocation classification compendium—2007: Orthopaedic trauma association classification, database and outcomes committee. J Orthop Trauma 21(10 Suppl): S1–S133
Matas J (2004) Robust wide-baseline stereo from maximally stable extremal regions. Image Vis Comput 22(10): 761–767
Mikolajczyk K, Schmid C (2005) A performance evaluation of local descriptors. IEEE Trans Pattern Anal Mach Intell 27(10): 1615–1630
Müller H, Deselaers T, Kim E, Kalpathy-Cramer J, Deserno TM, Clough P, Hersh W (2008) Overview of the ImageCLEFmed 2007 medical retrieval and annotation tasks. In: CLEF 2007 proceedings. Lecture notes in computer science (LNCS), vol 5152. Springer, Budapest, Hungary, pp 473–491
Müller H, Deselaers T, Lehmann T, Clough P, Kim E, Hersh W (2007) Overview of the ImageCLEFmed 2006 medical retrieval and medical annotation tasks. In: CLEF 2006 proceedings. Lecture notes in computer science (LNCS), vol 4730. Springer, Alicante, Spain, pp 595–608
Müller H, Fabry P, Lovis C, Geissbuhler A (2003) MedGIFT—retrieving medical image by their visual content. In: World summit of the information society, forum science and society. Geneva, Switzerland
Müller H, Kalpathy-Cramer J, Kahn CE Jr, Hatt W, Bedrick S, Hersh W (2009) Overview of the ImageCLEFmed 2008 medical image retrieval task. In: Peters C, Giampiccolo D, Ferro N, Petras V, Gonzalo J, Peñas A, Deselaers T, Mandl T, Jones G, Kurimo M (eds) Evaluating systems for multilingual and multimodal information access—9th workshop of the cross-language evaluation forum. Lecture notes in computer science.,vol 5706. Aarhus, Denmark, pp 500–510
Müller H, Michoux N, Bandon D, Geissbuhler A (2004) A review of content-based image retrieval systems in medicine-clinical benefits and future directions. Int J Med Inform 73(1): 1–23
Muller H, Zhou X, Depeursinge A, Pitkanen M, Iavindrasana J, Geissbuhler A (2007) Medical visual information retrieval: state of the art and challenges ahead. In: Proceedings of the 2007 IEEE international conference on multimedia and expo, ICME’07. IEEE, pp 683–686
Nehemiah HK, Khanna A, Kumar DS (2006) Intelligent fractured image retrieval from medical image databases. Asian J Inf Technol 5: 448–453
Quellec G, Lamard M, Cazuguel G, Roux C, Cochener B (2011) Case retrieval in medical databases by fusing heterogeneous information. IEEE Trans Med Imaging 30(1): 108–118
Rosset A, Müller H, Martins M, Dfouni N, Vallée JP, Ratib O (2004) Casimage project—a digital teaching files authoring environment. J Thorac Imaging 19(2): 1–6
Savoy J (2002) Report on CLEF—2001 experiments. In: Report on the CLEF conference 2001 (Cross Language Evaluation Forum). LNCS 2406, Springer, Darmstadt, Germany, pp 27–43
Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12): 1349–1380
Squire DM, Müller W, Müller H, Pun T (2000) Content-based query of image databases: inspirations from text retrieval. In: Ersboll BK, Johansen P (eds) Pattern recognition letters (selected papers from the 11th scandinavian conference on image analysis SCIA ‘99), vol 21, pp 1193–1198
Stern R, Hoffmeyer P, Rosset A, Garcia J (2003) Fractures. University of Geneva, Geneva, Switzerland
Swain MJ, Ballard DH (1991) Color indexing. Int J Comput Vis 7(1): 11–32
Tommasi T, Orabona F, Caputo B (2008) CLEF2008 image annotation task: an SVM confidence-based approach. In: Working notes of the 2008 CLEF workshop. Aarhus, Denmark
Weber B (1966) Die verletzungen des oberen sprungge-lenkes. Aktuelle Probleme in der Chirurgie. Huber, Stuttgart
Zhou X, Depeursinge A, Müller H (2010) Information fusion for combining visual and textual image retrieval. In: International conference on pattern recognition, ICPR’10. IEEE Computer Society, Los Alamitos, CA, USA
Zhou X, Eggel I, Müller H (2009) The MedGIFT group at ImageCLEF 2009. In: Working notes of CLEF 2009 (Cross Language Evaluation Forum). Corfu, Greece
Zhou X, Gobeill J, Müller H (2009) The MedGIFT group at ImageCLEF 2008. In: CLEF 2008 proceedings. Lecture notes in computer science (LNCS), vol 5706. Springer, Aarhus, Denmark, pp 712–718
Zhou X, Gobeill J, Ruch P, Müller H (2008) University and hospitals of geneva participating at imageclef 2007. In: CLEF 2007 proceedings. Lecture notes in computer science (LNCS), vol 5152. Springer, Budapest, Hungary, pp 649–656
Zhou X, Krabbenhöft H, Niinimäki M, Depeursinge A, Möller S, Müller H (2009) An easy setup for parallel medical image processing: using Taverna and ARC. In: Proceedings of HealthGrid 2009. Berlin, Germany
About this article
Cite this article
Zhou, X., Stern, R. & Müller, H. Case-based fracture image retrieval. Int J CARS 7, 401–411 (2012). https://doi.org/10.1007/s11548-011-0643-8
- Content-based image retrieval
- Feature selection
- Fracture database
- Medical imaging
- Decision support system