General Overview of ImageCLEF at the CLEF 2016 Labs

  • Mauricio Villegas
  • Henning Müller
  • Alba García Seco de Herrera
  • Roger Schaer
  • Stefano Bromuri
  • Andrew Gilbert
  • Luca Piras
  • Josiah Wang
  • Fei Yan
  • Arnau Ramisa
  • Emmanuel Dellandrea
  • Robert Gaizauskas
  • Krystian Mikolajczyk
  • Joan Puigcerver
  • Alejandro H. Toselli
  • Joan-Andreu Sánchez
  • Enrique Vidal
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9822)

Abstract

This paper presents an overview of the ImageCLEF 2016 evaluation campaign, an event that was organized as part of the CLEF (Conference and Labs of the Evaluation Forum) labs 2016. ImageCLEF is an ongoing initiative that promotes the evaluation of technologies for annotation, indexing and retrieval for providing information access to collections of images in various usage scenarios and domains. In 2016, the 14th edition of ImageCLEF, three main tasks were proposed: (1) identification, multi-label classification and separation of compound figures from biomedical literature; (2) automatic annotation of general web images; and (3) retrieval from collections of scanned handwritten documents. The handwritten retrieval task was the only completely novel task this year, although the other two tasks introduced several modifications to keep the proposed tasks challenging.

References

  1. 1.
    Aldavert, D., Rusinol, M., Toledo, R., Llados, J.: Integrating visual and textual cues for query-by-string word spotting. In: 2013 12th International Conference on Document Analysis and Recognition (ICDAR), pp. 511–515, August 2013Google Scholar
  2. 2.
    Balikas, G., Kosmopoulos, A., Krithara, A., Paliouras, G., Kakadiaris, I.A.: Results of the bioasq tasks of the question answering lab at CLEF 2015. In: Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum, Toulouse, France, 8–11 September 2015 (2015)Google Scholar
  3. 3.
    Causer, T., Wallace, V.: Building a volunteer community: results and findings from transcribe Bentham. Digit. Humanit. Q. 6(2) (2012). http://www.digitalhumanities.org/dhq/vol/6/2/000125/000125.html
  4. 4.
    Denkowski, M., Lavie, A.: Meteor universal: language specific translation evaluation for any target language. In: Proceedings of the EACL 2014 Workshop on Statistical Machine Translation (2014)Google Scholar
  5. 5.
    Fischer, A., Keller, A., Frinken, V., Bunke, H.: Lexicon-free handwritten word spotting using character HMMs. Pattern Recognit. Lett. 33(7), 934–942 (2012). Special Issue on Awards from (ICPR)CrossRefGoogle Scholar
  6. 6.
    Frinken, V., Fischer, A., Bunke, H.: A novel word spotting algorithm using bidirectional long short-term memory neural networks. In: Schwenker, F., El Gayar, N. (eds.) Artificial Neural Networks in Pattern Recognition. LNCS, vol. 5998, pp. 185–196. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    Gatos, B., Pratikakis, I.: Segmentation-free word spotting in historical printed documents. In: 10th International Conference on Document Analysis and Recognition (ICDAR 2009), pp. 271–275, July 2009Google Scholar
  8. 8.
    Gilbert, A., Piras, L., Wang, J., Yan, F., Dellandrea, E., Gaizauskas, R., Villegas, M., Mikolajczyk, K.: Overview of the ImageCLEF 2015 scalable image annotation, localization and sentence generation task. In: CLEF2015 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Toulouse, France, 8–11 September 2015 (2015)Google Scholar
  9. 9.
    Gilbert, A., Piras, L., Wang, J., Yan, F., Ramisa, A., Dellandrea, E., Gaizauskas, R., Villegas, M., Mikolajczyk, K.: Overview of the ImageCLEF 2016 scalable concept image annotation task. In: CLEF2016 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Évora, Portugal, 5–8 September 2016 (2016)Google Scholar
  10. 10.
    Giotis, A., Gerogiannis, D., Nikou, C.: Word spotting in handwritten text using contour-based models. In: 2014 14th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 399–404, September 2014Google Scholar
  11. 11.
    Goeuriot, L., Kelly, L., Suominen, H., Hanlen, L., Névéol, A., Grouin, C., Palotti, J.R.M., Zuccon, G.: Overview of the CLEF ehealth evaluation lab 2015. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction - 6th International Conference of the CLEF Association, CLEF 2015, Toulouse, France, 8–11 September 2015, Proceedings, pp. 429–443 (2015). doi:10.1007/978-3-319-24027-5_44
  12. 12.
    He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: The IEEE International Conference on Computer Vision (ICCV), December 2015Google Scholar
  13. 13.
    García Seco de Herrera, A., Kalpathy-Cramer, J., Demner Fushman, D., Antani, S., Müller, H.: Overview of the ImageCLEF 2013 medical tasks. In: Working Notes of CLEF 2013 (Cross Language Evaluation Forum) (2013). http://ceur-ws.org/Vol-1179/CLEF2013wn-ImageCLEF-SecoDeHerreraEt2013b.pdf
  14. 14.
    García Seco de Herrera, A., Müller, H., Bromuri, S.: Overview of the ImageCLEF 2015 medical classification task. In: Working Notes of CLEF 2015 (Cross Language Evaluation Forum), CEUR Workshop Proceedings, CEUR-WS.org, September 2015Google Scholar
  15. 15.
    García Seco de Herrera, A., Schaer, R., Bromuri, S., Müller, H.: Overview of the ImageCLEF 2016 medical task. In: CLEF2016 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Évora, Portugal, 5–8 September 2016 (2016)Google Scholar
  16. 16.
    Joly, A., Goëau, H., Glotin, H., Spampinato, C., Bonnet, P., Vellinga, W., Planquè, R., Rauber, A., Palazzo, S., Fisher, B., Müller, H.: LifeCLEF 2015: multimedia life species identification challenges. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction - 6th International Conference of the CLEF Association, CLEF 2015, Toulouse, France, 8–11 September 2015, Proceedings, pp. 462–483 (2015). doi:10.1007/978-3-319-24027-5_46
  17. 17.
    Kalpathy-Cramer, J., García Seco de Herrera, A., Demner-Fushman, D., Antani, S., Bedrick, S., Müller, H.: Evaluating performance of biomedical image retrieval systems an overview of the medical image retrieval task at ImageCLEF 2004–2014. Comput. Med. Imaging Graph. 39, 55–61 (2015). doi:10.1016/j.compmedimag.2014.03.004 CrossRefGoogle Scholar
  18. 18.
    Koitka, S., Friedrich, C.M.: Traditional feature engineering and deep learning approaches at medical classification task of ImageCLEF 2016. In: CLEF2016 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Évora, Portugal, 5–8 September 2016 (2016)Google Scholar
  19. 19.
    Lavrenko, V., Rath, T.M., Manmatha, R.: Holistic word recognition for handwritten historical documents. In: First International Workshop on Document Image Analysis for Libraries, Proceedings, pp. 278–287 (2004)Google Scholar
  20. 20.
    Müller, H., Clough, P., Deselaers, T., Caputo, B. (eds.): ImageCLEF: Experimental Evaluation in Visual Information Retrieval. The Information Retrieval Series, vol. 32. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15181-1 MATHGoogle Scholar
  21. 21.
    Müller, H., Kalpathy-Cramer, J., Demner-Fushman, D., Antani, S.: Creating a classification of image types in the medical literature for visual categorization. In: SPIE Medical Imaging (2012)Google Scholar
  22. 22.
    Pratikakis, I., Zagoris, K., Gatos, B., Louloudis, G., Stamatopoulos, N.: ICFHR 2014 competition on handwritten keyword spotting (H-KWS 2014). In: 2014 14th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 814–819, September 2014Google Scholar
  23. 23.
    Puigcerver, J., Toselli, A.H., Vidal, E.: ICDAR 2015 competition on keyword spotting for handwritten documents. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 1176–1180, August 2015Google Scholar
  24. 24.
    Ramisa, A., Yan, F., Moreno-Noguer, F., Mikolajczyk, K.: Breakingnews: article annotation by image and text processing. CoRR abs/1603.07141 (2016). http://arxiv.org/abs/1603.07141
  25. 25.
    Rodríguez-Serrano, J.A., Perronnin, F.: Handwritten word-spotting using hidden Markov models and universal vocabularies. Pattern Recognit. 42, 2106–2116 (2009)CrossRefMATHGoogle Scholar
  26. 26.
    Sorensen, S., Li, P., Kolagunda, A., Jiang, X., Wang, X., Shatkay, H., Kambhamettu, C.: UDEL CIS working notes in ImageCLEF 2016. In: CLEF2016 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Évora, Portugal, 5–8 September 2016 (2016)Google Scholar
  27. 27.
    Snchez, J.A., Romero, V., Toselli, A.H., Vidal, E.: ICFHR 2014 competition on handwritten text recognition on transcriptorium datasets (HTRTS). In: 2014 14th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 785–790, September 2014Google Scholar
  28. 28.
    Sánchez, J.A., Toselli, A.H., Romero, V., Vidal, E.: ICDAR 2015 competition HTRtS: Handwritten text recognition on the transcriptorium dataset. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 1166–1170, August 2015Google Scholar
  29. 29.
    Toselli, A.H., Puigcerver, J., Vidal, E.: Context-aware lattice based filler approach for key word spotting in handwritten documents. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 736–740, August 2015Google Scholar
  30. 30.
    Tsikrika, T., de Herrera, A.G.S., Müller, H.: Assessing the scholarly impact of ImageCLEF. In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., Rijke, M. (eds.) CLEF 2011. LNCS, vol. 6941, pp. 95–106. Springer, Heidelberg (2011). doi:10.1007/978-3-642-23708-9_12 CrossRefGoogle Scholar
  31. 31.
    Villegas, M., et al.: General overview of ImageCLEF at the CLEF 2015 labs. In: Mothe, J., et al. (eds.) CLEF 2015. LNCS, vol. 9283, pp. 441–461. Springer, Heidelberg (2015). doi:10.1007/978-3-319-24027-5_45 CrossRefGoogle Scholar
  32. 32.
    Villegas, M., Paredes, R.: Overview of the ImageCLEF 2012 scalable web image annotation task. In: Forner, P., Karlgren, J., Womser-Hacker, C. (eds.) CLEF 2012 Evaluation Labs and Workshop, Online Working Notes, Rome, Italy, 17–20 September 2012 (2012). http://ceur-ws.org/Vol-1178/CLEF2012wn-ImageCLEF-VillegasEt2012.pdf
  33. 33.
    Villegas, M., Paredes, R.: Overview of the ImageCLEF 2014 scalable concept image annotation task. In: CLEF2014 Working Notes, CEUR Workshop Proceedings, vol. 1180, pp. 308–328, CEUR-WS.org, Sheffield, UK, 15–18 September 2014 (2014). http://ceur-ws.org/Vol-1180/CLEF2014wn-Image-VillegasEt2014.pdf
  34. 34.
    Villegas, M., Paredes, R., Thomee, B.: Overview of the ImageCLEF 2013 scalable concept image annotation subtask. In: CLEF 2013 Evaluation Labs and Workshop, Online Working Notes, Valencia, Spain, 23–26 September 2013 (2013). http://ceur-ws.org/Vol-1179/CLEF2013wn-ImageCLEF-VillegasEt2013.pdf
  35. 35.
    Villegas, M., Puigcerver, J., Toselli, A.H.: ImageCLEF 2016 Bentham handwritten retrieval dataset (2016). doi:10.5281/zenodo.52994
  36. 36.
    Villegas, M., Puigcerver, J., Toselli, A.H., Sánchez, J.A., Vidal, E.: Overview of the ImageCLEF 2016 handwritten scanned document retrieval task. In: CLEF2016 Working Notes, CEUR Workshop Proceedings, CEUR-WS.org, Evora, Portugal, 5–8 September 2016 (2016)Google Scholar
  37. 37.
    Wang, J., Gaizauskas, R.: Generating image descriptions with gold standard visual inputs: motivation, evaluation and baselines. In: Proceedings of the 15th European Workshop on Natural Language Generation (ENLG), pp. 117–126. Association for Computational Linguistics, Brighton, UK (2015). http://www.aclweb.org/anthology/W15-4722
  38. 38.
    Zagoris, K., Ergina, K., Papamarkos, N.: Image retrieval systems based on compact shape descriptor and relevance feedback information. J. Vis. Commun. Image Represent. 22(5), 378–390 (2011)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Mauricio Villegas
    • 1
  • Henning Müller
    • 2
  • Alba García Seco de Herrera
    • 3
  • Roger Schaer
    • 2
  • Stefano Bromuri
    • 4
  • Andrew Gilbert
    • 5
  • Luca Piras
    • 6
  • Josiah Wang
    • 7
  • Fei Yan
    • 5
  • Arnau Ramisa
    • 8
  • Emmanuel Dellandrea
    • 9
  • Robert Gaizauskas
    • 7
  • Krystian Mikolajczyk
    • 10
  • Joan Puigcerver
    • 1
  • Alejandro H. Toselli
    • 1
  • Joan-Andreu Sánchez
    • 1
  • Enrique Vidal
    • 1
  1. 1.Universitat Politècnica de ValènciaValenciaSpain
  2. 2.University of Applied Sciences Western Switzerland (HES-SO)SierreSwitzerland
  3. 3.National Library of MedicineBethesdaUSA
  4. 4.Open University of the NetherlandsHeerlenThe Netherlands
  5. 5.University of SurreyGuildfordUK
  6. 6.University of CagliariCagliariItaly
  7. 7.University of SheffieldSheffieldUK
  8. 8.Institut de Robòtica i Informàtica Industrial (UPC-CSIC)BarcelonaSpain
  9. 9.École Centrale de LyonÉcullyFrance
  10. 10.Imperial College LondonLondonUK

Personalised recommendations