Challenges and Benchmarks in Bioimage Analysis

  • Michal Kozubek
Part of the Advances in Anatomy, Embryology and Cell Biology book series (ADVSANAT, volume 219)


Similar to the medical imaging community, the bioimaging community has recently realized the need to benchmark various image analysis methods to compare their performance and assess their suitability for specific applications. Challenges sponsored by prestigious conferences have proven to be an effective means of encouraging benchmarking and new algorithm development for a particular type of image data. Bioimage analysis challenges have recently complemented medical image analysis challenges, especially in the case of the International Symposium on Biomedical Imaging (ISBI). This review summarizes recent progress in this respect and describes the general process of designing a bioimage analysis benchmark or challenge, including the proper selection of datasets and evaluation metrics. It also presents examples of specific target applications and biological research tasks that have benefited from these challenges with respect to the performance of automatic image analysis methods that are crucial for the given task. Finally, available benchmarks and challenges in terms of common features, possible classification and implications drawn from the results are analysed.


Ground Truth Benchmark Dataset Target Application Dice Coefficient Cell Segmentation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The author would like to thank Martin Maška and David Svoboda for their feedback and useful comments. This work was supported by the Czech Science Foundation (Grant No. 302/12/G157).


  1. 2DSNEMI Challenge (2012) MIT. Accessed 17 May 2015
  2. 3DSNEMI Challenge (2013) MIT. Accessed 17 May 2015
  3. AMIDA (2013) University Medical Center Utrecht. Accessed 17 May 2015
  4. BBBC (2008) Broad Institute of Harvard and MIT. Accessed 17 May 2015
  5. BigNeuron (2015) Allen Institute for Brain Science. Accessed 17 May 2015
  6. Buck TE, Li J, Rohde GK, Murphy RF (2012) Toward the virtual cell: automated approaches to building models of subcellular organization “learned” from microscopy images. Bioessays 34:791–799CrossRefPubMedPubMedCentralGoogle Scholar
  7. Carpenter A, Kamentsky L, Eliceiri KW (2012) A call for bioimaging software usability. Nat Methods 9(7):666–670CrossRefPubMedPubMedCentralGoogle Scholar
  8. CCDB (2002) University of California, San Diego. Accessed 17 May 2015
  9. CellOrganizer (2012) Carnegie Mellon University, Pittsburgh. Accessed 17 May 2015
  10. Chenouard N et al (2014) Objective comparison of particle tracking methods. Nat Methods 11(3):281–289CrossRefPubMedPubMedCentralGoogle Scholar
  11. Coelho LP, Shariff A, Murphy RF (2009) Nuclear segmentation in microsope cell images: a hand-segmented dataset and comparison of algorithms. In: Proceedings of the 2009 IEEE International Symposium on Biomedical Imaging (ISBI 2009), IEEE, Boston, 28 June–1 July 2009, pp 518–521Google Scholar
  12. Computer Vision Homepage (1994) School of Computer Science, Carnegie Mellon University. Accessed 17 May 2015
  13. CIL (2010) American Society for Cell Biology. Accessed 17 May 2015
  14. CTC (2013) University of Navarra. Accessed 17 May 2015
  15. CytoPacq (2008) Masaryk University, Brno. Accessed 17 May 2015
  16. Deconvolution Benchmark Datasets (2010) EPFL. Accessed 17 May 2015
  17. DIADEM Challenge (2010) Howard Hughes Medical Institute. Accessed 17 May 2015
  18. DMC (2013) EPFL. Accessed 17 May 2015
  19. Drelie Gelasca E, Obara B, Fedorov D, Kvilekval K, Manjunath BS (2009) A biosegmentation benchmark for evaluation of bioimage analysis methods. BMC Bioinformatics 10:368CrossRefPubMedPubMedCentralGoogle Scholar
  20. Ferko MC, Patterson BW, Butler PJ (2006) High-resolution solid modeling of biological samples imaged with 3D fluorescence microscopy. Microsc Res Tech 69(8):648–655CrossRefPubMedPubMedCentralGoogle Scholar
  21. Foggia P, Percannella G, Soda P, Vento M (2013) Benchmarking HEp-2 cells classification methods. IEEE Trans Med Imaging 32(10):1878–1889CrossRefPubMedGoogle Scholar
  22. Gillette TA, Brown KM, Ascoli GA (2011) The DIADEM metric: comparing multiple reconstructions of the same neuron. Neuroinformatics 9(2-3):233–245CrossRefPubMedPubMedCentralGoogle Scholar
  23. GLAS (2015) Department of Computer Science, University of Warwick. Accessed 17 May 2015
  24. Griffa A, Garin N, Sage D (2010) Comparison of deconvolution software in 3D microscopy: a user point of view, part I and part II. G.I.T. Imaging Microscopy 1:43–45Google Scholar
  25. Gurcan MN, Madabhushi A, Rajpoot N (2010) Pattern recognition in histopathological images: an ICPR 2010 contest. In: Ünay D, Çataltepe Z, Aksoy S (eds) Recognizing patterns in signals, speech, images and videos. Lecture notes in computer science, vol 6388. Springer, Heidelberg, pp 226–234Google Scholar
  26. HEp-2 Images Dataset (2012) Mivia Lab, University of Salerno. Accessed 17 May 2015
  27. Hill E (2008) Announcing the JCB DataViewer, a browser-based application for viewing original image files. J Cell Biol 183:969–970CrossRefPubMedCentralGoogle Scholar
  28. JCB DataViewer (2008) Rockefeller University Press. Accessed 17 May 2015
  29. Jannin P, Grova C, Maurer C (2006) Model for defining and reporting reference-based validation protocols in medical image processing. Int J CARS 1(2):63–73CrossRefGoogle Scholar
  30. Kozubek M, Matula P (2000) An efficient algorithm for measurement and correction of chromatic aberrations in fluorescence microscopy. J Microsc 200(3):206–217CrossRefPubMedGoogle Scholar
  31. Kwan RK-S, Evans AC, Pike GB (1999) MRI simulation-based evaluation of image-processing and classification methods. IEEE Trans Med Imaging 18(11):1085–1097CrossRefPubMedGoogle Scholar
  32. Lehmussola A, Selinummi J, Ruusuvuori P, Niemist A, Yli-Harja O (2005) Simulating fluorescent microscope images of cell populations. In: Proceedings of the 27th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC’05), IEEE, Shanghai, 17–18 Jan 2006, pp 3153–3156Google Scholar
  33. Ljosa V, Sokolnicki KL, Carpenter AE (2012) Annotated high-throughput microscopy image sets for validation. Nat Methods 9(7):637CrossRefPubMedPubMedCentralGoogle Scholar
  34. LMC (2013) EPFL. Accessed 17 May 2015
  35. Lockett SJ, Sudar D, Thompson CT, Pinkel D, Gray JW (1998) Efficient, interactive, and three-dimensional segmentation of cell nuclei in thick tissue sections. Cytometry A 31:275–286CrossRefGoogle Scholar
  36. Malm P, Brun A, Bengtsson E (2015) Simulation of bright-field microscopy images depicting Pap-Smear specimen. Cytometry A 87A:212–226CrossRefGoogle Scholar
  37. Martone ME, Zhang S, Gupta A, Qian X, He H, Price DL, Wong M, Santini S, Ellisman MH (2003) The cell-centered database: a database for multiscale structural and protein localization data from light and electron microscopy. Neuroinformatics 1(4):379–395CrossRefPubMedGoogle Scholar
  38. Martone ME, Tran J, Wong WW, Sargis J, Fong L, Larson S, Lamont SP, Gupta A, Ellisman MH (2008) The Cell Centered Database project: an update on building community resources for managing and sharing 3D imaging data. J Struct Biol 161(3):220–231CrossRefPubMedPubMedCentralGoogle Scholar
  39. Maška M et al (2014) A benchmark for comparison of cell tracking algorithms. Bioinformatics 30(11):1609–1617CrossRefPubMedPubMedCentralGoogle Scholar
  40. McNally JG, Cogswell CJ, Fekete PW, Conchello JA (1997) Comparison of 3D microscopy methods by imaging a well characterized test object. In: Cogswell CJ, Conchello JA, Wilson T (eds) Three-dimensional microscopy: image acquisition and processing IV, San Jose, 8 Feb 1997. Proc SPIE, vol 2984, pp 52–63Google Scholar
  41. MITOS-ATYPIA (2014) Consortium for Open Medical Image Computing. Accessed 17 May 2015
  42. Murphy Lab Data (1999) Carnegie Mellon University. Accessed 17 May 2015
  43. OCCISC (2014) University of Adelaide. Accessed 17 May 2015
  44. Open Bio Image Alliance (2015) Accessed 17 May 2015
  45. Orloff DN, Iwasa JH, Martone ME, Ellisman MH, Kane CM (2012) The cell: an image library-CCDB: a curated repository of microscopy data. Nucleic Acids Res 41:D1241–D1250CrossRefPubMedPubMedCentralGoogle Scholar
  46. Ortiz-de-Solórzano C, Muñoz-Barrutia A, Meijering E, Kozubek M (2015) Toward a morphodynamic model of the cell: signal processing for cell modeling. IEEE Signal Proc Mag 32(1):20–29CrossRefGoogle Scholar
  47. Peng H, Hawrylycz M, Roskams J, Hill S, Spruston N, Meijering E, Ascoli GA (2015) BigNeuron: large-scale 3D neuron reconstruction from optical microscopy images. Neuron 87(2):252–256CrossRefPubMedGoogle Scholar
  48. Price K (1986) Anything you can do, I can do better (no you can’t). Comput Vision Graph 36:387–391CrossRefGoogle Scholar
  49. Rajaram S, Pavie B, Hac NE, Altschuler SJ, Wu LF (2012) SimuCell: a flexible framework for creating synthetic microscopy images. Nat Methods 9(7):634–635CrossRefPubMedGoogle Scholar
  50. Rosenberg C (1996) The Lenna Story. Accessed 17 May 2015
  51. Roux L, Racoceanu D, Loménie N, Kulikova M, Irshad H, Klossa J, Capron F, Genestie C, Le Naour G, Gurcan MN (2013) Mitosis detection in breast cancer histological images: an ICPR 2012 contest. J Pathol Inform 4:8CrossRefPubMedPubMedCentralGoogle Scholar
  52. Sage D, Kirshner H, Pengo T, Stuurman N, Min J, Manley S, Unser M (2015) Quantitative evaluation of software packages for single-molecule localization microscopy. Nat Methods 12(8):717–724CrossRefPubMedGoogle Scholar
  53. Sage D, Kirshner H, Vonesch C, Lefkimmiatis S, Unser M (2013) Benchmarking image-processing algorithms for biomicroscopy: reference datasets and perspectives. In: Proceedings of the 21st European Signal Processing Conference (EUSIPCO), IEEE, Marrakech, 9–13 Sept 2013, pp 1–4Google Scholar
  54. Shiraishi J, Katsuragawa S, Ikezoe J, Matsumoto T, Kobayashi T, Komatsu K, Matsui M, Fujita H, Kodera Y, Doi K (2000) Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules. AJR Am J Roentgenol 174(1):71–74CrossRefPubMedGoogle Scholar
  55. SimuCell (2012) Altschuler & Wu laboratories, University of California, San Francisco. Accessed 17 May 2015
  56. Svoboda D, Kozubek M, Stejskal S (2009) Generation of digital phantoms of cell nuclei and simulation of image formation in 3D image cytometry. Cytometry A 75A:494–509CrossRefGoogle Scholar
  57. UCSB Biosegmentation Benchmark (2008) University of California, Santa Barbara. Accessed 17 May 2015
  58. van Ginneken B, Kerkstra S (2015) Grand challenges in biomedical image analysis. Accessed 17 May 2015
  59. Veta M et al (2015) Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med Image Anal 20(1):237–248CrossRefPubMedGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Faculty of Informatics, Centre for Biomedical Image AnalysisMasaryk UniversityBrnoCzech Republic

Personalised recommendations