Multimedia Tools and Applications

, Volume 77, Issue 1, pp 519–548 | Cite as

Automatic separation of compound figures in scientific articles

  • Mario Taschwer
  • Oge MarquesEmail author


Content-based analysis and retrieval of digital images found in scientific articles is often hindered by images consisting of multiple subfigures (compound figures). We address this problem by proposing a method (ComFig) to automatically classify and separate compound figures, which consists of two main steps: (i) a supervised compound figure classifier (ComFig classifier) discriminates between compound and non-compound figures using task-specific image features; and (ii) an image processing algorithm is applied to predicted compound images to perform compound figure separation (ComFig separation). The proposed ComFig classifier is shown to achieve state-of-the-art classification performance on a published dataset. Our ComFig separation algorithm shows superior separation accuracy on two different datasets compared to other known automatic approaches. Finally, we propose a method to evaluate the effectiveness of the ComFig chain combining classifier and separation algorithm, and use it to optimize the misclassification loss of the ComFig classifier for maximal effectiveness in the chain.


Multipanel figure separation Document image understanding 



We thank Sameer Antani (NLM) and the authors of [1] for providing their compound figure separation dataset for evaluation purposes. We are grateful to Laszlo Böszörmenyi (ITEC, AAU) for valuable discussions and comments on this work.


  1. 1.
    Apostolova E, You D, Xue Z, Antani S, Demner-Fushman D, Thoma GR (2013) Image retrieval from scientific publications: Text and image content processing to separate multipanel figures. J Assoc Inf Sci Technol 64(5):893–908. doi: 10.1002/asi.22810 CrossRefGoogle Scholar
  2. 2.
    Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):1–127. doi: 10.1561/2200000006 CrossRefzbMATHGoogle Scholar
  3. 3.
    Bengio Y, Courville A, Vincent P (2013) Representation learning: A review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828. doi: 10.1109/TPAMI.2013.50 CrossRefGoogle Scholar
  4. 4.
    Bishop CM (2006) Pattern Recognition and Machine Learning, chap. 1.5 (Decision Theory). Springer, Secaucus, NJ, USA, pp 38–47Google Scholar
  5. 5.
    Caruana R, Niculescu-Mizil A (2006) An empirical comparison of supervised learning algorithms. In: Proceedings of the 23rd International Conference on Machine Learning, ICML ’06, pp. 161–168. ACM, New York, NY, USA . doi: 10.1145/1143844.1143865
  6. 6.
    Chatzichristofis SA, Boutalis YS (2008) CEDD: Color and edge directivity descriptor: A compact descriptor for image indexing and retrieval. In: Computer Vision Systems, LNCS. doi: 10.1007/978-3-540-79547-6_30, vol 5008. Springer, pp 312–322
  7. 7.
    Chen N, Blostein D (2007) A survey of document image classification: Problem statement, classifier architecture and performance evaluation. Int J Doc Anal Recognit 10(1):1–16. doi: 10.1007/s10032-006-0020-2 CrossRefGoogle Scholar
  8. 8.
    Chhatkuli A, Foncubierta-Rodríguez A, Markonis D, Meriaudeau F, Müller H (2013) Separating compound figures in journal articles to allow for subfigure classification. Proc SPIE 8674:86,740J–12. doi: 10.1117/12.2007897 CrossRefGoogle Scholar
  9. 9.
    Hastie T, Tibshirani R, Friedman J (2009) The Elements of Statistical Learning, 2nd edn., chap. 7 (Model Assessment and Selection). Springer, New York, pp 219–260Google Scholar
  10. 10.
    García Seco de Herrera A, Kalpathy-Cramer J, Demner-Fushman D, Antani S, Müller H (2013) Overview of the ImageCLEF 2013 medical tasks. In: CLEF 2013 Working Notes, CEUR Proc., vol. 1179.
  11. 11.
    García Seco de Herrera A, Müller H, Bromuri S (2015) Overview of the ImageCLEF 2015 medical classification task. In: CLEF 2015 Working Notes, CEUR Proc., vol. 1391.
  12. 12.
    Kalpathy-Cramer J, de Herrera AGS, Demner-Fushman D, Antani S, Bedrick S, Müller H (2015) Evaluating performance of biomedical image retrieval systems—an overview of the medical image retrieval task at ImageCLEF 2004–2013. Comput Med Imaging Graph 39(0):55–61. doi: 10.1016/j.compmedimag.2014.03.004. Medical visual information analysis and retrieval
  13. 13.
    Kitanovski I, Dimitrovski I, Loskovska S (2013) FCSE at medical tasks of ImageCLEF 2013. In: CLEF 2013 Working Notes, CEUR Proc., vol. 1179.
  14. 14.
    Kou G, Lu Y, Peng Y, Shi Y (2012) Evaluation of classification algorithms using MCDM and rank correlation. Int J Inf Technol Decis Mak 11(01):197–225. doi: 10.1142/S0219622012500095 CrossRefGoogle Scholar
  15. 15.
    Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Pereira F, Burges C, Bottou L, Weinberger K (eds) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc.Google Scholar
  16. 16.
    Mitchell TM (1997) Machine Learning, chap. 5 (Evaluating Hypotheses). McGraw-Hill, New York, pp 128–153Google Scholar
  17. 17.
    Murphy RF, Velliste M, Yao J, Porreca G (2001) Searching online journals for fluorescence microscope images depicting protein subcellular location patterns. In: Proceedings of the 2nd IEEE International Symposium on Bioinformatics and Bioengineering, BIBE ’01, pp. 119–128. IEEE Computer Society, Washington, DC, USAGoogle Scholar
  18. 18.
    Pelka O, Friedrich CM (2015) FHDO Biomedical Computer Science Group at medical classification task of ImageCLEF 2015. In: CLEF 2015 Working Notes, CEUR Workshop Proceedings, ISSN 1613-0073., vol 1391
  19. 19.
    Qian Y, Murphy RF (2008) Improved recognition of figures containing fluorescence microscope images in online journal articles using graphical models. Bioinformatics 24 (4):569–576. doi: 10.1093/bioinformatics/btm561 CrossRefGoogle Scholar
  20. 20.
    Santosh K, Xue Z, Antani S, Thoma G (2015) NLM at ImageCLEF 2015: Biomedical multipanel figure separation. In: CLEF 2015 Working Notes, CEUR Proc., vol. 1391.
  21. 21.
    Shatkay H, Chen N, Blostein D (2006) Integrating image data into biomedical text categorization. Bioinformatics 22(14):e446–e453. doi: 10.1093/bioinformatics/btl235 CrossRefGoogle Scholar
  22. 22.
    You D, Rahman MM, Xue Z, Demner-Fushman D, Antani S, Thoma G (2015) Literature-based biomedical image classification and retrieval. Comput Med Imag Graph 39:3–13. doi: 10.1016/j.compmedimag.2014.06.006 CrossRefGoogle Scholar
  23. 23.
    Sivic J, Zisserman A (2003) Video Google: A text retrieval approach to object matching in videos. In: Proceedings of the Ninth IEEE international conference on, Computer vision, 2003. IEEE, pp 1470–1477Google Scholar
  24. 24.
    Smith-Miles KA (2009) Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput Surv 41(1):6:1–6:25. doi: 10.1145/1456650.1456656 Google Scholar
  25. 25.
    Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958MathSciNetzbMATHGoogle Scholar
  26. 26.
    Taschwer M, Marques O (2015) AAUITEC at ImageCLEF 2015: Compound figure separation. In: CLEF 2015 Working Notes, CEUR Proc., vol. 1391.
  27. 27.
    Taschwer M, Marques O (2016) Compound figure separation combining edge and band separator detection. In: Tian Q, Sebe N, Qi GJ, Huet B, Hong R, Liu X (eds) MultiMedia Modeling, Lecture Notes in Computer Science. doi: 10.1007/978-3-319-27671-7_14, vol 9516. Springer International Publishing, pp 162–173
  28. 28.
    Wang X, Jiang X, Kolagunda A, Shatkay H, Kambhamettu C (2015) CIS UDEL working notes on ImageCLEF 2015: Compound figure detection task. In: CLEF 2015 Working Notes, CEUR Workshop Proceedings, ISSN 1613-0073., vol 1391
  29. 29.
    Yuan X, Ang D (2014) A novel figure panel classification and extraction method for document image understanding. Int J Data Min Bioinform 9(1):22–36. doi: 10.1504/IJDMB.2014.057779 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.ITEC, Klagenfurt University (AAU), UniversitaetsstrasseKlagenfurtAustria
  2. 2.Florida Atlantic University (FAU)Boca RatonUSA

Personalised recommendations