Skip to main content
SpringerLink
Log in

A Benchmark of PDF Information Extraction Tools Using a Multi-task and Multi-domain Evaluation Framework for Academic Documents

  • Conference paper
  • First Online:
Information for a Better World: Normality, Virtuality, Physicality, Inclusivity (iConference 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13972))

Included in the following conference series:

Abstract

Extracting information from academic PDF documents is crucial for numerous indexing, retrieval, and analysis use cases. Choosing the best tool to extract specific content elements is difficult because many, technically diverse tools are available, but recent performance benchmarks are rare. Moreover, such benchmarks typically cover only a few content elements like header metadata or bibliographic references and use smaller datasets from specific academic disciplines. We provide a large and diverse evaluation framework that supports more extraction tasks than most related datasets. Our framework builds upon DocBank, a multi-domain dataset of 1.5 M annotated content elements extracted from 500 K pages of research papers on arXiv. Using the new framework, we benchmark ten freely available tools in extracting document metadata, bibliographic references, tables, and other content elements from academic PDF documents. GROBID achieves the best metadata and reference extraction results, followed by CERMINE and Science Parse. For table extraction, Adobe Extract outperforms other tools, even though the performance is much lower than for other content elements. All tools struggle to extract lists, footers, and equations. We conclude that more research on improving and combining tools is necessary to achieve satisfactory extraction quality for most content elements. Evaluation datasets and frameworks like the one we present support this line of research. We make our data and code publicly available to contribute toward this goal.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For example author(s), title, affiliation(s), address(es), email(s).

  2. 2.

    Refers to extracting the components of bibliographic references, e.g., author(s), title, venue, editor(s), volume, issue, page range, year of publication, etc.

  3. 3.

    https://github.com/CrossRef/pdfextract.

  4. 4.

    https://github.com/BMKEG/lapdftext.

  5. 5.

    https://github.com/eliask/pdfssa4met.

  6. 6.

    https://github.com/dimatura/pdfmeat.

  7. 7.

    https://github.com/knmnyn/ParsCit.

  8. 8.

    https://github.com/WING-NUS/Neural-ParsCit.

  9. 9.

    https://github.com/abhinavkashyap/sciwing.

  10. 10.

    https://www.adobe.io/apis/documentcloud/dcsdk/pdf-extract.html.

  11. 11.

    https://www.adobe.com/de/sensei.html.

  12. 12.

    https://github.com/adobe/pdfservices-python-sdk-samples.

  13. 13.

    https://tika.apache.org/.

  14. 14.

    https://github.com/chrismattmann/tika-python.

  15. 15.

    https://github.com/camelot-dev/camelot.

  16. 16.

    https://github.com/pdfminer/pdfminer.six.

  17. 17.

    https://github.com/itext.

  18. 18.

    https://github.com/chulwoopack/docstrum.

  19. 19.

    https://github.com/cjlin1/libsvm.

  20. 20.

    http://mallet.cs.umass.edu/sequences.php.

  21. 21.

    https://www.crummy.com/software/BeautifulSoup/bs4/doc/.

  22. 22.

    https://github.com/kermitt2/grobid.

  23. 23.

    https://github.com/kermitt2/delft.

  24. 24.

    https://GROBID.readthedocs.io/en/latest/Troubleshooting/.

  25. 25.

    http://pdfbox.apache.org/.

  26. 26.

    https://github.com/jalan/pdftotext.

  27. 27.

    https://github.com/ad-freiburg/pdfact.

  28. 28.

    https://github.com/pymupdf/PyMuPDF.

  29. 29.

    https://mupdf.com/.

  30. 30.

    https://github.com/tesseract-ocr/tesseract.

  31. 31.

    https://github.com/inspirehep/refextract.

  32. 32.

    https://linux.die.net/man1/pdftotext.

  33. 33.

    https://github.com/allenai/science-parse.

  34. 34.

    https://github.com/chezou/tabula-py.

  35. 35.

    https://github.com/doc-analysis/DocBank.

  36. 36.

    https://github.com/jsvine/pdfplumber.

  37. 37.

    See Table 4 for more information on the tools’ extraction approaches.

  38. 38.

    https://grobid.readthedocs.io/en/latest/Principles/.

  39. 39.

    https://github.com/kermitt2/grobid/issues/340.

  40. 40.

    https://grobid.readthedocs.io/en/latest/Principles/.

  41. 41.

    https://github.com/elifesciences/sciencebeam-pipelines.

References

  1. Ahmed, M.W., Afzal, M.T.: FLAG-PDFe: features oriented metadata extraction framework for scientific publications. IEEE Access 8, 99458–99469 (2020). https://doi.org/10.1109/ACCESS.2020.2997907

    Article  Google Scholar 

  2. Aiello, M., Monz, C., Todoran, L., Worring, M.: Document understanding for a broad class of documents. Int. J. Doc. Anal. Recogn. 5(1),1–16 (2002). https://doi.org/10.1007/s10032-002-0080-x

  3. Anzaroot, S., Passos, A., Belanger, D., McCallum, A.: Learning soft linear constraints with application to citation field extraction. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 593–602. Association for Computational Linguistics, Baltimore, Maryland (2014). https://doi.org/10.3115/v1/P14-1056

  4. Azimjonov, J., Alikhanov, J.: Rule based metadata extraction framework from academic articles. arXiv CoRR 1807.09009v1 [cs.IR], pp. 1–10 (2018). https://doi.org/10.48550/arXiv.1807.09009

  5. Bast, H., Korzen, C.: The Icecite research paper management system. In: Lin, X., Manolopoulos, Y., Srivastava, D., Huang, G. (eds.) WISE 2013. LNCS, vol. 8181, pp. 396–409. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-41154-0_30

    Chapter  Google Scholar 

  6. Bast, H., Korzen, C.: A benchmark and evaluation for text extraction from PDF. In: 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pp. 1–10. IEEE, Toronto, ON, Canada (2017). https://doi.org/10.1109/JCDL.2017.7991564

  7. Bhardwaj, A., Mercier, D., Dengel, A., Ahmed, S.: DeepBIBX: deep learning for image based bibliographic data extraction. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, E.S. (eds.) Neural Information Processing. ICONIP 2017. LNCS, vol. 10635, pp. 286–293. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70096-0_30

  8. Borkar, V., Deshmukh, K., Sarawagi, S.: Automatic segmentation of text into structured records. SIGMOD Rec. 30(2), 175–186 (2001). https://doi.org/10.1145/376284.375682

    Article  Google Scholar 

  9. Clark, C., Divvala, S.: PDFFigures 2.0: mining figures from research papers. In: Proceedings of the 16th ACM/IEEE-CS on Joint Conference on Digital Libraries, pp. 143–152. JCDL 2016, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2910896.2910904

  10. Cortez, E., da Silva, A.S., Gonçalves, M.A., Mesquita, F., de Moura, E.S.: FLUX-CIM: flexible unsupervised extraction of citation metadata. In: Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries. pp. 215–224. JCDL 2007, Association for Computing Machinery, New York, NY, USA (2007). https://doi.org/10.1145/1255175.1255219

  11. Cortez, E., da Silva, A.S., Gonçalves, M.A., Mesquita, F., de Moura, E.S.: A flexible approach for extracting metadata from bibliographic citations. JASIST 60(6), 1144–1158 (2009). https://doi.org/10.1002/asi.21049

    Article  Google Scholar 

  12. Councill, I., Giles, C.L., Kan, M.Y.: ParsCit: an open-source CRF reference string parsing package. In: Proceedings of the Sixth International Conference on Language Resources and Evaluation. European Language Resources Association, Marrakech, Morocco (2008). https://aclanthology.org/L08-1291/

  13. Cui, B.-G., Chen, X.: An improved hidden Markov model for literature metadata extraction. In: Huang, D.-S., Zhao, Z., Bevilacqua, V., Figueroa, J.C. (eds.) ICIC 2010. LNCS, vol. 6215, pp. 205–212. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14922-1_26

    Chapter  Google Scholar 

  14. Day, M.Y., et al.: Reference metadata extraction using a hierarchical knowledge representation framework. Decis. Support Syst. 43(1), 152–167 (2007). https://doi.org/10.1016/j.dss.2006.08.006

    Article  Google Scholar 

  15. De La Torre, M., Aguirre, C., Anshutz, B., Hsu, W.: MATESC: metadata-analytic text extractor and section classifier for scientific publications. In: Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, vol. 1, pp. 261–267. SciTePress (2018). https://doi.org/10.5220/0006937702610267

  16. Fan, T., et al.: PARDA: a dataset for scholarly pdf document metadata extraction evaluation. In: Gao, H., Wang, X., Yin, Y., Iqbal, M. (eds.) CollaborateCom 2018. LNICST, vol. 268, pp. 417–431. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-12981-1_29

    Chapter  Google Scholar 

  17. Färber, M., Thiemann, A., Jatowt, A.: A high-quality gold standard for citation-based tasks. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation. European Language Resources Association, Miyazaki, Japan (2018). https://aclanthology.org/L18-1296

  18. Giuffrida, G., Shek, E.C., Yang, J.: Knowledge-based metadata extraction from postscript files. In: Proceedings of the Fifth ACM Conference on Digital Libraries, pp. 77–84. DL 2000, Association for Computing Machinery, New York, NY, USA (2000). https://doi.org/10.1145/336597.336639

  19. Granitzer, M., Hristakeva, M., Jack, K., Knight, R.: A comparison of metadata extraction techniques for crowdsourced bibliographic metadata management. In: Proceedings of the 27th Annual ACM Symposium on Applied Computing, pp. 962–964. SAC 2012, Association for Computing Machinery, New York, NY, USA (2012). https://doi.org/10.1145/2245276.2245462

  20. Grennan, M., Beel, J.: Synthetic vs. real reference strings for citation parsing, and the importance of re-training and out-of-sample data for meaningful evaluations: Experiments with GROBID, GIANT and CORA. In: Proceedings of the 8th International Workshop on Mining Scientific Publications, pp. 27–35. Association for Computational Linguistics, Wuhan, China (2020). https://aclanthology.org/2020.wosp-1.4

  21. Grennan, M., Schibel, M., Collins, A., Beel, J.: GIANT: the 1-billion annotated synthetic bibliographic-reference-string dataset for deep citation parsing [Data] (2019). https://doi.org/10.7910/DVN/LXQXAO

  22. Hashmi, A.M., Afzal, M.T., Rehman, S.U.: Rule based approach to extract metadata from scientific pdf documents. In: 2020 5th International Conference on Innovative Technologies in Intelligent Systems and Industrial Applications (CITISIA), pp. 1–4. IEEE, Sydney, Australia (2020). https://doi.org/10.1109/CITISIA50690.2020.9371784

  23. Hetzner, E.: A simple method for citation metadata extraction using hidden Markov models. In: Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 280–284. JCDL 2008, Association for Computing Machinery, New York, NY, USA (2008). https://doi.org/10.1145/1378889.1378937

  24. Kasdorf, W.E.: The Columbia guide to digital publishing. Columbia University Press, USA (2003)

    Google Scholar 

  25. Kern, R., Jack, K., Hristakeva, M.: TeamBeam - meta-data extraction from scientific literature. D-Lib Magaz. 18(7/8), 1045 (2012). https://doi.org/10.1045/july2012-kern

  26. Klein, D., Manning, C.D.: Conditional structure versus conditional estimation in NLP models. In: Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9–16. Association for Computational Linguistics, Pennsylvania, Philadelphia, PA, USA (2002). https://doi.org/10.3115/1118693.1118695

  27. Klink, S., Dengel, A., Kieninger, T.: Document structure analysis based on layout and textual features. In: IAPR International Workshop on Document Analysis Systems. IAPR, Rio de Janeiro, Brazil (2000)

    Google Scholar 

  28. Körner, M., Ghavimi, B., Mayr, P., Hartmann, H., Staab, S.: Evaluating reference string extraction using line-based conditional random fields: a case study with german language publications. In: Nørvåg, K., et al. (eds.) ADBIS 2017. CCIS, vol. 767, pp. 137–145. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67162-8_15

    Chapter  Google Scholar 

  29. Lafferty, J.D., McCallum, A., Pereira, F.C.N.: conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the Eighteenth International Conference on Machine Learning, pp. 282–289. ICML ’01, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (2001). https://dl.acm.org/doi/10.5555/645530.655813

  30. Ley, M.: DBLP: some lessons learned. Proc. VLDB Endow. 2(2), 1493–1500 (2009). https://doi.org/10.14778/1687553.1687577

  31. Li, M., et al.: DocBank: a benchmark dataset for document layout analysis. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 949–960. International Committee on Computational Linguistics, Barcelona, Spain (Online) (2020). https://doi.org/10.18653/v1/2020.coling-main.82

  32. Lipinski, M., Yao, K., Breitinger, C., Beel, J., Gipp, B.: Evaluation of header metadata extraction approaches and tools for scientific PDF documents. In: Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 385–386. JCDL 2013, Association for Computing Machinery, New York, NY, USA (2013). https://doi.org/10.1145/2467696.2467753

  33. Livathinos, N., et al.: Robust pdf document conversion using recurrent neural networks. Proceed. AAAI Conf. Artif. Intell. 35(17), 15137–15145 (2021). https://doi.org/10.1609/aaai.v35i17.17777

    Article  Google Scholar 

  34. Lo, K., Wang, L.L., Neumann, M., Kinney, R., Weld, D.: S2ORC: the semantic scholar open research corpus. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969–4983. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.447

  35. Lopez, P.: GROBID (2008). https://github.com/kermitt2/grobid

  36. Lopez, P.: GROBID: combining automatic bibliographic data recognition and term extraction for scholarship publications. In: Agosti, M., Borbinha, J., Kapidakis, S., Papatheodorou, C., Tsakonas, G. (eds.) ECDL 2009. LNCS, vol. 5714, pp. 473–474. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04346-8_62

    Chapter  Google Scholar 

  37. Mao, S., Kim, J., Thoma, G.R.: a dynamic feature generation system for automated metadata extraction in preservation of digital materials. In: 1st International Workshop on Document Image Analysis for Libraries, pp. 225–232. IEEE Computer Society, Palo Alto, CA, USA (2004). https://doi.org/10.1109/DIAL.2004.1263251

  38. Mao, S., Rosenfeld, A., Kanungo, T.: Document structure analysis algorithms: a literature survey. In: Proceedings Document Recognition and Retrieval X. SPIE Proceedings, vol. 5010, pp. 197–207. SPIE, Santa Clara, California, USA (2003). https://doi.org/10.1117/12.476326

  39. McCallum, A.K., Nigam, K., Rennie, J., Seymore, K.: Automating the construction of internet portals with machine learning. Inf. Retrieval 3(2), 127–163 (2000). https://doi.org/10.1023/A:1009953814988

    Article  Google Scholar 

  40. National Library of Medicine: PubMed. https://pubmed.ncbi.nlm.nih.gov/

  41. National Library of Medicine: PubMed Central. https://www.ncbi.nlm.nih.gov/pmc/

  42. Ojokoh, B., Zhang, M., Tang, J.: A trigram hidden Markov model for metadata extraction from heterogeneous references. Inf. Sci. 181(9), 1538–1551 (2011). https://doi.org/10.1016/j.ins.2011.01.014

    Article  Google Scholar 

  43. Ororbia, A.G., Wu, J., Khabsa, M., WIlliams, K., Giles, C.L.: Big scholarly data in CiteSeerX: information extraction from the web. In: Proceedings of the 24th International Conference on World Wide Web, pp. 597–602. WWW 2015 Companion, Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10.1145/2740908.2741736

  44. Palmero, G., Dimitriadis, Y.: Structured document labeling and rule extraction using a new recurrent fuzzy-neural system. In: Proceedings of the Fifth International Conference on Document Analysis and Recognition, pp. 181–184. Springer (1999). https://doi.org/10.1109/ICDAR.1999.791754

  45. Peng, F., McCallum, A.: Accurate information extraction from research papers using conditional random fields. In: Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL, pp. 329–336. Association for Computational Linguistics, Boston, Massachusetts, USA (2004). https://aclanthology.org/N04-1042

  46. Prasad, A., Kaur, M., Kan, M.-Y.: Neural ParsCit: a deep learning-based reference string parser. Int. J. Digit. Libr. 19(4), 323–337 (2018). https://doi.org/10.1007/s00799-018-0242-1

    Article  Google Scholar 

  47. Rizvi, S.T.R., Dengel, A., Ahmed, S.: A hybrid approach and unified framework for bibliographic reference extraction. IEEE Access 8, 217231–217245 (2020). https://doi.org/10.1109/ACCESS.2020.3042455

    Article  Google Scholar 

  48. Rodrigues Alves, D., Colavizza, G., Kaplan, F.: Deep reference mining from scholarly literature in the arts and humanities. Front. Res. Metr. Anal. 3, 21 (2018). https://doi.org/10.3389/frma.2018.00021

    Article  Google Scholar 

  49. Saier, T., Färber, M.: Bibliometric-enhanced arXiv: a data set for paper-based and citation-based tasks. In: Proceedings of the 8th International Workshop on Bibliometric-enhanced Information Retrieval (BIR). CEUR Workshop Proceedings, vol. 2345, pp. 14–26. CEUR-WS.org, Cologne, Germany (2019). http://ceur-ws.org/Vol-2345/paper2.pdf

  50. Saier, T., Färber, M.: unarXive: a large scholarly data set with publications’ full-text, annotated in-text citations, and links to metadata. Scientometrics 125(3), 3085–3108 (2020). https://doi.org/10.1007/s11192-020-03382-z

    Article  Google Scholar 

  51. Schloss Dagstuhl - Leibniz Center for Informatics, University of Trier: dblp: Computer Science Bibliography. https://dblp.org/

  52. Souza, A., Moreira, V., Heuser, C.: ARCTIC: metadata extraction from scientific papers in PDF using two-layer CRF. In: Proceedings of the 2014 ACM Symposium on Document Engineering, pp. 121–130. DocEng 2014, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2644866.2644872

  53. Tkaczyk, D., Collins, A., Sheridan, P., Beel, J.: Machine learning vs. rules and out-of-the-box vs. retrained: an evaluation of open-source bibliographic reference and citation parsers. In: Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries, pp. 99–108. JCDL 2018, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3197026.3197048

  54. Tkaczyk, D., Szostek, P., Fedoryszak, M., Dendek, P.J., Bolikowski, Ł.: CERMINE: automatic extraction of structured metadata from scientific literature. Int. J. Doc. Anal. Recogn. (IJDAR) 18(4), 317–335 (2015). https://doi.org/10.1007/s10032-015-0249-8

  55. Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000–6010. NIPS2017, Curran Associates Inc., Red Hook, NY, USA (2017). https://dl.acm.org/doi/10.5555/3295222.3295349

  56. Vilnis, L., Belanger, D., Sheldon, D., McCallum, A.: Bethe projections for non-local inference. In: Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, pp. 892–901. UAI2015, AUAI Press, Arlington, Virginia, USA (2015). https://doi.org/10.48550/arXiv.1503.01397

Download references

Author information

Authors and Affiliations

  1. University of Göttingen, 37073, Göttingen, Germany

    Norman Meuschke, Timo Spinde & Bela Gipp

  2. University of Passau, 94032, Passau, Germany

    Apurva Jagdale & Jelena Mitrović

  3. The Institute for Artificial Intelligence R &D of Serbia, 21000, Novi Sad, Serbia

    Jelena Mitrović

Authors
  1. Norman Meuschke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Apurva Jagdale
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Timo Spinde
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jelena Mitrović
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bela Gipp
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Norman Meuschke .

Editor information

Editors and Affiliations

  1. iSchool Organization, Berlin, Germany

    Isaac Sserwanga

  2. Victoria University of Wellington, Wellington, New Zealand

    Anne Goulding

  3. University of Missouri, Chicago, IL, USA

    Heather Moulaison-Sandy

  4. University of South Australia, Adelaide, SA, Australia

    Jia Tina Du

  5. University of Porto, Porto, Portugal

    António Lucas Soares

  6. Monash University, Clayton, VIC, Australia

    Viviane Hessami

  7. University of Tennessee at Knoxville, Knoxville, TN, USA

    Rebecca D. Frank

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Meuschke, N., Jagdale, A., Spinde, T., Mitrović, J., Gipp, B. (2023). A Benchmark of PDF Information Extraction Tools Using a Multi-task and Multi-domain Evaluation Framework for Academic Documents. In: Sserwanga, I., et al. Information for a Better World: Normality, Virtuality, Physicality, Inclusivity. iConference 2023. Lecture Notes in Computer Science, vol 13972. Springer, Cham. https://doi.org/10.1007/978-3-031-28032-0_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-28032-0_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-28031-3

  • Online ISBN: 978-3-031-28032-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation