Skip to main content

Evaluating author name disambiguation for digital libraries: a case of DBLP

Scientometrics volume 116, pages 1867–1886 (2018)Cite this article

A Correction to this article was published on 27 November 2018

This article has been updated

Abstract

Author name ambiguity in a digital library may affect the findings of research that mines authorship data of the library. This study evaluates author name disambiguation in DBLP, a widely used but insufficiently evaluated digital library for its disambiguation performance. In doing so, this study takes a triangulation approach that author name disambiguation for a digital library can be better evaluated when its performance is assessed on multiple labeled datasets with comparison to baselines. Tested on three types of labeled data containing 5000 to 6 M disambiguated names, DBLP is shown to assign author names quite accurately to distinct authors, resulting in pairwise precision, recall, and F1 measures around 0.90 or above overall. DBLP’s author name disambiguation performs well even on large ambiguous name blocks but deficiently on distinguishing authors with the same names. Compared to other disambiguation algorithms, DBLP’s disambiguation performance is quite competitive, possibly due to its hybrid disambiguation approach combining algorithmic disambiguation and manual error correction. A discussion follows on strengths and weaknesses of labeled datasets used in this study for future efforts to evaluate author name disambiguation on a digital library scale.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3

Change history

  • 27 November 2018

    In the original publication of the article, in Abstract, the size of labeled data was incorrectly reported.

  • 27 November 2018

    In the original publication of the article, in Abstract, the size of labeled data was incorrectly reported.

Notes

  1. DBLP-related papers were searched using the term “DBLP” with a Document Type filter (“Conference Paper and Article”) at https://www.scopus.com/search/form.uri?display=basic.

  2. An exception is Kim and Diesner (2015) in which the DBLP’s disambiguation performance as of May 2014 was measured on a sample of labeled data (474 distinct authors in 3921 papers) extracted from Shin et al. (2014). The evaluation results were Pairwise F1 = 0.96 and K-metric = 0.952.

  3. At the time of this study, the service is provided by Clarivate Analytics at https://clarivate.com/hcr/.

  4. The dataset in a XML format can be downloaded from dblp.org/xml/release/dblp-2017-09-03.xml.gz.

  5. DBLP team kindly provided the list of 39,152 name pairs in synonym relation for this study.

  6. http://clgiles.ist.psu.edu/data/nameset_author-disamb.tar.zip.

  7. http://www.lbd.dcc.ufmg.br/lbd/collections/disambiguation/DBLP.tar.gz/at_download/file.

  8. https://aminer.org/.

  9. http://arnetminer.org/lab-datasets/disambiguation/rich-author-disambiguation-data.zip.

  10. https://github.com/yaya213/DBLP-Name-Disambiguation-Dataset.

  11. https://figshare.com/articles/ORCID_Public_Data_File_2017/5479792.

  12. For details, see http://dblp.org/faq/17334571.

  13. https://static.aminer.org/lab-datasets/citation/dblp.v10.zip.

  14. Splitting can affect the pP by decreasing the denominator in Eq. (1) but also by decreasing the numerator, thus reducing the overall impact of splitting on pP.

  15. For comparison, decimal points of performance results in Table 4 were modified to be consistent with metric units in other studies. Also, B-Cubed metrics were calculated for QIAN on all names regardless of block size, following the referenced study.

References

  • Abbott, A., Cyranoski, D., Jones, N., Maher, B., Schiermeier, Q., & Van Noorden, R. (2010). Do metrics matter? Nature, 465(7300), 860–862.

    Article  Google Scholar 

  • Bilder, G. (2011). Disambiguation without de-duplication: Modeling authority and trust in the ORCID system. Retrieved from https://www.crossref.org/wp/labs/whitepapers/disambiguation-deduplication-wp-v4.pdf.

  • Bornmann, L., & Mutz, R. (2015). Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. Journal of the Association for Information Science and Technology, 66(11), 2215–2222.

    Article  Google Scholar 

  • Cota, R. G., Ferreira, A. A., Nascimento, C., Goncalves, M. A., & Laender, A. H. F. (2010). An unsupervised heuristic-based hierarchical method for name disambiguation in bibliographic citations. Journal of the American Society for Information Science and Technology, 61(9), 1853–1870.

    Article  Google Scholar 

  • D’Angelo, C. A., Giuffrida, C., & Abramo, G. (2011). A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments. Journal of the American Society for Information Science and Technology, 62(2), 257–269.

    Article  Google Scholar 

  • Erman, N., & Todorovski, L. (2015). The effects of measurement error in case of scientific network analysis. Scientometrics, 104(2), 453–473.

    Article  Google Scholar 

  • Fegley, B. D., & Torvik, V. I. (2013). Has large-scale named-entity network analysis been resting on a flawed assumption? PLoS ONE, 8(7), 670299.

    Article  Google Scholar 

  • Ferreira, A. A., Goncalves, M. A., & Laender, A. H. F. (2012). A brief survey of automatic methods for author name disambiguation. Sigmod Record, 41(2), 15–26.

    Article  Google Scholar 

  • Franceschet, M. (2011). Collaboration in computer science: A network science approach. Journal of the American Society for Information Science and Technology, 62(10), 1992–2012.

    Article  Google Scholar 

  • Haak, L. L., Fenner, M., Paglione, L., Pentz, E., & Ratner, H. (2012). ORCID: A system to uniquely identify researchers. Learned Publishing, 25(4), 259–264.

    Article  Google Scholar 

  • Han, H., Zha, H. Y., & Giles, C. L. (2005). Name disambiguation spectral in author citations using a K-way clustering method. In Proceedings of the 5th ACM/IEEE joint conference on digital libraries, proceedings, pp. 334–343.

  • Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251–261.

    Article  Google Scholar 

  • Ioannidis, J. P. A., Boyack, K. W., & Klavans, R. (2014). Estimates of the continuously publishing core in the scientific workforce. PLoS ONE, 9(7), 6101698.

    Article  Google Scholar 

  • Kang, I. S., Kim, P., Lee, S., Jung, H., & You, B. J. (2011). Construction of a large-scale test set for author disambiguation. Information Processing and Management, 47(3), 452–465.

    Article  Google Scholar 

  • Kawashima, H., & Tomizawa, H. (2015). Accuracy evaluation of Scopus author ID based on the largest funding database in Japan. Scientometrics, 103(3), 1061–1071.

    Article  Google Scholar 

  • Kim, J., & Diesner, J. (2015). The effect of data pre-processing on understanding the evolution of collaboration networks. Journal of Informetrics, 9(1), 226–236.

    Article  Google Scholar 

  • Kim, J., & Diesner, J. (2016). Distortive effects of initial-based name disambiguation on measurements of large-scale coauthorship networks. Journal of the Association for Information Science and Technology, 67(6), 1446–1461.

    Article  Google Scholar 

  • Lerchenmueller, M. J., & Sorenson, O. (2016). Author disambiguation in PubMed: Evidence on the precision and recall of authority among NIH-funded scientists. PLoS ONE, 11(7), e0158731.

    Article  Google Scholar 

  • Levin, M., Krawczyk, S., Bethard, S., & Jurafsky, D. (2012). Citation-based bootstrapping for large-scale author disambiguation. Journal of the American Society for Information Science and Technology, 63(5), 1030–1047.

    Article  Google Scholar 

  • Ley, M. (2002). The DBLP computer science bibliography: Evolution, research issues, Perspectives. In A. F. Laender & A. Oliveira (Eds.), String processing and information retrieval (Vol. 2476, pp. 1–10). Berlin: Springer.

    Chapter  Google Scholar 

  • Ley, M. (2009). DBLP: Some lessons learned. Proceedings of the VLDB Endowment, 2(2), 1493–1500.

    Article  Google Scholar 

  • Liu, W., Islamaj Dogan, R., Kim, S., Comeau, D. C., Kim, W., Yeganova, L., et al. (2014). Author name disambiguation for PubMed. Journal of the Association for Information Science and Technology, 65(4), 765–781.

    Article  Google Scholar 

  • Louppe, G., Al-Natsheh, H. T., Susik, M., & Maguire, E. J. (2016). Ethnicity sensitive author disambiguation using semi-supervised learning. Knowledge Engineering and Semantic Web, Kesw, 2016(649), 272–287.

    Article  Google Scholar 

  • Martin, T., Ball, B., Karrer, B., & Newman, M. E. J. (2013). Coauthorship and citation patterns in the physical review. Physical Review E, 88(1), 012814.

    Article  Google Scholar 

  • Menestrina, D., Whang, S. E., & Garcia-Molina, H. (2010). Evaluating entity resolution results. Proceedings of the VLDB Endowment, 3(1–2), 208–219.

    Article  Google Scholar 

  • Milojević, S. (2013). Accuracy of simple, initials-based methods for author name disambiguation. Journal of Informetrics, 7(4), 767–773.

    Article  Google Scholar 

  • Moed, H. F., Aisati, M., & Plume, A. (2013). Studying scientific migration in Scopus. Scientometrics, 94(3), 929–942.

    Article  Google Scholar 

  • Müller, M. C., Reitz, F., & Roy, N. (2017). Data sets for author name disambiguation: an empirical analysis and a new resource. Scientometrics, 111(3), 1467–1500.

    Article  Google Scholar 

  • Newman, M. E. J. (2001). The structure of scientific collaboration networks. Proceedings of the National Academy of Sciences USA, 98(2), 404–409.

    Article  MathSciNet  Google Scholar 

  • On, B. W., Lee, D., Kang, J., & Mitra, P. (2005). Comparative study of name disambiguation problem using a scalable blocking-based framework. In Proceedings of the 5th ACM/IEEE joint conference on digital libraries, proceedings, pp. 344–353.

  • Qian, Y., Zheng, Q., Sakai, T., Ye, J., & Liu, J. (2015). Dynamic author name disambiguation for growing digital libraries. Information Retrieval Journal, 18(5), 379–412.

    Article  Google Scholar 

  • Reijnhoudt, L., Costas, R., Noyons, E., Borner, K., & Scharnhorst, A. (2014). ‘Seed plus expand’: A general methodology for detecting publication oeuvres of individual researchers. Scientometrics, 101(2), 1403–1417.

    Article  Google Scholar 

  • Reitz, F., & Hoffmann, O. (2013). Learning from the past: An analysis of person name corrections in the DBLP collection and social network properties of affected entities. In T. Özyer, J. Rokne, G. Wagner, & A. H. P. Reuser (Eds.), The influence of technology on social network analysis and mining (pp. 427–453). Vienna: Springer.

    Chapter  Google Scholar 

  • Santana, A. F., Goncalves, M. A., Laender, A. H. F., & Ferreira, A. A. (2015). On the combination of domain-specific heuristics for author name disambiguation: The nearest cluster method. International Journal on Digital Libraries, 16(3–4), 229–246.

    Article  Google Scholar 

  • Schulz, C., Mazloumian, A., Petersen, A. M., Penner, O., & Helbing, D. (2014). Exploiting citation networks for large-scale author name disambiguation. EPJ Data Science, 3(1), 11.

    Article  Google Scholar 

  • Shin, D., Kim, T., Choi, J., & Kim, J. (2014). Author name disambiguation using a graph model with node splitting and merging based on bibliographic information. Scientometrics, 100(1), 15–50.

    Article  Google Scholar 

  • Sinatra, R., Wang, D., Deville, P., Song, C. M., & Barabasi, A. L. (2016). Quantifying the evolution of individual scientific impact. Science, 354(6312), aaf5239.

    Article  Google Scholar 

  • Smalheiser, N. R., & Torvik, V. I. (2009). Author name disambiguation. Annual Review of Information Science and Technology, 43, 287–313.

    Article  Google Scholar 

  • Strotmann, A., & Zhao, D. Z. (2012). Author name disambiguation: What difference does it make in author-based citation analysis? Journal of the American Society for Information Science and Technology, 63(9), 1820–1833.

    Article  Google Scholar 

  • Tang, J., Fong, A. C. M., Wang, B., & Zhang, J. (2012). A unified probabilistic framework for name disambiguation in digital library. IEEE Transactions on Knowledge and Data Engineering, 24(6), 975–987.

    Article  Google Scholar 

  • Torvik, V. I., & Smalheiser, N. R. (2009). Author name disambiguation in MEDLINE. ACM Transactions on Knowledge Discovery from Data, 3(3), 11.

    Article  Google Scholar 

  • Wang, D. J., Shi, X. L., McFarland, D. A., & Leskovec, J. (2012). Measurement error in network data: A re-classification. Social Networks, 34(4), 396–409.

    Article  Google Scholar 

  • Wang, X., Tang, J., Cheng, H., & Yu, P. S. (2011). ADANA: Active name disambiguation. In Paper presented at the 2011 IEEE 11th international conference on data mining. http://ieeexplore.ieee.org/document/6137284/.

Download references

Acknowledgements

I would like to thank Florian Reitz (Leibniz Center for Informatics, Schloss Dagstuhl, Germany) for providing the list of synonyms in DBLP and Alan Filipe Santana (Departamento de Ciência da Computação, Universidade Federal de Minas Gerais, Brazil) for sharing the raw KISTI dataset. I am also thankful to anonymous reviewers for their comments. This work was supported by grants from the National Science Foundation (Grants #1561687 and #1535370), the Alfred P. Sloan Foundation, and the Ewing Marion Kauffman Foundation.

Author information

Authors and Affiliations

  1. Institute for Research on Innovation and Science, University of Michigan, 330 Packard Street, Ann Arbor, MI, 48104-2910, USA

    Jinseok Kim

Authors
  1. Jinseok Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinseok Kim.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kim, J. Evaluating author name disambiguation for digital libraries: a case of DBLP. Scientometrics 116, 1867–1886 (2018). https://doi.org/10.1007/s11192-018-2824-5

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-018-2824-5

Keywords

  • Author name disambiguation
  • Digital library
  • Triangulation
  • Disambiguation evaluation
  • DBLP