Evaluating author name disambiguation for digital libraries: a case of DBLP

Abstract

Author name ambiguity in a digital library may affect the findings of research that mines authorship data of the library. This study evaluates author name disambiguation in DBLP, a widely used but insufficiently evaluated digital library for its disambiguation performance. In doing so, this study takes a triangulation approach that author name disambiguation for a digital library can be better evaluated when its performance is assessed on multiple labeled datasets with comparison to baselines. Tested on three types of labeled data containing 5000 to 6 M disambiguated names, DBLP is shown to assign author names quite accurately to distinct authors, resulting in pairwise precision, recall, and F1 measures around 0.90 or above overall. DBLP’s author name disambiguation performs well even on large ambiguous name blocks but deficiently on distinguishing authors with the same names. Compared to other disambiguation algorithms, DBLP’s disambiguation performance is quite competitive, possibly due to its hybrid disambiguation approach combining algorithmic disambiguation and manual error correction. A discussion follows on strengths and weaknesses of labeled datasets used in this study for future efforts to evaluate author name disambiguation on a digital library scale.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3

Change history

  • 27 November 2018

    In the original publication of the article, in Abstract, the size of labeled data was incorrectly reported.

  • 27 November 2018

    In the original publication of the article, in Abstract, the size of labeled data was incorrectly reported.

Notes

  1. 1.

    DBLP-related papers were searched using the term “DBLP” with a Document Type filter (“Conference Paper and Article”) at https://www.scopus.com/search/form.uri?display=basic.

  2. 2.

    An exception is Kim and Diesner (2015) in which the DBLP’s disambiguation performance as of May 2014 was measured on a sample of labeled data (474 distinct authors in 3921 papers) extracted from Shin et al. (2014). The evaluation results were Pairwise F1 = 0.96 and K-metric = 0.952.

  3. 3.

    At the time of this study, the service is provided by Clarivate Analytics at https://clarivate.com/hcr/.

  4. 4.

    The dataset in a XML format can be downloaded from dblp.org/xml/release/dblp-2017-09-03.xml.gz.

  5. 5.

    DBLP team kindly provided the list of 39,152 name pairs in synonym relation for this study.

  6. 6.

    http://clgiles.ist.psu.edu/data/nameset_author-disamb.tar.zip.

  7. 7.

    http://www.lbd.dcc.ufmg.br/lbd/collections/disambiguation/DBLP.tar.gz/at_download/file.

  8. 8.

    https://aminer.org/.

  9. 9.

    http://arnetminer.org/lab-datasets/disambiguation/rich-author-disambiguation-data.zip.

  10. 10.

    https://github.com/yaya213/DBLP-Name-Disambiguation-Dataset.

  11. 11.

    https://figshare.com/articles/ORCID_Public_Data_File_2017/5479792.

  12. 12.

    For details, see http://dblp.org/faq/17334571.

  13. 13.

    https://static.aminer.org/lab-datasets/citation/dblp.v10.zip.

  14. 14.

    Splitting can affect the pP by decreasing the denominator in Eq. (1) but also by decreasing the numerator, thus reducing the overall impact of splitting on pP.

  15. 15.

    For comparison, decimal points of performance results in Table 4 were modified to be consistent with metric units in other studies. Also, B-Cubed metrics were calculated for QIAN on all names regardless of block size, following the referenced study.

References

  1. Abbott, A., Cyranoski, D., Jones, N., Maher, B., Schiermeier, Q., & Van Noorden, R. (2010). Do metrics matter? Nature, 465(7300), 860–862.

    Article  Google Scholar 

  2. Bilder, G. (2011). Disambiguation without de-duplication: Modeling authority and trust in the ORCID system. Retrieved from https://www.crossref.org/wp/labs/whitepapers/disambiguation-deduplication-wp-v4.pdf.

  3. Bornmann, L., & Mutz, R. (2015). Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. Journal of the Association for Information Science and Technology, 66(11), 2215–2222.

    Article  Google Scholar 

  4. Cota, R. G., Ferreira, A. A., Nascimento, C., Goncalves, M. A., & Laender, A. H. F. (2010). An unsupervised heuristic-based hierarchical method for name disambiguation in bibliographic citations. Journal of the American Society for Information Science and Technology, 61(9), 1853–1870.

    Article  Google Scholar 

  5. D’Angelo, C. A., Giuffrida, C., & Abramo, G. (2011). A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments. Journal of the American Society for Information Science and Technology, 62(2), 257–269.

    Article  Google Scholar 

  6. Erman, N., & Todorovski, L. (2015). The effects of measurement error in case of scientific network analysis. Scientometrics, 104(2), 453–473.

    Article  Google Scholar 

  7. Fegley, B. D., & Torvik, V. I. (2013). Has large-scale named-entity network analysis been resting on a flawed assumption? PLoS ONE, 8(7), 670299.

    Article  Google Scholar 

  8. Ferreira, A. A., Goncalves, M. A., & Laender, A. H. F. (2012). A brief survey of automatic methods for author name disambiguation. Sigmod Record, 41(2), 15–26.

    Article  Google Scholar 

  9. Franceschet, M. (2011). Collaboration in computer science: A network science approach. Journal of the American Society for Information Science and Technology, 62(10), 1992–2012.

    Article  Google Scholar 

  10. Haak, L. L., Fenner, M., Paglione, L., Pentz, E., & Ratner, H. (2012). ORCID: A system to uniquely identify researchers. Learned Publishing, 25(4), 259–264.

    Article  Google Scholar 

  11. Han, H., Zha, H. Y., & Giles, C. L. (2005). Name disambiguation spectral in author citations using a K-way clustering method. In Proceedings of the 5th ACM/IEEE joint conference on digital libraries, proceedings, pp. 334–343.

  12. Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251–261.

    Article  Google Scholar 

  13. Ioannidis, J. P. A., Boyack, K. W., & Klavans, R. (2014). Estimates of the continuously publishing core in the scientific workforce. PLoS ONE, 9(7), 6101698.

    Article  Google Scholar 

  14. Kang, I. S., Kim, P., Lee, S., Jung, H., & You, B. J. (2011). Construction of a large-scale test set for author disambiguation. Information Processing and Management, 47(3), 452–465.

    Article  Google Scholar 

  15. Kawashima, H., & Tomizawa, H. (2015). Accuracy evaluation of Scopus author ID based on the largest funding database in Japan. Scientometrics, 103(3), 1061–1071.

    Article  Google Scholar 

  16. Kim, J., & Diesner, J. (2015). The effect of data pre-processing on understanding the evolution of collaboration networks. Journal of Informetrics, 9(1), 226–236.

    Article  Google Scholar 

  17. Kim, J., & Diesner, J. (2016). Distortive effects of initial-based name disambiguation on measurements of large-scale coauthorship networks. Journal of the Association for Information Science and Technology, 67(6), 1446–1461.

    Article  Google Scholar 

  18. Lerchenmueller, M. J., & Sorenson, O. (2016). Author disambiguation in PubMed: Evidence on the precision and recall of authority among NIH-funded scientists. PLoS ONE, 11(7), e0158731.

    Article  Google Scholar 

  19. Levin, M., Krawczyk, S., Bethard, S., & Jurafsky, D. (2012). Citation-based bootstrapping for large-scale author disambiguation. Journal of the American Society for Information Science and Technology, 63(5), 1030–1047.

    Article  Google Scholar 

  20. Ley, M. (2002). The DBLP computer science bibliography: Evolution, research issues, Perspectives. In A. F. Laender & A. Oliveira (Eds.), String processing and information retrieval (Vol. 2476, pp. 1–10). Berlin: Springer.

    Google Scholar 

  21. Ley, M. (2009). DBLP: Some lessons learned. Proceedings of the VLDB Endowment, 2(2), 1493–1500.

    Article  Google Scholar 

  22. Liu, W., Islamaj Dogan, R., Kim, S., Comeau, D. C., Kim, W., Yeganova, L., et al. (2014). Author name disambiguation for PubMed. Journal of the Association for Information Science and Technology, 65(4), 765–781.

    Article  Google Scholar 

  23. Louppe, G., Al-Natsheh, H. T., Susik, M., & Maguire, E. J. (2016). Ethnicity sensitive author disambiguation using semi-supervised learning. Knowledge Engineering and Semantic Web, Kesw, 2016(649), 272–287.

    Article  Google Scholar 

  24. Martin, T., Ball, B., Karrer, B., & Newman, M. E. J. (2013). Coauthorship and citation patterns in the physical review. Physical Review E, 88(1), 012814.

    Article  Google Scholar 

  25. Menestrina, D., Whang, S. E., & Garcia-Molina, H. (2010). Evaluating entity resolution results. Proceedings of the VLDB Endowment, 3(1–2), 208–219.

    Article  Google Scholar 

  26. Milojević, S. (2013). Accuracy of simple, initials-based methods for author name disambiguation. Journal of Informetrics, 7(4), 767–773.

    Article  Google Scholar 

  27. Moed, H. F., Aisati, M., & Plume, A. (2013). Studying scientific migration in Scopus. Scientometrics, 94(3), 929–942.

    Article  Google Scholar 

  28. Müller, M. C., Reitz, F., & Roy, N. (2017). Data sets for author name disambiguation: an empirical analysis and a new resource. Scientometrics, 111(3), 1467–1500.

    Article  Google Scholar 

  29. Newman, M. E. J. (2001). The structure of scientific collaboration networks. Proceedings of the National Academy of Sciences USA, 98(2), 404–409.

    MathSciNet  Article  Google Scholar 

  30. On, B. W., Lee, D., Kang, J., & Mitra, P. (2005). Comparative study of name disambiguation problem using a scalable blocking-based framework. In Proceedings of the 5th ACM/IEEE joint conference on digital libraries, proceedings, pp. 344–353.

  31. Qian, Y., Zheng, Q., Sakai, T., Ye, J., & Liu, J. (2015). Dynamic author name disambiguation for growing digital libraries. Information Retrieval Journal, 18(5), 379–412.

    Article  Google Scholar 

  32. Reijnhoudt, L., Costas, R., Noyons, E., Borner, K., & Scharnhorst, A. (2014). ‘Seed plus expand’: A general methodology for detecting publication oeuvres of individual researchers. Scientometrics, 101(2), 1403–1417.

    Article  Google Scholar 

  33. Reitz, F., & Hoffmann, O. (2013). Learning from the past: An analysis of person name corrections in the DBLP collection and social network properties of affected entities. In T. Özyer, J. Rokne, G. Wagner, & A. H. P. Reuser (Eds.), The influence of technology on social network analysis and mining (pp. 427–453). Vienna: Springer.

    Google Scholar 

  34. Santana, A. F., Goncalves, M. A., Laender, A. H. F., & Ferreira, A. A. (2015). On the combination of domain-specific heuristics for author name disambiguation: The nearest cluster method. International Journal on Digital Libraries, 16(3–4), 229–246.

    Article  Google Scholar 

  35. Schulz, C., Mazloumian, A., Petersen, A. M., Penner, O., & Helbing, D. (2014). Exploiting citation networks for large-scale author name disambiguation. EPJ Data Science, 3(1), 11.

    Article  Google Scholar 

  36. Shin, D., Kim, T., Choi, J., & Kim, J. (2014). Author name disambiguation using a graph model with node splitting and merging based on bibliographic information. Scientometrics, 100(1), 15–50.

    Article  Google Scholar 

  37. Sinatra, R., Wang, D., Deville, P., Song, C. M., & Barabasi, A. L. (2016). Quantifying the evolution of individual scientific impact. Science, 354(6312), aaf5239.

    Article  Google Scholar 

  38. Smalheiser, N. R., & Torvik, V. I. (2009). Author name disambiguation. Annual Review of Information Science and Technology, 43, 287–313.

    Article  Google Scholar 

  39. Strotmann, A., & Zhao, D. Z. (2012). Author name disambiguation: What difference does it make in author-based citation analysis? Journal of the American Society for Information Science and Technology, 63(9), 1820–1833.

    Article  Google Scholar 

  40. Tang, J., Fong, A. C. M., Wang, B., & Zhang, J. (2012). A unified probabilistic framework for name disambiguation in digital library. IEEE Transactions on Knowledge and Data Engineering, 24(6), 975–987.

    Article  Google Scholar 

  41. Torvik, V. I., & Smalheiser, N. R. (2009). Author name disambiguation in MEDLINE. ACM Transactions on Knowledge Discovery from Data, 3(3), 11.

    Article  Google Scholar 

  42. Wang, D. J., Shi, X. L., McFarland, D. A., & Leskovec, J. (2012). Measurement error in network data: A re-classification. Social Networks, 34(4), 396–409.

    Article  Google Scholar 

  43. Wang, X., Tang, J., Cheng, H., & Yu, P. S. (2011). ADANA: Active name disambiguation. In Paper presented at the 2011 IEEE 11th international conference on data mining. http://ieeexplore.ieee.org/document/6137284/.

Download references

Acknowledgements

I would like to thank Florian Reitz (Leibniz Center for Informatics, Schloss Dagstuhl, Germany) for providing the list of synonyms in DBLP and Alan Filipe Santana (Departamento de Ciência da Computação, Universidade Federal de Minas Gerais, Brazil) for sharing the raw KISTI dataset. I am also thankful to anonymous reviewers for their comments. This work was supported by grants from the National Science Foundation (Grants #1561687 and #1535370), the Alfred P. Sloan Foundation, and the Ewing Marion Kauffman Foundation.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jinseok Kim.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kim, J. Evaluating author name disambiguation for digital libraries: a case of DBLP. Scientometrics 116, 1867–1886 (2018). https://doi.org/10.1007/s11192-018-2824-5

Download citation

Keywords

  • Author name disambiguation
  • Digital library
  • Triangulation
  • Disambiguation evaluation
  • DBLP