Discovering the structure and impact of the digital library evaluation domain

  • Leonidas Papachristopoulos
  • Giannis Tsakonas
  • Moses Boudourides
  • Michalis Sfakakis
  • Nikos Kleidis
  • Sergios Lenis
  • Christos Papatheodorou


The multidimensional nature of digital libraries evaluation domain poses several challenges to the research communities that intend to assess criteria, methods, products and tools, and also practice them. The amount of scientific production that is published in the domain hinders and disorientates the interested researchers. These researchers need guidance to exploit effectively the considerable amount of data and the diversity of methods, as well as to identify new research goals and develop their plans for future studies. This paper proposes a methodological pathway to investigate the core topics that structure the digital library evaluation domain and their impact. Further to the exploration of these topical entities, this study investigates also the researchers that contribute substantially to key topics, their communities and their relationships. The proposed methodology exploits topic modeling and network analysis in combination with citation and altmetrics analysis on a corpus consisting of the digital library evaluation papers presented in JCDL, ECDL/TDPL and ICADL conferences in the period 2001–2013.


Digital libraries Evaluation Topic modeling Network analysis Latent dirichlet allocation 



We would like to thank Akrivi Athanasopoulou and Aggeliki Giannopoulou for assisting in data management processes.


  1. 1.
    Borgman, C.L.: What are digital libraries? Competing visions. Inf. Process. Manag. 35(3), 227–243 (1999)CrossRefGoogle Scholar
  2. 2.
    Saracevic, T.: Digital library evaluation: toward evolution of concepts. Libr. Trends 49(2), 350–369 (2000)Google Scholar
  3. 3.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)MATHGoogle Scholar
  4. 4.
    Haythornthwaite, C.: Social network analysis: an approach and technique for the study of information exchange. Libr. Inf. Sci. Res. 18(4), 323–342 (1996)CrossRefGoogle Scholar
  5. 5.
    NISO, Altmetrics definitions and use cases (2016)Google Scholar
  6. 6.
    Holland, A.G.: Information science: an interdisciplinary effort? J. Doc. 64(1), 7–23 (2008)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Nguyen, S.H., Chowdhury, G.: Digital library research (1990–2010): a knowledge map of core topics and subtopics. In: Proceedings of the 13th International Conference on Asia-Pacific Digital Libraries, ICADL 2011, Beijing, China, October 24–27, 2011, vol. 7008, pp. 367–371 (2011)Google Scholar
  8. 8.
    Pomerantz, J., Wildemuth, B.M., Yang, S., Fox, E.A.: Curriculum development for digital libraries. In: Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries, p. 175 (2006)Google Scholar
  9. 9.
    Finlay, C.S., Sugimoto, C.R., Li, D., Russell, T.G.: LIS dissertation titles and abstracts (1930–2009): where have all the Librar* gone? Libr. Q. 82(1), 29–46 (2012)CrossRefGoogle Scholar
  10. 10.
    Milojević, S., Sugimoto, C.R., Yan, E., Ding, Y.: The cognitive structure of library and information science: analysis of article title words. J. Am. Soc. Inf. Sci. Technol. 62(10), 1933–1953 (2011)CrossRefGoogle Scholar
  11. 11.
    Alhoori, H., Furuta, R.: Do altmetrics follow the crowd or does the crowd follow altmetrics? In: Proceedings of the 14th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 375–378 (2014)Google Scholar
  12. 12.
    Bolelli, L., Ertekin, S., Zhou, D., Giles, C.L.: Finding topic trends in digital libraries. In Proceedings of the 9th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 69–72 (2009)Google Scholar
  13. 13.
    Kim, H.: Domain analysis with text mining: Analysis of digital library research trends using profiling methods. J. Inf. Sci. 36(2), 144–161 (2010)CrossRefGoogle Scholar
  14. 14.
    Noh, Y., Hagedorn, K., Newman, D.: Are learned topics more useful than subject headings. In: Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries, pp. 411–412 (2011)Google Scholar
  15. 15.
    Papachristopoulos, L., Kleidis, N., Sfakakis, M., Tsakonas, G., Papatheodorou, C.: Discovering the topical evolution of the digital library evaluation community. In: Proceedings of the 9th Metadata and Semantics Research Conference, 9–11 September 2015, Manchester, UK, pp. 101–112 (2015)Google Scholar
  16. 16.
    Newman, D., Noh, Y., Talley, E., Karimi, S., Baldwin, T.: Evaluating topic models for digital libraries. In: Proceedings of the 10th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 215–224 (2010)Google Scholar
  17. 17.
    Newman, D., Karimi, S., Cavedon, L.: Using topic models to interpret MEDLINE’s medical subject headings. In: Proceedings of Advances in Artificial Intelligence (AI 2009), vol. 5866, pp. 270–279 (2009)Google Scholar
  18. 18.
    Newman, D., Hagedorn, K., Chemudugunta, C., Smyth, P.: Subject metadata enrichment using statistical topic models. In: Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 366–375 (2007)Google Scholar
  19. 19.
    Aletras, N., Stevenson, M., Clough, P.: Computing similarity between items in a digital library of cultural heritage. J. Comput. Cult. Herit. 5(4), 1–19 (2012)CrossRefGoogle Scholar
  20. 20.
    Tuarob, S., Pouchard, L.C., Mitra, P., Giles, C.L.: A generalized topic modeling approach for automatic document annotation. Int. J. Digit. Libr. 16(2), 111–128 (2015)CrossRefGoogle Scholar
  21. 21.
    Song, M., Heo, G.E., Kim, S.Y.: Analyzing topic evolution in bioinformatics: investigation of dynamics of the field with conference data in DBLP. Scientometrics 101(1), 397–428 (2014)CrossRefGoogle Scholar
  22. 22.
    Daud, A., Li, J., Zhou, L., Muhammad, F.: Conference mining via generalized topic modeling. In: Proceedings of the European Conference on Machine Learning (ECML) and Principles and Practice of Knowledge Discovery in Databases (PKDD) 2009, Bled, Slovenia, September 7–11, (2009)Google Scholar
  23. 23.
    Wuehrer, G.A., Smejkal, A.E.: The knowledge domain of the academy of international business studies (AIB) conferences: a longitudinal scientometric perspective for the years 2006–2011. Scientometrics 95(2), 541–561 (2013)CrossRefGoogle Scholar
  24. 24.
    Lisée, C., Larivière, V., Archambault, É.: Conference proceedings as a source of scientific information: a bibliometric analysis. J. Am. Soc. Inf. Sci. Technol. 59(11), 1776–1784 (2008)CrossRefGoogle Scholar
  25. 25.
    Martins, W.S., Gonçalves, M.A., Laender, A.H.F., Ziviani, N.: Assessing the quality of scientific conferences based on bibliographic citations. Scientometrics 83(1), 133–155 (2009)CrossRefGoogle Scholar
  26. 26.
    Jahja, I., Effendy, S., Yap, R.H.C.: Experiments on rating conferences with CORE and DBLP. DLib agazine (Online). (2014). Accessed 03 Jan 2015
  27. 27.
    Bellotti, E.: Getting funded. Multi-level network of physicists in Italy. Soc. Networks 34(2), 215–229 (2012)CrossRefGoogle Scholar
  28. 28.
    Singh, V., Perdigones, A., Garcia, J.L., Cañas-Guerroro, I., Mazarrón, F.R.: Analyzing worldwide research in hardware architecture, 1997–2011. Commun. ACM 58(1), 76–85 (2014)CrossRefGoogle Scholar
  29. 29.
    Catalá-López, F., Alonso-Arroyo, A., Aleixandre-Benavent, R., Ridao, M., Bolaños, M., García-Altés, A., Sanfélix-Gimeno, G., Peiró, S.: Coauthorship and institutional collaborations on cost-effectiveness analyses: a systematic network analysis. PLoS One 7(5), e38012 (2012)CrossRefGoogle Scholar
  30. 30.
    Liu, X., Bollen, J., Nelson, M.L., de Sompel, H.: Co-authorship networks in the digital library research community. Inf. Process. Manag. 41(6), 1462–1480 (2005)CrossRefGoogle Scholar
  31. 31.
    Tsakonas, G., Mitrelis, A., Papachristopoulos, L., Papatheodorou, C.: An exploration of the research trends in the digital library evaluation domain. In: Proceedings of the 12th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 347–348 (2012)Google Scholar
  32. 32.
    Afiontzi, E., Kazadeis, G., Papachristopoulos, L., Sfakakis, M., Tsakonas, G., Papatheodorou, C.: Charting the digital library evaluation domain with a semantically enhanced mining methodology. In: Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries, pp. 125–134 (2013)Google Scholar
  33. 33.
    Fox, C.: A stop list for general text. SIGIR Forum 24(1–2), 19–21 (1989)CrossRefGoogle Scholar
  34. 34.
    Mimno, D.: jsLDA: an implementation of Latent Dirichlet Allocation in javascript (Online).
  35. 35.
    Church, K.W., Hanks, P.: Word association norms, mutual information, and lexicography. Comput. Linguist. 16(1), 22–29 (1990)Google Scholar
  36. 36.
    Webometric Analyst 2.0, Statistical Cybermetrics Research Group (Online). (2016)
  37. 37.
    Haustein, S., Larivière, V., Thelwall, M., Amyot, D., Peters, I.: Tweets vs. Mendeley readers: How do these two social media metrics differ. IT Inf. Technol. 56(5), 2–13 (2014)Google Scholar
  38. 38.
    Kraker, P., Weissensteiner, P., Brusilovsky, P., Weißensteiner, P., Brusilovsky, P.: Altmetrics-based visualizations depicting the evolution of a knowledge domain. In: Proceedings of the 19th International Conference on Science and Technology Indicators (STI 2014), vol. 8010, pp. 330–333 (2014)Google Scholar
  39. 39.
    Papachristopoulos, L., Papatheodorou, C., Mitrelis, A., Tsakonas, G.: Where and how knowledge on digital library evaluation spreads? A case study on conference literature. In: Proceedings of Libraries In the Digital Age 2014, Zadar, Croatia, 16–20 June (2014)Google Scholar
  40. 40.
    Freeman, L.C.: Centrality in social networks conceptual clarification. Soc. Networks 1(3), 215–239 (1978)CrossRefGoogle Scholar
  41. 41.
    Opsahl, T., Agneessens, F., Skvoretz, J.: Node centrality in weighted networks: Generalizing degree and shortest paths. Soc. Networks 32(3), 245–251 (2010)CrossRefGoogle Scholar
  42. 42.
    Watts, D.J., Strogatz, S.H.: Collective dynamics of ‘small-world’ networks. Nature 393(6684), 440–442 (1998)CrossRefGoogle Scholar
  43. 43.
    Mislove, A., Marcon, M., Gummadi, K.P., Druschel, P., Bhattacharjee, B.: Measurement and analysis of online social networks. In: Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement–IMC ’07, p. 29 (2007)Google Scholar
  44. 44.
    Cherven, K.: Mastering Gephi Network Visualization. PACKT Publishing, Birmingham (2015)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  1. 1.Department of Archives, Library Science and MuseologyIonian UniversityCorfuGreece
  2. 2.Library and Information CenterUniversity of PatrasPatrasGreece
  3. 3.Department of MathematicsUniversity of PatrasPatrasGreece
  4. 4.Department of InformaticsAthens University of Economics and BusinessAthensGreece

Personalised recommendations