, Volume 112, Issue 2, pp 731–746 | Cite as

A relevance ranking method for citation-based search results

  • Christopher W. BelterEmail author


A growing number of researchers are exploring the use of citation relationships such as direct citation, bibliographic coupling, and co-citation for information retrieval in scientific databases and digital libraries. In this paper, I propose a method of ranking the relevance of citation-based search results to a set of key, or seed, papers by measuring the number of citation relationships they share with those key papers. I tested the method against 23 published systematic reviews and found that the method retrieved 87% of the studies included in these reviews. The relevance ranking approach identified a subset of the citation search results that comprised 27% of the total documents retrieved by the method, and 7% of the documents retrieved by these reviews, but that contained 75% of the studies included in these reviews. Additional testing suggested that the method may be less appropriate for reviews that combine literature in ways that are not reflected in the literature itself. These results suggest that this ranking method could be useful in a range of information retrieval contexts.


Bibliometrics Information retrieval Citation analysis Systematic reviews 



Thanks to Cindy Clark of the NIH Library for reviewing and editing a previous draft of this manuscript. Thanks also to two anonymous reviewers whose comments improved the manuscript.

Supplementary material

11192_2017_2406_MOESM1_ESM.docx (33 kb)
Supplementary material 1 (DOCX 15 kb)


  1. Belter, C. W. (2016). Citation analysis as a literature search method for systematic reviews. Journal of the Association for Information Science and Technology, 67(11), 2766–2777. doi: 10.1002/asi.23605.CrossRefGoogle Scholar
  2. Bichteler, J., & Eaton, E. A. (1980). The combined use of bibliographic coupling and cocitation for document retrieval. Journal of the American Society for Information Science, 31(4), 278–282. doi: 10.1002/asi.4630310408.CrossRefGoogle Scholar
  3. Carevic, Z., & Schaer, P. (2014). On the connection between citation-based and topical relevance ranking: results of a pretest using I search. Paper presented at the proceedings of the first workshop on bibliometric-enhanced information retrieval, Amsterdam, 13 April.Google Scholar
  4. Cribbin, T. (2014). Augmenting citation chain aggregation with article maps. Paper presented at the proceedings of the first workshop on knowledge maps and information retrieval, LondonGoogle Scholar
  5. Eto, M. (2013). Evaluations of context-based co-citation searching. Scientometrics, 94(2), 651–673. doi: 10.1007/s11192-012-0756-z.CrossRefGoogle Scholar
  6. Garfield, E. (1955). Citation Indexes for science: A new dimension in documentation through association of ideas. Science, 122(3159), 108–111. doi: 10.1126/science.122.3159.108.CrossRefGoogle Scholar
  7. Garfield, E. (1964). “Science Citation Index”–A new dimension in Indexing. Science, 144(3619), 649–654. doi: 10.1126/science.144.3619.649.CrossRefGoogle Scholar
  8. Heck, T., & Schaer, P. (2013). Performing informetric analysis on information retrieval test collections: Preliminary experiments in the physics domain. arXiv:1306.1743.
  9. Hinde, S., & Spackman, E. (2015). Bidirectional citation searching to completion: An exploration of literature searching methods. Pharmacoeconomics, 33(1), 5–11. doi: 10.1007/s40273-014-0205-3.CrossRefGoogle Scholar
  10. Horsley, T., Dingwall, O., & Sampson, M. (2011). Checking reference lists to find additional studies for systematic reviews. Cochrane database of systematic reviews (8), MR000026. doi: 10.1002/14651858.MR000026.pub2.
  11. Ingwersen, P. (1996). Cognitive perspectives of information retrieval interaction: elements of a cognitive IR theory. Journal of Documentation, 52(1), 3–50. doi: 10.1108/eb026960.CrossRefGoogle Scholar
  12. Janssens, A. C. J. W., & Gwinn, M. (2015). Novel citation-based search method for scientific literature: Application to meta-analyses. BMC Medical Research Methodology, 15(1), 1–11. doi: 10.1186/s12874-015-0077-z.CrossRefGoogle Scholar
  13. Larsen, B. (2002). Exploiting citation overlaps for information retrieval: Generating a boomerang effect from the network of scientific papers. Scientometrics, 54(2), 155–178. doi: 10.1023/A:1016011326300.CrossRefGoogle Scholar
  14. Linder, S. K., Kamath, G. R., Pratt, G. F., Saraykar, S. S., & Volk, R. J. (2015). Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. Journal of Clinical Epidemiology, 68(4), 412–417. doi: 10.1016/j.jclinepi.2014.10.008.CrossRefGoogle Scholar
  15. Mayr, P., & Scharnhorst, A. (2015). Scientometrics and information retrieval: Weak-links revitalized. Scientometrics, 102(3), 2193–2199. doi: 10.1007/s11192-014-1484-3.CrossRefGoogle Scholar
  16. National Research Council. (2011). Finding what works in health care: Standards for systematic reviews. Washington, DC: The National Academies Press.Google Scholar
  17. Ortuno, F., Rojas, I., Andrade-Navarro, M., & Fontaine, J.-F. (2013). Using cited references to improve the retrieval of related biomedical documents. BMC Bioinformatics, 14(1), 113. doi: 10.1186/1471-2105-14-113.CrossRefGoogle Scholar
  18. Pao, M. L., & Worthen, D. B. (1989). Retrieval effectiveness by semantic and citation searching. Journal of the American Society for Information Science, 40(4), 226–235. doi: 10.1002/(sici)1097-4571(198907)40:4<226:aid-asi2>;2-6.CrossRefGoogle Scholar
  19. Robinson, K. A., Dunn, A. G., Tsafnat, G., & Glasziou, P. (2014). Citation networks of related trials are often disconnected: Implications for bidirectional citation searches. Journal of Clinical Epidemiology, 67(7), 793–799. doi: 10.1016/j.jclinepi.2013.11.015.CrossRefGoogle Scholar
  20. Sci2 Team (2009). Science of science (Sci2) tool. Indiana University and SciTech strategies.Google Scholar
  21. Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., et al. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. British Medical Journal, 349, g7647. doi: 10.1136/bmj.g7647.CrossRefGoogle Scholar
  22. Van Rijsbergen, C. J. (1979). Information retrieval. London: Butterworths.zbMATHGoogle Scholar
  23. Wright, K., Golder, S., & Rodriguez-Lopez, R. (2014). Citation searching: A systematic review case study of multiple risk behaviour interventions. BMC Medical Research Methodology, 14, 73. doi: 10.1186/1471-2288-14-73.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2017

Authors and Affiliations

  1. 1.NIH Library, Office of Research ServicesNational Institutes of HealthBethesdaUSA

Personalised recommendations