, Volume 33, Issue 1, pp 5–11 | Cite as

Bidirectional Citation Searching to Completion: An Exploration of Literature Searching Methods

Practical Application


Literature reviews underpin the majority of research projects in the health sciences, and yet relatively little analysis has been published as to the most appropriate method to identify relevant literature, outside of specialist information journals. The method of applying keyword search queries to bibliographic databases using Boolean logic dominates literature reviews due to its easy application to the major online databases. However, it is recognised increasingly as being problematic where the research question cannot be clearly defined or requires an element of exploration, due to its reliance on author’s use of titling and keywords and is unable to identify topics other than those defined in the search query. This paper discusses the relative merits of a systematic citation searching approach as both an alternative and a concurrent method to keyword searching. A method of citation searching, both forwards and backwards, which is iterated to form a closed loop solution, is discussed. An illustrative example is presented of both methods, applying them to the topic of the UK National Institute for Health and Care Excellence (NICE) cost-effectiveness threshold. The case study finds the citation searching approach dominates the traditional keyword searching approach, finding 76 papers of relevance, including all 15 found by the alternative approach. Conceptually, and in the example presented, it is demonstrated that the proposed method can represent a dominant strategy to the more traditional approach in some situations, highlighting that, wherever possible, it is preferential to employ multiple methods of searching. However, it is clear that a better understanding is required as to how we can most efficiently search the ever-growing sea of literature.



The authors would like to acknowledge the contributions of Su Golder, Prof. Karl Claxton and Prof. Mark Sculpher, all of whom helped with the conceptualisation and editing of this research.

Research was conducted as part of funding received from the Medical Research Council/National Institute for Health Research through its Methodology Research Programme (Award G0901498/1), both researchers having received funding from the sponsor for the duration of the project.

Author contribution

Sebastian Hinde conducted the literature searches and was first author of the paper. Eldon Spackman provided supervision during the literature searches, defining the aims of the reviews, as well as helping to draft the paper.

Conflict of interest

Neither author has any conflicts of interest to declare.


  1. 1.
    NICE, Guide to the methods of technology appraisal 2013. London: NICE, 2013.Google Scholar
  2. 2.
    Weinstein MC, et al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices-Modeling Studies. Value Health. 2003;6(1):9–17.PubMedCrossRefGoogle Scholar
  3. 3.
    Schlosser RW, et al. Use of information-seeking strategies for developing systematic reviews and engaging in evidence-based practice: the application of traditional and comprehensive Pearl Growing: a review. Int J Lang Commun Disord. 2006;41(5):567–82.PubMedCrossRefGoogle Scholar
  4. 4.
    Garfield E. Citation indexes for science. Science. 1955;122:108–11.PubMedCrossRefGoogle Scholar
  5. 5.
    Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005;331(7524):1064–5.PubMedCentralPubMedCrossRefGoogle Scholar
  6. 6.
    Higgins J, Green S. Cochrane handbook for systematic reviews for interventions. The Cochrane Collaboration, 2008.Google Scholar
  7. 7.
    Golder S, Loke Y, Zorzela L. Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews. Health Information Libraries Journal, 2014. Forthcoming.Google Scholar
  8. 8.
    Golder S, Loke YK, Zorzela L. Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011. J Clin Epidemiol. 2013;66(3):253–60.PubMedCrossRefGoogle Scholar
  9. 9.
    Kulkarni AV, et al. Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA. 2009;302(10):1092–6.PubMedCrossRefGoogle Scholar
  10. 10.
    Bakkalbasi N, et al. Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomed Digit Libr. 2006;3:7.PubMedCentralPubMedCrossRefGoogle Scholar
  11. 11.
    Robinson KA, et al. Citation networks of related trials are often disconnected: implications for bidirectional citation searches. J Clin Epidemiol. 2014;67(7):793–9.PubMedCrossRefGoogle Scholar
  12. 12.
    Claxton K, et al. Methods for the estimation of the NICE cost effectiveness threshold. CHE Research Paper, 2013. 81.Google Scholar
  13. 13.
    Papaioannou D, et al. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Libr J. 2010;27(2):114–22.PubMedCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  1. 1.Centre for Health EconomicsUniversity of YorkHeslingtonUK

Personalised recommendations