Background

Systematic reviews identify, appraise, and synthesize evidence that meets pre-specified criteria to answer a research question[1]. Comprehensive search strategies are critical; however, they can produce thousands of citations, with only a small number ultimately being included in the review[1–3]. This results in "inefficient use of valuable resources in terms of time involved in screening…a large number of records"[4]. Searching for information on the efficacy of an intervention that is mainly evaluated in observational studies is particularly challenging for indexers and searchers as methodologies are often poorly reported[5].

Prior research has indicated that although MEDLINE is capable of identifying the majority of relevant studies, searching this database alone for the purposes of conducting a comprehensive literature search is insufficient[6–18]. Interestingly, Golder and Loke found that MEDLINE was not necessary to retrieve all the relevant studies on drug adverse events[19]. However, the true value of the contribution of citations from other databases has often been questioned[20, 21]. There are varying levels of coverage for different topics and subjects in different databases. From their analysis of the effectiveness of different databases in identifying studies for the WHO systematic review of maternal morbidity and mortality, Betrán et al. concluded that there is a "need for extending the search to other sources beyond well-known electronic databases" when conducting systematic reviews on this topic[6]. By assessing what is not retrieved when only one database is searched for literature on injury prevention and safety promotion, Lawrence reported that "no single database included all of the relevant articles on any topic and the database with the broadest coverage differed by topic"[12]. Golder et al. also worked on a topic without standardized terminology and concluded that "even sensitive search strategies with a broad range of synonyms may not identify all the references meeting the inclusion criteria that are available in a particular database" suggesting that searching a variety of sources is probably the most effective way to compensate for this[18]. Lorenzetti et al. explored the extent of contributions made by databases other than MEDLINE in rapid health technology assessments, and reached a similar conclusion in 2014[17]. Likewise, Lemeshow et al. discovered that limiting the search for observational studies to one or two databases will retrieve only 60%–80% of relevant publications[8].

The purpose of our research was to audit the sources of studies used in an unpublished systematic review on the efficacy of ward closure as an infection outbreak control intervention measure, prepared by Wong H et al. (unpublished work) for Alberta Health Services. Our primary objective was to assess the effectiveness of searching additional scholarly biomedical databases beyond MEDLINE: what is the value of searching additional databases, and do the results from these databases impact the final conclusions of a systematic review of observational studies? Ultimately, we were unable to determine the impact of searching each database, but we did discover that hand searching is critical.

Methods

Data collection

We conducted a systematic review on the efficacy of ward closure as an infection control practice. In consultation with the primary investigators who are subject experts, databases were selected. The search strategy was developed by librarians. YK and SEP searched Ovid MEDLINE including In-Process & Other Non-Indexed Citations, Ovid Embase, CINAHL Plus, Cochrane Database of Systematic reviews (CDSR), LILACS, and IndMED for any study type discussing the implementation of ward closure in the case of an outbreak. CDSR was searched to identify studies that may not have been retrieved through the searches. The primary investigators were particularly concerned that the searches would retrieve international studies. LILACS, which indexes the scientific and technical literature of Latin America and the Caribbean, is recommended in the Cochrane Infectious Disease Group Guide to Search Strategy[22]. A Spanish speaking co-investigator provided lists of terms in Spanish and Portuguese and reviewed the abstracts in these languages. IndMED, which indexes selected peer-reviewed medical journals published in India, covering approximately 100 journals from 1985 onwards, was specifically requested by one of the primary investigators. YK and SEP also searched bibliographies of included articles, websites of the Centers for Disease Control and Prevention, the International Centre for Infectious Diseases, and the World Health Organization, and meeting abstracts from Community and Hospital Infection Control Association (2012 and 2013), International Consortium for Prevention & Infection Control (2012 and 2013), International Conference on Emerging Infectious Diseases (2010), Infectious Diseases Society of America (2012), European Society of Clinical Microbiology and Infectious Diseases (2012 and 2013), International Society for Infectious Diseases (2012), and Association of Medical Microbiology (2012). All citations were exported to RefWorks for bibliographic management. Studies were screened by two reviewers, including HW, first by title and abstract and then by full text using a pre-defined set of inclusion and exclusion criteria. To be included, a study had to meet the following inclusion criteria: (1) be set in tertiary acute care hospitals/facilities or long-term acute care hospitals; (2) indicate that complete or partial ward closure took place for at least 48 h for outbreak control. Publication types such as surveys, secondary data analysis, non-original reports, grey literature, editorials, letters, cost analyses, and reviews were excluded. No language or publication date range limit was applied.

Data analysis

Using the same methodology as Betrán et al., we recorded the source of each citation and tracked the number of references identified in each database as well as the number remaining after the removal of duplicates, all references included in the systematic review, and the databases that contained these references[6]. To determine whether the included studies were unique to the database from which they were retrieved or whether they were also present in other databases, we carried out title searches and recorded this information on a spreadsheet using Microsoft Excel 2010. We analyzed our search results using a relative recall method. As discussed by Lorenzetti et al., relative recall is "the proportion of…relevant articles that any specific system, filter, or tool retrieves"[17]. For our study, the relative recall of each source was calculated by dividing the number of included citations retrieved from each database by the total number of citations included in the systematic review. In addition, all of the selected databases were searched for all titles in the set of included articles. The sensitivity of each search was calculated by dividing the number of included citations retrieved from each database by the total number of included articles indexed in each database.

Results

A total of 2,095 unique citations were identified and screened for inclusion in the systematic review: 2,060 from database searching and 35 from hand searching and other sources. Out of these citations, 97 were included in the final review.

The yield of included studies from the selected databases ranged from 0% to 82.5% (Table 1). The MEDLINE search retrieved 80 (82.5%) of the articles included in the systematic review (Table 1). Out of these included articles, four were unique to MEDLINE (Table 1). However, 93 of the 97 articles included were indexed in MEDLINE (Table 1). The sensitivity of our search was 86% (Table 1). Our search in Embase was slightly more sensitive than MEDLINE; 91 of the included articles indexed in Embase, and searching the database also retrieved 82.5% of the articles included in the systematic review, which means that the sensitivity of our search in this database was 87.9% (Table 1). Seventy-six included articles overlapped between Embase and MEDLINE. Four were unique to Embase (Table 1). CINAHL retrieved 35 included articles and contributed four unique articles (Table 1). The sensitivity of the CINAHL search was 79.5% (Table 1). The LILACS search did not retrieve any of the included or unique articles but this database did index 75 of the 97 articles included in the systematic review (Table 1). Searching in IndMED did not retrieve any included articles or make any unique contributions (Table 1). Also, this database did not index any included article (Table 1). Even though all 97 included articles were indexed in at least one of the selected databases, eight were retrieved due to hand searching (Table 1).

Table 1 Number of articles identified by search strategies and included in systematic review
Full size table

Discussion

We assessed the efficacy of searching beyond MEDLINE by reviewing the literature and analyzing our results from a systematic review of ward closure as an infection control intervention to control outbreaks, prepared by Wong H et al. (unpublished work). As this systematic review did not find any published controlled studies, the authors concluded that the implementation of ward closure for control of outbreaks should neither be actively encouraged nor discouraged, or formalized as a policy.

While Betrán et al. concluded that it is important to search regional databases to locate studies from journals not indexed in MEDLINE, Egger et al. argued that doing so "raises the worrying possibility that rather than preventing bias through extensive literature searches, bias could be introduced by including trials of low methodological quality"[6, 20]. Egger et al. claimed, "trials that are difficult to locate tend to be of lower methodological quality than trials that are easily accessible and published in English," defining them as trials published in languages other than English, and in journals not indexed in MEDLINE[20].

Betrán et al. reported that their searches in MEDLINE, Embase, CINAHL, and LILACS were sensitive by 61.6%, 43.9%, 10.2%, and 5.3% respectively[6]. Our searches in MEDLINE, Embase, CINAHL, and LILACS, on the other hand, were sensitive by 86%, 87.9%, 79.5%, and 0% respectively (Table 1). In contrast to Betrán et al.’s recommendation about searching regional databases, the results of our data analysis in this specific subject area revealed that searching in databases other than MEDLINE, Embase, and CINAHL did not make such significant contributions; at only four unique citations each, Embase and CINAHL might also be considered to have contributed marginally. Our discovery of articles indexed in the databases but not retrieved by our search strategies supports Golder et al.’s findings that even the most sensitive search strategies will not retrieve all the relevant results when working with a subject lacking standardized terms and the "failure of search strategies…to identify all the relevant references available on each database"[18, 19].

The LILACS results were particularly interesting. While LILACS indexed 75 articles included in the systematic review, none of our search strategies retrieved them. We initially searched this database using the same strategies as the other databases; however, as this did not yield any relevant articles, we modified our search using a few keywords provided by a Spanish speaker on the review team. The unique and potentially relevant articles located did not meet the inclusion criteria so were excluded from the systematic review. The true value of searching in LILACS may lie with the ability to carry out a search in Spanish or Portuguese. Clark and Castro do not explicitly address searching in English, Spanish, or Portuguese when claiming that LILACS could have added further information to 70% of the systematic reviews they surveyed, arguing that the "database should be used as a routine source of studies in the preparation of SR"[23]. According to Egger et al., "systematic reviews that are based on a search of English language literature that is accessible in the major bibliographic databases will often produce results that are close to those obtained from reviews based on more comprehensive searches that are free of language restrictions"[20]. Future studies could investigate how to improve the specificity of a LILACS search and compare searching LILACS in English, Spanish, and Portuguese to confirm or refute Egger et al.’s claim.

One of the most noteworthy discoveries made during our data analysis was that database searches alone would have missed 8.3% of relevant citations. Although they were all indexed in one of the selected databases, we were only able to retrieve these citations through hand searching and reference checking. In general, this was because the focus of these articles was mostly on describing the overall experience of an infection outbreak and ward closure was not mentioned in the title, abstract, or keywords list. This suggests that time spent on "serendipitous means of identifying relevant information," rather than expanding the range of databases, was an efficient way to compensate for the citations that search in major databases failed to identify and was relevant to the topic area under review in this circumstance[18]. Specifically, strategies "such as…asking colleagues, pursuing references that look interesting, and simply being alert to serendipitous discovery," as Greenhalgh and Peacock argued, "may have a better yield per hour spent and are likely to identify important sources that would otherwise be missed"[24].

Although we demonstrate that the value in searching beyond the mainstream databases of MEDLINE, Embase, and CINAHL is marginal, our study does have limitations. In particular, as indicated previously, our review did not include any controlled study that could be used for a meta-analysis. Thus, quantifying the impact of each citation on the final result of the review and testing Betrán et al.’s or Egger et al.’s findings were not possible[6, 20]. In addition, because our conclusions are specific to one systematic review of an intervention that was only described in observational studies, outcomes of interest may vary when searching for studies and reviews of other topics.

Conclusions

Due to indexing and abstracting issues, using conventional protocol-driven search techniques in major bibliographic databases may lead to an inadvertent omission of significant articles in a systematic review of an intervention that requires extensive keyword searching. The results from our data analysis suggest that expanding the range of databases to search beyond MEDLINE, Embase, and CINAHL, however, may not be the most effective way to address this problem. Rather, redirecting effort into serendipitous discoveries may be a more efficient usage of the review team’s resources.