Abstract
Background
The aim of the present study was to determine empirically which electronic databases contribute best to a literature search in surgical systematic reviews.
Methods
For ten published systematic reviews, the systematic literature searches were repeated in the databases MEDLINE, Web of Science, CENTRAL, and EMBASE. On the basis of these reviews, a gold standard set of eligible articles was created. Recall (%), precision (%), unique contribution (%), and numbers needed to read (NNR) were calculated for each database, as well as for searches of citing references and of the reference lists of related systematic reviews (hand search).
Results
CENTRAL yielded the highest recall (88.4%) and precision (8.3%) for randomized controlled trials (RCT), MEDLINE for non-randomized studies (NRS; recall 92.6%, precision 5.2%). The most effective combination of two databases plus hand searching for RCT was MEDLINE/CENTRAL (98.6% recall, NNR 97). Adding EMBASE marginally increased the recall to 99.3%, but with an NNR of 152. For NRS, the most effective combination was MEDLINE/Web of Science (99.5% recall, NNR 60).
Conclusions
For surgical systematic reviews, the optimal literature search for RCT employs MEDLINE and CENTRAL. For surgical systematic reviews of NRS, Web of Science instead of CENTRAL should be searched. EMBASE does not contribute substantially to reviews with a surgical intervention.
Similar content being viewed by others
Abbreviations
- CE:
-
CENTRAL
- EM:
-
EMBASE
- HS:
-
Hand search
- ME:
-
MEDLINE accessed via Pubmed
- NNR :
-
Numbers needed to read
- NRS:
-
Non-randomized study/studies
- RCT:
-
Randomized controlled trial(s)
- SR :
-
Systematic review(s)
- WoS:
-
Web of Science
References
OCEBM Levels of Evidence Working Group*: “The Oxford 2011 levels of evidence”. Oxford Centre for Evidence-Based Medicine (*OCEBM Levels of Evidence Working Group*=Jeremy Howick ICJLL, Paul Glasziou, Trish Greenhalgh, Carl Heneghan, Alessandro Liberati, Ivan Moschetti, Bob Phillips HT, Olive Goddard and Mary Hodgkinson). http://www.cebm.net/index.aspx?o=5653. Accessed 5 Jul 2016
McKenzie S, Mailey B, Artinyan A, Kim J, Ellenhorn JD (2010) The incidence and outcomes of pancreatectomy in patients with metastatic pancreatic adenocarcinoma. JOP: Journal of the Pancreas [Electric Resource] 11(4):341–347
Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097
Irwig L, Tosteson AN, Gatsonis C, Lau J, Colditz G, Chalmers TC, Mosteller F (1994) Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med 120(8):667–676. https://doi.org/10.7326/0003-4819-120-8-199404150-00008
Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E (2015) Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking. Syst Rev 4(1):82. https://doi.org/10.1186/s13643-015-0074-7
Betran AP, Say L, Gulmezoglu AM, Allen T, Hampson L (2005) Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol 5(1):6. https://doi.org/10.1186/1471-2288-5-6
Ogilvie D, Hamilton V, Egan M, Petticrew M (2005) Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health 59(9):804–808. https://doi.org/10.1136/jech.2005.034181
Royle P, Bain L, Waugh N (2005) Systematic reviews of epidemiology in diabetes: finding the evidence. BMC Med Res Methodol 5(1):2. https://doi.org/10.1186/1471-2288-5-2
Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, St John PD, Viola R, Raina P (2003) Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol 56(10):943–955. https://doi.org/10.1016/S0895-4356(03)00110-0
Vassar M, Carr B, Kash-Holley M, DeWitt E, Koller C, Day J, Day K, Herrmann D, Holzmann M (2015) Database choices in endocrine systematic reviews. J Med Libr Assoc 103(4):189–192. https://doi.org/10.3163/1536-5050.103.4.005
Contin P, Goossen K, Grummich K, Jensen K, Schmitz-Winnenthal H, Buchler MW et al (2013) ENERgized vessel sealing systems versus CONventional hemostasis techniques in thyroid surgery—the ENERCON systematic review and network meta-analysis. Langenbeck’s Arch Surg 398(8):1039–1056. https://doi.org/10.1007/s00423-013-1137-7
Harnoss JC, Ulrich AB, Harnoss JM, Diener MK, Buchler MW, Welsch T (2014) Use and results of consensus definitions in pancreatic surgery: a systematic review. Surgery 155(1):47–57. https://doi.org/10.1016/j.surg.2013.05.035
Harnoss JM, Harnoss JC, Diener MK, Contin P, Ulrich AB, Buchler MW et al (2014) Portal annular pancreas: a systematic review of a clinical challenge. Pancreas 43(7):981–986. https://doi.org/10.1097/MPA.0000000000000186
Mehrabi A, Fischer L, Hafezi M, Dirlewanger A, Grenacher L, Diener MK, Fonouni H, Golriz M, Garoussi C, Fard N, Rahbari NN, Werner J, Büchler MW (2014) A systematic review of localization, surgical treatment options, and outcome of insulinoma. Pancreas 43(5):675–686. https://doi.org/10.1097/MPA.0000000000000110
Huttner FJ, Koessler-Ebs J, Hackert T, Ulrich A, Buchler MW, Diener MK (2015) Meta-analysis of surgical outcome after enucleation versus standard resection for pancreatic neoplasms. Br J Surg 102(9):1026–1036. https://doi.org/10.1002/bjs.9819
Huttner FJ, Tenckhoff S, Jensen K, Uhlmann L, Kulu Y, Buchler MW et al (2015) Meta-analysis of reconstruction techniques after low anterior resection for rectal cancer. Br J Surg 102(7):735–745. https://doi.org/10.1002/bjs.9782
Klaiber U, Probst P, Knebel P, Contin P, Diener MK, Buchler MW et al (2015) Meta-analysis of complication rates for single-loop versus dual-loop (Roux-en-Y) with isolated pancreaticojejunostomy reconstruction after pancreaticoduodenectomy. Br J Surg 102(4):331–340. https://doi.org/10.1002/bjs.9703
Loffler T, Rossion I, Goossen K, Saure D, Weitz J, Ulrich A et al (2015) Hand suture versus stapler for closure of loop ileostomy—a systematic review and meta-analysis of randomized controlled trials. Langenbeck's Arch Surg 400(2):193–205. https://doi.org/10.1007/s00423-014-1265-8
Mehrabi A, Hafezi M, Arvin J, Esmaeilzadeh M, Garoussi C, Emami G, Kössler-Ebs J, Müller-Stich BP, Büchler MW, Hackert T, Diener MK (2015) A systematic review and meta-analysis of laparoscopic versus open distal pancreatectomy for benign and malignant lesions of the pancreas: it's time to randomize. Surgery 157(1):45–55. https://doi.org/10.1016/j.surg.2014.06.081
Muller-Stich BP, Kenngott HG, Gondan M, Stock C, Linke GR, Fritz F et al (2015) Use of mesh in laparoscopic paraesophageal hernia repair: a meta-analysis and risk-benefit analysis. PLoS One 10(10):e0139547. https://doi.org/10.1371/journal.pone.0139547
Muller-Stich BP, Senft JD, Warschkow R, Kenngott HG, Billeter AT, Vit G et al (2015) Surgical versus medical treatment of type 2 diabetes mellitus in nonseverely obese patients: a systematic review and meta-analysis. Ann Surg 261(3):421–429. https://doi.org/10.1097/SLA.0000000000001014
Probst P, Huttner FJ, Klaiber U, Knebel P, Ulrich A, Buchler MW et al (2015) Stapler versus scalpel resection followed by hand-sewn closure of the pancreatic remnant for distal pancreatectomy. Cochrane Database Syst Rev 11:Cd008688
Huttner FJ, Fitzmaurice C, Schwarzer G, Seiler CM, Antes G, Buchler MW et al (2016) Pylorus-preserving pancreaticoduodenectomy (pp Whipple) versus pancreaticoduodenectomy (classic Whipple) for surgical treatment of periampullary and pancreatic carcinoma, Cd006053. Cochrane Database Syst Rev 2
Huttner FJ, Mihaljevic AL, Hackert T, Ulrich A, Buchler MW, Diener MK (2016) Effectiveness of Tachosil((R)) in the prevention of postoperative pancreatic fistula after distal pancreatectomy: a systematic review and meta-analysis. Langenbeck’s Arch Surg 401(2):151–159. https://doi.org/10.1007/s00423-016-1382-7
Kossler-Ebs JB, Grummich K, Jensen K, Huttner FJ, Muller-Stich B, Seiler CM et al (2016) Incisional hernia rates after laparoscopic or open abdominal surgery—a systematic review and meta-analysis. World J Surg 40(10):2319–2330. https://doi.org/10.1007/s00268-016-3520-3
Probst P, Grummich K, Harnoss JC, Huttner FJ, Jensen K, Braun S et al (2016) Placebo-controlled trials in surgery: a systematic review and meta-analysis. Medicine (Baltimore) 95(17):e3516. https://doi.org/10.1097/MD.0000000000003516
Probst P, Knebel P, Grummich K, Tenckhoff S, Ulrich A, Buchler MW et al (2016) Industry bias in randomized controlled trials in general and abdominal surgery: an empirical study. Ann Surg 264(1):87–92. https://doi.org/10.1097/SLA.0000000000001372
Fraser C, Murray A, Burr J (2006) Identifying observational studies of surgical interventions in MEDLINE and EMBASE. BMC Med Res Methodol 6(1):41. https://doi.org/10.1186/1471-2288-6-41
Beyer FR, Wright K (2013) Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Inf Libr J 30(1):49–58. https://doi.org/10.1111/hir.12009
Booth A (2010) How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. Int J Technol Assess Health Care 26(4):431–435. https://doi.org/10.1017/S0266462310000966
Egger M, Juni P, Bartlett C, Holenstein F, Sterne J (2003) How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess 7(1):1–76
Bachmann LM, Estermann P, Kronenberg C, ter Riet G (2003) Identifying diagnostic accuracy studies in EMBASE. J Med Libr Assoc 91(3):341–346
Journal Citation Reports® (Thomson Reuters, 2016)
Bhangu A, Nepogodiev D, Gupta A, Torrance A, Singh P (2012) Systematic review and meta-analysis of outcomes following emergency surgery for Clostridium difficile colitis. Br J Surg 99(11):1501–1513. https://doi.org/10.1002/bjs.8868
Constantinides VA, Christakis I, Touska P, Palazzo FF (2012) Systematic review and meta-analysis of retroperitoneoscopic versus laparoscopic adrenalectomy. Br J Surg 99(12):1639–1648. https://doi.org/10.1002/bjs.8921
Fung AK, Aly EH (2012) Systematic review of single-incision laparoscopic colonic surgery. Br J Surg 99(10):1353–1364. https://doi.org/10.1002/bjs.8834
Marimuthu K, Varadhan KK, Ljungqvist O, Lobo DN (2012) A meta-analysis of the effect of combinations of immune modulating nutrients on outcome in patients undergoing major open gastrointestinal surgery. Ann Surg 255(6):1060–1068. https://doi.org/10.1097/SLA.0b013e318252edf8
O’Reilly EA, Burke JP, O'Connell PR (2012) A meta-analysis of surgical morbidity and recurrence after laparoscopic and open repair of primary unilateral inguinal hernia. Ann Surg 255(5):846–853. https://doi.org/10.1097/SLA.0b013e31824e96cf
Shabanzadeh DM, Sorensen LT (2012) Laparoscopic surgery compared with open surgery decreases surgical site infection in obese patients: a systematic review and meta-analysis. Ann Surg 256(6):934–945. https://doi.org/10.1097/SLA.0b013e318269a46b
Venkat R, Edil BH, Schulick RD, Lidor AO, Makary MA, Wolfgang CL (2012) Laparoscopic distal pancreatectomy is associated with significantly less overall morbidity compared to the open technique: a systematic review and meta-analysis. Ann Surg 255(6):1048–1059. https://doi.org/10.1097/SLA.0b013e318251ee09
Xiong JJ, Altaf K, Mukherjee R, Huang W, WM H, Li A et al (2012) Systematic review and meta-analysis of outcomes after intraoperative pancreatic duct stent placement during pancreaticoduodenectomy. Br J Surg 99(8):1050–1061. https://doi.org/10.1002/bjs.8788
Zeng YK, Yang ZL, Peng JS, Lin HS, Cai L (2012) Laparoscopy-assisted versus open distal gastrectomy for early gastric cancer: evidence from randomized and nonrandomized clinical trials. Ann Surg 256(1):39–52. https://doi.org/10.1097/SLA.0b013e3182583e2e
Trastulli S, Cirocchi R, Desiderio J, Guarino S, Santoro A, Parisi A, Noya G, Boselli C (2013) Systematic review and meta-analysis of randomized clinical trials comparing single-incision versus conventional laparoscopic cholecystectomy. Br J Surg 100(2):191–208. https://doi.org/10.1002/bjs.8937
Boeker M, Vach W, Motschall E (2013) Google scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Med Res Methodol 13(1). https://doi.org/10.1186/1471-2288-13-131
Levay P, Ainsworth N, Kettle R, Morgan A (2016) Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Research Synthesis Methods 7(1):34–45. https://doi.org/10.1002/jrsm.1158
Glas AS, Lijmer JG, Prins MH, Bonsel GJ, Bossuyt PM (2003) The diagnostic odds ratio: a single indicator of test performance. J Clin Epidemiol 56(11):1129–1135. https://doi.org/10.1016/S0895-4356(03)00177-X
Hausner E, Waffenschmidt S, Kaiser T, Simon M (2012) Routine development of objectively derived search strategies. Syst Rev 1(1):19. https://doi.org/10.1186/2046-4053-1-19
Rathbone J, Hoffmann T, Glasziou P (2015) Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev 4(1):80. https://doi.org/10.1186/s13643-015-0067-6
Shemilt I, Simon A, Hollands GJ, Marteau TM, Ogilvie D, O'Mara-Eves A et al: Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews. Res Synth Methods 2014, 5(1):31–49, DOI: https://doi.org/10.1002/jrsm.1093
Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH (2010) Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics 11(1):55. https://doi.org/10.1186/1471-2105-11-55
O'Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S (2015) Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev 4(1):5. https://doi.org/10.1186/2046-4053-4-5
Polisena J, Garritty C, Umscheid CA, Kamel C, Samra K, Smith J, Vosilla A (2015) Rapid review summit: an overview and initiation of a research agenda. Syst Rev 4:111. https://doi.org/10.1186/s13643-015-0111-6
Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, Perrier L, Hutton B, Moher D, Straus SE (2015) A scoping review of rapid review methods. BMC Med 13(1):224. https://doi.org/10.1186/s12916-015-0465-6
Lefebvre C, Eisinga A, McDonald S, Paul N (2008) Enhancing access to reports of randomized trials published world-wide—the contribution of EMBASE records to the Cochrane Central Register of Controlled Trials (CENTRAL) in the Cochrane Library. Emerg Themes Epidemiol 5(1):13. https://doi.org/10.1186/1742-7622-5-13
Hallam E, Plaice C (1999) An evaluation of EMBASE within the NHS: findings of the Database Access Project working partnership to extend the knowledge base of healthcare. Health Libr Rev 16(3):192–203. https://doi.org/10.1046/j.1365-2532.1999.0225a-1.x
Robinson KA (2005) Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol 58(3):320; author reply 1. https://doi.org/10.1016/j.jclinepi.2004.08.001
Sampson M, Tetzlaff J, Urquhart C (2011) Precision of healthcare systematic review searches in a cross-sectional sample. Research Synthesis Methods 2(2):119–125. https://doi.org/10.1002/jrsm.42
Sampson M, McGowan J (2011) Inquisitio validus index Medicus: a simple method of validating MEDLINE systematic review searches. Research Synthesis Methods 2(2):103–109. https://doi.org/10.1002/jrsm.40
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7(1):10. https://doi.org/10.1186/1471-2288-7-10
Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C (2009) An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol 62(9):944–952. https://doi.org/10.1016/j.jclinepi.2008.10.012
Wright K, Golder S, Rodriguez-Lopez R (2014) Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol 14(1):73. https://doi.org/10.1186/1471-2288-14-73
Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B (2016) The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol 16(1):127. https://doi.org/10.1186/s12874-016-0232-1
Whiting P, Westwood M, Burke M, Sterne J, Glanville J (2008) Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol 61(4):357–364. https://doi.org/10.1016/j.jclinepi.2007.05.013
Reveiz L, Ospina E, Zorrilla AF (2004) Should we consider Embase in Latin America? J Clin Epidemiol 57(8):866; author reply 7-8–866; author reply 868. https://doi.org/10.1016/j.jclinepi.2004.02.005
Egger M, Smith GD (1998) Bias in location and selection of studies. BMJ 316(7124):61–66. https://doi.org/10.1136/bmj.316.7124.61
McKibbon KA, Haynes RB, Dilks CJ, Ramsden MF, Ryan NC, Baker L et al (1990) How good are clinical MEDLINE searches? A comparative study of clinical end-user and librarian searches. Comput Biomed Res 23(6):583–593. https://doi.org/10.1016/0010-4809(90)90042-B
For an up-to-date list of search filters, see: The InterTASC Information Specialists’ Sub-Group Search Filter Resource, Search Filters by Study Design. https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/filters-to-identify-randomized-controlled-trials-and. Accessed 4 Jul 2016. Scottish Intercollegiate Guidelines Network (SIGN) search filters, http://www.sign.ac.uk/methodology/filters.html. Accessed 4 Jul 2016
For a non-validated Web of Science RCT filter, see: University of Alberta Libraries, Systematic Reviews-Searching the Literature, Randomized Controlled Trial Filters, Web of Science. http://guides.library.ualberta.ca/c.php?g=248586&p=1655962. Accessed 4 Jul 2016
Funding
The resources and facilities of the University of Heidelberg were used in conducting this study. There was no external source of funding.
Author information
Authors and Affiliations
Contributions
MD and KGo conceived the work. KGo, KGr, PP, and MD designed the protocol. KGo, ST, and KGr acquired the data. KGo and ST performed the analysis. KGo, ST, PP, AM, MB, and MD interpreted the results. KGo and PP drafted the manuscript. ST, KGr, AM, MB, and MD revised it critically. All authors approved the final version of the manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Availability of data and material
The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.
Competing interests
The authors declare that they have no competing interests.
Sources of financial support
The resources and facilities of the University of Heidelberg were used in conducting this study. There was no external source of funding.
Electronic supplementary material
Additional file 1
Study protocol (PDF 547 kb).
Additional file 2
(DOCX 37 kb).
Additional file 3
(DOCX 31 kb).
Additional file 4
(DOCX 29 kb).
Additional file 5
(DOCX 33 kb).
Additional file 6
(DOCX 34 kb).
Additional file 7
(DOCX 36 kb).
Additional file 8
(DOCX 35 kb).
Rights and permissions
About this article
Cite this article
Goossen, K., Tenckhoff, S., Probst, P. et al. Optimal literature search for systematic reviews in surgery. Langenbecks Arch Surg 403, 119–129 (2018). https://doi.org/10.1007/s00423-017-1646-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00423-017-1646-x