Skip to main content
SpringerLink
Log in

Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews

  • Published:
Journal of Experimental Criminology Aims and scope Submit manuscript

Abstract

A review based on a biased collection of studies is likely to produce biased conclusions. As such, a central component of a quality systematic review is a comprehensive document search. Three issues are explored in this paper that underpin the importance of such a search. First, the evidence of publication selection bias clearly establishes that peer-reviewed journal articles are more likely to show statistically significant results than less formally published studies. Second, the presence of grey literature studies that cannot be discovered through a search of bibliographic databases is demonstrated using Campbell Collaboration systematic reviews, and third, the logical and empirical problems with using publication status as a proxy for methodological quality are examined. More robust methods of identifying research conducted in the social sciences are needed if we are to take seriously the accumulation of knowledge, such as the development of a priori registries of research studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Braga, A. (2007). Hot spots policing. Campbell Collaboration Library, http://www.campbellcollaboration.org.

  • Callaham, M. L. Baxt, W. G. Waeckerle, J. F. & Wears, R. L. (1998). Reliability of editors' subjective quality ratings of peer-reviews of manuscripts. JAMA, 280(3), 229–231.

    Article  Google Scholar 

  • Cicchetti, D. (1991). The reliability of peer-review for manuscript and grant submission. Behavioral and Brain Sciences, 14(1), 119–186.

    Google Scholar 

  • Cooper, H. M. (1998). Synthesizing research: A guide for literature reviews. Thousand Oaks: Sage.

    Google Scholar 

  • Cooper, H. DeNeve, K. & Charlton, K. (1997). Finding the missing science: the fate of studies submitted for review by a human subjects committee. Psychological Methods, 2(4), 447–452.

    Article  Google Scholar 

  • De Angelis, C. Drazen, J. M. Frizelle, F. A. Haug, C. Hoey, J. Horton, R. et al. (2004). Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Annals of Internal Medicine, 141(6), 477–478.

    Google Scholar 

  • Dickersin, K. (2005). Publication bias: recognizing the problem, understanding its origins and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 11–33). Chichester: Wiley.

    Google Scholar 

  • Dickersin, K. Scherer, R. & Lefebvre, C. (1994). Systematic reviews: Identifying relevant studies for systematic reviews. BMJ, 309(6964), 1286–1291.

    Google Scholar 

  • Garrido, V., & Morales, L. A. (2007). Programs for serious (violent and chronic) juvenile offenders in secure corrections. Campbell Collaboration Library, http://www.campbellcollaboration.org.

  • Grayson, L. & Gomersall, A. (2003). A difficult business: Finding the evidence for social science reviews (ESRC Working Paper 19). London: ESRC UK Centre for Evidence Based Policy and Practice.

    Google Scholar 

  • Greenhalgh, T. & Peacock, R. (2005). Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ, 331, 1064–1065.

    Article  Google Scholar 

  • Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis. Statistical Science, 7, 246–255.

    Article  Google Scholar 

  • Higgins, J. P. & Green, S. (Eds.) (2008). Cochrane handbook for systematic reviews of interventions (5th ed.). The Cochrane Collaboration. Retrieved from http://www.cochrane-handbook.org/.

  • Hopewell, S. Clarke, M. Stewart, L. & Tieney, J. (2001). Time to publication for results of clinical trials. Cochrane Database of Methodological Reviews, 3, MR000011.

    Google Scholar 

  • Hopewell, S. McDonald, S. Clarke, M. J. & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews, 2, MR000010.

    Google Scholar 

  • Horton, R. & Smith, R. (1999). Time to register randomised trials. BMJ, 319, 856–866.

    Google Scholar 

  • Hunter, J. E. & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks: Sage.

    Google Scholar 

  • Iyengar, S. & Greenhouse, J. B. (1988). Selection models and the file drawer problem. Statistical Science, 3(1), 109–117.

    Article  Google Scholar 

  • Jefferson, T. Rudin, M. Folse, S. B. & Davidoff, F. (2006). Editorial peer review for improving the quality of repots of biomedical studies. Cochrane Database of Methodological Reviews, 1, MR000016.

    Google Scholar 

  • Killias, M., & Villettaz, P. (2006). The effects of custodial vs. non-custodial sentences on reoffending. Campbell Collaboration Library, http://www.campbellcollaboration.org.

  • Lipsey, M. W. & Wilson, D. B. (1993). The efficacy of psychological, educational, and behavioral treatment: confirmation from meta-analysis. American Psychologist, 48(12), 1181–1209.

    Article  Google Scholar 

  • Lipsey, M. W. & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks: Sage.

    Google Scholar 

  • Lipsey, M., Landenberger, N., & Wilson, S. J. (2007). Cognitive-behavioral programs for juvenile and adult offenders: a meta-analysis of controlled intervention studies. Campbell Collaboration Library, http://www.campbellcollaboration.org.

  • Littell, J. H. (2006). The case for multisystemic therapy: evidence or orthodoxy? Children and Youth Services Review, 28, 458–472.

    Article  Google Scholar 

  • Lum, C., Kennedy, L., & Sherley, A. (2006). The effectiveness of counter-terrorism strategies. Campbell Collaboration Library, http://www.campbellcollaboration.org.

  • Mazerolle, L., Soole, D. W., & Rombouts, S. (2007). Police-led drug enforcement strategies. Campbell Collaboration Library, http://www.campbellcollaboration.org.

  • McDougall, C., Cohen, M., Swaray, R., & Perry, A. (2008). Cost-benefit analysis and cost-effectiveness of sentencing: a systematic review of the literature. Campbell Collaboration Library, http://www.campbellcollaboration.org.

  • McLeod, B. D. & Weisz, J. R. (2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72(2), 235–251.

    Article  Google Scholar 

  • Mitchell, O. J., Wilson, D. B., & MacKenzie, D. L. (2006). Incarceration-based drug treatment programs. Campbell Collaboration Library, http://www.campbellcollaboration.org.

  • Petrosino, A., Turpin-Petrosino, C., & Buehler, J. (2005). ‘Scared Straight’ and other juvenile awareness programs for preventing delinquency. Campbell Collaboration Library, http://www.campbellcollaboration.org.

  • Reed, J. G. & Baxter, P. M. (2009). Using reference databases. In H. Cooper, L. V. Hedges & J. C. Valentine (Eds.), Handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation.

    Google Scholar 

  • Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641.

    Article  Google Scholar 

  • Rothstein, H. & Hopewell, S. (2009). Grey literature. In H. Cooper, L. V. Hedges & J. C. Valentine (Eds.), Handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation.

    Google Scholar 

  • Rothstein, H. Sutton, A. J. & Borenstein, M. (eds). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Chichester: Wiley.

    Google Scholar 

  • Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47(10), 1173–1181.

    Article  Google Scholar 

  • Smith, M. L. (1980). Publication bias and meta-analysis. Evaluation in Education, 4, 22–24.

    Article  Google Scholar 

  • Stern, J. M. & Simes, R. J. (1997). Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ, 315, 640–645.

    Google Scholar 

  • Sterne, J. A. C. Jüni, P. Schulz, K. F. Altman, D. G. Bartlett, C. & Egger, M. (2002). Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Statistics in Medicine, 21(11), 1513–1524.

    Article  Google Scholar 

  • Tonks, A. (2002). A clinical trials register for Europe. BMJ, 325, 1314–1315.

    Google Scholar 

  • Visher, C., Winterfield, L., & Coggeshall, M. (2006). The effects of non-custodial employment programs on the recidivism rates of ex-offenders. Campbell Collaboration Library, http://www.campbellcollaboration.org.

  • Weisburd, D. Lum, C. M. & Petrosino, A. (2001). Does research design affect study outcomes in criminal justice? Annals of the American Academy of Political and Social Science, 578, 50–70.

    Article  Google Scholar 

  • White, H. D. (2009). Scientific communication and literature retrieval. In H. Cooper, L. V. Hedges & J. C. Valentine (Eds.), Handbook of research synthesis and meta-analysis (2nd ed.). New York: Russell Sage Foundation.

    Google Scholar 

  • WHO (World Health Organization). Welcome to the WHO International Clinical Trials Registry Platform. Retrieved June 23, 2008, from http://www.who.int/ictrp/en/.

  • Wilson, D. B. & Lipsey, M. W. (2001). The role of method in treatment effectiveness research: evidence from meta-analysis. Psychological Methods, 6(4), 413–429.

    Article  Google Scholar 

  • Wilson, D. B., MacKenzie, D. L., & Mitchell, F. N. (2005). The effects of correctional boot camps on offending. Campbell Collaboration Library, http://www.campbellcollaboration.org.

Download references

Author information

Authors and Affiliations

  1. George Mason University, 10900 University Blvd., MS4F4, Manassas, VA, 20110, USA

    David B. Wilson

Authors
  1. David B. Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David B. Wilson.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wilson, D.B. Missing a critical piece of the pie: simple document search strategies inadequate for systematic reviews. J Exp Criminol 5, 429–440 (2009). https://doi.org/10.1007/s11292-009-9085-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11292-009-9085-5

Keywords

Search

Navigation