EPSA Philosophy of Science: Amsterdam 2009 pp 123-135 | Cite as

# Meta-analysis as Judgment Aggregation

## Abstract

For several decades now, a new inductive method, meta-analysis, is all the rage in social and medical sciences. Meta-analyses, that is, statistical reviews of the results of primary studies concerning a test hypothesis, set new standards of excellence on what counts as strong evidence. In the current prevailing mood in medical and behavioural sciences, it is only a properly conducted, up-to-date meta-analysis that licenses detachment of hypotheses from the host of evidential claims made in individual studies, which claims may be inconclusive or contradictory with each other. My goal in this chapter is to see the extent to which judgment aggregation methods subsume meta-analytic ones. To this end, I derive a generalized version of the classical Condorcet Jury Theorem, and I contend that one can model at least some meta-analytic procedures using this theorem.

## Keywords

Rational Agent Primary Study Primary Research Rational Choice Theory Background Assumption## References

- Cartwright, Nancy. 2007.
*Hunting causes and using them*. Cambridge: Cambridge University Press.CrossRefGoogle Scholar - Cooper, Harris, and L.V. Hedges, eds. 1994.
*The handbook of research synthesis*. New York: Russell Sage Foundation.Google Scholar - Dietrich, Franz, and Christian List. 2004. A model of jury decisions where all jurors have the same evidence.
*Synthese*142(2): 175–202.CrossRefGoogle Scholar - Donoho, David L. 2000. High-dimensional data analysis: The curses and blessings of dimensionality. Lecture delivered at the American Mathematical Society Conference, “Mathematical Challenges of the Twenty-first Century”. http://www-stat.stanford.edu/~donoho/Lectures/AMS2000/AMS2000.html
- Duval, S., and R. Tweedie. 2000. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis.
*Biometrics*56: 455–463.CrossRefGoogle Scholar - Fiske, Donald W. 1983. The meta-analytic revolution in outcome research.
*Journal of Consulting and Clinical Psychology*51(1): 65–70.CrossRefGoogle Scholar - Haenni, R., and S. Hartmann. 2006. Modeling partially reliable information sources: A general approach based on Dempster-Shafer theory.
*Information Fusion*7: 361–379.CrossRefGoogle Scholar - Hammersley, Martyn. 2001. On ‘systematic’ reviews of research literatures: A ‘narrative’ response to Evans&Benefield.
*British Educational Research Journal*27(5): 543–554.CrossRefGoogle Scholar - Hawthorne, James, unpublished manuscript (circulated beginning 2001). Voting in search of the public good: The probabilistic logic of majority judgments. http://faculty-staff.ou.edu/H/James.A.Hawthorne-1/Hawthorne-{}-Jury-Theorems.pdf
- Hedges, Larry V., and Ingram Olkin. 1985.
*Statistical methods for meta-analysis*. San Diego, CA: Academic.Google Scholar - Higgins, J.P.T., and S. Green, eds. 2009.
*Cochrane handbook for systematic reviews of interventions*Version 5.0.2 [updated September 2009]. The Cochrane Collaboration, 2009. http://www.cochrane-handbook.org. - Hunt, Morton. 1997.
*How science takes stock: The story of meta-analysis*. New York: Russell Sage Foundation.Google Scholar - Hunter, J.E., and F.L. Schmidt. 1990.
*Methods of meta-analysis*. Newbury Park: Sage.Google Scholar - Ioannidis, John P.A. 2005. Contradicted and initially stronger effects in highly cited clinical research.
*JAMA*294(2): 218–228.CrossRefGoogle Scholar - Lewis, D. 1980. A subjectivist’s guide to objective chance. In
*Studies in inductive logic and probability*, Vol. II, ed. Richard C. Jeffrey. Berkeley, CA, and Los Angeles: University of California Press.Google Scholar - Light, Richard J., and David B. Pillemer. 1984.
*Summing up: The science of reviewing research*. Cambridge, MA, and London: Harvard University Press.Google Scholar - Resnick, Sidney I. 1998.
*A probability path*. Basel: Birkhauser Verlag AG.Google Scholar - Rubin, Donald B. 1992. Meta-analysis: Literature synthesis or effect-size surface estimation?
*Journal of Educational and Behavioral Statistics*17: 363–374.CrossRefGoogle Scholar - Sackett, David L. et al. 2000.
*Evidence-based medicine*. Edinburgh: Churchill Livingstone.Google Scholar - Schmidt, Frank L. 1996. Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers.
*Psychological Methods*1(2): 115–129.CrossRefGoogle Scholar - Smith, Mary L., and Gene V. Glass. 1977. Meta-analysis of psychotherapy outcome studies.
*American Psychologist*32(9): 752–760.CrossRefGoogle Scholar