Skip to main content
SpringerLink
Log in

Blending Bayesian and frequentist methods according to the precision of prior information with applications to hypothesis testing

  • Published:
Statistical Methods & Applications Aims and scope Submit manuscript

Abstract

The proposed minimax procedure blends strict Bayesian methods with p values and confidence intervals or with default-prior methods. Two applications to hypothesis testing bring some implications to light. First, the blended probability that a point null hypothesis is true is equal to the p value or a lower bound of an unknown posterior probability, whichever is greater. As a result, the p value is reported instead of any posterior probability in the case of complete prior ignorance but is ignored in the case of a fully known prior. In the case of partial knowledge about the prior, the possible posterior probability that is closest to the p value is used for inference. The second application provides guidance on the choice of methods used for small numbers of tests as opposed to those appropriate for large numbers. Whereas statisticians tend to prefer a multiple comparison procedure that adjusts each p value for small numbers of tests, large numbers instead lead many to estimate the local false discovery rate (LFDR), a posterior probability of hypothesis truth. Each blended probability reduces to the LFDR estimate if it can be estimated with sufficient accuracy or to the adjusted p value otherwise.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Alba R, Payton P, Fei Z, McQuinn R, Debbie P, Martin GB, Tanksley SD, Giovannoni JJ (2005) Transcriptome and selected metabolite analyses reveal multiple points of ethylene control during tomato fruit development. Plant Cell 17:2954–2965

    Article  Google Scholar 

  • Augustin T (2002) Expected utility within a generalized concept of probability—a comprehensive framework for decision making under ambiguity. Stat Pap 43(1):5–22

    Article  MATH  MathSciNet  Google Scholar 

  • Augustin T (2004) Optimal decisions under complex uncertainty—basic notions and a general algorithm for data-based decision making with partial prior knowledge described by interval probability. Zeitschrift fur Angewandte Mathematik und Mechanik 84(10–11):678–687

    Article  MATH  MathSciNet  Google Scholar 

  • Balch MS (2012) Mathematical foundations for a theory of confidence structures. Int J Approx Reason 53(7):1003–1019

    Article  MATH  MathSciNet  Google Scholar 

  • Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B 57:289–300

    MATH  MathSciNet  Google Scholar 

  • Berger JO (1984) Robustness of Bayesian analyses. Studies in Bayesian econometrics. North-Holland, Ch. The robust Bayesian viewpoint, pp 63–124

  • Berger JO (1985) Statistical decision theory and Bayesian analysis. Springer, New York

    Book  MATH  Google Scholar 

  • Berger JO (1990) Robust Bayesian analysis: sensitivity to the prior. J Stat Plan Inference 25:303–328

    Article  MATH  Google Scholar 

  • Berger JO, Sellke T (1987) Testing a point null hypothesis: the irreconcilability of p values and evidence. J Am Stat Assoc 82:112–122

    MATH  MathSciNet  Google Scholar 

  • Berger JO, Brown L, Wolpert R (1994) A unified conditional frequentist and Bayesian test for fixed and sequential simple hypothesis-testing. Ann Stat 22(4):1787–1807

    Article  MATH  MathSciNet  Google Scholar 

  • Berger JO, Bernardo J, Sun D (2009) The formal definition of reference priors. Ann Stat 37(2):905–938

    Article  MATH  MathSciNet  Google Scholar 

  • Bernardo JM (1979) Reference posterior distributions for Bayesian inference. J R Stat Soc B 41:113–147

    MATH  MathSciNet  Google Scholar 

  • Bickel DR (2011a) Estimating the null distribution to adjust observed confidence levels for genome-scale screening. Biometrics 67:363–370

    Article  MATH  MathSciNet  Google Scholar 

  • Bickel DR (2011b) A predictive approach to measuring the strength of statistical evidence for single and multiple comparisons. Can J Stat 39:610–631

    Article  MATH  MathSciNet  Google Scholar 

  • Bickel DR (2012a) Coherent frequentism: a decision theory based on confidence sets. Commun Stat Theory Methods 41:1478–1496

    Article  MATH  MathSciNet  Google Scholar 

  • Bickel DR (2012b) Controlling the degree of caution in statistical inference with the Bayesian and frequentist approaches as opposite extremes. Electron J Statist 6:686–709

    Article  MATH  MathSciNet  Google Scholar 

  • Bickel DR (2012c) Game-theoretic probability combination with applications to resolving conflicts between statistical methods. Int J Approx Reason 53:880–891

    Article  MATH  MathSciNet  Google Scholar 

  • Bickel DR (2013) Simple estimators of false discovery rates given as few as one or two p-values without strong parametric assumptions. Stat Appl Genet Mol Biol 12:529–543

    MathSciNet  Google Scholar 

  • Bickel DR (2014a) A fiducial continuum from confidence sets to empirical Bayes set estimates as the number of comparisons increases. Working Paper, University of Ottawa, deposited in uO Research at http://hdl.handle.net/10393/31898

  • Bickel DR (2014b) Model fusion and multiple testing in the likelihood paradigm: shrinkage and evidence supporting a point null hypothesis. Working paper, University of Ottawa, deposited in uO Research at http://hdl.handle.net/10393/31897

  • Bickel DR (2014c) Small-scale inference: empirical Bayes and confidence methods for as few as a single comparison. Int Stat Rev 82:457–476

    Article  MathSciNet  Google Scholar 

  • Bickel DR, Padilla M (2014) A prior-free framework of coherent inference and its derivation of simple shrinkage estimators. J Stat Plan Inference 145:204–221

    Article  MATH  MathSciNet  Google Scholar 

  • Carlin BP, Louis TA (2009) Bayesian methods for data analysis, 3rd edn. Chapman and Hall/CRC, New York

    Google Scholar 

  • Coletti C, Scozzafava R (2002) Probabilistic logic in a coherent setting. Kluwer, Amsterdam

    Book  Google Scholar 

  • Cover T, Thomas J (2006) Elements of information theory. Wiley, New York

    MATH  Google Scholar 

  • Cox DR (2006) Principles of statistical inference. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  • Csiszár I (1991) Why least squares and maximum entropy? an axiomatic approach to inference for linear inverse problems. Ann Stat 19:2032–2066

    Article  MATH  Google Scholar 

  • DasGupta A, Studden W (1989) Frequentist behavior of robust Bayes estimates of normal means. Stat Decis 7:333–361

    MATH  MathSciNet  Google Scholar 

  • Dawid AP, Stone M (1982) The functional-model basis of fiducial inference (with discussion). Ann Stat 10:1054–1074

    Article  MATH  MathSciNet  Google Scholar 

  • Efron B (2010a) Large-scale inference: empirical Bayes methods for estimation, testing, and prediction. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Efron B (2010b) Rejoinder to comments on B. Efron, “Correlated z-values and the accuracy of large-scale statistical estimates”. J Am Stat Assoc 105:1067–1069

    Article  MathSciNet  Google Scholar 

  • Fisher RA (1973) Statistical methods and scientific inference. Hafner Press, New York

    MATH  Google Scholar 

  • Fraser DAS (2004) Ancillaries and conditional inference. Stat Sci 19:333–351

    Article  MATH  MathSciNet  Google Scholar 

  • Fraser DAS, Reid N (1990) Discussion: an ancillarity paradox which appears in multiple linear regression. Ann Stat 18:503–507

    Article  Google Scholar 

  • Gärdenfors P, Sahlin N-E (1982) Unreliable probabilities, risk taking, and decision making. Synthese 53:361–386

  • Genest C, Zidek JV (1986) Combining probability distributions: a critique and an annotated bibliography. Stat Sci 1:114–135

    Article  MathSciNet  Google Scholar 

  • Gilboa I, Schmeidler D (1989) Maxmin expected utility with non-unique prior. J Math Econ 18(2):141–153

    Article  MATH  MathSciNet  Google Scholar 

  • Good IJ (1952) Rational decisions. J R Stat Soc B 14:107–114

    MathSciNet  Google Scholar 

  • Good IJ (1983) Good thinking: the foundations of probability and its applications. G—reference, information and interdisciplinary subjects series. University of Minnesota Press, USA

    Google Scholar 

  • Grünwald P, Dawid AP (2004) Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory. Ann Stat 32:1367–1433

    Article  MATH  Google Scholar 

  • Hannig J (2009) On generalized fiducial inference. Stat Sinica 19:491–544

    MATH  MathSciNet  Google Scholar 

  • Harremoës P, Topsøe F (2001) Maximum entropy fundamentals. Entropy 3(3):191–226

    Article  MathSciNet  Google Scholar 

  • Jaynes E (2003) Probability theory: the logic of science. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Jozani M, Marchand É, Parsian A (2012) Bayesian and robust bayesian analysis under a general class of balanced loss function. Stat Pap 53(1):51–60

    Article  MATH  Google Scholar 

  • Kakihara Y (1999) Abstract methods in information theory (series on multivariate analysis, volume 4). World Scientific, Singapore

    Google Scholar 

  • Kass RE, Wasserman L (1996) The selection of prior distributions by formal rules. J Am Stat Assoc 91:1343–1370

    Article  MATH  Google Scholar 

  • Kracík J (2011) Combining marginal probability distributions via minimization of weighted sum of Kullback–Leibler divergences. Int J Approx Reason 52:659–671

    Article  MATH  Google Scholar 

  • Lavine M (1991) Sensitivity in Bayesian statistics: the prior and the likelihood. J Am Stat Assoc 86:396–399

    Article  MATH  MathSciNet  Google Scholar 

  • Levi I (1986a) Hard choices: decision making under unresolved conflict. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Levi I (1986b) The paradoxes of Allais and Ellsberg. Econ Philos 2:23–53

    Article  Google Scholar 

  • Lindley DV (1957) A statistical paradox. Biometrika 44:187–192

    Article  MATH  MathSciNet  Google Scholar 

  • Liu C, Aitkin M (2008) Bayes factors: prior sensitivity and model generalizability. J Math Psychol 52:362–375

  • Nadarajah S, Bityukov S, Krasnikov N (2015) Confidence distributions: a review. Stat Methodol 22:23–46

    Article  MathSciNet  Google Scholar 

  • Neuhaus KL, von Essen R, Tebbe U, Vogt A, Roth M, Riess M, Niederer W, Forycki F, Wirtzfeld A, Maeurer W (1992) Improved thrombolysis in acute myocardial infarction with front-loaded administration of alteplase: results of the rt-PA-APSAC patency study (TAPS). J Am Coll Cardiol 19:885–891

    Article  Google Scholar 

  • Padilla M, Bickel D. R. (2012) Estimators of the local false discovery rate designed for small numbers of tests. Stat Appl Genet Mol Biol 11(5), art. 4

  • Paris JB (1994) The uncertain reasoner’s companion: a mathematical perspective. Cambridge University Press, New York

    MATH  Google Scholar 

  • Paris J, Vencovská A (1997) In defense of the maximum entropy inference process. Int J Approx Reason 17(1):77–103

    Article  MATH  Google Scholar 

  • Pfaffelhuber E (1977) Minimax information gain and minimum discrimination principle. In: Csiszár I, Elias P (eds) Topics in information theory. Vol. 16 of Colloquia Mathematica Societatis János Bolyai. János Bolyai Mathematical Society and North-Holland, Amsterdam, pp 493–519

    Google Scholar 

  • Polansky AM (2007) Observed confidence levels: theory and application. Chapman and Hall, New York

    Book  Google Scholar 

  • Robbins H (1951) Asymptotically subminimax solutions of compound statistical decision problems. Proc Second Berkeley Symp Math Stat Probab 1:131–148

    Google Scholar 

  • Robinson GK (1991) That BLUP is a good thing: the estimation of random effects. Stat Sci 6:15–32

    Article  MATH  Google Scholar 

  • Savage LJ (1954) The foundations of statistics. Wiley, New York

    MATH  Google Scholar 

  • Schweder T, Hjort NL (2002) Confidence and likelihood. Scand J Stat 29:309–332

    Article  MATH  MathSciNet  Google Scholar 

  • Seidenfeld T (1988) Decision theory without independence or without ordering: what is the difference? Econ Philos 4(2):267–290

    Article  Google Scholar 

  • Seidenfeld T (2004) A contrast between two decision rules for use with (convex) sets of probabilities: Gamma-maximin versus E-admissibility. Synthese 140(1–2):69–88

    Article  MathSciNet  Google Scholar 

  • Sellke T, Bayarri MJ, Berger JO (2001) Calibration of p values for testing precise null hypotheses. Am Stat 55:62–71

    Article  MATH  MathSciNet  Google Scholar 

  • Sidak Z (1967) Rectangular confidence regions for means of multivariate normal distributions. J Am Stat Assoc 62(318):626–633

    MATH  MathSciNet  Google Scholar 

  • Singh K, Xie M, Strawderman WE (2007) Confidence distribution (CD)—distribution estimator of a parameter. IMS Lect Notes Monogr Ser 2007(54):132–150

    Article  MathSciNet  Google Scholar 

  • Topsøe F (1979) Information theoretical optimization techniques. Kybernetika 15(1):8–27

    MathSciNet  Google Scholar 

  • Topsøe F (2004) Entropy and equilibrium via games of complexity. Phys A 340(1–3):11–31

    Article  MathSciNet  Google Scholar 

  • Topsøe F (2007) Information theory at the service of science. In: Csiszár I, Katona GOH, Tardos G, Wiener G (eds) Entropy, search, complexity. Bolyai society mathematical studies. Springer, Berlin, pp 179–207

    Chapter  Google Scholar 

  • van Berkum E, Linssen H, Overdijk D (1996) Inference rules and inferential distributions. J Stat Plan Inference 49:305–317

    Article  MATH  Google Scholar 

  • Vidakovic B (2000) Gamma-minimax: a paradigm for conservative robust Bayesians. Robust Bayesian analysis. Springer, New York

    Google Scholar 

  • Walley P (1991) Statistical reasoning with imprecise probabilities. Chapman and Hall, London

    Book  MATH  Google Scholar 

  • Weichselberger K (2000) The theory of interval-probability as a unifying concept for uncertainty. Int J Approx Reason 24(2–3):149–170

    Article  MATH  MathSciNet  Google Scholar 

  • Weichselberger K (2001) Elementare Grundbegriffe einer allgemeineren Wahrscheinlichkeitsrechnung I: Intervallwahrscheinlichkeit als umfassendes Konzept. Physica-Verlag, Heidelberg

    Book  MATH  Google Scholar 

  • Westfall PH, Johnson WO, Utts JM (1997) A Bayesian perspective on the Bonferroni adjustment. Biometrika 84:419–427

    Article  MATH  MathSciNet  Google Scholar 

  • Westfall PH (2010) Comment on B. Efron, “Correlated z-values and the accuracy of large-scale statistical estimates”. J Am Stat Assoc 105:1063–1066

    Article  MathSciNet  Google Scholar 

  • Williamson J (2010) In defence of objective bayesianism. Oxford University Press, Oxford

    Book  MATH  Google Scholar 

  • Yuan B (2009) Bayesian frequentist hybrid inference. Ann Stat 37:2458–2501

    Article  MATH  Google Scholar 

Download references

Acknowledgments

The comments of the two anonymous reviewers and of the editor-in-chief are gratefully acknowledged for improving the clarity of presentation. In addition, I thank Xuemei Tang for providing the fruit-development microarray data. This research was partially supported by the Canada Foundation for Innovation, by the Ministry of Research and Innovation of Ontario, and by the Faculty of Medicine of the University of Ottawa.

Author information

Authors and Affiliations

  1. Ottawa Institute of Systems Biology, Department of Biochemistry, Microbiology, and Immunology, University of Ottawa, 451 Smyth Road, Ottawa, ON, K1H 8M5, Canada

    David R. Bickel

  2. Department of Mathematics and Statistics, 451 Smyth Road, Ottawa, ON, K1H 8M5, Canada

    David R. Bickel

Authors
  1. David R. Bickel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David R. Bickel.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bickel, D.R. Blending Bayesian and frequentist methods according to the precision of prior information with applications to hypothesis testing. Stat Methods Appl 24, 523–546 (2015). https://doi.org/10.1007/s10260-015-0299-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10260-015-0299-6

Keywords

Mathematics Subject Classification

Search

Navigation