Statistical Methods & Applications

, Volume 24, Issue 4, pp 523–546 | Cite as

Blending Bayesian and frequentist methods according to the precision of prior information with applications to hypothesis testing

  • David R. Bickel


The proposed minimax procedure blends strict Bayesian methods with p values and confidence intervals or with default-prior methods. Two applications to hypothesis testing bring some implications to light. First, the blended probability that a point null hypothesis is true is equal to the p value or a lower bound of an unknown posterior probability, whichever is greater. As a result, the p value is reported instead of any posterior probability in the case of complete prior ignorance but is ignored in the case of a fully known prior. In the case of partial knowledge about the prior, the possible posterior probability that is closest to the p value is used for inference. The second application provides guidance on the choice of methods used for small numbers of tests as opposed to those appropriate for large numbers. Whereas statisticians tend to prefer a multiple comparison procedure that adjusts each p value for small numbers of tests, large numbers instead lead many to estimate the local false discovery rate (LFDR), a posterior probability of hypothesis truth. Each blended probability reduces to the LFDR estimate if it can be estimated with sufficient accuracy or to the adjusted p value otherwise.


Confidence distribution Fiducial inference Imprecise probability Maximum entropy Multiple hypothesis testing Multiple comparison procedure Robust Bayesian analysis 

Mathematics Subject Classification

Primary: 62F15 Secondary: 62A01 



The comments of the two anonymous reviewers and of the editor-in-chief are gratefully acknowledged for improving the clarity of presentation. In addition, I thank Xuemei Tang for providing the fruit-development microarray data. This research was partially supported by the Canada Foundation for Innovation, by the Ministry of Research and Innovation of Ontario, and by the Faculty of Medicine of the University of Ottawa.


  1. Alba R, Payton P, Fei Z, McQuinn R, Debbie P, Martin GB, Tanksley SD, Giovannoni JJ (2005) Transcriptome and selected metabolite analyses reveal multiple points of ethylene control during tomato fruit development. Plant Cell 17:2954–2965CrossRefGoogle Scholar
  2. Augustin T (2002) Expected utility within a generalized concept of probability—a comprehensive framework for decision making under ambiguity. Stat Pap 43(1):5–22zbMATHMathSciNetCrossRefGoogle Scholar
  3. Augustin T (2004) Optimal decisions under complex uncertainty—basic notions and a general algorithm for data-based decision making with partial prior knowledge described by interval probability. Zeitschrift fur Angewandte Mathematik und Mechanik 84(10–11):678–687zbMATHMathSciNetCrossRefGoogle Scholar
  4. Balch MS (2012) Mathematical foundations for a theory of confidence structures. Int J Approx Reason 53(7):1003–1019zbMATHMathSciNetCrossRefGoogle Scholar
  5. Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B 57:289–300zbMATHMathSciNetGoogle Scholar
  6. Berger JO (1984) Robustness of Bayesian analyses. Studies in Bayesian econometrics. North-Holland, Ch. The robust Bayesian viewpoint, pp 63–124Google Scholar
  7. Berger JO (1985) Statistical decision theory and Bayesian analysis. Springer, New YorkzbMATHCrossRefGoogle Scholar
  8. Berger JO (1990) Robust Bayesian analysis: sensitivity to the prior. J Stat Plan Inference 25:303–328zbMATHCrossRefGoogle Scholar
  9. Berger JO, Sellke T (1987) Testing a point null hypothesis: the irreconcilability of p values and evidence. J Am Stat Assoc 82:112–122zbMATHMathSciNetGoogle Scholar
  10. Berger JO, Brown L, Wolpert R (1994) A unified conditional frequentist and Bayesian test for fixed and sequential simple hypothesis-testing. Ann Stat 22(4):1787–1807zbMATHMathSciNetCrossRefGoogle Scholar
  11. Berger JO, Bernardo J, Sun D (2009) The formal definition of reference priors. Ann Stat 37(2):905–938zbMATHMathSciNetCrossRefGoogle Scholar
  12. Bernardo JM (1979) Reference posterior distributions for Bayesian inference. J R Stat Soc B 41:113–147zbMATHMathSciNetGoogle Scholar
  13. Bickel DR (2011a) Estimating the null distribution to adjust observed confidence levels for genome-scale screening. Biometrics 67:363–370zbMATHMathSciNetCrossRefGoogle Scholar
  14. Bickel DR (2011b) A predictive approach to measuring the strength of statistical evidence for single and multiple comparisons. Can J Stat 39:610–631zbMATHMathSciNetCrossRefGoogle Scholar
  15. Bickel DR (2012a) Coherent frequentism: a decision theory based on confidence sets. Commun Stat Theory Methods 41:1478–1496zbMATHMathSciNetCrossRefGoogle Scholar
  16. Bickel DR (2012b) Controlling the degree of caution in statistical inference with the Bayesian and frequentist approaches as opposite extremes. Electron J Statist 6:686–709zbMATHMathSciNetCrossRefGoogle Scholar
  17. Bickel DR (2012c) Game-theoretic probability combination with applications to resolving conflicts between statistical methods. Int J Approx Reason 53:880–891zbMATHMathSciNetCrossRefGoogle Scholar
  18. Bickel DR (2013) Simple estimators of false discovery rates given as few as one or two p-values without strong parametric assumptions. Stat Appl Genet Mol Biol 12:529–543MathSciNetGoogle Scholar
  19. Bickel DR (2014a) A fiducial continuum from confidence sets to empirical Bayes set estimates as the number of comparisons increases. Working Paper, University of Ottawa, deposited in uO Research at
  20. Bickel DR (2014b) Model fusion and multiple testing in the likelihood paradigm: shrinkage and evidence supporting a point null hypothesis. Working paper, University of Ottawa, deposited in uO Research at
  21. Bickel DR (2014c) Small-scale inference: empirical Bayes and confidence methods for as few as a single comparison. Int Stat Rev 82:457–476MathSciNetCrossRefGoogle Scholar
  22. Bickel DR, Padilla M (2014) A prior-free framework of coherent inference and its derivation of simple shrinkage estimators. J Stat Plan Inference 145:204–221zbMATHMathSciNetCrossRefGoogle Scholar
  23. Carlin BP, Louis TA (2009) Bayesian methods for data analysis, 3rd edn. Chapman and Hall/CRC, New YorkGoogle Scholar
  24. Coletti C, Scozzafava R (2002) Probabilistic logic in a coherent setting. Kluwer, AmsterdamCrossRefGoogle Scholar
  25. Cover T, Thomas J (2006) Elements of information theory. Wiley, New YorkzbMATHGoogle Scholar
  26. Cox DR (2006) Principles of statistical inference. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  27. Csiszár I (1991) Why least squares and maximum entropy? an axiomatic approach to inference for linear inverse problems. Ann Stat 19:2032–2066zbMATHCrossRefGoogle Scholar
  28. DasGupta A, Studden W (1989) Frequentist behavior of robust Bayes estimates of normal means. Stat Decis 7:333–361zbMATHMathSciNetGoogle Scholar
  29. Dawid AP, Stone M (1982) The functional-model basis of fiducial inference (with discussion). Ann Stat 10:1054–1074zbMATHMathSciNetCrossRefGoogle Scholar
  30. Efron B (2010a) Large-scale inference: empirical Bayes methods for estimation, testing, and prediction. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  31. Efron B (2010b) Rejoinder to comments on B. Efron, “Correlated z-values and the accuracy of large-scale statistical estimates”. J Am Stat Assoc 105:1067–1069MathSciNetCrossRefGoogle Scholar
  32. Fisher RA (1973) Statistical methods and scientific inference. Hafner Press, New YorkzbMATHGoogle Scholar
  33. Fraser DAS (2004) Ancillaries and conditional inference. Stat Sci 19:333–351zbMATHMathSciNetCrossRefGoogle Scholar
  34. Fraser DAS, Reid N (1990) Discussion: an ancillarity paradox which appears in multiple linear regression. Ann Stat 18:503–507CrossRefGoogle Scholar
  35. Gärdenfors P, Sahlin N-E (1982) Unreliable probabilities, risk taking, and decision making. Synthese 53:361–386Google Scholar
  36. Genest C, Zidek JV (1986) Combining probability distributions: a critique and an annotated bibliography. Stat Sci 1:114–135MathSciNetCrossRefGoogle Scholar
  37. Gilboa I, Schmeidler D (1989) Maxmin expected utility with non-unique prior. J Math Econ 18(2):141–153zbMATHMathSciNetCrossRefGoogle Scholar
  38. Good IJ (1952) Rational decisions. J R Stat Soc B 14:107–114MathSciNetGoogle Scholar
  39. Good IJ (1983) Good thinking: the foundations of probability and its applications. G—reference, information and interdisciplinary subjects series. University of Minnesota Press, USAGoogle Scholar
  40. Grünwald P, Dawid AP (2004) Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory. Ann Stat 32:1367–1433zbMATHCrossRefGoogle Scholar
  41. Hannig J (2009) On generalized fiducial inference. Stat Sinica 19:491–544zbMATHMathSciNetGoogle Scholar
  42. Harremoës P, Topsøe F (2001) Maximum entropy fundamentals. Entropy 3(3):191–226MathSciNetCrossRefGoogle Scholar
  43. Jaynes E (2003) Probability theory: the logic of science. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  44. Jozani M, Marchand É, Parsian A (2012) Bayesian and robust bayesian analysis under a general class of balanced loss function. Stat Pap 53(1):51–60zbMATHCrossRefGoogle Scholar
  45. Kakihara Y (1999) Abstract methods in information theory (series on multivariate analysis, volume 4). World Scientific, SingaporeGoogle Scholar
  46. Kass RE, Wasserman L (1996) The selection of prior distributions by formal rules. J Am Stat Assoc 91:1343–1370zbMATHCrossRefGoogle Scholar
  47. Kracík J (2011) Combining marginal probability distributions via minimization of weighted sum of Kullback–Leibler divergences. Int J Approx Reason 52:659–671zbMATHCrossRefGoogle Scholar
  48. Lavine M (1991) Sensitivity in Bayesian statistics: the prior and the likelihood. J Am Stat Assoc 86:396–399zbMATHMathSciNetCrossRefGoogle Scholar
  49. Levi I (1986a) Hard choices: decision making under unresolved conflict. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  50. Levi I (1986b) The paradoxes of Allais and Ellsberg. Econ Philos 2:23–53CrossRefGoogle Scholar
  51. Lindley DV (1957) A statistical paradox. Biometrika 44:187–192zbMATHMathSciNetCrossRefGoogle Scholar
  52. Liu C, Aitkin M (2008) Bayes factors: prior sensitivity and model generalizability. J Math Psychol 52:362–375Google Scholar
  53. Nadarajah S, Bityukov S, Krasnikov N (2015) Confidence distributions: a review. Stat Methodol 22:23–46MathSciNetCrossRefGoogle Scholar
  54. Neuhaus KL, von Essen R, Tebbe U, Vogt A, Roth M, Riess M, Niederer W, Forycki F, Wirtzfeld A, Maeurer W (1992) Improved thrombolysis in acute myocardial infarction with front-loaded administration of alteplase: results of the rt-PA-APSAC patency study (TAPS). J Am Coll Cardiol 19:885–891CrossRefGoogle Scholar
  55. Padilla M, Bickel D. R. (2012) Estimators of the local false discovery rate designed for small numbers of tests. Stat Appl Genet Mol Biol 11(5), art. 4Google Scholar
  56. Paris JB (1994) The uncertain reasoner’s companion: a mathematical perspective. Cambridge University Press, New YorkzbMATHGoogle Scholar
  57. Paris J, Vencovská A (1997) In defense of the maximum entropy inference process. Int J Approx Reason 17(1):77–103zbMATHCrossRefGoogle Scholar
  58. Pfaffelhuber E (1977) Minimax information gain and minimum discrimination principle. In: Csiszár I, Elias P (eds) Topics in information theory. Vol. 16 of Colloquia Mathematica Societatis János Bolyai. János Bolyai Mathematical Society and North-Holland, Amsterdam, pp 493–519Google Scholar
  59. Polansky AM (2007) Observed confidence levels: theory and application. Chapman and Hall, New YorkCrossRefGoogle Scholar
  60. Robbins H (1951) Asymptotically subminimax solutions of compound statistical decision problems. Proc Second Berkeley Symp Math Stat Probab 1:131–148Google Scholar
  61. Robinson GK (1991) That BLUP is a good thing: the estimation of random effects. Stat Sci 6:15–32zbMATHCrossRefGoogle Scholar
  62. Savage LJ (1954) The foundations of statistics. Wiley, New YorkzbMATHGoogle Scholar
  63. Schweder T, Hjort NL (2002) Confidence and likelihood. Scand J Stat 29:309–332zbMATHMathSciNetCrossRefGoogle Scholar
  64. Seidenfeld T (1988) Decision theory without independence or without ordering: what is the difference? Econ Philos 4(2):267–290CrossRefGoogle Scholar
  65. Seidenfeld T (2004) A contrast between two decision rules for use with (convex) sets of probabilities: Gamma-maximin versus E-admissibility. Synthese 140(1–2):69–88MathSciNetCrossRefGoogle Scholar
  66. Sellke T, Bayarri MJ, Berger JO (2001) Calibration of p values for testing precise null hypotheses. Am Stat 55:62–71zbMATHMathSciNetCrossRefGoogle Scholar
  67. Sidak Z (1967) Rectangular confidence regions for means of multivariate normal distributions. J Am Stat Assoc 62(318):626–633zbMATHMathSciNetGoogle Scholar
  68. Singh K, Xie M, Strawderman WE (2007) Confidence distribution (CD)—distribution estimator of a parameter. IMS Lect Notes Monogr Ser 2007(54):132–150MathSciNetCrossRefGoogle Scholar
  69. Topsøe F (1979) Information theoretical optimization techniques. Kybernetika 15(1):8–27MathSciNetGoogle Scholar
  70. Topsøe F (2004) Entropy and equilibrium via games of complexity. Phys A 340(1–3):11–31MathSciNetCrossRefGoogle Scholar
  71. Topsøe F (2007) Information theory at the service of science. In: Csiszár I, Katona GOH, Tardos G, Wiener G (eds) Entropy, search, complexity. Bolyai society mathematical studies. Springer, Berlin, pp 179–207CrossRefGoogle Scholar
  72. van Berkum E, Linssen H, Overdijk D (1996) Inference rules and inferential distributions. J Stat Plan Inference 49:305–317zbMATHCrossRefGoogle Scholar
  73. Vidakovic B (2000) Gamma-minimax: a paradigm for conservative robust Bayesians. Robust Bayesian analysis. Springer, New YorkGoogle Scholar
  74. Walley P (1991) Statistical reasoning with imprecise probabilities. Chapman and Hall, LondonzbMATHCrossRefGoogle Scholar
  75. Weichselberger K (2000) The theory of interval-probability as a unifying concept for uncertainty. Int J Approx Reason 24(2–3):149–170zbMATHMathSciNetCrossRefGoogle Scholar
  76. Weichselberger K (2001) Elementare Grundbegriffe einer allgemeineren Wahrscheinlichkeitsrechnung I: Intervallwahrscheinlichkeit als umfassendes Konzept. Physica-Verlag, HeidelbergzbMATHCrossRefGoogle Scholar
  77. Westfall PH, Johnson WO, Utts JM (1997) A Bayesian perspective on the Bonferroni adjustment. Biometrika 84:419–427zbMATHMathSciNetCrossRefGoogle Scholar
  78. Westfall PH (2010) Comment on B. Efron, “Correlated z-values and the accuracy of large-scale statistical estimates”. J Am Stat Assoc 105:1063–1066MathSciNetCrossRefGoogle Scholar
  79. Williamson J (2010) In defence of objective bayesianism. Oxford University Press, OxfordzbMATHCrossRefGoogle Scholar
  80. Yuan B (2009) Bayesian frequentist hybrid inference. Ann Stat 37:2458–2501zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  1. 1.Ottawa Institute of Systems Biology, Department of Biochemistry, Microbiology, and ImmunologyUniversity of OttawaOttawaCanada
  2. 2.Department of Mathematics and StatisticsOttawaCanada

Personalised recommendations