Sankhya A

, Volume 74, Issue 2, pp 141–169 | Cite as

A frequentist framework of inductive reasoning

  • David R. BickelEmail author


A betting game establishes a sense in which confidence measures, confidence distributions in the form of probability measures, are the only reliable inferential probability distributions. In addition, because confidence measures are Kolmogorov probability distributions, they are as coherent as Bayesian posterior distributions in their avoidance of sure loss under the usual Dutch-book betting game.

Although a confidence measure can be computed without any prior, previous knowledge can be incorporated into confidence-based reasoning by combining the confidence measure from the observed data with one or more independent confidence measures representing previous agent opinion. The representation of subjective knowledge in terms of confidence measures rather than more general priors preserves approximate frequentist validity and thus reliability in the first game.

Keywords and phrases.

Artificial intelligence betting coherence confidence distribution confidence posterior expert system foundations of statistics inductive reasoning interpretation of probability machine learning personal probability prior elicitation subjective probability 

AMS (2000) subject classification.

Primary 62A01 Secondary 62C99 



I am grateful to the anonymous referee for comments that lead to several improvements in clarity and completeness, most notably the inclusion of Sections 3.2.2 and 5.2. Matthias Kohl kindly provided R code (R Development Core Team, 2004) used to compute the convolution of the double exponential distribution (“R-help” list message posted on 12/23/05). I thank Mark Cooper for helpful feedback and Jean Peccoud, Mark Whitsitt, Chris Martin, and Bob Merrill for their support of the seed of this paper (Bickel, 2006) at Pioneer Hi-Bred, International. Subsequent developments were partially supported by the Canada Foundation for Innovation and by the Ministry of Research and Innovation of Ontario.


  1. Armendt, B. (1992). Dutch strategies for diachronic rules: when believers see the sure loss coming. In PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association 1992, pp. 217–229.Google Scholar
  2. Barnard, G.A. (1987). R. A. Fisher: a true Bayesian? International Statistical Review, 55, 183–189.MathSciNetzbMATHCrossRefGoogle Scholar
  3. Barndorff-nielsen, O.E. and Cox, D.R. (1994). Inference and Asymptotics. CRC Press, London.zbMATHGoogle Scholar
  4. Berger, J.O., Bernardo, J.M. and Sun, D. (2009). The formal definition of reference priors. Ann. Statist., 37, 905–938.MathSciNetzbMATHCrossRefGoogle Scholar
  5. Berger, J.O. (2004). The case for objective Bayesian analysis. Bayesian Anal., 1, 1–17.Google Scholar
  6. Bernardo, J.M. (1979). Reference posterior distributions for Bayesian inference. J. R. Stat. Soc. Ser. B, 41, 113–147.MathSciNetzbMATHGoogle Scholar
  7. Bernardo, J.M. (1997). Noninformative priors do not exist: a discussion. J. Statist. Plann. Inference, 65, 159–189.MathSciNetCrossRefGoogle Scholar
  8. Bickel, D.R. (2006). Incorporating expert knowledge into frequentist results by combining subjective prior and objective posterior distributions: a generalization of confidence distribution combination. Technical Report, Pioneer Hi-Bred International, arXiv:math.ST/0602377v2.Google Scholar
  9. Bickel, D.R. (2011). Estimating the null distribution to adjust observed confidence levels for genome-scale screening. Biometrics, 67, 363–370.MathSciNetzbMATHCrossRefGoogle Scholar
  10. Bickel, D.R. (2012a). Blending Bayesian and frequentist methods according to the precision of prior information with applications to hypothesis testing. Working Paper, University of Ottawa, deposited in uO Research at
  11. Bickel, D.R. (2012b). Coherent frequentism: a decision theory based on confidence sets. Comm. Statist. Theory Methods, 41, 1478–1496.MathSciNetzbMATHCrossRefGoogle Scholar
  12. Bickel, D.R. (2012c). Controlling the degree of caution in statistical inference with the Bayesian and frequentist approaches as opposite extremes. Electron. J. Stat., 6, 686–709.CrossRefGoogle Scholar
  13. Bickel, D.R. (2012d). Empirical Bayes interval estimates that are conditionally equal to unadjusted confidence intervals or to default prior credibility intervals. Stat. Appl. Genet. Mol. Biol., 11, art. 3.Google Scholar
  14. Bickel, D.R. (2012e). A prior-free framework of coherent inference and its derivation of simple shrinkage estimators. Working Paper, University of Ottawa, deposited in uO Research at
  15. Bickel, D.R. (2012f). The strength of statistical evidence for composite hypotheses: Inference to the best explanation. Statistica Sinica, 22, 1147–1198.MathSciNetzbMATHGoogle Scholar
  16. Brazzale, A.R., Davison, A.C. and Reid, N. (2007). Applied Asymptotics: Case Studies in Small-sample Statistics. Cambridge University Press, Cambridge.zbMATHCrossRefGoogle Scholar
  17. Buehler, R.J. (1977). Conditional confidence statements and confidence estimators: Comment. J. Amer. Statist. Assoc., 72, 813–814.MathSciNetGoogle Scholar
  18. Carnap, R. (1971). A basic system of inductive logic, part 1. Studies in Inductive Logic and Probability, Vol. 1. University of California Press, Berkeley, pp. 3–165.Google Scholar
  19. Chaloner, K. (1996). The elicitation of prior distributions. Bayesian Biostatistics. Marcel Dekker, New York.Google Scholar
  20. Clarke, B. (2007). Information optimality and Bayesian modelling. J. Econometrics, 138, 405–429.MathSciNetCrossRefGoogle Scholar
  21. Cornfield, J. (1969). The Bayesian outlook and its application. Biometrics, 25, 617–657.MathSciNetCrossRefGoogle Scholar
  22. Cox, D.R. (1958). Some problems connected with statistical inference. Annals of Mathematical Statistics, 29, 357–372.MathSciNetzbMATHCrossRefGoogle Scholar
  23. Craig, P.S., Goldstein, M., Seheult, A.H. and Smith, J.A. (1998). Constructing partial prior specifications for models of complex physical systems. The Statistician, 47, 37–53.Google Scholar
  24. Datta, G.S. and Mukerjee, R. (2004). Probability Matching Priors: Higher Order Asymptotics. Springer, New York.zbMATHCrossRefGoogle Scholar
  25. De Finetti, B. (1970). Theory of Probability: a Critical Introductory Treatment, 1st Edition. John Wiley and Sons Ltd, New York.Google Scholar
  26. Dempster, A.P. (2008). The Dempster-Shafer calculus for statisticians. Internat. J. Approx. Reason., 48, 365–377.MathSciNetzbMATHCrossRefGoogle Scholar
  27. Edwards, A.W.F. (1992). Likelihood. Johns Hopkins Press, Baltimore.zbMATHGoogle Scholar
  28. Efron, B. (1993). Bayes and likelihood calculations from confidence intervals. Biometrika, 80, 3–26.MathSciNetzbMATHCrossRefGoogle Scholar
  29. Efron, B. (1998). R. A. Fisher in the 21st century, invited paper presented at the 1996 R. A. Fisher lecture. Statist. Sci., 13, 95–114.MathSciNetzbMATHCrossRefGoogle Scholar
  30. Efron, B. and Tibshirani, R. (1998). The problem of regions. Ann. Statist. 26, 1687–1718.MathSciNetzbMATHCrossRefGoogle Scholar
  31. Fisher, R.A. (1960). Scientific thought and the refinement of human reasoning. J. Oper. Res. Soc. Japan, 3, 1–10.Google Scholar
  32. Fisher, R.A. (1973). Statistical Methods and Scientific Inference. Hafner Press, New York.zbMATHGoogle Scholar
  33. Fraser, D. (1978). Inference and Linear Models. McGraw-Hill, New York.Google Scholar
  34. Fraser, D.A.S. (1968). The Structure of Inference. John Wiley, New York.zbMATHGoogle Scholar
  35. Fraser, D.A.S. (1977). Confidence, posterior probability, and the buehler example. Ann. Statist., 5, 892–898.MathSciNetzbMATHCrossRefGoogle Scholar
  36. Fraser, D.A.S. (1991). Statistical inference: likelihood to significance. J. Amer. Statist. Assoc., 86, 258–265.MathSciNetzbMATHCrossRefGoogle Scholar
  37. Fraser, D.A.S. (2004). Ancillaries and conditional inference. Statist. Sci., 19, 333–351.MathSciNetzbMATHCrossRefGoogle Scholar
  38. Fraser, D.A.S. (2006). Did Lindley get the argument the wrong way around? Technical Report, Department of Statistics, University of Toronto.Google Scholar
  39. Fraser, D.A.S. and Reid, N. (2002). Strong matching of frequentist and Bayesian parametric inference. J. Statist. Plann. Inference, 103, 263–285.MathSciNetzbMATHCrossRefGoogle Scholar
  40. Freedman, D.A. and Purves, R.A. (1969). Bayes’ method for bookies. Annals of Mathematical Statistics, 40, 1177–1186.MathSciNetzbMATHCrossRefGoogle Scholar
  41. Garthwaite, P.H., Kadane, J.B. and O’hagan, A. (2005). Statistical methods for eliciting probability distributions. J. Amer. Statist. Assoc., 100, 680–700.MathSciNetzbMATHCrossRefGoogle Scholar
  42. Gleser, L.J. (2002). [setting confidence intervals for bounded parameters]: Comment. Statist. Sci., 17, 161–163.Google Scholar
  43. Goldstein, M. (1997). Prior inferences for posterior judgements. In Structures and Norms in Science: Volume Two of the Tenth International Congress of Logic, Methodology and Philosophy of Science, Florence, August 1995 (M. L. D. Chiara, K. Doets, D. Mundici, & J. van Benthem Eds.). New York, Springer, pp. 55–71.Google Scholar
  44. Goldstein, M. (2001). Avoiding foregone conclusions: Geometric and foundational analysis of paradoxes of finite additivity. J. Statist. Plann. Inference, 94, 73–87.MathSciNetzbMATHCrossRefGoogle Scholar
  45. Goldstein, M. (2006). Subjective Bayesian analysis: principles and practice. Bayesian Anal., 1, 403–420.MathSciNetCrossRefGoogle Scholar
  46. Grundy, P.M. (1956). Fiducial distributions and prior distributions: an example in which the former cannot be associated with the latter. J. R. Stat. Soc. Ser. B, 18, 217–221.MathSciNetzbMATHGoogle Scholar
  47. Hacking, I. (1965). Logic of Statistical Inference. Cambridge University Press, Cambridge.zbMATHGoogle Scholar
  48. Hacking, I. (2001). An introduction to probability and inductive logic. Cambridge University Press, Cambridge.zbMATHCrossRefGoogle Scholar
  49. Hannig, J. (2009). On generalized fiducial inference. Statist. Sinica, 19, 491–544.MathSciNetzbMATHGoogle Scholar
  50. Hannig, J. and Xie, M. (2009). A note on Dempster-Shafer recombination of confidence distributions. Electron. J. Stat., 6, 1943–1966.CrossRefGoogle Scholar
  51. Heath, D. and Sudderth, W. (1978). On finitely additive priors, coherence, and extended admissibility. Ann. Statist., 6, 333–345.MathSciNetzbMATHCrossRefGoogle Scholar
  52. Heath, D. and Sudderth, W. (1989). Coherent inference from improper priors and from finitely additive priors. Ann. Statist., 17, 907–919.MathSciNetzbMATHCrossRefGoogle Scholar
  53. Helland, I.S. (2004). Statistical inference under symmetry. International Statistical Review, 72, 409–422.CrossRefGoogle Scholar
  54. Hurwicz, L. (1951). The generalized Bayes-minimax principle: a criterion for decision-making under uncertainty. Cowles Commission Discussion Paper 355.Google Scholar
  55. Hwang, J.T., Casella, G., Robert, C., Wells, M.T. and Farrell, R.H. (1992). Estimation of accuracy in testing. Ann. Statist., 20, 490–509.MathSciNetzbMATHCrossRefGoogle Scholar
  56. Jaffray, J.-Y. (1989). Linear utility theory for belief functions. Oper. Res. Lett., 8, 107–112.MathSciNetzbMATHCrossRefGoogle Scholar
  57. Jeffrey, R. (1986). Probabilism and induction. Topoi, 5, 51–58.MathSciNetCrossRefGoogle Scholar
  58. Kempthorne, O. (1976). Comment on E. T. Jaynes, ‘Confidence intervals vs Bayesian intervals’. Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science. D. Reidel, Dordrecht-Holland, Ch. Confidence intervals vs Bayesian intervals, pp. 220–228.Google Scholar
  59. Kiefer, J. (1977). Conditional confidence statements and confidence estimators: rejoinder. J. Amer. Statist. Assoc., 72, 822–827.MathSciNetGoogle Scholar
  60. Kohlas, J. and Monney, P.-A. (2008). An algebraic theory for statistical information based on the theory of hints. Internat. J. Approx. Reason., 48, 378–398.MathSciNetzbMATHCrossRefGoogle Scholar
  61. Kyburg, H.E. (2007). Probability and Inference. Texts in Philosophy 2. College Publications, London, Ch. Bayesian inference with evidential probability, pp. 281–296.Google Scholar
  62. Kyburg, H.E. and Teng, C.M. (2001). Uncertain Inference. Cambridge University Press, Cambridge.zbMATHCrossRefGoogle Scholar
  63. Lele, S.R. (2004). Elicit data, not prior: On using expert opinion in ecological studies. The Nature of Scientific Evidence: Statistical, Philosophical, and Empirical Considerations. University of Chicago Press, Chicago, pp. 410–436.Google Scholar
  64. Lindley, D.V. (1958). Fiducial distributions and Bayes’ theorem. J. R. Stat. Soc. Ser. B, 20, 102–107.MathSciNetzbMATHGoogle Scholar
  65. Liu, R. and Singh, K. (1997). Notions of limiting P values based on data depth and bootstrap. J. Amer. Statist. Assoc., 92, 266–277.MathSciNetzbMATHGoogle Scholar
  66. Maher, P. (1992). Diachronic rationality. Philos. Sci., 59, 120–141.MathSciNetCrossRefGoogle Scholar
  67. Mccullagh, P. (2002). What is a statistical model? Ann. Statist., 30, 1225–1267.MathSciNetzbMATHCrossRefGoogle Scholar
  68. Paris, J.B. (1994). The Uncertain Reasoner’s Companion: A Mathematical Perspective. Cambridge University Press, New York.zbMATHGoogle Scholar
  69. Polansky, A.M. (2007). Observed Confidence Levels: Theory and Application. Chapman and Hall, New York.CrossRefGoogle Scholar
  70. Robins, J. and Wasserman, L. (2000). Conditioning, likelihood, and coherence: A review of some foundational concepts. J. Amer. Statist. Assoc., 95, 1340–1346.MathSciNetzbMATHCrossRefGoogle Scholar
  71. Royall, R. (1997). Statistical Evidence: A Likelihood Paradigm. CRC Press, New York.zbMATHGoogle Scholar
  72. Royall, R. (2000). On the probability of observing misleading statistical evidence. J. Amer. Statist. Assoc., 95, 760–768.MathSciNetzbMATHCrossRefGoogle Scholar
  73. Savage, L.J. (1954). The Foundations of Statistics. John Wiley and Sons, New York.zbMATHGoogle Scholar
  74. Scheffe, H. (1977). A note on a reformulation of the s-method of multiple comparison. J. Amer. Statist. Assoc., 72, 143–146.MathSciNetzbMATHGoogle Scholar
  75. Schervish, M.J. (1995). Theory of Statistics. Springer-Verlag, New York.zbMATHCrossRefGoogle Scholar
  76. Schweder, T., Hjort, N.L. (2002). Confidence and likelihood. Scand. J. Stat., 29, 309–332.MathSciNetzbMATHCrossRefGoogle Scholar
  77. Seidenfeld, T. (2007). Probability and Inference. Texts in Philosophy 2. College Publications, London, Ch. Forbidden fruit: When epistemological probability may not take a bite of the Bayesian apple, pp. 267–279.Google Scholar
  78. Shafer, G. (2011). A betting interpretation for probabilities and Dempster-Shafer degrees of belief. Internat. J. Approx. Reason., 52, 127–136.MathSciNetzbMATHCrossRefGoogle Scholar
  79. Sharma, S.S. (1980). On hacking’s fiducial theory of inference. Canad. J. Statist., 8, 227–233.MathSciNetzbMATHCrossRefGoogle Scholar
  80. Singh, K., Xie, M. and Strawderman, W.E. (2005). Combining information from independent sources through confidence distributions. Ann. Statist., 33, 159–183.MathSciNetzbMATHCrossRefGoogle Scholar
  81. Sprott, D.A. (2000). Statistical Inference in Science. Springer, New York.zbMATHGoogle Scholar
  82. Troffaes, M.C.M. (2007). Decision making under uncertainty using imprecise probabilities. Internat. J. Approx. Reason., 45, 17–29.MathSciNetzbMATHCrossRefGoogle Scholar
  83. Ville, J. (1939). Gauthier-Villars, Paris.Google Scholar
  84. Vos, P. (2008). Boyles, R.A. (2008), “The role of likelihood in interval estimation,” The American Statistician, 62, 22–26: Comment by Vos and reply. Amer. Statist., 62, 274–275.CrossRefGoogle Scholar
  85. Wilkinson, G.N. (1977). On resolving the controversy in statistical inference (with discussion). J. R. Stat. Soc. Ser. B, 39, 119–171.MathSciNetzbMATHGoogle Scholar
  86. Williamson, J. (2009). Objective Bayesianism, Bayesian conditionalisation and voluntarism. Synthese, 178, 1–19.Google Scholar
  87. Zabell, S.L. (1992). R. A. Fisher and the fiducial argument. Statist. Sci., 7, 369–387.MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Indian Statistical Institute 2012

Authors and Affiliations

  1. 1.Ottawa Institute of Systems Biology, Department of Biochemistry, Microbiology and Immunology, Department of Mathematics and StatisticsUniversity of OttawaOttawaCanada

Personalised recommendations