Statistical Papers

, Volume 51, Issue 3, pp 613–628 | Cite as

Eliciting vague but proper maximal entropy priors in Bayesian experiments

  • Nicolas BousquetEmail author
Regular Article


Priors elicited according to maximal entropy rules have been used for years in objective and subjective Bayesian analysis. However, when the prior knowledge remains fuzzy or dubious, they often suffer from impropriety which can make them uncomfortable to use. In this article we suggest the formal elicitation of an encompassing family for the standard maximal entropy (ME) priors and the maximal data information (MDI) priors, which can lead to obtain proper families. An interpretation is given in the objective framework of channel coding. In a subjective framework, the performance of the method is shown in a reliability context when flat but proper priors are elicited for the Weibull lifetime distributions. Such priors appear as practical tools for sensitivity studies.


Bayesian inference Expert opinion Kullback–Leibler distance Shannon’s entropy Noninformative priors Channel coding Sensitivity study Weibull 

Mathematics Subject Classification (2000)

65K05 90C35 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Andrieu C, Doucet A, Fitzgerald WJ, Pérez JM (2001) Bayesian computational approaches to model selection. In: Smith RL, Young PC, Walkden A(eds) Nonlinear and non-Gaussian signal processing. Cambridge University Press, LondonGoogle Scholar
  2. Bacha M, Celeux G, Idée E, Lannoy A, Vasseur D (1998) Estimation de modèles de durées de vie fortement censurées. EyrollesGoogle Scholar
  3. Beirlant J, Dudewicz E, Gyorfi L, van der Meulen E (1997) Nonparametric entropy estimation: an overview. Int J Math Stat Sci 6: 17–39zbMATHMathSciNetGoogle Scholar
  4. Berger JO (1985) Statistical decision theory and Bayesian analysis. Springer, New YorkzbMATHGoogle Scholar
  5. Berger J (2006) The Case for objective bayesian analysis. Bayesian Anal 1: 385–402CrossRefMathSciNetGoogle Scholar
  6. Berger JO, Bernardo JM (1992) On the development of reference priors (with discussion). In: Bernardo JM, Berger JO, Dawid AP, Smith AFM(eds) Bayesian statistics, vol 4. Oxford University Press, USA, pp 35–60Google Scholar
  7. Berger JO, Sun D (1993) Bayesian analysis for the poly-Weibull distribution. J Am Stat Assoc 88: 1412–1418zbMATHCrossRefMathSciNetGoogle Scholar
  8. Berger JO, Sun D (1994) Bayesian sequential reliability for Weibull and related distributions. Ann Inst Stat Math 46: 221–249zbMATHCrossRefMathSciNetGoogle Scholar
  9. Bernardo JM (1997) Noninformative priors do not exist: a discussion. J Stat Plan Inference 65: 159–189CrossRefMathSciNetGoogle Scholar
  10. Billy F, Bousquet N, Celeux G (2006) Modelling and eliciting expert knowledge with fictitious data. In: Proceedings of the workshop on the use of expert judgement for decision-making, CEA CadaracheGoogle Scholar
  11. Bousquet N (2006a) A Bayesian analysis of industrial lifetime data with Weibull distributions, HAL-INRIA research report RR-6025Google Scholar
  12. Bousquet N (2006b) Analyse bayésienne de la durée de vie de composants industriels (Elements of Bayesian analysis for the prediction of the lifetime of industrial components). Ph.D. thesis, Parix XI UniversityGoogle Scholar
  13. Bousquet N, Celeux G (2006) Bayesian agreement between prior and data. In: Proceedings of the ISBA congress, Benidorm, SpainGoogle Scholar
  14. Celeux G, Marin J-M, Robert CP (2006) Iterated importance sampling in missing data problems. Comput Stat Data An 12: 3386–3404CrossRefMathSciNetGoogle Scholar
  15. Clarke B, Barron AR (1994) Jeffreys’ prior is asymptotically least favorable under entropy risk. J Stat Plan Inference 41: 37–60zbMATHCrossRefMathSciNetGoogle Scholar
  16. Clarke B (2007) Information optimality and Bayesian modelling. J Econom 138: 405–449CrossRefGoogle Scholar
  17. Cooke R (1991) Experts in uncertainty: opinion and subjective probability in science. Oxford University Press, New YorkGoogle Scholar
  18. Cover TM, Thomas JA (1991) Elements of information theory. Wiley, New YorkzbMATHCrossRefGoogle Scholar
  19. Craig PS, Goldstein M, Seheult AH, Smith JA (1998) Constructing partial prior specifications for models of complex physical systems. The Statistician 47: 37–53Google Scholar
  20. Forrester Y (2005) The quality of expert judgement: an interdisciplinary investigation. Ph.D. thesis, University of MarylandGoogle Scholar
  21. Friedlander MP, Gupta MR (2006) On minimizing distortion and relative entropy. IEEE Trans Inform Theory 52: 238–245CrossRefMathSciNetGoogle Scholar
  22. Dai Y-S, Xie M, Long Q, Ng S-H (2007) Uncertainty analysis in software reliability modeling by bayesian analysis with maximum-entropy principle. Trans Soft Eng 33: 781–795CrossRefGoogle Scholar
  23. Garthwaite PH, Kadane JB, O’Hagan A (2005) Statistical methods for eliciting probability distributions. J Am Stat Assoc 100: 680–701zbMATHCrossRefMathSciNetGoogle Scholar
  24. Gelfand AE, Mallick BK, Dey DK (1995) Modeling expert opinion arising as a partial probabilistic specification. J Am Stat Assoc 90: 598–604zbMATHCrossRefMathSciNetGoogle Scholar
  25. Goldstein M (2006) Subjective Bayesian analysis: principles and practice. Bayesian Anal 1: 403–420CrossRefMathSciNetGoogle Scholar
  26. Goossens LHJ, Cooke RM (2006) Expert judgement—calibration and combination. In: Proceedings of the workshop on the use of expert judgement for decision-making, CEA CadaracheGoogle Scholar
  27. Guida M, Calabria R, Pulcini G (1989) Bayes inference for a non-homogeneous Poisson process with power intensity law reliability. IEEE Trans Inform Theory 5: 603–609Google Scholar
  28. Hill SD, Spall JC (1994) Sensitivity of a Bayesian analysis to the prior distribution. IEEE Trans Inform Theory 24: 216–221MathSciNetGoogle Scholar
  29. Jaynes ET (1957) Information theory and statistical mechanics. Phys Rev 106, 620–630, and 108, 171–190Google Scholar
  30. Jaynes ET (1982) On the rationale of maximum entropy methods. Proc IEEE 70: 939–952CrossRefGoogle Scholar
  31. Jaynes ET (1988) The Relation of bayesian and maximum entropy methods (510Kb). In: Erickson GJ, Smith CR(eds) Maximum-entropy and Bayesian methods in science and engineering, vol 1. Kluwer, Dordrecht, pp 25–29Google Scholar
  32. Jaynes ET (2003) Probability theory: the logic of science. Cambridge University Press, LondonzbMATHGoogle Scholar
  33. Kadane JB, Wolfson J (1998) A experiences in elicitation. The Statistician 47: 3–19Google Scholar
  34. Kaminskiy MP, Krivtsov VV (2005) A simple procedure for Bayesian estimation of the Weibull distribution. IEEE Trans Inform Theory 54: 612–616Google Scholar
  35. Kass RE, Wasserman L (1996) The selection of prior distributions by formal rules. J Am Stat Assoc 91: 1343–1370zbMATHCrossRefGoogle Scholar
  36. Lannoy A, Procaccia H (2003) L’utilisation du jugement d’expert en sûreté de fonctionnement. Tec & DocGoogle Scholar
  37. Lawless JF (2003) Statistical models and methods for lifetime data, 2nd edn. Wiley, LondonzbMATHGoogle Scholar
  38. Le Besnerais G, Bercher J-F, Demoment G (1999) A new look at entropy for solving linear inverse problems. IEEE Trans Inform Theory 45: 1565–1578zbMATHCrossRefMathSciNetGoogle Scholar
  39. Linnemann JT (2000) Upper limits and priors. FNAL CL Worshop (with notes added in January 2003)Google Scholar
  40. Luttrell SP (1985) The use of transinformation in the design of data sampling schemes for inverse problems. Inverse Probl 1: 199–218zbMATHCrossRefMathSciNetGoogle Scholar
  41. Miller DJ, Yan L (2000) Approximate maximum entropy joint feature inference consistent with arbitrary lower order probability constraints: application to statistical classification. Neural Comput 12: 2175–2208CrossRefGoogle Scholar
  42. Natarajan R, McCulloch CE (1998) Gibbs sampling with diffuse proper priors: a valid approach to data-driven inference?. J Comput Graph Stat 7: 267–277CrossRefGoogle Scholar
  43. Oakley JE, O’Hagan A (2007) Uncertainty in prior elicitations: a nonparametric approach. Biometrika 94: 427–441zbMATHCrossRefMathSciNetGoogle Scholar
  44. O’Hagan A (1998) Eliciting expert beliefs in substantial practical applications. The Statistician 47(1): 21–35MathSciNetGoogle Scholar
  45. O’Hagan A (2006) Research in elicitation. In: Upadhyay SK, Singh U, Dey DK(eds) Bayesian statistics and its applications. Anamaya, New Delhi, pp 375–382Google Scholar
  46. O’Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, Oakley JE, Rakow T (2006) Uncertain judgements: eliciting expert probabilities. Wiley, ChichesterzbMATHCrossRefGoogle Scholar
  47. Press SJ (2003) Subjective and objective Bayesian statistics, 2nd edn. Wiley, New YorkzbMATHGoogle Scholar
  48. Robert CP (2001) The Bayesian choice. a decision-theoretic motivation, 2nd edn. Springer, HeidelbergGoogle Scholar
  49. Shannon CE (1948) A mathematical theory of communication. Bell Syst Technol J 27:379–423, 623–656Google Scholar
  50. Shulman N, Feder M (2004) The uniform distribution as a universal prior. IEEE Trans Inform Theory 50: 1356–1362CrossRefMathSciNetGoogle Scholar
  51. Singpurwalla ND (2006) Reliability and risk: a Bayesian perspective. Wiley, LondonzbMATHGoogle Scholar
  52. Skilling J (1989) Maximum entropy and Bayesian methods. Kluwer, DordrechtzbMATHGoogle Scholar
  53. Smith, CR, Erickson, G, Neudorfer, PO (eds) (1992) Maximum entropy and Bayesian methods (fundamental theories of physics). Kluwer Academic Publisher, DordrechtGoogle Scholar
  54. Soofi ES (1992) Information theory and Bayesian statistics. In: Berry DA, Chaloner KM, Geweke JK(eds) Bayesian analysis in statistics and econometrics in honor of Arnold Zellner. Wiley, New York, pp 179–189Google Scholar
  55. Soofi ES (1994) Capturing the intangible concept of information. J Am Stat Assoc 89: 1243–1254zbMATHCrossRefMathSciNetGoogle Scholar
  56. Soofi ES (2000) Principal information theoretic approaches. J Am Stat Assoc 95: 1349–1353zbMATHCrossRefMathSciNetGoogle Scholar
  57. Sun D (1997) A note on noninformative priors for Weibull distributions. J Stat Plan Inference 61: 319–338zbMATHCrossRefGoogle Scholar
  58. van Noortwijk JM, Dekker R, Cooke RM, Mazzuchi TA (1992) Expert judgment in maintenance optimization. IEEE Trans Reliab 41: 427–431CrossRefGoogle Scholar
  59. Venegaz-Martinez F (2004) On information measures and prior distributions: a synthesis. Morfismos 8: 27–50Google Scholar
  60. Zellner A (1977) Maximal data information prior distributions. In: Aykae A, Brumat C (eds) New developments in the applications of Bayesian methods, AmsterdamGoogle Scholar
  61. Zellner A, Sinha SK (1990) A note on the prior distributions of Weibull parameters. SCIMA 19: 5–13zbMATHGoogle Scholar
  62. Zellner A (1991) Bayesian methods and entropy in economics and econometrics. In: Grandy WT Jr, Schick LH(eds) Maximum entropy and Bayesian methods. Kluwer, Boston, pp 17–31Google Scholar
  63. Zellner A (1996) Models, prior information and Bayesian analysis. J Econom 75: 51–68zbMATHCrossRefMathSciNetGoogle Scholar
  64. Zellner A (1997) Bayesian analysis in econometrics and statistics: the Zellner view and papers. Edward Elgar, Cheltenham, UKGoogle Scholar
  65. Zellner A (2002) Information processing and Bayesian analysis. J Econom 107: 41–50zbMATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag 2008

Authors and Affiliations

  1. 1.Department of Mathematics and StatisticsLaval UniversityQuebecCanada

Personalised recommendations