Theory and Decision

, Volume 69, Issue 2, pp 257–288 | Cite as

Simple methods for evaluating and comparing binary experiments

Article
First Online:
Received:
Accepted:

Abstract

We consider a confidence parametrization of binary information sources in terms of appropriate likelihood ratios. This parametrization is used for Bayesian belief updates and for the equivalent comparison of binary experiments. In contrast to the standard parametrization of a binary information source in terms of its specificity and its sensitivity, one of the two confidence parameters is sufficient for a Bayesian belief update conditional on a signal realization. We introduce a confidence-augmented receiver operating characteristic for comparisons of binary experiments for a class of “balanced” decision problems, relative to which the confidence order offers a higher resolution than Blackwell’s informativeness order.

Where observation is concerned, Chance favors only the prepared mind.

—Louis Pasteur (1822–1895).

Keywords

Classification problems Decision making under uncertainty Informativeness Value of information 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Athey, S. (2000). Investment and information value for a risk averse firm. Working paper 00-30, Department of Economics, Massachusetts Institute of Technology, Cambridge, MA.Google Scholar
  2. Athey, S., & Levin, J. (2001). The value of information in monotone decision problems. Working paper 98-24, Department of Economics, Massachusetts Institute of Technology, Cambridge, MA.Google Scholar
  3. Barnard G.A. (1949) Statistical inference. Journal of the Royal Statistical Society: Series B (Methodological) 11(2): 115–149Google Scholar
  4. Bayes, T. (1764). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society London, 53, 370–418. (Reprinted in: Barnard, G. A. (1958). Biometrika, 45(3/4), 293–315).Google Scholar
  5. Berger, J.O., & Wolpert, R. (1984). The likelihood principle. Lecture notes, monograph series, Vol. 61, Hayward, CA: Institute of Mathematical Statistics.Google Scholar
  6. Birnbaum A. (1961) On the foundations of statistical inference: Binary experiments. Annals of Mathematical Statistics 32(2): 414–435CrossRefGoogle Scholar
  7. Birnbaum A. (1962) On the foundations of statistical inference. Journal of the American Statistical Association 57(298): 269–306CrossRefGoogle Scholar
  8. Birnbaum A., (1960) Classification procedures based on Bayes’s formula. Applied Statistics 9(3): 152–169CrossRefGoogle Scholar
  9. Blackwell, D. (1951). Comparison of experiments. In Proceedings of the second Berkeley symposium on mathematical statistics and probability (pp. 93–102). Berkeley: University of California Press.Google Scholar
  10. Blackwell D. (1953) Equivalent comparison of experiments. Annals of Mathematical Statistics 24(2): 265–272CrossRefGoogle Scholar
  11. Bohnenblust, H., Shapley, L., & Sherman, S. (1949). Reconnaissance in game theory. Research memorandum RM-208, RAND Corporation.Google Scholar
  12. Cavusoglu H., (2004) Configuration of detection software: A comparison of decision and game theory approaches. Decision Analysis 1(9): 131–148CrossRefGoogle Scholar
  13. Clemen R.T., Winkler R.L. (1993) Aggregating point estimates: A flexible modeling approach. Management Science 39(4): 501–515CrossRefGoogle Scholar
  14. DeGroot M. (1962) Uncertainty, information, and sequential experiments. Annals of Mathematical Statistics 33(2): 404–419CrossRefGoogle Scholar
  15. Eves H.W. (1988) Return to mathematical circles. Prindle, Weber, and Schmidt, Boston, MAGoogle Scholar
  16. Fawcett T. (2006) An introduction to ROC analysis. Pattern Recognition Letters 27(8): 861–874CrossRefGoogle Scholar
  17. Fishburn P.C. (1970) Utility theory for decision making. Wiley, New York, NYGoogle Scholar
  18. Flach, P. A., & Wu, S. (2003). Reparing concavities in ROC curves. In Proceedings of the 2003 UK workshop on computational intelligence (pp. 38–44). Bristol, UK: University of Bristol.Google Scholar
  19. Froeb L.M., Kobayashi B.H. (1996) Naive, biased, yet Bayesian: Can juries interpret selectively produced evidence?. Journal of Law, Economics, and Organization 12(1): 257–276Google Scholar
  20. Hastie T., Tibshirani R., (2001) The elements of statistical learning. Springer, New York, NYGoogle Scholar
  21. Hill B.M. (1987) The validity of the likelihood principle. American Statistician 41(2): 95–100CrossRefGoogle Scholar
  22. Hoel P.G., Peterson R.P. (1949) A solution to the problem of optimum classification. Annals of Mathematical Statistics. 20(3): 433–438CrossRefGoogle Scholar
  23. Kaye D.H., Koehler J.J. (1991) Can jurors understand probabilistic evidence?. Journal of the Royal Statistical Society: Series A (Statistics in Society) 154(1): 75–81CrossRefGoogle Scholar
  24. Keisler J. (2004) Value of information in portfolio decision analysis. Decision Analysis 1(3): 177–189CrossRefGoogle Scholar
  25. Kihlstrom R.E. (1984) A Bayesian exposition of Blackwell’s theorem on the comparison of experiments. In: Boyer M., Kihlstrom R.E. (eds) Bayesian models in economic theory. Elsevier Science, North-Holland, Amsterdam, NL, pp 13–31Google Scholar
  26. Kreps D. (1988) Notes on the theory of choice. Westview Press, Boulder, COGoogle Scholar
  27. Kriege M., Brekelmans C.T., Boetes C. et al (2004) Efficacy of MRI and mammography for breast-cancer screening in women with a familial or genetic predisposition. New England Journal of Medicine 351(5): 427–437CrossRefGoogle Scholar
  28. Lasko T.A., Bhagwat J.G., Zou K.H., Ohno-Machado L. (2005) The use of receiver operating characteristic curves in biomedical informatics. Journal of Biomedical Informatics 38(5): 404–415CrossRefGoogle Scholar
  29. LaValle I.H. (1968) On cash equivalents and information evaluation in decisions under uncertainty. Journal of the American Statistical Association 63(321): 252–290CrossRefGoogle Scholar
  30. Leach M.O., Boggis C.R.M., Dixon A.F. et al (2005) Screening with magnetic resonance imaging and mammography of a UK population at high familial risk of breast cancer: A prospective multicentre cohort study (MARIBS). Lancet 365(9473): 1769–1778CrossRefGoogle Scholar
  31. Lehmann E.L. (1988) Comparing location experiments. Annals of Statistics 16(2): 521–533CrossRefGoogle Scholar
  32. Lindley D.V. (1956) On a measure of the information provided by an experiment. Annals of Mathematical Statistics 27(4): 986–1005CrossRefGoogle Scholar
  33. Lindley D.V. (1964) The Bayesian analysis of contingency tables. Annals of Mathematical Statistics 35(4): 1622–1643CrossRefGoogle Scholar
  34. Milgrom P., Segal I. (2002) Envelope theorems for arbitrary choice sets. Econometrica 70(2): 583–601CrossRefGoogle Scholar
  35. Morris P.A. (1977) Combining expert judgments: A Bayesian approach. Management Science 23(7): 679–693CrossRefGoogle Scholar
  36. Nemec C.F., Listinsky J., (2007) How should we screen for breast cancer? Mammography, ultrasonography, MRI. Cleveland Clinic Journal of Medicine 74(12): 897–904CrossRefGoogle Scholar
  37. Neyman J., (1933) On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London: Series A 231: 289–337CrossRefGoogle Scholar
  38. Pepe M.S. (2003) The statistical evaluation of medical tests for classification and prediction. Oxford University Press, Oxford, UKGoogle Scholar
  39. Radner R., Stiglitz J.E. (1984) A nonconcavity in the value of information. In: Boyer M., Kihlstrom R.E. (eds) Bayesian models in economic theory. Elsevier Science, North-Holland, Amsterdam, NL, pp 33–52Google Scholar
  40. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423, 623–656.Google Scholar
  41. Shannon C.E., Weaver W. (1948) The mathematical theory of communication. University of Illinois Press, Chicago, ILGoogle Scholar
  42. Strulovici B.H., Weber T.A. (2008) Monotone comparative statics: Geometric approach. Journal of Optimization Theory and Applications 137(3): 641–673CrossRefGoogle Scholar
  43. Wald A. (1939) Contributions to the theory of statistical estimation and testing hypotheses. Annals of Mathematical Statistics 10(4): 299–326CrossRefGoogle Scholar
  44. Weber T.A., Croson D.C. (2003) Selling less information for more: Garbling with benefits. Economics Letters 83(2): 165–171‘CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC. 2009

Authors and Affiliations

  1. 1.Department of Management Science and Engineering, 442 Terman Engineering CenterStanford UniversityStanfordUSA

Personalised recommendations