Skip to main content
SpringerLink
Log in

Simple methods for evaluating and comparing binary experiments

  • Published:
Theory and Decision Aims and scope Submit manuscript

Abstract

We consider a confidence parametrization of binary information sources in terms of appropriate likelihood ratios. This parametrization is used for Bayesian belief updates and for the equivalent comparison of binary experiments. In contrast to the standard parametrization of a binary information source in terms of its specificity and its sensitivity, one of the two confidence parameters is sufficient for a Bayesian belief update conditional on a signal realization. We introduce a confidence-augmented receiver operating characteristic for comparisons of binary experiments for a class of “balanced” decision problems, relative to which the confidence order offers a higher resolution than Blackwell’s informativeness order.

Where observation is concerned, Chance favors only the prepared mind.

—Louis Pasteur (1822–1895).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Athey, S. (2000). Investment and information value for a risk averse firm. Working paper 00-30, Department of Economics, Massachusetts Institute of Technology, Cambridge, MA.

  • Athey, S., & Levin, J. (2001). The value of information in monotone decision problems. Working paper 98-24, Department of Economics, Massachusetts Institute of Technology, Cambridge, MA.

  • Barnard G.A. (1949) Statistical inference. Journal of the Royal Statistical Society: Series B (Methodological) 11(2): 115–149

    Google Scholar 

  • Bayes, T. (1764). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society London, 53, 370–418. (Reprinted in: Barnard, G. A. (1958). Biometrika, 45(3/4), 293–315).

    Google Scholar 

  • Berger, J.O., & Wolpert, R. (1984). The likelihood principle. Lecture notes, monograph series, Vol. 61, Hayward, CA: Institute of Mathematical Statistics.

  • Birnbaum A. (1961) On the foundations of statistical inference: Binary experiments. Annals of Mathematical Statistics 32(2): 414–435

    Article  Google Scholar 

  • Birnbaum A. (1962) On the foundations of statistical inference. Journal of the American Statistical Association 57(298): 269–306

    Article  Google Scholar 

  • Birnbaum A., (1960) Classification procedures based on Bayes’s formula. Applied Statistics 9(3): 152–169

    Article  Google Scholar 

  • Blackwell, D. (1951). Comparison of experiments. In Proceedings of the second Berkeley symposium on mathematical statistics and probability (pp. 93–102). Berkeley: University of California Press.

  • Blackwell D. (1953) Equivalent comparison of experiments. Annals of Mathematical Statistics 24(2): 265–272

    Article  Google Scholar 

  • Bohnenblust, H., Shapley, L., & Sherman, S. (1949). Reconnaissance in game theory. Research memorandum RM-208, RAND Corporation.

  • Cavusoglu H., (2004) Configuration of detection software: A comparison of decision and game theory approaches. Decision Analysis 1(9): 131–148

    Article  Google Scholar 

  • Clemen R.T., Winkler R.L. (1993) Aggregating point estimates: A flexible modeling approach. Management Science 39(4): 501–515

    Article  Google Scholar 

  • DeGroot M. (1962) Uncertainty, information, and sequential experiments. Annals of Mathematical Statistics 33(2): 404–419

    Article  Google Scholar 

  • Eves H.W. (1988) Return to mathematical circles. Prindle, Weber, and Schmidt, Boston, MA

    Google Scholar 

  • Fawcett T. (2006) An introduction to ROC analysis. Pattern Recognition Letters 27(8): 861–874

    Article  Google Scholar 

  • Fishburn P.C. (1970) Utility theory for decision making. Wiley, New York, NY

    Google Scholar 

  • Flach, P. A., & Wu, S. (2003). Reparing concavities in ROC curves. In Proceedings of the 2003 UK workshop on computational intelligence (pp. 38–44). Bristol, UK: University of Bristol.

  • Froeb L.M., Kobayashi B.H. (1996) Naive, biased, yet Bayesian: Can juries interpret selectively produced evidence?. Journal of Law, Economics, and Organization 12(1): 257–276

    Google Scholar 

  • Hastie T., Tibshirani R., (2001) The elements of statistical learning. Springer, New York, NY

    Google Scholar 

  • Hill B.M. (1987) The validity of the likelihood principle. American Statistician 41(2): 95–100

    Article  Google Scholar 

  • Hoel P.G., Peterson R.P. (1949) A solution to the problem of optimum classification. Annals of Mathematical Statistics. 20(3): 433–438

    Article  Google Scholar 

  • Kaye D.H., Koehler J.J. (1991) Can jurors understand probabilistic evidence?. Journal of the Royal Statistical Society: Series A (Statistics in Society) 154(1): 75–81

    Article  Google Scholar 

  • Keisler J. (2004) Value of information in portfolio decision analysis. Decision Analysis 1(3): 177–189

    Article  Google Scholar 

  • Kihlstrom R.E. (1984) A Bayesian exposition of Blackwell’s theorem on the comparison of experiments. In: Boyer M., Kihlstrom R.E. (eds) Bayesian models in economic theory. Elsevier Science, North-Holland, Amsterdam, NL, pp 13–31

    Google Scholar 

  • Kreps D. (1988) Notes on the theory of choice. Westview Press, Boulder, CO

    Google Scholar 

  • Kriege M., Brekelmans C.T., Boetes C. et al (2004) Efficacy of MRI and mammography for breast-cancer screening in women with a familial or genetic predisposition. New England Journal of Medicine 351(5): 427–437

    Article  Google Scholar 

  • Lasko T.A., Bhagwat J.G., Zou K.H., Ohno-Machado L. (2005) The use of receiver operating characteristic curves in biomedical informatics. Journal of Biomedical Informatics 38(5): 404–415

    Article  Google Scholar 

  • LaValle I.H. (1968) On cash equivalents and information evaluation in decisions under uncertainty. Journal of the American Statistical Association 63(321): 252–290

    Article  Google Scholar 

  • Leach M.O., Boggis C.R.M., Dixon A.F. et al (2005) Screening with magnetic resonance imaging and mammography of a UK population at high familial risk of breast cancer: A prospective multicentre cohort study (MARIBS). Lancet 365(9473): 1769–1778

    Article  Google Scholar 

  • Lehmann E.L. (1988) Comparing location experiments. Annals of Statistics 16(2): 521–533

    Article  Google Scholar 

  • Lindley D.V. (1956) On a measure of the information provided by an experiment. Annals of Mathematical Statistics 27(4): 986–1005

    Article  Google Scholar 

  • Lindley D.V. (1964) The Bayesian analysis of contingency tables. Annals of Mathematical Statistics 35(4): 1622–1643

    Article  Google Scholar 

  • Milgrom P., Segal I. (2002) Envelope theorems for arbitrary choice sets. Econometrica 70(2): 583–601

    Article  Google Scholar 

  • Morris P.A. (1977) Combining expert judgments: A Bayesian approach. Management Science 23(7): 679–693

    Article  Google Scholar 

  • Nemec C.F., Listinsky J., (2007) How should we screen for breast cancer? Mammography, ultrasonography, MRI. Cleveland Clinic Journal of Medicine 74(12): 897–904

    Article  Google Scholar 

  • Neyman J., (1933) On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London: Series A 231: 289–337

    Article  Google Scholar 

  • Pepe M.S. (2003) The statistical evaluation of medical tests for classification and prediction. Oxford University Press, Oxford, UK

    Google Scholar 

  • Radner R., Stiglitz J.E. (1984) A nonconcavity in the value of information. In: Boyer M., Kihlstrom R.E. (eds) Bayesian models in economic theory. Elsevier Science, North-Holland, Amsterdam, NL, pp 33–52

    Google Scholar 

  • Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423, 623–656.

    Google Scholar 

  • Shannon C.E., Weaver W. (1948) The mathematical theory of communication. University of Illinois Press, Chicago, IL

    Google Scholar 

  • Strulovici B.H., Weber T.A. (2008) Monotone comparative statics: Geometric approach. Journal of Optimization Theory and Applications 137(3): 641–673

    Article  Google Scholar 

  • Wald A. (1939) Contributions to the theory of statistical estimation and testing hypotheses. Annals of Mathematical Statistics 10(4): 299–326

    Article  Google Scholar 

  • Weber T.A., Croson D.C. (2003) Selling less information for more: Garbling with benefits. Economics Letters 83(2): 165–171‘

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Management Science and Engineering, 442 Terman Engineering Center, Stanford University, Stanford, CA, 94305-4026, USA

    Thomas A. Weber

Authors
  1. Thomas A. Weber
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas A. Weber.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Weber, T.A. Simple methods for evaluating and comparing binary experiments. Theory Decis 69, 257–288 (2010). https://doi.org/10.1007/s11238-009-9136-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11238-009-9136-4

Keywords

Search

Navigation