Probabilistic coherence measures: a psychological study of coherence assessment

Abstract

Over the years several non-equivalent probabilistic measures of coherence have been discussed in the philosophical literature. In this paper we examine these measures with respect to their empirical adequacy. Using test cases from the coherence literature as vignettes for psychological experiments we investigate whether the measures can predict the subjective coherence assessments of the participants. It turns out that the participants’ coherence assessments are best described by Roche’s (Insights from philosophy, jurisprudence and artificial intelligence, 2013) coherence measure based on Douven and Meijs’ (Synthese 156:405–425, 2007) average mutual support approach and the conditional probability.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. 1.

    We excluded the Siebel and Schippers’ inconsistent testimony case from all analyses because most measures have undefined function values in this test case (see Appendix 2).

  2. 2.

    Six participants did not answer the questionnaire prior to the lab study and were therefore excluded from this analysis.

References

  1. Akiba, K. (2000). Shogenji’s probabilistic measure of coherence is incoherent. Analysis, 60, 356–359.

    Article  Google Scholar 

  2. BonJour, L. (1985). The structure of empirical knowledge. Cambridge: Harvard University Press.

    Google Scholar 

  3. Bovens, L., & Hartmann, S. (2003). Bayesian epistemology. Oxford: Oxford University Press.

    Google Scholar 

  4. Carnap, R. (1950). Logical foundations of probability. Chicago: University of Chicago Press.

    Google Scholar 

  5. Cheng, P. W. (1997). From covariation to causation: A causal power theory. Psychological Review, 104, 367–405.

    Article  Google Scholar 

  6. Cialdini, R. B., Trost, M. R., & Newsom, J. T. (1995). Preference for consistency: The development of a valid measure and the discovery of surprising behavioral implications. Journal of Personality and Social Psychology, 69, 318–328.

    Article  Google Scholar 

  7. Crupi, V., Tentori, K., & Gonzales, M. (2007). On Bayesian measures of evidential support: Theoretical and empirical issues. Philosophy of Science, 74, 229–252.

    Article  Google Scholar 

  8. Douven, I., & Meijs, W. (2007). Measuring coherence. Synthese, 156, 405–425.

    Article  Google Scholar 

  9. Festa, R. (2012). For unto every one that hath shall be given. Matthew properties for incremental confirmation. Synthese, 184, 89–100.

    Article  Google Scholar 

  10. Finch, H. A. (1960). Confirming power of observations metricized for decisions among hypotheses. Philosophy of Science, 27, 293–307.

    Article  Google Scholar 

  11. Fitelson, B. (2004). Two technical corrections to my coherence measure. http://fitelson.org/coherence2.

  12. Fitelson, B. (2003). A probabilistic theory of coherence. Analysis, 63, 194–199.

    Article  Google Scholar 

  13. Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25–42.

    Article  Google Scholar 

  14. Gaifman, H. (1979). Subjective probability, natural predicates and Hempel’s ravens. Erkenntnis, 21, 105–147.

    Google Scholar 

  15. Glass, D. H. (2002). Coherence, explanation, and Bayesian networks. In O’Neill, M., Sutcliffe, R. F. E., Ryan, C., Eaton, M., & Griffith, N. J. L. (Eds.), Artificial intelligence and cognitive science. 13th Irish conference, AICS 2002, Limerick, Ireland, September 2002 (pp. 177–182). Berlin: Springer.

  16. Glass, D. H. (2005). Problems with priors in probabilistic measures of coherence. Erkenntnis, 63, 375–385.

    Article  Google Scholar 

  17. Good, I. J. (1984). The best explicatum for weight of evidence. Journal of Statistical Computation and Simulation, 19, 294–299.

    Article  Google Scholar 

  18. Greiner, B. (2004). An online recruitment system for economic experiments. In K. Kremer & V. Macho (Eds.), Forschung und wissenschaftliches Rechnen 2003, GWDG Bericht 63 (pp. 79–93). Goettingen: Ges. fuer Wiss. Datenverarbeitung.

    Google Scholar 

  19. Harris, A., & Hahn, U. (2009). Bayesian rationality in evaluating multiple testimonies: Incorporating the role of coherence. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(5), 1366–1373.

    Google Scholar 

  20. Jeffreys, H. (1961). Theory of probability. Oxford: Oxford University Press.

    Google Scholar 

  21. Joyce, J. (2008). Bayes’ theorem. http://plato.stanford.edu/archives/fall2008/entries/bayes-theorem/.

  22. Kemeny, J., & Oppenheim, P. (1952). Degrees of factual support. Philosophy of Science, 1952, 307–324.

    Article  Google Scholar 

  23. Keynes, J. (1921). A treatise on probability. London: Macmillan.

    Google Scholar 

  24. Kolmogorov, A. (1956). Foundations of the theory of probability. New York: AMS Chelsea Publishing.

    Google Scholar 

  25. Koscholke, J. (2015). Evaluating test cases for probabilistic measures of coherence. Erkenntnis. doi:10.1007/s10670-015-9734-1.

  26. Kuipers, T. A. F. (2000). From instrumentalism to constructive realism. Dordrecht: Reidel.

    Google Scholar 

  27. Levi, I. (1962). Corroboration and rules of acceptance. British Journal for the Philosophy of Science, 13, 307–313.

    Google Scholar 

  28. Meijs, W. (2005). Probabilistic measures of coherence. PhD thesis, Erasmus University, Rotterdam.

  29. Meijs, W. (2006). Coherence as generalized logical equivalence. Erkenntnis, 64, 231–252.

    Article  Google Scholar 

  30. Meijs, W., & Douven, I. (2007). On the alleged impossibility of coherence. Synthese, 157(3), 347–360.

    Article  Google Scholar 

  31. Mortimer, H. (1988). The logic of induction. Paramus: Prentice Hall.

    Google Scholar 

  32. Nozick, R. (1981). Philosophical explanations. Oxford: Clarendon.

    Google Scholar 

  33. Olsson, E. J. (2002). What is the problem of coherence and truth? The Journal of Philosophy, 94, 246–272.

    Article  Google Scholar 

  34. Olsson, E. J. (2005). Against coherence: Truth, probability and justification. Oxford: Oxford University Press.

    Google Scholar 

  35. Pfeiffer, P. (1990). Probability for applications. New York: Springer.

    Google Scholar 

  36. Pinheiro, J., Bates, D., DebRoy, S., Sarkar, D., & R Core Team (2013). nlme: Linear and nonlinear mixed effects models. http://CRAN.R-project.org/package=nlme.

  37. Popper, K. R. (1954). Degree of confirmation. British Journal for the Philosophy of Science, 5, 143–149.

    Article  Google Scholar 

  38. R Core Team (2015). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.

  39. Rescher, N. (1958). Theory of evidence. Philosophy of Science, 25, 83–94.

    Article  Google Scholar 

  40. Rescher, N. (1973). The coherence theory of truth. Oxford: Oxford University Press.

    Google Scholar 

  41. Rips, L. J. (2001). Two kinds of reasoning. Psychological Science, 12, 129–134.

    Article  Google Scholar 

  42. Roche, W. (2013). Coherence and probability: A probabilistic account of coherence. In M. Araszkiewicz & J. Savelka (Eds.), Coherence: Insights from philosophy, jurisprudence and artificial intelligence (pp. 59–91). Dordrecht: Springer.

    Google Scholar 

  43. Schippers, M. (2014). Probabilistic measures of coherence: From adequacy constraints towards pluralism. Synthese, 191(16), 3821–3845.

    Article  Google Scholar 

  44. Schippers, M., & Siebel, M. (2015). Inconsistency as a touchstone for coherence measures. Theoria, 30, 11–41.

    Article  Google Scholar 

  45. Schupbach, J. N. (2011). New hope for Shogenji’s coherence measure. British Journal for the Philosophy of Science, 62(1), 125–142.

    Article  Google Scholar 

  46. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464.

    Article  Google Scholar 

  47. Shogenji, T. (1999). Is coherence truth conducive? Analysis, 59, 338–345.

    Article  Google Scholar 

  48. Shogenji, T. (2012). The degree of epistemic justification and the conjunction fallacy. Synthese, 184, 29–48.

    Article  Google Scholar 

  49. Siebel, M. (2004). On Fitelson’s measure of coherence. Analysis, 64, 189–190.

    Article  Google Scholar 

  50. Siebel, M., & Wolff, W. (2008). Equivalent testimonies as a touchstone of coherence measures. Synthese, 161, 167–182.

    Article  Google Scholar 

  51. Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of \(p\) values. Psychonomic Bulletin & Review, 14, 779–804.

    Article  Google Scholar 

  52. Weller, J. A., Dieckmann, N. F., Tusler, M., Mertz, C. K., Burns, W. J., & Peters, E. (2013). Development and testing of an abbreviated numeracy scale: A Rasch analysis approach. Journal of Behavioral Decision Making, 26, 198–212.

    Article  Google Scholar 

Download references

Acknowledgments

We would like to thank (in alphabetical order) Arndt Bröder, Andreas Glöckner, Björn Meder, Michael Schippers and Mark Siebel for their contributions. We would also like to thank the participants of the Operationalization Workshop 2013 in Freiburg for helpful comments. This work was supported by grant SI 1731/1-1 to Mark Siebel and grant GL 632/3-1 and BR 2130/8-1 to Andreas Glöckner and Arndt Bröder from the Deutsche Forschungsgemeinschaft (DFG) as part of the priority program “New Frameworks of Rationality” (SPP 1516).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jakob Koscholke.

Appendices

Appendix 1: Test cases

Akiba’s (2000) Die case

Imagine rolling a fair die and consider the following three statements:

  • \(S_1:\) The die comes up 2.

  • \(S_2:\) The die comes up 2 or 4.

  • \(S_3:\) The die comes up 2 or 4 or 6.

Which pair of statements fits together better. Statement 1 and 2 or statement 1 and 3?

Bovens and Hartmann’s (2003) tweety case

  1. Situation 1:

    Consider a population of 100 animals. 50 out of 100 animals are birds and 50 out of 100 animals cannot fly. Among these 100 animals there is exactly one animal that is a bird and cannot fly. Randomly pick one animal and consider the following two statements:

    • \(S_1:\) The picked animal is a bird.

    • \(S_2:\) The picked animal cannot fly.

  2. Situation 2:

    Consider a population of 100 animals. 50 out of 100 animals are birds and 50 out of 100 animals cannot fly. Among these 100 animals there is exactly one penguin and therefore a bird that cannot fly. Randomly pick one animal and consider the following three statements:

    • \(S_1:\) The picked animal is a bird.

    • \(S_2:\) The picked animal cannot fly.

    • \(S_3:\) The picked animal is a penguin.

In which of the two situations do the respective sets of statements fit together better?

Bovens and Hartmann’s (2003) Tokyo murder case

Imagine that a murder has occurred in Toyko and the corpse is still to be found. In order to search more efficiently the map of Tokyo is separated into 100 equally-sized where the probability of finding the corpse is the same for each square. Now, 5 pairs of equally reliable and independent witnesses give the following statements as witness reports:

  • Pair 1: 

    • \(S_1:\) The corpse is in squares 50 to 60.

    • \(S_2:\) The corpse is in squares 51 to 61.

  • Pair 2: 

    • \(S_1:\) The corpse is in squares 22 to 55.

    • \(S_2:\) The corpse is in squares 55 to 90.

  • Pair 3: 

    • \(S_1:\) The corpse is in squares 20 to 61.

    • \(S_2:\) The corpse is in squares 50 to 91.

  • Pair 4: 

    • \(S_1:\) The corpse is in squares 41 to 60.

    • \(S_2:\) The corpse is in squares 51 to 70.

  • Pair 5: 

    • \(S_1:\) The corpse is in squares 39 to 61.

    • \(S_2:\) The corpse is in squares 50 to 72.

Which pair of statements fits together best, which worst? Can you give and ordering where the first pair is the best and the last pair the worst?

Glass’ (2005) Dodecahedron case

  • Situation 1: You are rolling a fair die.

  • Situation 2: You are rolling a fair dodecahedron.

Now consider the following two statements:

  • \(S_1:\) The result will be 2.

  • \(S_2:\) The result will be 2 or 4.

In which of the two situation do these two statements fit together better?

Meijs’ (2005) Samurai case

  1. Situation 1:

    There are 10,000,000 suspects in a murder case. 1059 of them are Japanese and also 1059 own a samurai sword such that in total there are 9 suspects who are Japanese and own a samurai sword at the same time.

  2. Situation 2:

    There are 100 suspects in a murder case. 10 of them are Japanese and also 10 own a samurai sword such that in total there are 9 suspects who are Japanese and own a samurai sword at the same time.

Now consider the following two statements:

  • \(S_1:\) The murderer is Japanese.

  • \(S_2:\) The murderer owns a samurai sword.

In which of the two situations do the two statements fit together better?

Meijs’ (2006) Albino rabbit case

  1. Situation 1:

    There are 102 rabbits on the first island. 101 out of these 102 rabbits are grey. Also, 101 out of 102 rabbits have two ears. In total there are 100 out of 102 rabbits which are grey and have two ears at the same time. Consequently, there is exactly one rabbit which is grey but does not have two ears and exactly one rabbits which is not grey but has two ears.

  2. Situation 2:

    There are 102 rabbits on the second island, too. 100 out of these 102 rabbits are grey. Also, 100 out of 102 rabbits have two ears. In total there are 100 out of 102 rabbits which are grey and have two ears at the same time. Consequently, every grey rabbit has two ears and every rabbit that has two ears is also grey.

Now, randomly pick one rabbit and consider the following two statements:

  • \(S_1:\) The rabbit is grey.

  • \(S_2:\) The rabbits has two ears.

In which of the two situations do these two statements fit together better?

Meijs and Douven’s (2007) plane lottery case

Imagine the following lottery. The chances are 4 / 100 for flying to the North pole, 49 / 100 for flying to the South pole and 47 / 100 for flying to New Zealand. The probability for seeing a penguin at the North pole is 0, at the South pole it is 10 / 49 and in New Zealand it is 1 / 47. Now consider the following two situations in which when having landed one is confronted with two statements.

  • Situation 1: 

    • \(S_1:\) You are landing at the North pole.

    • \(S_2:\) The animal you see is a penguin.

  • Situation 2: 

    • \(S_1:\) You are landing at the South pole.

    • \(S_2:\) The animal you see is a penguin.

In which of the two situations do the respective statements fit together better?

Schippers and Siebel’s (2015) inconsistent testimony case

Imagine there are 8 suspects for a robbery. It is certain that exactly one of them is the robber. Consider the following two situations in which two independent and equally reliable witnesses give statements about the robber:

  • Situation 1: 

    • \(S_1:\) The robbery was committed by suspect 1 or 2.

    • \(S_2:\) The robbery was committed by suspect 2 or 3.

    • \(S_3:\) The robbery was committed by suspect 1 or 3.

  • Situation 2: 

    • \(S_1:\) The robbery was committed by suspect 1 or 2.

    • \(S_2:\) The robbery was committed by suspect 3 or 4.

    • \(S_3:\) The robbery was committed by suspect 5 or 6.

In which of these two situations do the respective sets of statements fit together better?

Schupbach’s (2011) Robber case

Imagine there are 10 suspects for a robbery. It is certain that exactly one of them is the robber. Consider the following two situations in which two independent and equally reliable witnesses make give statements about the robber:

  • Situation 1: 

    • \(S_1:\) The robbery was committed by suspect 1 or 2 or 3.

    • \(S_2:\) The robbery was committed by suspect 1 or 2 or 4.

    • \(S_3:\) The robbery was committed by suspect 1 or 3 or 4.

  • Situation 2: 

    • \(S_1:\) The robbery was committed by suspect 1 or 2 or 3.

    • \(S_2:\) The robbery was committed by suspect 1 or 4 or 5.

    • \(S_3:\) The robbery was committed by suspect 1 or 6 or 7.

In which of these two situations do the respective sets of statements fit together better?

Siebel’s (2004) pickpocketing robber case

Imagine the following situation. There are 10 equally likely suspects for a murder. 8 out of 10 have committed a pickpocketing before, 8 out of 10 have committed a robbery and in total 6 out of 10 have committed a pickpocketing and a robbery. Now consider the following two statements:

  • \(S_1:\) The murderer has committed a robbery.

  • \(S_2:\) The murderer has committed a pickpocketing.

Do these two statements fit together or not?

Appendix 2: Test case results

See Table 2.

Table 2 Summary of the results

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Koscholke, J., Jekel, M. Probabilistic coherence measures: a psychological study of coherence assessment. Synthese 194, 1303–1322 (2017). https://doi.org/10.1007/s11229-015-0996-6

Download citation

Keywords

  • Bayesian coherentism
  • Probabilistic coherence measures
  • Probabilistic support measures
  • Test cases
  • Experimental philosophy