Ordering effects, updating effects, and the specter of global skepticism

Abstract

One widely-endorsed argument in the experimental philosophy literature maintains that intuitive judgments are unreliable because they are influenced by the order in which thought experiments prompting those judgments are presented. Here, we explicitly state this argument from ordering effects and show that any plausible understanding of the argument leads to an untenable conclusion. First, we show that the normative principle is ambiguous. On one reading of the principle, the empirical observation is well-supported, but the normative principle is false. On the other reading, the empirical observation has only weak support, and the normative principle, if correct, would impugn the reliability of deliberative reasoning, testimony, memory, and perception, since judgments in all these areas are sensitive to ordering in the relevant sense. We then reflect on what goes wrong with the argument.

This is a preview of subscription content, access via your institution.

Notes

  1. 1.

    Throughout this paper, we will talk about intuitive judgments and the (propositional) contents of intuitive judgments. In our experience, the term “intuition” too easily slips back and forth between a mental state and the (propositional) content of a mental state. Readers who think that intuitions are non-propositional intellectual seemings, inclinations to make judgments, or something similar may make suitable substitutions in our arguments and in the arguments that we attribute to various experimental philosophers.

  2. 2.

    We are here treating “unreliable” as having normative content, in a sense similar to “untrustworthy.” Anyone bothered by this choice could rephrase the principle to indicate that judgments systematically affected by ordering should not be trusted or ought not to be relied on.

  3. 3.

    In their conclusion, Swain et al. write: “Specifically, we found that intuitions about the Truetemp Case vary depending on whether, and which, other cases are presented before it. Such variability calls into question the legitimacy of using the intuitions generated by the Truetemp Case as evidence against reliabilism. But it is unclear what about this case makes it susceptible to these effects, which raises questions about the reliance on intuitions about thought-experiments more generally, especially given that this is not the only case called into question by empirical research” (153, emphasis added). They go on to write: “We certainly do not take ourselves to have offered anything like a general proof of the unreliability of all intuitions (nor do we think that any such proof would be either possible or desirable). But we do take ourselves to have raised a serious empirical worry that philosophers need to begin deciding how to address.”

  4. 4.

    An anonymous referee worried that our choice of labels here is likely to introduce more confusion into the literature. In response, let us be very clear about our terminological choice. Consistent with standard usage in the field, we will use “order effect” to indicate any case where participants’ responses are systematically affected by the order in which stimuli are presented. The category of order effects then divides into two sub-types: the genuine or proper order effects and the updating effects. We retain the label “order effects” for one sub-type in part because that is consistent with the usage in psychology (for which see Hogarth and Einhorn 1992), in part because that sub-type is plausibly prima facie epistemically problematic in a way that the other species is not, and in part because we do not have any better, more evocative label to provide. The distinction is purely descriptive. There are two structurally different kinds of ordering effect. We label one of them “genuine ordering effects” and the other “updating effects.”

  5. 5.

    We are not assuming that judgments or cognitive processes that exhibit updating effects are reliable. What we are claiming is that the fact that a judgment or cognitive process exhibits an updating effect is not prima facie reason to think that judgment or cognitive process is not reliable. Put another way: If all one knows about a judgment is that it exhibits an updating effect, that doesn’t provide a reason to think that it is unreliable. It might be unreliable, but whether or not it is unreliable depends on much more than sensitivity to ordering. For example, perhaps a judgment is sensitive to an irrelevant feature. However, we think it is too hasty to conclude that prior cases are an irrelevant feature as we will discuss in the next few pages of the paper.

  6. 6.

    One point of interest is that if researchers have participants’ retrieve a source story from long term memory, and both the target and the source stories are particularly long, stories with superficial similarity are more likely to influence people’s judgments about the target story than stories that are structurally similar to each other. However, in the empirical studies of philosophical ordering effects, the thought experiments presented to people are both short and presented without delay between each thought experiment (see Hogarth and Einhorn’s meta-analysis for why length of vignette hinders ordering effects).

  7. 7.

    Hummel (personal communication) tells us: “There is no precise definition [of a schema]. Generally, it is a structured (i.e., explicitly relational; formally more powerful than a simple feature list) representation that summarizes the core/common properties of some domain of knowledge, especially the most important relations (including higher-order relations) governing that domain.” See Table 3 in Hummel and Holyoak (2003) for a short list of functional properties that schemas have.

  8. 8.

    See Horne et al. (2013) for a similar account.

  9. 9.

    However, at this point it is unknown whether people update their schemas explicitly or implicitly.

  10. 10.

    In typical cases, what we are calling the background belief will actually be a network of related beliefs.

  11. 11.

    The Control Case was the Coupon Case: the 9th non-moral dilemma from the supplementary materials of Greene et al. (2001).

  12. 12.

    In the Control Condition, the mean response was \(\hbox {M}=5.8, \hbox {SD}=1.2\).

  13. 13.

    In Condition 2, the mean response was \(\hbox {M}=5.1, \hbox {SD}= 1.56\). In Condition 3, the mean response was \(\hbox {M}=4.1, \hbox {SD}= 1.52\).

  14. 14.

    Here we note several other measures these researchers took to rule out alternative hypotheses for their results. In light of well-known difficulties with spontaneous recall of source problems (Gick and Holyoak 1983), we have every reason to believe these measures were effective. First, the scale participants responded on only had the endpoints labeled. Consequently, participants had very little information to go on about the degree of agreement they were assigning to their belief (e.g., 73 out of 100 agreement with the belief). For this reason, it would be extremely difficult for participants to recover the exact position of their response on the scale during the first phase of the study, especially because this belief was one of many controversial beliefs that participants had to consider. Finally, the moral dilemma used to manipulate people’s beliefs (i.e., the Transplant dilemma) was embedded among many other moral dilemmas, half of which were related to distractor beliefs noted above. In sum, the presence of belief distractors and dilemma distractors, as well as the scale Horne and colleagues used, make it much more likely that people merely reported their beliefs during phases 1 and 2 of the study, rather than remembering their rating in the first phase of the study and altering their agreement rating accordingly during the second phase of the study.

  15. 15.

    Whereas ordinary Bayesian conditionalization is given by the equation \(Cr_{New} (p)=Cr_{Old} (p|e)\), the simplest case of Jeffrey conditionalization is given by the equation \(Cr_{New} (p)=Cr_{Old} (p|e)\cdot Cr_{New} (e)+Cr_{Old} (p|\sim e)\cdot Cr_{New} (\sim e)\) to account for the fact that we might not be certain about our evidence e for the proposition p.

  16. 16.

    Although we will not focus on it here, the way that Lange describes the evidence also problematizes, we think, how to count the number of pieces of evidence one has. When a child sees her mother laugh and then scowl, does she see and experience a single thing (a laugh-to-scowl) or two things (a laugh and a scowl)? If she experiences two things, does she also experience temporal succession or is each event indexed by time? And if the time index matters to the nature of her experience, isn’t that just to sneak in the supposedly irrelevant information about the order of her pieces of evidence? We have no answers to these questions at present, but surely anyone interested in defending an order-invariance principle must answer them.

  17. 17.

    And that same response problematizes conclusions drawn from the psychological research we surveyed in Sect. 3 as well. Thanks to Conor Mayo-Wilson for pointing this out.

  18. 18.

    We are not here endorsing the view that differences in the judgments participants make are actually due to differences in how vividly they imagine the cases; our goal here is merely to give a how-possibly reconstruction. But as far as we know, the experimental evidence is consistent with such a view. For example, differences in vividness, as opposed to differences in personal contact, might be consistent with Schwitzgebel and Cushman ’s (2015) results respecting the Footbridge, Trapdoor, and Switch variants of the Trolley dilemma.

References

  1. Allan, L. (1977). The time-order error in judgments of duration. Canadian Journal of Psychology, 31, 24–31.

    Article  Google Scholar 

  2. Anderson, N. (1971). Integration theory and attitude change. Psychological Review, 78, 171–206.

    Article  Google Scholar 

  3. Anderson, J., Bothell, D., Lebiere, C., & Matessa, M. (1998). An Integrated Theory of List Memory. Journal of Memory and Language, 38(4), 341–380.

    Article  Google Scholar 

  4. Ashton, A., & Ashton, R. (1988). Sequential belief revision in auditing. The Accounting Review, 63, 623–641.

    Google Scholar 

  5. Bengson, J. (2013). Experimental Attacks on Intuitions and Answers. Philosophy and Phenomenological Research, 86(3), 495–532.

    Article  Google Scholar 

  6. Cromwell, H. (1950). The relative effect on audience attitude of the first versus the second argumentative speech of a series. Speech Monographs, 17, 105–122.

    Article  Google Scholar 

  7. Cullen, S. (2010). Survey-Driven Romanticism. Review of Philosophy and Psychology, 1(2), 275–296.

    Article  Google Scholar 

  8. Davidson, D. (2001). “A Coherence Theory of Truth and Knowledge,” In Subjective and Intersubjective, Objective. Oxford: Clarendon Press.

  9. Dean, M. (1980). Presentation order effects in product taste tests. The Journal of Psychology, 105, 107–110.

    Article  Google Scholar 

  10. Döring, F. (1999). Why Bayesian psychology is incomplete. Philosophy of Science, 379-389.

  11. Doumas, L. A., Hummel, J. E., & Sandhofer, C. M. (2008). A theory of the discovery and predication of relational concepts. Psychological Review, 115(1), 1–43.

    Article  Google Scholar 

  12. Englund, M., & Hellström, Å. (2012). Presentation-order effects for aesthetic stimulus preference. Attention, Perception, & Psychophysics, 74, 1499–1511.

    Article  Google Scholar 

  13. Eskine, K., Kacinik, N., & Prinz, J. (2011). A bad taste in the mouth: gustatory disgust influences moral judgment. Psychological Science, 22(3), 295–299.

    Article  Google Scholar 

  14. Feigenbaum, E., & Simon, H. (1962). A theory of the serial position effect. British Journal of Psychology, 53(3), 307–320.

    Article  Google Scholar 

  15. Feldman, R. (1988). Having Evidence. In D. Austin (Ed.), Philosophical Analysis. Dordrecht: Kluwer.

    Google Scholar 

  16. Feltz, A., & Cokely, E. (2011). Individual differences in theory-of-mind judgments: Order effects and side effects. Philosophical Psychology, 24, 343–355.

    Article  Google Scholar 

  17. Feltz, A., & Cokely, E. (2012). The Philosophical Personality Argument. Philosophical Studies, 161(2), 227–246.

    Article  Google Scholar 

  18. Gentner, D., Rattermann, M. J., & Forbus, K. D. (1993). The roles of similarity in transfer: Separating retrievability from inferential soundness. Cognitive psychology, 25(4), 524–575.

    Article  Google Scholar 

  19. Gettier, E. (1963). Is justified true belief knowledge? Analysis, 23(6), 121–123.

    Article  Google Scholar 

  20. Gick, M., & Holyoak, K. (1983). Schema Induction and Analogical Transfer. Cognitive Psychology, 15(1), 1–38.

    Article  Google Scholar 

  21. Girotto, V., Mazzocco, A., & Tasso, A. (1997). The effect of premise order in conditional reasoning: a test of the mental model theory. Cognition, 63, 1–28.

    Article  Google Scholar 

  22. Greene, J., et al. (2001). An fMRI Investigation of Emotional Engagement in Moral Judgment. Science, 293(5537), 2105–2108.

    Article  Google Scholar 

  23. Haidt, J., et al. (1993). Affect, Culture, and Morality, or Is It Wrong to Eat Your Dog? Journal of Personality and Social Psychology, 65(4), 613–628.

    Article  Google Scholar 

  24. Haarmann, H., & Usher, M. (2001). Maintenance of semantic information in capacity-limited item short-term memory. Psychonomic Bulletin & Review, 8, 568–578.

    Article  Google Scholar 

  25. Hogarth, R., & Einhorn, H. (1992). Order Effects in Belief Updating: The Belief-Adjustment Model. Cognitive Psychology, 24(1), 1–55.

    Article  Google Scholar 

  26. Holyoak, K., & Simon, D. (1999). Bidirectional Reasoning in decision making by constraint satisfaction. Journal of Experimental Psychology: General, 128, 3–31.

    Article  Google Scholar 

  27. Horne, Z., Powell, D., & Spino, J. (2013). Belief Updating in Moral Dilemmas. Review of Philosophy and Psychology, 4(4), 705–714.

    Article  Google Scholar 

  28. Horne, Z., Powell, D., & Hummel, J. (2015). A single counterexample leads to stable moral belief updating. Cognitive Science.

  29. Howard, M., & Kahana, M. (1999). Contextual variability and serial position effects in free recall. Journal of experimental psychology Learning memory and cognition, 25(4), 923–941.

    Article  Google Scholar 

  30. Hummel, J., & Holyoak, K. (2003). A Symbolic-Connectionist Theory of Relational Inference and Generalization. Psychological Review, 110(2), 220–264.

    Article  Google Scholar 

  31. Jamieson, D., & Petrusic, W. (1975). Presentation order effects in duration discrimination. Perception & Psychophysics, 17, 197–202.

    Article  Google Scholar 

  32. Lewis, D. (1996). Elusive Knowledge. Australasian Journal of Philosophy, 74, 549–567.

    Article  Google Scholar 

  33. Liao, S. M., Wiegmann, A., Alexander, J., & Vong, G. (2012). Putting the trolley in order: Experimental philosophy and the loop case. Philosophical Psychology, 25(5), 661–671.

    Article  Google Scholar 

  34. Machery, E., Mallon, R., Nichols, S., & Stich, S. P. (2004). Semantics, cross-cultural style. Cognition,92(3), B1–B12.

  35. Markman, A. (1997). Constraints on analogical inference. Cognitive Science, 21(4), 373–418.

    Article  Google Scholar 

  36. McDermott, M. (1995). Redundant causation. British Journal for the Philosophy of Science, 46(4), 523–544.

    Article  Google Scholar 

  37. Murdock, B. (1962). The serial position effect of free recall. Journal of Experimental Psychology, 64(5), 482–488.

    Article  Google Scholar 

  38. Nadelhoffer, T., & Feltz, A. (2008). The Actor-Observer Bias and Moral Intuitions. Neuroethics, 1(2), 133–144.

    Article  Google Scholar 

  39. Neta, R. (2008). What Evidence Do You Have? British Journal for the Philosophy of Science, 59, 89–119.

    Article  Google Scholar 

  40. Newell, A., & Simon, H. (1972). Human Problem Solving. Englewood Cliffs, NJ: Prentice-Hall.

    Google Scholar 

  41. Nichols, S., & Knobe, J. (2007). Moral Responsibility and Determinism. Nous, 41(4), 663–685.

    Article  Google Scholar 

  42. Oberauer, K., Hörnig, R., Weidenfeld, A., & Wilhelm, O. (2005). Effects of directionality in deductive reasoning: II. Premise integration and conclusion evaluation. The Quarterly Journal of Experimental Psychology, 58(7), 1225–1247.

    Article  Google Scholar 

  43. Parducci, A. (1965). Category judgment: a range-frequency model. Psychological review, 72(6), 407–418.

    Article  Google Scholar 

  44. Penn, D. C., Holyoak, K. J., & Povinelli, D. J. (2008). Darwin’s mistake: Explaining the discontinuity between human and nonhuman minds. Behavioral and Brain Sciences, 31(2), 109–130.

    Google Scholar 

  45. Pennington, D. (1982). Witnesses and Their Testimony: Effects of Ordering on Jurors Verdicts. Journal of Applied Social Psychology, 4, 318–333.

    Article  Google Scholar 

  46. Petrinovich, L., & O’Neill, P. (1996). Influence of wording and framing effects on moral intuitions. Ethology and Sociobiology, 17(3), 145–171.

    Article  Google Scholar 

  47. Prinz, J. (2007). The Emotional Construction of Morals. Oxford: Oxford University Press.

    Google Scholar 

  48. Quine, W. (1969). “Epistemology Naturalized,” In Ontological Relativity & Other Essays. New York: Columbia University Press.

  49. Rachels, J. (1975). Active and Passive Euthanasia. New England Journal of Medicine, 292(2), 78–80.

    Article  Google Scholar 

  50. Roediger, H., & Crowder, R. (1976). A serial position effect in recall of United States presidents. Bulletin of the Psychonomic Society, 8(4), 275–278.

    Article  Google Scholar 

  51. Scarpi, D. (2004). Effects of Presentation Order on Product Evaluation: An Empirical Analysis. The International Review of Retail, Distribution and Consumer Research, 14, 309–319.

    Article  Google Scholar 

  52. Schulz, E., Cokely, E. T., & Feltz, A. (2011). Persistent bias in expert judgments about free will and moral responsibility: A test of the expertise defense. Consciousness and Cognition, 20(4), 1722–1731.

    Article  Google Scholar 

  53. Schwitzgebel, E., & Cushman, F. (2012). Expertise in Moral Reasoning? Mind & Language, 27(2), 135–153.

    Article  Google Scholar 

  54. Schwitzgebel, E., & Cushman, F. (2015). Philosophers’ biased judgments persist despite training, expertise and reflection. Cognition, 141, 127–137.

    Article  Google Scholar 

  55. Sinnott-Armstrong, W. (2008). Framing Moral Intuitions In Moral Psychology, Vol. 2: The cognitive science of morality: Intuition and diversity. Cambridge: MIT Press.

  56. Sosa, E. (2007). Experimental Philosophy and Philosophical Intuition. Philosophical Studies, 132(1), 99–107.

    Article  Google Scholar 

  57. Stevens, S. S., & Galanter, E. H. (1957). Ratio scales and category scales for a dozen perceptual continua. Journal of Experimental Psychology, 54(6), 377–411.

    Article  Google Scholar 

  58. Swain, S., Alexander, J., & Weinberg, J. M. (2008). The instability of philosophical intuitions: Running hot and cold on truetemp. Philosophy and Phenomenological Research, 76(1), 138–155.

    Article  Google Scholar 

  59. Sytsma, J. et al. (2015). Gödel in the Land of the Rising Sun. Review of Philosophy and Psychology, 6(2), 213–230.

  60. Tobia, K., Buckwalter, W., & Stich, S. (2012). Moral intuitions: Are philosophers experts? Philosophical Psychology, 1-10.

  61. Walker, L., Thibaut, J., & Andreoli, V. (1972). Order of presentation at trial. Yale Law Journal, 82, 216–226.

    Article  Google Scholar 

  62. Weiss, D., & Anderson, N. (1969). Subjective averaging of length with serial presentation. Journal of Experimental Psychology, 82, 52–63.

    Article  Google Scholar 

  63. Wiegmann, A., & Okan, J. (2012). Order effects in moral judgments. Searching for an explanation. In Miyake, N., Peebles, D., & Cooper, R.P. (Eds.), Proceedings of the \(34^{th}\) Annual Conference of the Cognitive Science Society (pp. 1143-1148). Austin, TX. Cognitive Science Society.

  64. Wiegmann, A., Okan, J., & Nagel, J. (2012). Order effects in moral judgment. Philosophical Psychology, 1-24.

  65. Wiegmann, A., & Waldmann, M. R. (2014). Transfer effects between moral dilemmas: A causal model theory. Cognition, 131(1), 28–43.

    Article  Google Scholar 

  66. Williamson, T. (2000). Knowledge and its Limits. Oxford: Oxford University Press.

    Google Scholar 

  67. Williamson, T. (2004). Philosphical ‘Intuitions’ and Scepticism about Judgement. dialectica, 58(1), 109–153.

    Article  Google Scholar 

  68. Williamson, T. (2008). The philosophy of philosophy. Oxford: John Wiley & Sons.

    Google Scholar 

  69. Williamson, T. (forthcoming). “Philosophical Criticisms of Experimental Philosophy,” In Sytsma and Buck-walter.

  70. Wright, J. C. (2010). On intuitional stability: The clear, the strong, and the paradigmatic. Cognition, 115(3), 491–503.

    Article  Google Scholar 

Download references

Acknowledgments

Thanks to Josh Alexander, Wes Buckwalter, Greg Gandenberger, Balazs Gyenis, John Hummel, Josh Knobe, Dan Korman, Conor Mayo-Wilson, Derek Powell, David Rose, Jonah Schupbach, Eric Schwitzgebel, John Turri, Jonathan Waskan, Dan Malinsky, Shaun Nichols, and two anonymous referees for comments on earlier drafts.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Zachary Horne.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Horne, Z., Livengood, J. Ordering effects, updating effects, and the specter of global skepticism. Synthese 194, 1189–1218 (2017). https://doi.org/10.1007/s11229-015-0985-9

Download citation

Keywords

  • Ordering effects
  • Updating effects
  • Skepticism
  • Thought experiments
  • Experimental philosophy