pp 1–15 | Cite as

Exclusion Criteria in Experimental Philosophy

  • Carsten Bergenholtz
  • Jacob BuschEmail author
  • Sara Kier Praëm
Original Research


When experimental philosophers carry out studies on thought experiments, some participants are excluded based on certain exclusion criteria, mirroring standard social science vignette methodology. This involves excluding people that do not pay attention or who miscomprehend the scenario presented in thought experiments. However, experimental philosophy studies sometimes exclude an alarmingly high number of participants. We argue that this threatens the external and internal validity of the conclusions being drawn and we show how a simple visualization of thought experiments can reduce exclusion rates significantly. Furthermore, we argue that focus should not merely be on how many are excluded, but also why they are excluded, and we highlight the role of comprehension questions in this regard. Philosophical thought experiments often rely on the acceptance of certain key premises that may be regarded contestable, and asking comprehension questions involving such key assumptions could be problematic as that may result in some participants being inadvertently excluded from the study, potentially creating a selection bias.



The authors wish to thank Vilius Dranseika, Oana Vuculescu and two anonymous reviewers for valuable comments and suggestions. Research for this publication was supported by grants from the Carlsberg Foundation (CF15-0943) and the Independent Research Fund Denmark (DFF-4180-00071).


  1. Alexander, C. S., & Becker, H. J. (1978). The use of vignettes in survey research. Public Opinion Quarterly, 42, 93–104.CrossRefGoogle Scholar
  2. Buckwalter, W. (2013). Gettier made ESEE. Philosophical Psychology, 27(3), 368–383.CrossRefGoogle Scholar
  3. Chisholm, R. (1966). Theory of knowledge. N.J.: Englewood Cliffs.Google Scholar
  4. Clark, C. J., Winegard, B. M., & Baumeister, R. F. (2019). Forget the folk: Moral responsibility preservation motives and other conditions for compatibilism. Frontiers in Psychology, 10, 215.CrossRefGoogle Scholar
  5. Cohen, M. (1999). 101 philosophy problems. London: Routledge.Google Scholar
  6. Colaço, D., Buckwalter, W., Stich, S., & Machery, E. (2014). Epistemic intuitions in fake-barn thought experiments. Episteme, 11(02), 199–212.CrossRefGoogle Scholar
  7. Douglas, K., Otto, R., Desmarais, S., & Borum, R. (2001). Clinical forensic psychology. In I. B. Weiner, J. A. Schinka, & W. F. Velicer (Eds.), Handbook of psychology, research methods in psychology. New York: Wiley.Google Scholar
  8. Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4), 25–42.CrossRefGoogle Scholar
  9. Horvath, J. (2010). How (not) to react to experimental philosophy. Philosophical Psychology, 23, 447–480.CrossRefGoogle Scholar
  10. Jackson, F. (1986). What Mary didn’t know. Journal of Philosophy, 83(5), 291–295.CrossRefGoogle Scholar
  11. Kim, M., & Yuan, Y. (2014). No cross-cultural differences in the Gettier car case intuition: A replication study. Episteme, 12(3), 355–361.CrossRefGoogle Scholar
  12. Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Jr., Alper, S., et al. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1, 443–490. Scholar
  13. Levi, W. H., & Lentz, R. (1982). Effects of text illustrations: A review of research. Educational Communication and Technology Journal, 30(4), 195–232.Google Scholar
  14. Little, R. J. A., & Rubin, D. B. (2014). Statistical analysis with missing data. New York: Wiley.Google Scholar
  15. Machery, E., Stich, S., Rose, D., Chatterjee, A., Karasawa, K., Struchiner, N., et al. (2015). Gettier across cultures. Noûs, 51(3), 645–664.CrossRefGoogle Scholar
  16. Maniaci, M., & Rogge, R. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48(1), 61–83.CrossRefGoogle Scholar
  17. Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.CrossRefGoogle Scholar
  18. Murray, D., & Nahmias, E. (2014). Explaining away incompatibilist intuitions. Philosophy and Phenomenological Research, 88(2), 434–467.CrossRefGoogle Scholar
  19. Nahmias, E., Coates, D. J., & Kvaran, T. (2007). Free will, moral responsibility, and mechanism: Experiments on folk intuitions. Midwest Studies in Philosophy, 31(1), 214–242.CrossRefGoogle Scholar
  20. Nahmias, E., Morris, S., Nadelhoffer, T., & Turner, J. (2005). Surveying freedom: Folk intuitions about free will and moral responsibility. Philosophical Psychology, 18(5), 561–584.CrossRefGoogle Scholar
  21. Nichols, S., & Bruno, M. (2010). Intuitions about personal identity: An empirical study. Philosophical Psychology, 23, 293–312.CrossRefGoogle Scholar
  22. Nichols, S., & Knobe, J. (2007). Moral responsibility and determinism: The cognitive science of folk intuitions. Nous, 41, 663–685.CrossRefGoogle Scholar
  23. Oppenheimer, D., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872.CrossRefGoogle Scholar
  24. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.Google Scholar
  25. Philips, J. & Cushman, F. (2017) Replication of Nahmias, Coates, & Kavarn (2007) Free will, moral responsibility, and mechanism: Experiments on folk intuitions.
  26. Polonioli, A., Vega-Mendoza, M., Blankinship, B., & Carmel, D. (2018). Reporting in experimental philosophy: Current standards and recommendations for future practice. Review of Philosophy and Psychology. Scholar
  27. Searle, J. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, 3(3), 417–457.CrossRefGoogle Scholar
  28. Seyedsayamdost, H. (2015). On normativity and epistemic intuitions: Failure of replication. Episteme, 12(1), 95–116.CrossRefGoogle Scholar
  29. Sorensen, R. (1992). Thought experiments. New York: Oxford University Press.Google Scholar
  30. Sosa, E. (2009). A defense of the use of intuitions in philosophy. In M. Bishop & D. Murphy (Eds.), Stich and his critics (pp. 101–112). Oxford: Blackwell.Google Scholar
  31. Starmans, C., & Friedman, O. (2012). The folk conception of knowledge. Cognition, 124, 272–283.CrossRefGoogle Scholar
  32. Thomas, K., & Clifford, S. (2017). Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments. Computers in Human Behavior, 77, 184–197.CrossRefGoogle Scholar
  33. Turri, J. (2013). A conspicuous art: Putting Gettier to the test. Philosophers’ Imprint, 13, 1–16.Google Scholar
  34. van’t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12.CrossRefGoogle Scholar
  35. Weinberg, J. M., Nichols, S., & Stich, S. (2001). Normativity and epistemic intuitions. Philosophical topics, 49(1&2), 429–460.CrossRefGoogle Scholar
  36. Williams, B. (1973). Problems of the self. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  37. Williamson, T. (2011). Philosophical expertise and the burden of proof. Metaphilosophy, 42, 215–229.CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Department of ManagementAarhus UniversityAarhus VDenmark
  2. 2.Department of PhilosophyAarhus UniversityAarhus CDenmark
  3. 3.Department of Mathematics (Centre for Science Studies)Aarhus UniversityAarhus CDenmark

Personalised recommendations