Exclusion Criteria in Experimental Philosophy
When experimental philosophers carry out studies on thought experiments, some participants are excluded based on certain exclusion criteria, mirroring standard social science vignette methodology. This involves excluding people that do not pay attention or who miscomprehend the scenario presented in thought experiments. However, experimental philosophy studies sometimes exclude an alarmingly high number of participants. We argue that this threatens the external and internal validity of the conclusions being drawn and we show how a simple visualization of thought experiments can reduce exclusion rates significantly. Furthermore, we argue that focus should not merely be on how many are excluded, but also why they are excluded, and we highlight the role of comprehension questions in this regard. Philosophical thought experiments often rely on the acceptance of certain key premises that may be regarded contestable, and asking comprehension questions involving such key assumptions could be problematic as that may result in some participants being inadvertently excluded from the study, potentially creating a selection bias.
The authors wish to thank Vilius Dranseika, Oana Vuculescu and two anonymous reviewers for valuable comments and suggestions. Research for this publication was supported by grants from the Carlsberg Foundation (CF15-0943) and the Independent Research Fund Denmark (DFF-4180-00071).
- Chisholm, R. (1966). Theory of knowledge. N.J.: Englewood Cliffs.Google Scholar
- Cohen, M. (1999). 101 philosophy problems. London: Routledge.Google Scholar
- Douglas, K., Otto, R., Desmarais, S., & Borum, R. (2001). Clinical forensic psychology. In I. B. Weiner, J. A. Schinka, & W. F. Velicer (Eds.), Handbook of psychology, research methods in psychology. New York: Wiley.Google Scholar
- Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Jr., Alper, S., et al. (2018). Many Labs 2: Investigating variation in replicability across samples and settings. Advances in Methods and Practices in Psychological Science, 1, 443–490. https://doi.org/10.1177/2515245918810225.CrossRefGoogle Scholar
- Levi, W. H., & Lentz, R. (1982). Effects of text illustrations: A review of research. Educational Communication and Technology Journal, 30(4), 195–232.Google Scholar
- Little, R. J. A., & Rubin, D. B. (2014). Statistical analysis with missing data. New York: Wiley.Google Scholar
- Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.Google Scholar
- Philips, J. & Cushman, F. (2017) Replication of Nahmias, Coates, & Kavarn (2007) Free will, moral responsibility, and mechanism: Experiments on folk intuitions. https://osf.io/n4573/.
- Sorensen, R. (1992). Thought experiments. New York: Oxford University Press.Google Scholar
- Sosa, E. (2009). A defense of the use of intuitions in philosophy. In M. Bishop & D. Murphy (Eds.), Stich and his critics (pp. 101–112). Oxford: Blackwell.Google Scholar
- Turri, J. (2013). A conspicuous art: Putting Gettier to the test. Philosophers’ Imprint, 13, 1–16.Google Scholar