Skip to main content

Experimenter Philosophy: the Problem of Experimenter Bias in Experimental Philosophy


It has long been known that scientists have a tendency to conduct experiments in a way that brings about the expected outcome. Here, we provide the first direct demonstration of this type of experimenter bias in experimental philosophy. Opposed to previously discovered types of experimenter bias mediated by face-to-face interactions between experimenters and participants, here we show that experimenters also have a tendency to create stimuli in a way that brings about expected outcomes. We randomly assigned undergraduate experimenters to receive two different hypotheses about folk intuitions of consciousness, and then asked them to design experiments based on their hypothesis. Specifically, experimenters generated sentences ascribing intentional and phenomenal mental states to groups, which were later rated by online participants for naturalness. We found a significant interaction between experimenter hypothesis and participant ratings indicating a general tendency for experimenters to obtain the result that they expected. These results indicate that experimenter bias is a real problem in experimental philosophy since the methods and design employed here mirror the predominant survey methods of the field as a whole. The bearing of the current results on Knobe and Prinz’s (Phenomenology and Cognitive Science 7(1):67–83, 2008) group mind hypothesis is discussed, and new methods for avoiding experimenter bias are proposed.

This is a preview of subscription content, access via your institution.


  1. 1.

    This is true across many domains in the natural sciences, but we will limit the discussion here to empirical psychology.

  2. 2.

    Thanks to an anonymous reviewer for this point.


  1. Adair, J.G., and J.S. Epstein. 1968. Verbal cues in the mediation of experimenter bias. Psychological Reports 22: 1045–1053.

    Article  Google Scholar 

  2. Arico, A. 2007. Deregulating corporate consciousness: A critique of Knobe and Prinz’s intuitions about consciousness. Toronto: Poster presented at the Society for Philosophy and Psychology.

    Google Scholar 

  3. Barber, T.X., and M.J. Silver. 1968. Pitfalls in data analysis and interpretation: a reply to Rosenthal. Psychological Bulletin 70: 48–62.

    Article  Google Scholar 

  4. Cullen, S. 2010. Survey-driven romanticism. Review of Philosophy and Psychology 1: 275–96.

    Article  Google Scholar 

  5. Cushman, F.A., L. Young, and M.D. Hauser. 2006. The role of reasoning and intuition in moral judgments: testing three principles of harm. Psychological Science 17(12): 1082–1089.

    Article  Google Scholar 

  6. Dickersin, K. 1990. The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association 263: 1385–9.

    Article  Google Scholar 

  7. Doyen, S., O. Klein, C. Pichon, and A. Cleeremans. 2012. Behavioral priming: it is all in the brain, but whose brain? PLoS One 7(1): e29081.

    Article  Google Scholar 

  8. Feltz, A., E.T. Cokely, and T. Thomas Nadelhoffer. 2009. Natural compatibilism versus natural incompatibilism: back to the drawing board. Mind and Language 24(1): 1–23.

    Article  Google Scholar 

  9. Feltz, A., Harris, M., and Perez, A. (2012). Perspective in intentional action attribution. Philosophical Psychology.

  10. Hamlin, J., K. Wynn, and P. Bloom. 2007. Social evaluation by preverbal infants. Nature 450: 557–559.

    Article  Google Scholar 

  11. Ioannidis, J.P.A. 2005. Why most published research findings are false. Public Library of Science, Medicine 2: e124.

    Google Scholar 

  12. Khemlani, S.S., A.B. Sussman, and D.M. Oppenheimer. 2011. Harry Potter and the sorcerer’s scope: latent scope biases in explanatory reasoning. Memory & Cognition 39(1).

  13. Knobe, J. 2003. Intentional action and side effects in ordinary language. Analysis 63: 190–193.

    Article  Google Scholar 

  14. Knobe, J., and J. Prinz. 2008. Intuitions about consciousness: experimental studies. Phenomenology and Cognitive Science 7(1): 67–83.

    Article  Google Scholar 

  15. Knobe, J., Buckwalter, W., Nichols, S., Robbins, P., Sarkissian, H., and Sommers, T. (2011). Experimental philosophy. Annual Review of Psychology, 63.

  16. Monroe, A.E., and B.F. Malle. 2010. From uncaused will to conscious choice: the need to study, not speculate about people’s folk concept of free will. Review of Philosophy and Psychology 1(2): 211–224.

    Article  Google Scholar 

  17. Phelan M., Arico A., and Nichols S. (2012). Thinking things and feeling things: On an alleged discontinuity in folk metaphysics of mind.

  18. Rosenthal, R. 1979. The “file drawer problem” and tolerance for null results. Psychological Bulletin 86: 638–641.

    Article  Google Scholar 

  19. Rosenthal, R., and K. Fode. 1963. The effect of experimenter bias on performance of the albino rat. Behavioral Science 8: 183–189.

    Article  Google Scholar 

  20. Rosenthal, R., and I. Jacobson. 1968. Pygmalion in the classroom. New York: Holt, Rinehart and Winston.

    Google Scholar 

  21. Rosenthal, R., and D.B. Rubin. 1978. Interpersonal expectancy effects: the first 345 studies. The Behavioral and Brain Sciences 3: 377–386.

    Article  Google Scholar 

  22. Scholl, B.J. 2008. Two kinds of experimental philosophy, and their methodological dangers. Talk given at the SPP Workshop on Experimental Philosophy. Philadelphia: University of Pennsylvania.

    Google Scholar 

  23. Strickland, B., M. Fisher, and J. Knobe. 2012. Moral structure falls out of general event structure. Psychological Inquiry 23(2): 198–205.

    Article  Google Scholar 

  24. Sytsma, J., and E. Machery. 2009. How to study folk intuitions about phenomenal consciousness. Philosophical Psychology 22: 21–35.

    Article  Google Scholar 

  25. Young, L., and J. Phillips. 2011. The paradox of moral focus. Cognition 119: 166–178.

    Article  Google Scholar 

Download references


We would like to thank Joshua Knobe, Chaz Firestone, Alex Shaw, Brandon Liverence, Hugo Mercier and Brian Scholl for useful comments and feedback. We would also like to thank Julie Van Dyke and Laurie Santos for help finding experimenters. Finally, we thank the Yale undergraduate experimenters for designing the experiments.

Author information



Corresponding author

Correspondence to Brent Strickland.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Strickland, B., Suben, A. Experimenter Philosophy: the Problem of Experimenter Bias in Experimental Philosophy. Rev.Phil.Psych. 3, 457–467 (2012).

Download citation


  • Mental State
  • Sentence Type
  • Experimenter Bias
  • Feeling Condition
  • Experimental Philosophy