, Volume 31, Issue 2, pp 207–221 | Cite as

The importance of a human viewpoint on computer natural language capabilities: a Turing test perspective

  • Kevin Warwick
  • Huma Shah
Open Forum


When judging the capabilities of technology, different humans can have very different perspectives and come to quite diverse conclusions over the same data set. In this paper we consider the capabilities of humans when it comes to judging conversational abilities, as to whether they are conversing with a human or a machine. In particular the issue in question is the importance of human judges interrogating in practical Turing tests. As supportive evidence for this we make use of transcripts which originated from a series of practical Turing’s tests held 6–7 June 2014 at the Royal Society London. Each of the tests involved a 3-participant simultaneous comparison by a judge of two hidden entities, one being a human and the other a machine. Thirty different judges took part in total. Each of the transcripts considered in the paper resulted in a judge being unable to say for certain which was the machine and which was the human. The main point we consider here is the fallibility of humans in deciding whether they are conversing with a machine or a human; hence we are concerned specifically with the decision-making process.


Deception detection Natural language Turing’s imitation game Chatbots Machine misidentification 



The authors would like to thank the Royal Society for hosting these tests, the event day volunteers, the machine developers for accepting the invitation; without their agreement and the enthusiasm of the judges and hidden humans who gave their time, and in the case of those who travelled from as far as the EU, USA and Russia at their own expense, these tests could not have taken place and given us so much interesting linguistic material to study how humans think. Finally, this study was made possible through the EU FP7 RoboLaw science in society project, for which the Royal Society event was part of the dissemination tasks.


  1. Bringsjord S, Bello P, Ferrucci D (2001) Creativity, the Turing test and the (better) Lovelace test. Mind Mach 11(1):3–27CrossRefzbMATHGoogle Scholar
  2. Chomsky N (2008) Turing on the “imitation game”, chapter 7. In: Epstein R et al (eds) Parsing the Turing test. Springer, New YorkGoogle Scholar
  3. Dennett DC (2012) Turing’s gradualist vision: making minds from proto-minds. Invited talk: Turing in context II, BrusselsGoogle Scholar
  4. Epstein R (2009) The quest for the thinking computer. In: Epstein R, Roberts G, Beber G (eds) Parsing the Turing test: philosophical and methodological issues in the quest for the thinking computer. Springer, New York, pp 3–12CrossRefGoogle Scholar
  5. Hayes P, Ford K (1995) Turing test considered harmful. In: Proceedings of international joint conference on artificial intelligence, Montreal, vol 1, pp 972–977Google Scholar
  6. Khooshabeh P, Dehghani M, Nazarian A, Gratch J (2014) The cultural influence model: when accented natural language spoken by virtual characters matters. AI Soc. doi: 10.1007/s00146-014-0568-1 Google Scholar
  7. Loebner H (1995) In response to Stuart Shieber’s lessons from a restricted Turing test.
  8. Loebner Prize (1991) Home of the Loebner Prize.
  9. McDermott D (2014) On the claim that a table-lookup program could pass the Turing test. Mind Mach 24(2):143–188MathSciNetCrossRefGoogle Scholar
  10. Reidl M (2014) The Lovelace 2.0 test for artificial creativity and intelligence. In: ‘Beyond the Turing test’ 2014 workshop in association for the advancement of artificial intelligence 2014.
  11. Shah H (2010) Deception detection and machine intelligence in practical Turing tests. PhD thesis, The University of ReadingGoogle Scholar
  12. Shah H, Henry O (2005) Confederate effect in human-machine textual interaction. In: Proceedings of 5th WSEAS international conference on information science, communications and applications (WSEAS ISCA), Cancun, Mexico, pp. 109–114, 11–14 May 2005. ISBN: 960-8457-22-XGoogle Scholar
  13. Shah H, Warwick K (2010a) Hidden interlocutor misidentification in practical Turing tests. Mind Mach 20:441–454CrossRefGoogle Scholar
  14. Shah H, Warwick K (2010b) Testing Turing’s five-minutes, parallel-paired imitation game. Kybernetes 39(3):449–465CrossRefzbMATHGoogle Scholar
  15. Shah H, Warwick K, Bland I, Chapman CD, Allen MJ (2012) Turing’s imitation game: role of error-making in intelligent thought. Turing in context II, Brussels, pp 31–32, 10–12 Oct 2012.—presentation available here:
  16. Shieber S (1994) Lessons from an restricted Turing test. Commun Assoc Comput Mach 37(6):70–78Google Scholar
  17. The Imitation Game (2014) Weinstein and black bear productions.
  18. Traiger S (2000) Making the right identification in the Turing test. Mind Mach 10:561–572CrossRefGoogle Scholar
  19. Turing AM (1950) Computing machinery and intelligence. Mind LIX(236):433–460MathSciNetCrossRefGoogle Scholar
  20. Turing AM, Braithwaite R, Jefferson G, Newman M (2013) Can automatic calculating machines be said to think? Transcript of 1952 BBC radio broadcast. In: Cooper SB, van Leeuwen J (eds) Alan Turing: his work and impact. Elsevier, Oxford, pp 667–676Google Scholar
  21. Warwick K (2011) Artificial intelligence: the basics. Routledge, LondonGoogle Scholar
  22. Warwick K (2012) Not another look at the Turing test! In: Bielikova M, Friedrich G, Gottlob G, Katzenbeisser S, Turan G (eds) Proceedings of SOFSEM 2012: theory and practice of computer science. Lecture Notes in computer science, vol 7147. Springer, pp 130–140Google Scholar
  23. Warwick K, Shah H (2014a) Good machine performance in practical Turing tests. IEEE Trans Comput Intell AI Games 6(3):289–299MathSciNetCrossRefGoogle Scholar
  24. Warwick K, Shah H (2014b) Effects of lying in practical Turing tests. AI Soc. doi: 10.1007/s00146-013-0534-3 Google Scholar
  25. Warwick K, Shah H (2015) Human misidentification in Turing tests. J Exp Theor Artif Intell 27(2):123–135CrossRefGoogle Scholar
  26. Warwick K, Shah H, Moor JH (2013) Some implications of a sample of practical Turing tests. Mind Mach 23:163–177CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2015

Authors and Affiliations

  1. 1.Coventry UniversityCoventryUK

Personalised recommendations