Abstract
When judging the capabilities of technology, different humans can have very different perspectives and come to quite diverse conclusions over the same data set. In this paper we consider the capabilities of humans when it comes to judging conversational abilities, as to whether they are conversing with a human or a machine. In particular the issue in question is the importance of human judges interrogating in practical Turing tests. As supportive evidence for this we make use of transcripts which originated from a series of practical Turing’s tests held 6–7 June 2014 at the Royal Society London. Each of the tests involved a 3-participant simultaneous comparison by a judge of two hidden entities, one being a human and the other a machine. Thirty different judges took part in total. Each of the transcripts considered in the paper resulted in a judge being unable to say for certain which was the machine and which was the human. The main point we consider here is the fallibility of humans in deciding whether they are conversing with a machine or a human; hence we are concerned specifically with the decision-making process.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bringsjord S, Bello P, Ferrucci D (2001) Creativity, the Turing test and the (better) Lovelace test. Mind Mach 11(1):3–27
Chomsky N (2008) Turing on the “imitation game”, chapter 7. In: Epstein R et al (eds) Parsing the Turing test. Springer, New York
Dennett DC (2012) Turing’s gradualist vision: making minds from proto-minds. Invited talk: Turing in context II, Brussels
Epstein R (2009) The quest for the thinking computer. In: Epstein R, Roberts G, Beber G (eds) Parsing the Turing test: philosophical and methodological issues in the quest for the thinking computer. Springer, New York, pp 3–12
Hayes P, Ford K (1995) Turing test considered harmful. In: Proceedings of international joint conference on artificial intelligence, Montreal, vol 1, pp 972–977
Khooshabeh P, Dehghani M, Nazarian A, Gratch J (2014) The cultural influence model: when accented natural language spoken by virtual characters matters. AI Soc. doi:10.1007/s00146-014-0568-1
Loebner H (1995) In response to Stuart Shieber’s lessons from a restricted Turing test. http://www.loebner.net/Prizef/In-response.html
Loebner Prize (1991) Home of the Loebner Prize. http://www.loebner.net/Prizef/loebner-prize.html
McDermott D (2014) On the claim that a table-lookup program could pass the Turing test. Mind Mach 24(2):143–188
Reidl M (2014) The Lovelace 2.0 test for artificial creativity and intelligence. In: ‘Beyond the Turing test’ 2014 workshop in association for the advancement of artificial intelligence 2014. http://arxiv.org/pdf/1410.6142v1.pdf
Shah H (2010) Deception detection and machine intelligence in practical Turing tests. PhD thesis, The University of Reading
Shah H, Henry O (2005) Confederate effect in human-machine textual interaction. In: Proceedings of 5th WSEAS international conference on information science, communications and applications (WSEAS ISCA), Cancun, Mexico, pp. 109–114, 11–14 May 2005. ISBN: 960-8457-22-X
Shah H, Warwick K (2010a) Hidden interlocutor misidentification in practical Turing tests. Mind Mach 20:441–454
Shah H, Warwick K (2010b) Testing Turing’s five-minutes, parallel-paired imitation game. Kybernetes 39(3):449–465
Shah H, Warwick K, Bland I, Chapman CD, Allen MJ (2012) Turing’s imitation game: role of error-making in intelligent thought. Turing in context II, Brussels, pp 31–32, 10–12 Oct 2012. http://www.computing-conference.ugent.be/file/14—presentation available here: http://www.academia.edu/1916866/Turings_Imitation_Game_Role_of_Error-making_in_Intelligent_Thought
Shieber S (1994) Lessons from an restricted Turing test. Commun Assoc Comput Mach 37(6):70–78
The Imitation Game (2014) Weinstein and black bear productions. http://theimitationgamemovie.com/
Traiger S (2000) Making the right identification in the Turing test. Mind Mach 10:561–572
Turing AM (1950) Computing machinery and intelligence. Mind LIX(236):433–460
Turing AM, Braithwaite R, Jefferson G, Newman M (2013) Can automatic calculating machines be said to think? Transcript of 1952 BBC radio broadcast. In: Cooper SB, van Leeuwen J (eds) Alan Turing: his work and impact. Elsevier, Oxford, pp 667–676
Warwick K (2011) Artificial intelligence: the basics. Routledge, London
Warwick K (2012) Not another look at the Turing test! In: Bielikova M, Friedrich G, Gottlob G, Katzenbeisser S, Turan G (eds) Proceedings of SOFSEM 2012: theory and practice of computer science. Lecture Notes in computer science, vol 7147. Springer, pp 130–140
Warwick K, Shah H (2014a) Good machine performance in practical Turing tests. IEEE Trans Comput Intell AI Games 6(3):289–299
Warwick K, Shah H (2014b) Effects of lying in practical Turing tests. AI Soc. doi:10.1007/s00146-013-0534-3
Warwick K, Shah H (2015) Human misidentification in Turing tests. J Exp Theor Artif Intell 27(2):123–135
Warwick K, Shah H, Moor JH (2013) Some implications of a sample of practical Turing tests. Mind Mach 23:163–177
Acknowledgments
The authors would like to thank the Royal Society for hosting these tests, the event day volunteers, the machine developers for accepting the invitation; without their agreement and the enthusiasm of the judges and hidden humans who gave their time, and in the case of those who travelled from as far as the EU, USA and Russia at their own expense, these tests could not have taken place and given us so much interesting linguistic material to study how humans think. Finally, this study was made possible through the EU FP7 RoboLaw science in society project, for which the Royal Society event was part of the dissemination tasks.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Warwick, K., Shah, H. The importance of a human viewpoint on computer natural language capabilities: a Turing test perspective. AI & Soc 31, 207–221 (2016). https://doi.org/10.1007/s00146-015-0588-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00146-015-0588-5