A Turing Test to Evaluate a Complex Summarization Task

  • Alejandro Molina
  • Eric SanJuan
  • Juan-Manuel Torres-Moreno
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8138)


This paper deals with a new strategy to evaluate a Natural Language Processing (NLP) complex task using the Turing test. Automatic summarization based on sentence compression requires to asses informativeness and modify inner sentence structures. This is much more intrinsically related with real rephrasing than plain sentence extraction and ranking paradigm so new evaluation methods are needed. We propose a novel imitation game to evaluate Automatic Summarization by Compression (ASC). Rationale of this Turing-like evaluation could be applied to many other NLP complex tasks like Machine translation or Text Generation. We show that a state of the art ASC system can pass such a test and simulate a human summary in 60% of the cases.


Natural Language Processing Machine Translation Turing Test Simulation Game Human Player 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Tratz, S., Hovy, E.: Summarisation Evaluation Using Transformed Basic Elements. In: Workshop Text Analysis Conference (TAC 2008), Gaithersburg, MD, USA (2008)Google Scholar
  2. 2.
    Louis, A., Nenkova, A.: Automatically Evaluating Content Selection in Summarization without Human Models. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Singapour, August 6-7, pp. 306–314. ACL (2009)Google Scholar
  3. 3.
    Saggion, H., Torres-Moreno, J.-M., da Cunha, I., SanJuan, E.: Multilingual summarization evaluation without human models. In: Proceedings of the 23rd International Conference on Computational Linguistics: Posters (COLING 2010), Beijing, Chine, pp. 1059–1067. ACL (2010)Google Scholar
  4. 4.
    Torres-Moreno, J.-M., Saggion, H., da Cunha, I., SanJuan, E.: Summary Evaluation With and Without References. Polibits: Research Journal on Computer Science and Computer Engineering with Applications 42, 13–19 (2010)Google Scholar
  5. 5.
    Molina, A., Torres-Moreno, J.-M., SanJuan, E., da Cunha, I., Sierra, G., Velázquez-Morales, P.: Discourse segmentation for sentence compression. In: Batyrshin, I., Sidorov, G. (eds.) MICAI 2011, Part I. LNCS, vol. 7094, pp. 316–327. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  6. 6.
    Molina, A., Torres-Moreno, J.-M., SanJuan, E., da Cunha, I., Martínez, G.E.S.: Discursive sentence compression. In: Gelbukh, A. (ed.) CICLing 2013, Part II. LNCS, vol. 7817, pp. 394–407. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  7. 7.
    Turing, A.M.: Computing machinery and intelligence. Mind 59(236), 433–460 (1950)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Harnad, S.: Minds, Machines and Turing. Journal of Logic, Language and Information 9(4), 425–445 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    da Cunha, I., Torres-Moreno, J.-M., Sierra, G.: On the Development of the RST Spanish Treebank. In: Linguistic Annotation Workshop, pp. 1–10. The Association for Computer Linguistics (2011)Google Scholar
  10. 10.
    da Cunha, I., SanJuan, E., Torres-Moreno, J.-M., Lloberes, M., Castellón, I.: DiSeg 1.0: The first system for Spanish Discourse Segmentation. Expert Systems with Applications 39(2), 1671–1678 (2012)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Alejandro Molina
    • 1
  • Eric SanJuan
    • 1
    • 2
  • Juan-Manuel Torres-Moreno
    • 1
    • 2
    • 3
  1. 1.LIAUniversité d’Avignon et des Pays de VaucluseAvignon Cedex 9France
  2. 2.Brain & Language Research InstituteAix-en-Provence Cedex 1France
  3. 3.École Polytechnique de MontréalMontréalCanada

Personalised recommendations