Skip to main content
View expanded cover

International Workshop on Future and Emerging Trends in Language Technology

FETLT 2015: Future and Emergent Trends in Language Technology pp 139–148Cite as

Adopting Semantic Similarity for Utterance Candidates Discovery from Human-to-Human Dialogue Corpus

Part of the Lecture Notes in Computer Science book series (LNAI,volume 9577)

Abstract

Having appropriate utterances in response to user input is an essential element to sustain the flow of conversation in dialogue systems, and a basic and fundamental element for maintaining such conversation coherence is an adjacency pair. To find appropriate candidates for adjacency pairs completion, and thus contribute to avoiding conversational disrupt in casual chatbot systems, we suggest an approach that utilizes human-to-human chat logs, and combines standard Information Retrieval methods and semantic similarity measures based on distributed word representations. The experimental results show the approach improves the quality of utterance pairs compared to standard IR-based methods.

Keywords

  • Semantic similarity
  • Utterance candidates
  • Continuous word embeddings
  • Word2vec
  • Human-to-human dialogue corpus

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-33500-1_12
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   44.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-33500-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   59.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.

Notes

  1. 1.

    As described in Sect. 3.

  2. 2.

    CyberAgent, Inc. https://www.cyberagent.co.jp/en/.

  3. 3.

    Apache Lucene is used for the experiments.

  4. 4.

    https://pypi.python.org/pypi/gensim implementation.

  5. 5.

    We don’t consider an extra crowdsourcing step proposed in the paper though.

References

  1. Gandhe, S., Traum, D.: I’ve Said It Before, and I’ll Say It Again: an empirical investigation of the upper bound of the selection approach to dialogue. In: Proceedings of the SIGdial 2010 Conference, Tokyo, Japan, pp. 245–248 (2010)

    Google Scholar 

  2. Schegloff, E.A.: Sequence Organization in Interaction: A Primer in Conversation Analysis I. Cambridge University Press, Cambridge (2007)

    CrossRef  Google Scholar 

  3. Huang, J., Zhou, M., Yang, D.: Extracting chatbot knowledge from online discussion forums. In: Proceedings of the 20th International Joint Conference on Artifical Intelligence, Hyderabad, India, pp. 423–428 (2007)

    Google Scholar 

  4. Wu, Y., Wang, G., Li, W., Li, Z.: Automatic chatbot knowledge acquisition from online forum via rough set and ensemble learning. In: Proceedings of the IFIP International Conference on Network and Parallel Computing, Shanghai, China, pp. 242–246 (2008)

    Google Scholar 

  5. Higashinaka, R., Kobayashi, N., Hirano, T., Miyazaki, C., Meguro, T., Makino, T., Matsuo, Y.: Syntactic filtering and content-based retrieval of Twitter sentences for the generation of system utterances in dialogue systems. In: Proceedings of International Workshop Series on Spoken Dialogue Systems Technology, Napa, USA, pp. 113–123 (2014)

    Google Scholar 

  6. Bessho, F., Harada, T., Kuniyoshi, Y.: Dialog system using real-time crowdsourcing and Twitter large-scale corpus. In: Proceedings of the SIGdial 2012 Conference, Stroudsburg, PA, USA, pp. 227–231 (2012)

    Google Scholar 

  7. Gandhe, S., Traum, D.: Creating spoken dialogue characters from corpora without annotations. In: Proceedings of Interspeech-2007, Antwerp, Belgium, pp. 2201–2204 (2007)

    Google Scholar 

  8. Nio, L., Sakti, S., Neubig, G., Toda, T., Nakamura, S.: Utilizing human-to-human conversation examples for a multi domain chat-oriented dialog system. IEICE Trans. 97–D(6), 1497–1505 (2014)

    Google Scholar 

  9. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of ICLR Workshop (2013)

    Google Scholar 

  10. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)

    Google Scholar 

  11. Baroni, M., Dinu, G., Kruszewski, G.: A systematic comparison of context-counting vs. context-predicting semantic vectors. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, Maryland, USA, pp. 238–247 (2014)

    Google Scholar 

  12. Hearst, M.A.: TextTiling: segmenting text into multi-paragraph subtopic passages. Comput. Linguist. 23(1), 33–64 (1997)

    Google Scholar 

  13. Firth, J.R.: A synopsis of linguistic theory 1930–1955. Studies in Linguistic Analysis, pp. 1–32 (1957)

    Google Scholar 

  14. Blei, D., Ng, A., Jordan, M.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)

    MATH  Google Scholar 

  15. Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: Proceedings of the 31st International Conference on Machine Learning (ICML-2014), pp. 1188–1196 (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CyberAgent, Inc., Akihabara Dai Bld., Sotokanda 1-18-13, Chiyoda-ku, Tokyo, 101-8608, Japan

    Roman Y. Shtykh & Mitsuharu Makita

  2. Media Research Institute, Waseda University, 2-579-15 Mikajima, Tokorozawa, 359-1192, Japan

    Roman Y. Shtykh

Authors
  1. Roman Y. Shtykh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mitsuharu Makita
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roman Y. Shtykh .

Editor information

Editors and Affiliations

  1. University of Seville, Seville, Spain

    José F. Quesada

  2. University of Seville, Seville, Spain

    Francisco-Jesús Martín Mateos

  3. University of Seville, Seville, Spain

    Teresa Lopez-Soto

Rights and permissions

Reprints and Permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Shtykh, R.Y., Makita, M. (2016). Adopting Semantic Similarity for Utterance Candidates Discovery from Human-to-Human Dialogue Corpus. In: Quesada, J., Martín Mateos, FJ., Lopez-Soto, T. (eds) Future and Emergent Trends in Language Technology. FETLT 2015. Lecture Notes in Computer Science(), vol 9577. Springer, Cham. https://doi.org/10.1007/978-3-319-33500-1_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-33500-1_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-33499-8

  • Online ISBN: 978-3-319-33500-1

  • eBook Packages: Computer ScienceComputer Science (R0)