Skip to main content

Dialogue Act Recognition Using Visual Information

  • Conference paper
  • First Online:
Document Analysis and Recognition – ICDAR 2021 (ICDAR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12822))

Included in the following conference series:

  • 3515 Accesses


Automatic dialogue management including dialogue act (DA) recognition is usually focused on dialogues in the audio signal. However, some dialogues are also available in a written form and their automatic analysis is also very important.

The main goal of this paper thus consists in the dialogue act recognition from printed documents. For visual DA recognition, we propose a novel deep model that combines two recurrent neural networks.

The approach is evaluated on a newly created dataset containing printed dialogues from the English VERBMOBIL corpus. We have shown that visual information does not have any positive impact on DA recognition using good quality images where the OCR result is excellent. We have also demonstrated that visual information can significantly improve the DA recognition score on low-quality images with erroneous OCR.

To the best of our knowledge, this is the first attempt focused on DA recognition from visual data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others


  1. 1.

    The noise is not artificial (i.e. we didn’t perform any image transformation), but we have created the noise by real usage of the scanner. We put a blank piece of paper in the scanner and we changed the scanning quality by different scanning options and the amount of light.

  2. 2.


  1. Bunt, H.: Context and dialogue control. Think Quarterly 3(1), 19–31 (1994)

    Google Scholar 

  2. Stolcke, A., et al.: Dialogue act modeling for automatic tagging and recognition of conversational speech. Comput. Linguist. 26(3), 339–373 (2000)

    Article  Google Scholar 

  3. Frankel, J., King, S.: ASR-articulatory speech recognition. In: Seventh European Conference on Speech Communication and Technology (2001)

    Google Scholar 

  4. Cerisara, C., Král, P., Lenc, L.: On the effects of using word2vec representations in neural networks for dialogue act recognition. Comput. Speech Lang. 47, 175–193 (2018)

    Article  Google Scholar 

  5. Jekat, S., Klein, A., Maier, E., Maleck, I., Mast, M., Quantz, J.J.: Dialogue acts in verbmobil (1995)

    Google Scholar 

  6. Godfrey, J.J., Holliman, E.C., McDaniel, J.: Switchboard: telephone speech corpus for research and development. In: Proceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing - Volume 1, ser. ICASSP 1992, pp. 517–520. IEEE Computer Society, USA (1992)

    Google Scholar 

  7. Shriberg, E., Dhillon, R., Bhagat, S., Ang, J., Carvey, H.: The ICSI meeting recorder dialog act (MRDA) corpus. Technical report, International Computer Science Institute, Berkely (2004)

    Google Scholar 

  8. Benedı, J.-M., et al.: Design and acquisition of a telephone spontaneous speech dialogue corpus in Spanish: Dihana. In: Fifth International Conference on Language Resources and Evaluation (LREC), pp. 1636–1639 (2006)

    Google Scholar 

  9. Colombo, P., Chapuis, E., Manica, M., Vignon, E., Varni, G., Clavel, C.: Guiding attention in sequence-to-sequence models for dialogue act prediction. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, pp. 7594–7601 (2020)

    Google Scholar 

  10. Shang, G., Tixier, A.J.-P., Vazirgiannis, M., Lorré, J.-P.: Speaker-change aware CRF for dialogue act classification, arXiv preprint arXiv:2004.02913 (2020)

  11. Alexandersson, J., et al.: Dialogue acts in Verbmobil 2. DFKI Saarbrücken (1998)

    Google Scholar 

  12. Reithinger, N., Klesen, M.: Dialogue act classification using language models. In: Fifth European Conference on Speech Communication and Technology (1997)

    Google Scholar 

  13. Samuel, K., Carberry, S., Vijay-Shanker, K.: Dialogue act tagging with transformation-based learning, arXiv preprint cmp-lg/9806006 (1998)

    Google Scholar 

  14. Martínek, J., Král, P., Lenc, L., Cerisara, C.: Multi-lingual dialogue act recognition with deep learning methods, arXiv preprint arXiv:1904.05606 (2019)

  15. Cerisara, C., Jafaritazehjani, S., Oluokun, A., Le, H.: Multi-task dialog act and sentiment recognition on mastodon, arXiv preprint arXiv:1807.05013 (2018)

  16. Li, J., Fei, H., Ji, D.: Modeling local contexts for joint dialogue act recognition and sentiment classification with bi-channel dynamic convolutions. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 616–626 (2020)

    Google Scholar 

  17. Zhang, Q., Fu, J., Liu, X., Huang, X.: Adaptive co-attention network for named entity recognition in tweets. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1 (2018)

    Google Scholar 

  18. Audebert, N., Herold, C., Slimani, K., Vidal, C.: Multimodal deep networks for text and image-based document classification. In: Cellier, P., Driessens, K. (eds.) ECML PKDD 2019. CCIS, vol. 1167, pp. 427–443. Springer, Cham (2020).

    Chapter  Google Scholar 

  19. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T.: compressing text classification models, arXiv preprint arXiv:1612.03651 (2016)

  20. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.-C.: Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

    Google Scholar 

  21. Jain, R., Wigington, C.: Multimodal document image classification. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 71–77. IEEE (2019)

    Google Scholar 

  22. Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2017)

    Article  Google Scholar 

  23. Martínek, J., Lenc, L., Král, P., Nicolaou, A., Christlein, V.: Hybrid training data for historical text OCR. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 565–570. IEEE (2019)

    Google Scholar 

  24. Han, L., Kamdar, M.R.: MRI to MGMT: predicting methylation status in glioblastoma patients using convolutional recurrent neural networks (2017)

    Google Scholar 

  25. Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 369–376. ACM (2006)

    Google Scholar 

  26. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  27. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781 (2013)

  28. Sauvola, J., Pietikäinen, M.: Adaptive document image binarization. Pattern Recogn. 33(2), 225–236 (2000)

    Article  Google Scholar 

Download references


This work has been partly supported from ERDF “Research and Development of Intelligent Components of Advanced Technologies for the Pilsen Metropolitan Area (InteCom)” (no.: CZ.02.1.01/0.0/0.0/17_048/0007267) and by Grant No. SGS-2019-018 Processing of heterogeneous data and its specialized applications. We would like to thank also Mr. Matěj Zeman for some implementation work.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Jiří Martínek .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Martínek, J., Král, P., Lenc, L. (2021). Dialogue Act Recognition Using Visual Information. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12822. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86330-2

  • Online ISBN: 978-3-030-86331-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics