QRFA: A Data-Driven Model of Information-Seeking Dialogues
Understanding the structure of interaction processes helps us to improve information-seeking dialogue systems. Analyzing an interaction process boils down to discovering patterns in sequences of alternating utterances exchanged between a user and an agent. Process mining techniques have been successfully applied to analyze structured event logs, discovering the underlying process models or evaluating whether the observed behavior is in conformance with the known process. In this paper, we apply process mining techniques to discover patterns in conversational transcripts and extract a new model of information-seeking dialogues, QRFA, for Query, Request, Feedback, Answer. Our results are grounded in an empirical evaluation across multiple conversational datasets from different domains, which was never attempted before. We show that the QRFA model better reflects conversation flows observed in real information-seeking conversations than models proposed previously. Moreover, QRFA allows us to identify malfunctioning in dialogue system transcripts as deviations from the expected conversation flow described by the model via conformance analysis.
KeywordsConversational search Log analysis Process mining
The work of S. Vakulenko and C. Di Ciccio has received funding from the EU H2020 program under MSCA-RISE agreement 645751 (RISE_BPM) and the Austrian Research Promotion Agency (FFG) under grant 861213 (CitySPIN). S. Vakulenko was also supported by project 855407 “Open Data for Local Communities” (CommuniData) of the Austrian Federal Ministry of Transport, Innovation and Technology (BMVIT) under the program “ICT of the Future.” M. de Rijke was supported by Ahold Delhaize, the Association of Universities in the Netherlands (VSNU), and the Innovation Center for Artificial Intelligence (ICAI).
All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.
- 1.Adriansyah, A., van Dongen, B.F., van der Aalst, W.M.: Conformance checking using cost-based fitness analysis. In: 15th International Enterprise Distributed Object Computing Conference, pp. 55–64. IEEE Computer Society (2011)Google Scholar
- 2.Austin, J.L.: How to do Things with Words. Oxford Paperbacks, Oxford University Press, Oxford (1976)Google Scholar
- 4.Cohen, W.W., Carvalho, V.R., Mitchell, T.M.: Learning to classify email into “speech acts”. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 309–316 (2004)Google Scholar
- 5.Core, M.: Coding dialogs with the DAMSL annotation scheme. In: Working Notes of the AAAI Fall Symposium on Communicative Action in Humans and Machines, pp. 1–8 (1997)Google Scholar
- 7.Di Ciccio, C., Mecella, M.: A two-step fast algorithm for the automated discovery of declarative workflows. In: Symposium on Computational Intelligence and Data Mining, pp. 135–142. IEEE (2013)Google Scholar
- 9.Goffman, E.: Erving Goffman: Exploring the Interaction Order. Polity Press, Cambridge (1988)Google Scholar
- 10.Jeong, M., Lin, C., Lee, G.G.: Semi-supervised speech act recognition in emails and forums. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pp. 1250–1259 (2009)Google Scholar
- 11.Jo, Y., Yoder, M., Jang, H., Rosé, C.P.: Modeling dialogue acts with content word filtering and speaker preferences. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2179–2189 (2017)Google Scholar
- 13.Leemans, S.J.J., Fahland, D., van der Aalst, W.M.P.: Process and deviation exploration with inductive visual miner. In: Proceedings of the BPM Demo Sessions 2014 Co-located with the 12th International Conference on Business Process Management, p. 46 (2014)Google Scholar
- 14.Li, Z., Kiseleva, J., de Rijke, M.: Dialogue generation: from imitation learning to inverse reinforcement learning. In: 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, January 2019Google Scholar
- 15.Radlinski, F., Craswell, N.: A theoretical framework for conversational search. In: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, pp. 117–126. ACM (2017)Google Scholar
- 16.Richetti, P.H.P., de A.R. Gonçalves, J.C., Baião, F.A., Santoro, F.M.: Analysis of knowledge-intensive processes focused on the communication perspective. In: Carmona, J., Engels, G., Kumar, A. (eds.) BPM 2017. LNCS, vol. 10445, pp. 269–285. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65000-5_16CrossRefGoogle Scholar
- 18.Schiffrin, D.: Approaches to Discourse: Language as Social Interaction. Blackwell Textbooks in Linguistics. Wiley, Hoboken (1994)Google Scholar
- 23.Trippas, J.R., Spina, D., Cavedon, L., Sanderson, M.: How do people interact in conversational speech-only search tasks: a preliminary analysis. In: Proceedings of the 2017 ACM on Conference on Human Information Interaction and Retrieval, pp. 325–328. ACM (2017)Google Scholar
- 24.Trippas, J.R., Spina, D., Cavedon, L., Joho, H., Sanderson, M.: Informing the design of spoken conversational search: perspective paper. In: Proceedings of the 2018 Conference on Human Information Interaction and Retrieval, pp. 32–41. ACM (2018)Google Scholar
- 26.van der Aalst, W.M., Adriansyah, A., van Dongen, B.F.: Replaying history on process models for conformance checking and performance analysis. Wiley Interdisciplinary Reviews: Data Min. Knowl. Discov. 2(2), 182–192 (2012)Google Scholar
- 29.Wang, G.A., Wang, H.J., Li, J., Abrahams, A.S., Fan, W.: An analytical framework for understanding knowledge-sharing processes in online Q&A communities. ACM Trans. Manag. Inf. Syst. 5(4), 18 (2015)Google Scholar
- 30.Williams, J., Raux, A., Henderson, M.: The dialog state tracking challenge series: a review. Dialogue Discourse 7(3), 4–33 (2016)Google Scholar