Crowd Knowledge Enhanced Multimodal Conversational Assistant in Travel Domain

Abstract

We present a new solution towards building a crowd knowledge enhanced multimodal conversational system for travel. It aims to assist users in completing various travel-related tasks, such as searching for restaurants or things to do, in a multimodal conversation manner involving both text and images. In order to achieve this goal, we ground this research on the combination of multimodal understanding and recommendation techniques which explores the possibility of a more convenient information seeking paradigm. Specifically, we build the system in a modular manner where each modular construction is enriched with crowd knowledge from social sites. To the best of our knowledge, this is the first work that attempts to build intelligent multimodal conversational systems for travel, and moves an important step towards developing human-like assistants for completion of daily life tasks. Several current challenges are also pointed out as our future directions.

Appendices

Appendix A: State Tracking

State tracking refers to the maintenance of the dialogue state $\mathcal {S}_t$ which is the representation of the conversation session until time t. Based on the state $\mathcal {S}_{t-1}$ in former time step and the multimodal understanding result $\mathcal {U}_t$ for utterance in time step t, the dialogue state is obtained as follows:

$$\begin{aligned} \mathcal {S}_t = \mathcal {G}(\mathcal {S}_{t-1},{<}\mathcal {M}_t,\mathcal {D}_t,\mathcal {C}_t,\mathcal {A}_t,\mathcal {I}_t{>}), \end{aligned}$$

(6)

where $\mathcal {G}$ refers to a set of rules. We generally summarize the rules as below:

• (1) if $\mathcal {M}_t = Chitchat$, then $\mathcal {S}_t = \mathcal {S}_{t-1}$;

• (2) if domain $\mathcal {D}_t$ is changed: $\mathcal {S}_t$ will be updated totally based on $\mathcal {U}_t$;

• (3) if domain $\mathcal {D}_t$ is not changed: if $\mathcal {M}_t \ne Negation$, $\mathcal {S}_t$ will inherit information stored in $\mathcal {S}_{t-1}$;

• (4) if domain $\mathcal {D}_t$ is not changed: if $\mathcal {M}_t = Negation$, $\mathcal {S}_t$ will inherit information stored in $\mathcal {S}_{t-1}$ while update the parts according $\mathcal {U}_t$;

• (5) if the time interval between two consecutive utterances exceeds a pre-defined length at time t, then $\mathcal {S}_t$ will be cleaned.

Tracking dialogue states is the key to elevate user experience on multi-turn conversation. The main reason that we do not follow the previous works to learn models is because of the lack of dialogue data to train statistical tracking models. We leave leveraging session-level labeled dialogue data to improve the state tracking task as our future work.

Appendix B: Action Decision

At each turn of conversation between user and agent, the dialogue management module takes the current state tracking results as input, and outputs the corresponding actions. Due to the lack of large-scale dialogue training data, we also resort to a set of rules. Intuitively, the main action types considered and the conditions for triggering it are as below:

• Proactive Questioning. This action will be triggered when (a) a recommendation intent is detected, (b) a domain is detected, and (c) no enough constraints or attributes is detected in $\mathcal {S}_t$. This action is often used to obtain more constraints or attributes to narrow down the search space.

• Candidate Listing. This action is often triggered when recommendation results are obtained or the Show more intent is detected. As each venue in our dataset is associated with Foursquare photos, we implement candidate listing via a list of images where each image corresponds to a venue. In the interface, user can conveniently choose a venue by simply clicking its corresponding image.

• Venue Recommendation. This action is triggered when the intent $\mathcal {I}_t$ in $\mathcal {S}_t$ is Recommendation, and it will retrieve results from the recommendation module.

• Question Answering. It will be triggered when a venue is selected and one of its slot names are detected in $\mathcal {U}_t$ without value around. It returns the missing attribute value by looking up the venue database.

• Review Summary. This action will be triggered when a venue is selected and the intent $\mathcal {I}_t$ in $\mathcal {S}_t$ is Ask opinion. It will summarize the reviews of the target venue and present in organized form.

• API call. It will be triggered when a venue is selected and the Map direction intent is present in $\mathcal {I}_t$. The Google Map API will be called with the start position and destination. Currently, only the Map API is integrated. However, other APIs such as weather report can also be integrated with proper modifications.

• Chitchat. This action will be triggered when none travel venue seeking related intent is detected. As pointed out by [18], nearly 80% of utterances are chitchat queries for e-commerce bots. If the system cannot reply to them, then the conversation may not be able to continue. Thus, it will activate the chitchat response generation to obtain a reply.