Skip to main content
SpringerLink
Log in

Tell me what you Like: introducing natural language preference elicitation strategies in a virtual assistant for the movie domain

  • Research
  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

Preference elicitation is a crucial step for every recommendation algorithm. In this paper, we present a strategy that allows users to express their preferences and needs through natural language statements. In particular, our natural language preference elicitation pipeline allows users to express preferences on objective movie features (e.g., actors, directors, etc.) as well as on subjective features that are collected by mining user-written movie reviews. To validate our claims, we carried out a user study in the movie domain (\(N=114\)). The main finding of our experiment is that users tend to express their preferences by using objective features, whose usage largely overcomes that of subjective features, which are more complicated to be expressed. However, when the users are able to express their preferences also in terms of subjective features, they obtain better recommendations in a lower number of conversation turns. We have also identified the main challenges that arise when users talk to the virtual assistant by using subjective features, and this paves the way for future developments of our methodology.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Availability of supporting data

The source of the framework used in the experiments is available on Github.

Notes

  1. https://www.wikidata.org/wiki/Q83495

  2. https://www.w3.org/TR/rdf-sparql-query/

  3. https://en.wikipedia.org/wiki/Levenshtein_distance

  4. https://snap.stanford.edu/data/web-Movies.html

  5. https://stanfordnlp.github.io/CoreNLP/

  6. https://github.com/machetegrapefruit/fictional-waddle

  7. https://cloud.google.com/dialogflow

  8. https://stanfordnlp.github.io/CoreNLP/ner.html

  9. http://jung.sourceforge.net/

  10. https://radimrehurek.com/gensim/models/doc2vec.html

  11. https://github.com/machetegrapefruit/glowing-pancake

  12. https://pingouin-stats.org/

  13. This is confirmed by a recent Gartner study: https://www.gartner.com/en/newsroom/press-releases/2022-07-27-gartner-predicts-chatbots-will-become-a-primary-customer-service-channel-within-five-years

  14. https://github.com/swapUniba/Content_based_recommender_conveRSE-with-Aspects

  15. https://dl.acm.org/doi/10.1145/3511047.3536407

References

  • Alabdulkareem, F., Cercone, N., & Liaskos, S. (2015). Goal and preference identification through natural language. In: 2015 IEEE 23rd international requirements engineering conference (RE), IEEE, pp. 56–65. https://doi.org/10.1109/re.2015.7320408

  • Alqahtani, F., Meier, S., & Orji, R. (2022). Personality-based approach for tailoring persuasive mental health applications. UMUAI, 32(3), 253–295. https://doi.org/10.1007/s11257-021-09289-5

    Article  Google Scholar 

  • Basile, P., Musto, C., de Gemmis, M., & et al. (2014). Content-based recommender systems+ DBpedia knowledge= semantics-aware recommender systems. In: Semantic web evaluation challenge. Springer, pp. 163–169. https://doi.org/10.1007/978-3-319-12024-9_21

  • Bogers, T., & Koolen, M. (2017). Defining and supporting narrative-driven recommendation. In: RecSys ’17. ACM, pp. 238–242, https://doi.org/10.1145/3109859.3109893

  • Bogers, T., & Koolen, M. (2018). "I’m looking for something like ...": Combining narratives and example items for narrative-driven book recommendation. In: Anelli, V. W., Noia, T. D., Lops, P., & et al. (eds.) Proceedings of the workshop on knowledge-aware and conversational recommender systems 2018 co-located with 12th ACM Conference on recommender systems, KaRS@RecSys 2018, Vancouver, Canada, October 7, 2018, CEUR Workshop Proceedings, vol. 2290. CEUR-WS.org, pp. 35–43, https://ceur-ws.org/Vol-2290/kars2018_paper7.pdf

  • Brown, T. B., Mann, B., Ryder, N., & et al. (2020). Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., & et al. (eds.) Advances in neural information processing systems 33: Annual conference on neural information processing systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html

  • Cai, W., Jin, Y., & Chen, L. (2022). Impacts of personal characteristics on user trust in conversational recommender systems. In: CHI conference on human factors in computing systems, pp. 1–14. https://doi.org/10.1145/3491102.3517471

  • Cambria, E., Schuller, B., Xia, Y., et al. (2013). New avenues in opinion mining and sentiment analysis. IEEE Int Systems, 28(2), 15–21. https://doi.org/10.1109/mis.2013.30

    Article  Google Scholar 

  • Carterette, B., Bennett, P. N., Chickering, D. M., & et al. (2008). Here or there. In: ECIR, Springer, pp. 16–27, https://doi.org/10.1007/978-3-540-78646-7_5

  • Chen, L., & Pu, P. (2012). Critiquing-based recommenders: survey and emerging trends. UMUAI, 22(1), 125–150. https://doi.org/10.1007/s11257-011-9108-6

    Article  Google Scholar 

  • De Filippo, A., Lombardi, M., & Milano, M. (2016). Non-linear optimization of business models in the electricity market. In: Integration of AI and OR techniques in constraint programming: 13th international conference, CPAIOR 2016, Banff, AB, Canada, May 29-June 1, 2016, Proceedings 13, Springer, pp. 81–97. https://doi.org/10.1007/978-3-319-33954-2_7

  • Di Palma, D., Biancofiore, G. M., Anelli, V. W., & et al. (2023). Evaluating chatgpt as a recommender system: A rigorous approach. arXiv:2309.03613

  • Dong, R., Schaal, M., O’Mahony, M. P., & et al. (2013). Topic extraction from online reviews for classification and recommendation. In: Rossi, F. (ed.) IJCAI 2013, Proceedings of the 23rd international joint conference on artificial intelligence, Beijing, China, August 3-9, 2013. IJCAI/AAAI, pp 1310–1316, http://www.aaai.org/ocs/index.php/IJCAI/IJCAI13/paper/view/6640

  • Elahi, M., Ricci, F., & Rubens, N. (2016). A survey of active learning in collaborative filtering recommender systems. Computer Science Review, 20, 29–50. https://doi.org/10.1016/j.cosrev.2016.05.002 publisher: Elsevier

  • Filippo, A. D., Lombardi, M., & Milano, M. (2021). Integrated offline and online decision making under uncertainty. J Artif Intell Res, 70, 77–117. https://doi.org/10.1613/JAIR.1.12333

  • Gemmis, Md., Iaquinta, L., Lops, P., & et al. (2011). Learning preference models in recommender systems. Preference Learning pp. 387–407. https://doi.org/10.1007/978-3-642-14125-6_18

  • Glass, J., Polifroni, J., Seneff, S., & et al. (2000). Data collection and performance evaluation of spoken dialogue systems: The MIT experience. In: Sixth international conference on spoken language processing, https://doi.org/10.21437/icslp.2000-737

  • Haveliwala, T. H. (2003). Topic-sensitive pagerank: A context-sensitive ranking algorithm for web search. TKDE, 15(4), 784–796. https://doi.org/10.1109/tkde.2003.1208999

    Article  Google Scholar 

  • He, Z., Xie, Z., Jha, R., & et al. (2023). Large language models as zero-shot conversational recommenders. In: Proceedings of the 32nd ACM international conference on information and knowledge management, pp. 720–730, https://doi.org/10.1145/3583780.3614949

  • Iovine, A., Narducci, F., de Gemmis, M., et al. (2020a). A comparison of services for intent and entity recognition for conversational recommender systems. In: Brusilovsky, P., de Gemmis, M., Felfernig, A., et al. (eds.) Proceedings of the 7th Joint workshop on interfaces and human decision making for recommender systems co-located with 14th ACM conference on recommender systems (RecSys 2020), Online Event, September 26, 2020, CEUR Workshop Proceedings, vol. 2682. CEUR-WS.org, pp. 37–47, https://ceur-ws.org/Vol-2682/paper4.pdf

  • Iovine, A., Narducci, F., Semeraro, G. (2020b). Conversational Recommender Systems and natural language:: A study through the ConveRSE framework. Decision Support Systems p. 113250. https://doi.org/10.1016/j.dss.2020.113250

  • Jannach, D., Manzoor, A., Cai, W., & et al. (2022). A survey on conversational recommender systems. ACM Comput Surv, 54(5), 105:1-105:36. https://doi.org/10.1145/3453154

  • Kang, J., Condiff, K., Chang, S., & et al. (2017). Understanding How People Use Natural Language to Ask for Recommendations. In: Proceedings of the eleventh ACM conference on recommender systems - RecSys ’17. ACM, pp 229–237, https://doi.org/10.1145/3109859.3109873

  • Le, Q. V., & Mikolov, T. (2014). Distributed representations of sentences and documents. In: Proceedings of the 31th international conference on machine learning, ICML 2014, Beijing, China, 21-26 June 2014, JMLR Workshop and Conference Proceedings, vol. 32. JMLR.org, pp. 1188–1196, http://proceedings.mlr.press/v32/le14.html

  • Lei, W., He, X., Miao, Y., & et al. (2020). Estimation-action-reflection: Towards deep interaction between conversational and recommender systems. In: WSDM, pp. 304–312, https://doi.org/10.1145/3336191.3371769

  • Li, R., Kahou, S. E., Schulz, H., & et al. (2018). Towards deep conversational recommendations. https://proceedings.neurips.cc/paper/2018/hash/800de15c79c8d840f4e78d3af937d4d4-Abstract.html

  • Lops, P., de Gemmis, M., Semeraro, G., & et al. (2009). A semantic content-based recommender system integrating folksonomies for personalized access. Web Personalization in Intelligent Environments, pp. 27–47. https://doi.org/10.1007/978-3-642-02794-9_2

  • Lops, P., De Gemmis, M., Semeraro, G., & et al. (2011). Leveraging the linkedin social network data for extracting content-based user profiles. In: Proceedings of the fifth ACM conference on recommender systems, pp. 293–296, https://doi.org/10.1145/2043932.2043986

  • Manzoor, A., & Jannach, D. (2021). Conversational recommendation based on end-to-end learning: How far are we? Computers in Human Behavior Reports, 4(100), 139. https://doi.org/10.1016/j.chbr.2021.100139

    Article  Google Scholar 

  • Manzoor, A., & Jannach, D. (2022). Towards retrieval-based conversational recommendation. Inf Syst, 109(102), 083. https://doi.org/10.1016/J.IS.2022.102083

    Article  Google Scholar 

  • Musto, C., Gemmis, Md., Lops, P., & et al. (2012a). Semantics and Content-Based Recommendations, Springer US, pp. 251–298. https://doi.org/10.1007/978-1-0716-2197-4_7

  • Musto C, Semeraro G, Lops P, & et al (2012b). Leveraging social media sources to generate personalized music playlists. In: E-Commerce and web technologies: 13th international conference, EC-Web 2012, Vienna, Austria, September 4-5, 2012. Proceedings 13, Springer, pp. 112–123, https://doi.org/10.1007/978-3-642-32273-0_10

  • Musto, C., Basile, P., Lops, P., & et al. (2014a). Linked open data-enabled strategies for top-n recommendations. In: Bogers, T., Koolen, M., & Cantador, I. (eds.) Proceedings of the 1st workshop on new trends in content-based recommender systems co-located with the 8th ACM conference on recommender systems, CBRecSys@RecSys 2014, Foster City, Silicon Valley, California, USA, October 6, 2014, CEUR Workshop Proceedings, vol. 1245. CEUR-WS.org, pp. 49–56, https://ceur-ws.org/Vol-1245/cbrecsys2014-paper08.pdf

  • Musto, C., Semeraro, G., Lops, P., & et al. (2014b). Combining distributional semantics and entity linking for context-aware content-based recommendation. In: Dimitrova, V., Kuflik, T., Chin, D., & et al. (eds.) User Modeling, Adaptation, and Personalization - 22nd International Conference, UMAP 2014, Aalborg, Denmark, July 7-11, 2014. Proceedings, Lecture Notes in Computer Science, vol. 8538. Springer, pp. 381–392, https://doi.org/10.1007/978-3-319-08786-3_34

  • Musto, C., de Gemmis, M., Lops, P., et al. (2021). Generating post hoc review-based natural language justifications for recommender systems. User Model User Adapt Interact, 31(3), 629–673. https://doi.org/10.1007/S11257-020-09270-8

    Article  Google Scholar 

  • Musto, C., Lops, P., de Gemmis, M., et al. (2021). Context-aware graph-based recommendations exploiting personalized pagerank. Knowl Based Syst, 216(106), 806. https://doi.org/10.1016/J.KNOSYS.2021.106806

    Article  Google Scholar 

  • Nakagawa, H., & Mori, T. (2002). A simple but powerful automatic term extraction method. In: COLING-02 on COMPUTERM 2002, ACL, pp. 1–7, https://doi.org/10.3115/1118771.1118778

  • Nguyen, H., & Haddawy, P. (1999). The decision-theoretic interactive video advisor. In: Laskey, K. B., & Prade, H. (eds.) UAI ’99: Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence, Stockholm, Sweden, July 30 - August 1, 1999. Morgan Kaufmann, pp 494–501, https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1 &smnu=2 &article_id=202 &proceeding_id=15

  • Pu, P., Chen, L., & Hu, R. (2011). A user-centric evaluation framework for recommender systems, pp. 157–164. https://doi.org/10.1145/2043932.2043962

  • Rafailidis, D., & Manolopoulos, Y. (2019a). Can Virtual Assistants Produce Recommendations? In: Proceedings of the 9th international conference on web intelligence, Mining and Semantics, pp. 1–6. https://doi.org/10.1145/3326467.3326468

  • Rafailidis, D., & Manolopoulos, Y. (2019b) The technological gap between virtual assistants and recommendation systems. arXiv:1901.00431

  • Roumeliotis, K. I., & Tselikas, N. D. (2023). Chatgpt and open-ai models: A preliminary review. Future Internet, 15(6), 192. https://doi.org/10.3390/fi15060192

    Article  Google Scholar 

  • Roy, A., & Ludwig, S. A. (2021). Genre based hybrid filtering for movie recommendation engine. Journal of Intelligent Information Systems 56(3):485–507. https://doi.org/10.1007/s10844-021-00637-w

  • Sacenti, J. A. P., Fileto, R., & Willrich, R. (2021). Knowledge graph summarization impacts on movie recommendations. Journal of Intelligent Information Systems, 58(1), 43–66. https://doi.org/10.1007/s10844-021-00650-z

    Article  Google Scholar 

  • Sanner, S., Balog, K., Radlinski, F., & et al. (2023). Large language models are competitive near cold-start recommenders for language-and item-based preferences. In: Proceedings of the 17th ACM conference on recommender systems, pp. 890–896. https://doi.org/10.1145/3604915.3608845

  • Schnabel, T., Ramos, G., & Amershi, S. (2020). Who doesn’t like dinosaurs? Finding and eliciting richer preferences for recommendation. In: Fourteenth ACM conference on recommender systems, pp. 398–407. https://doi.org/10.1145/3383313.3412267

  • Shokeen, J., & Rana, C. (2019). Social recommender systems: techniques, domains, metrics, datasets and future scope. Journal of Intelligent Information Systems, 54(3), 633–667. https://doi.org/10.1007/s10844-019-00578-5

    Article  Google Scholar 

  • Touvron, H., Lavril, T., Izacard, G., & et al. (2023). Llama: Open and efficient foundation language models. arXiv:2302.13971

  • Tulshan, A.S., & Dhage, S.N. (2018). Survey on virtual assistant: Google assistant, Siri, Cortana, Alexa. In: International symposium on signal processing and intelligent recognition systems, Springer, pp. 190–201. https://doi.org/10.1007/978-981-13-5758-9_17

  • Vrandecic, D., & Krötzsch, M. (2014). Wikidata: a free collaborative knowledge base. Commun ACM, 57(10), 78–85. https://doi.org/10.1145/2629489

    Article  Google Scholar 

  • Zhou, K., Zhao, W. X., Bian, S., & et al. (2020). Improving conversational recommender systems via knowledge graph based semantic fusion. In: KDD, pp. 1006–1014. https://doi.org/10.1145/3394486.3403143

Download references

Acknowledgements

We acknowledge the support of the PNRR project FAIR - Future AI Research (PE00000013), Spoke 6 - Symbiotic AI (CUP H97G22000210007) under the NRRP MUR program funded by the NextGenerationEU.

Funding

Not applicable.

Author information

Authors and Affiliations

  1. University of Bari Aldo Moro, Bari, Italy

    Cataldo Musto, Alessandro Francesco Maria Martina, Andrea Iovine, Marco de Gemmis & Giovanni Semeraro

  2. Polytechnic University of Bari, Bari, Italy

    Fedelucio Narducci

Authors
  1. Cataldo Musto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessandro Francesco Maria Martina
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Iovine
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fedelucio Narducci
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marco de Gemmis
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Giovanni Semeraro
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Musto contributed to the writing of the entire manuscript, supervised the design of the pipeline and was responsible of the experimental setting. Martina was responsible of the development of the same component and followed the execution of the experiments. Iovine developed the paper’s main idea and was responsible for the development of the whole framework for conversational recommendations. Narducci helped with the development of initial idea. de Gemmis and Semeraro helped with the finalization of the paper.

Corresponding author

Correspondence to Cataldo Musto.

Ethics declarations

This paper or a similar version is not currently under review by a journal or conference. This paper is void of plagiarism or self-plagiarism as defined by the Committee on Publication Ethics and Springer Guidelines. A previous version of the article has been published as a LBR at ACM UMAP 2022.In particular, the current paper provides the following novel contributions: (a) an extended description of the knowledge extraction and knowledge exploitation pipelines, that more easily allow to reproduce and replicate the pipeline; (b) the evaluation of two different recommendation algorithms (i.e., Doc2Vec and PageRank); (c) a different experiment setting and research questions, aiming at assessing whether and how subjective properties impact the quality of the recommendations; (d) an extended version of the questionnaire that we used in the user study.

Competing Interests

The authors declare no competing interests.

Ethical Approval

All the participants gave their consent to participate to the user study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Musto, C., Martina, A.F.M., Iovine, A. et al. Tell me what you Like: introducing natural language preference elicitation strategies in a virtual assistant for the movie domain. J Intell Inf Syst 62, 575–599 (2024). https://doi.org/10.1007/s10844-023-00835-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-023-00835-8

Keywords

Search

Navigation