Multimedia Tools and Applications

, Volume 74, Issue 17, pp 6871–6896 | Cite as

The impact of hesitation, a social signal, on a user’s quality of experience in multimedia content retrieval

  • Tomaž Vodlan
  • Marko Tkalčič
  • Andrej Košir


The social signal (SS) of hesitation is commonly manifested through a multiplicity of nonverbal behavioural cues when a user is faced with a variety of decision choices. The aim of this study is to show that the utilization of the SS of hesitation in a conversational recommender system (RS) can improve the user quality of experience (QoE) when interacting with a video-on-demand system. An appropriate experimental design was modelled to detect the impact of the SS. The experimental scenario was a manual video-on-demand system with a conversational RS where the user selected one video clip among several presented on the screen. The system adjusted the list of the video items to be recommended according to the extracted SS class {hesitation, no hesitation}. To detect if the user was hesitating, we used hand movements, eye behaviour and time between two selections. Two user groups were tested to allow realistic estimation of the impact of the SS. In the user test group, the SS of hesitation was considered, while in the control group it was not. The evaluation of impact of the SS on QoE was based on pre- and post-interaction questionnaires. Our results showed a significant difference in user satisfaction with the system between those two groups, indicating that the use of SS of hesitation in conversational RS improves the QoE when the user interacts with a video-on-demand system.


Social signals Hesitation Human–computer interaction Video-on-demand Recommender system 



Operation partially financed by the European Union, European Social Fund. This work was supported by the EU Seventh Framework Programme FP7 / 2007–2013 through the project PHENICX (grant no. 601166).


  1. 1.
    Aggarwall JK, Ryoo MS (2011) Human activity analysis: a review. ACM Comput Surv 43(3). doi: 10.1145/1922649.1922649.1922653
  2. 2.
    Bewick V, Cheek L, Ball J (2005) Statistics review 14: logistic regression. Crit Care 9(1):112–118. doi: 10.1186/cc3045 CrossRefGoogle Scholar
  3. 3.
    Bousmalis K, Morency L, Pantic M (2011) Modeling hidden dynamics of multimodal cues for spontaneous agreement and disagreement recognition. In Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition, pp 746–752Google Scholar
  4. 4.
    Branco N, Zagalo N, Branco P, Otero N, Centre A (2011) Blink: observing thin slices of behavior to determine users’ expectation towards task difficulty. In Proceedings of the International Conference on Human Factors in Computing Systems - CHI 2011, pp 2299–2304Google Scholar
  5. 5.
    Brooke J (1996) SUS: a “quick and dirty” usability scale. In: Jordan PW et al (eds) Usability evaluation in industry. CRC Press, Taylor and Francis, LondonGoogle Scholar
  6. 6.
    Bruzgiene R, Narbutaite L, Adomkus T, Cibulskis R (2013) Subjective and objective MOS evaluation of user’s perceived quality assessment for IPTV service: a study of the experimental investigations. Elektronika ir Elektrotechnika 19(7):110–113. doi: 10.5755/j01.eee.19.7.5178 CrossRefGoogle Scholar
  7. 7.
    Carmel M, Kuflik T (2010) Social signal processing: detecting small group interaction in leisure activity. In Proceedings of the 15th international conference on Intelligent user interfaces IUI’10, pp 309–312Google Scholar
  8. 8.
    COMMIT (2011) Sensing for natural interaction. Accessed 12 Dec 2013
  9. 9.
    Cronbach LJ (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16(3):297–334. doi: 10.1007/BF02310555 CrossRefGoogle Scholar
  10. 10.
    Dawes JG (2008) Do data characteristics change according to the number of scale points used? An experiment using 5 point, 7 point and 10 point scales. Int J Mark Res 50(1):61–78MathSciNetGoogle Scholar
  11. 11.
    Diefendorff JM, Richard EM, Gosserand RH (2006) Examination of situational and attitudinal moderators of the hesitation and performance relation. Pers Psychol 59(2):365–393. doi: 10.1111/j.1744-6570.2006.00641.x CrossRefGoogle Scholar
  12. 12.
    Ferreira JP, Noronha e Sousa M, Branco N, Ferreira MJ, Otero N, Zagalo N, Branco P (2012) Thin slices of interaction: predicting users’ task difficulty within 60 sec. In Proceedings of the CHI ‘12, Extended Abstracts on Human Factors in Computing Systems, pp 171–180Google Scholar
  13. 13.
    Field A (2009) Discovering statistics using SPSS, 3rd edn. SAGE Publications Ltd, LondonGoogle Scholar
  14. 14.
    Finstad K (2010) Response interpolation and scale sensitivity: evidence against 5-point scales. JUS 5(3):104–110Google Scholar
  15. 15.
    Fornell C, Larcker DF (1981) Evaluating structural equation models with unobservable variables and measurement error. JMKR 18(1):39–50CrossRefGoogle Scholar
  16. 16.
    Gino F, Schweiitzer ME (2008) Blinded by anger or feeling the love: how emotions influence advice taking. J Appl Psychol 93(5):1165–1173. doi: 10.1037/0021-9010.93.5.1165 CrossRefGoogle Scholar
  17. 17.
    Håkansson M (2012) Human-computer interaction. Accessed 19 July 2013
  18. 18.
    Hollnagel E, Woods DD (2005) Joint cognitive systems. Foundations of cognitive systems engineering. CRC Press, Taylor & Francis Group, London, p 219CrossRefGoogle Scholar
  19. 19.
    Hu A (2001) Video-on-demand broadcasting protocols: a comprehensive study. In Proceedings of the 20th Annual Joint Conference of the IEEE Computer and Communications Societies, pp 508–517Google Scholar
  20. 20.
    Hung H, Chittaranjan G (2010) The idiap wolf corpus: exploring group behaviour in a competitive role-playing game. In Proceedings of the international conference on Multimedia, MM ‘10, pp 879–882Google Scholar
  21. 21.
    IBM Corp. (2012) IBM SPSS statistics for windows, version 21.0. IBM Corp., ArmonkGoogle Scholar
  22. 22.
    IDRE-UCLA (2012) What is dummy coding? Accessed 17 June 2013
  23. 23.
    International Telecommunication Union (2007) Consideration on channel zapping time in IPTV performace monitoring. 4th FG IPTV meeting. Bled, Slovenia, 7–11 May 2007. Accessed 15 Dec 2013
  24. 24.
    Jokinen K, Allwood J (2010) Hesitation in intercultural communication: some observations and analyses on interpreting shoulder shrugging. In: Ishida T (ed) Culture and computing: computing and communication for crosscultural interaction. Springer, BerlinGoogle Scholar
  25. 25.
    Justin T, Pobar M, Ipšić I, Mihelič F, Žibert J (2012) A bilingual HMM-based speech synthesis system for closely related languages. LNCS 7499:543–550. doi: 10.1007/978-3-642-32790-2-66 Google Scholar
  26. 26.
    Kanji GK (2006) 100 statistical tests. SAGE Publications, LondonGoogle Scholar
  27. 27.
    Karam M, Schraefel MC (2005) A taxonomy of gestures in human computer interaction, Faculty of Physical Sciences and Engineering University of Southampton, Southampton. Accessed 6 Dec 2013
  28. 28.
    Knijnenburg BP, Kobsa A (2012) Making decisions about privacy: information disclosure in context-aware recommender systems. Institute for Software Research, University of California, IrvineGoogle Scholar
  29. 29.
    Knijnenburg BP, Rao N, Kobsa A (2012) Experimental materials used in the study on inspectability and control in social recommender systems. Institute for Software Research, University of California, IrvineGoogle Scholar
  30. 30.
    Kooij R, Ahmed K, Brunnström K (2006) Perceived quality of channel zapping. In Proceedings of 5th IASTED International Conference Communication Systems and Networks, Palma de Mallorca, Spain, 28–30 August 2006, pp 155–158Google Scholar
  31. 31.
    Koren Y (2008) Factorization meets the neighborhood: a multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD, pp 426–434Google Scholar
  32. 32.
    Košir A, Odić A, Kunaver M, Tkalčič M, Tasič JF (2011) Database for contextual personalization. Elektrotehniški Vestnik 78(5):270–274Google Scholar
  33. 33.
    Lerner JS, Small DA, Loewenstein G (2004) Heart strings and purse strings: carry-over effects of emotion on economic transactions. Psychol Sci 15(5):337–340CrossRefGoogle Scholar
  34. 34.
    Leung R, McGrenere J, Graf P (2011) Age-related differences in the initial usability of mobile device icons. Behav Inf Technol 30(5):629–642. doi: 10.1080/01449290903171308 CrossRefGoogle Scholar
  35. 35.
    Lew M, Bakker EM, Sebe N, Huang TS (2007) Human-computer intelligent interaction: a survey. In Proceedings of the 2007 I.E. international conference on Human-computer interaction, pp 1–5Google Scholar
  36. 36.
    Montgomery DC (2009) Design and analysis of experiments. John Wiley & Sons, HobokenGoogle Scholar
  37. 37.
    Moon AJ, Panton B, HFM, Van der Loos M, Croft E (2010) Using hesitation gestures for safe and ethical human-robot interaction. In Proceedings of the ICRA 2010, pp 11–13Google Scholar
  38. 38.
    Moon A, Parker CAC, Croft EA, Van der Loos HFM (2011) Did you see it hesitate?—Empirically grounded design of hesitation trajectories for collaborative robots. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 1994–1999Google Scholar
  39. 39.
    Mu X, Chen Y, Yang J, Jiang J (2010) An improved similarity algorithm based on hesitation degree for user-based collaborative filtering. In: Cai Z et al (eds) Advances in computation and intelligence. Sprineger-Verlag, BerlinGoogle Scholar
  40. 40.
    Nicholas D (2005) How age impacts website usability for teens and seniors. Accessed 20 July 2013
  41. 41.
    Nielsen J (1994) Response times: the three important limits. Accessed 17 Dec 2013
  42. 42.
    Nunnally JC (1967) Psycohmetric theory, 1st edn. McGraw-Hill, New YorkGoogle Scholar
  43. 43.
    Odić A, Tkalčič M, Tasič JF, Košir A (2013) Predicting and detecting the relevant contextual information in a movie-recommender system. Interact Comput 25(1):74–90. doi: 10.1093/iwc/iws003 Google Scholar
  44. 44.
    Pantic M, Nijholt A, Pentland A, Huang TS (2008) Human-centred intelligent human-computer interaction (HCI2): how far are we from attaining it? IJAACS 1(2):168–187CrossRefGoogle Scholar
  45. 45.
    Pentland A (2007) Social signal processing. IEEE Signal Proc Mag 24(4):108–111. doi: 10.1109/msp.2007.4286569 CrossRefGoogle Scholar
  46. 46.
    Ranne R (2008) Usability and system intelligence. In: Hämäläinen RP, Saarinen E (eds) Systems intelligence: a new lens on human engagement and action. University of Technology, Helsinki, pp 141–157Google Scholar
  47. 47.
    Ricci F et al (eds) (2011) Recommender systems handbook. Springer, New York. doi: 10.1007/978-0-387-85820-3 zbMATHGoogle Scholar
  48. 48.
    Sauro J (2012) Asking the right user experience questions. Accessed 19 June 2013
  49. 49.
    Seow SS (2008) Designing and engineering time: the psychology of time perception in software. Addison-Wesley Professional, BostonGoogle Scholar
  50. 50.
    Song Y, Demerirdjian D, Davis R (2012) Continuous body and hand gesture recognition for natural human-computer interaction. ACM Trans Interact Intell Syst Spec Issue Affect Interact Nat Environ 2(1):11–118. doi: 10.1145/2133366.2133371 Google Scholar
  51. 51.
    Sun X, Nijholt A, Truong KP, Pantic M (2012) Automatic visual mimicry expression analysis in interpersonal interaction. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR-W’11), Workshop on CVPR for Human Behaviour Analysis, pp 40–46Google Scholar
  52. 52.
    Tkalčič M, Odić A, Košir A, Tasič JF (2013) Affective labelling in a content-based recommender system for images. IEEE Trans Multimedia 15(2):391–400. doi: 10.1109/TMM.2012.2229970 CrossRefGoogle Scholar
  53. 53.
    Vinciarelli A, Dielmann A, Favre S, Salamin H (2009) Canal9: a database of political debates for analysis of social interactions. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction, pp 1–4Google Scholar
  54. 54.
    Vinciarelli A, Pantic M, Bourlard H (2009) Social signal processing: survey of an emerging domain. Image Vision Comput 27(12):1743–1759. doi: 10.1016/j.imavis.2008.11.007 CrossRefGoogle Scholar
  55. 55.
    Vinciarelli A, Slamin H, Pantic M (2009) Social signal processing: understanding social interactions through nonverbal behavior analysis. In Proceedings of the Computer Vision and Pattern Recognition Workshops, pp 42–49Google Scholar
  56. 56.
    Vinciarelli A, Pantic M, Heylen D, Pelachaud C, Poggi I, D’Errico F, Schroeder M (2012) Bridging the gap between social animal and unsocial machine: a survey of social signal processing. IEEE Trans Affect Comput 3(1):69–87. doi: 10.1109/t-affc.2011.27 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Agila d.o.o.LjubljanaSlovenia
  2. 2.Johannes Kepler UniversityLinzAustria
  3. 3.Faculty of Electrical EngineeringUniversity of LjubljanaLjubljanaSlovenia

Personalised recommendations