InApp Questions – an Approach for Contextual Evaluation of Applications

  • Jorge Ferraz de AbreuEmail author
  • Pedro Almeida
  • Pedro Beça
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 689)


The growing success of second-screen devices is changing the way users relate and interact with the television bringing a new dimension to the TV ecosystem. Nevertheless, the specificities of mobile devices are creating numerous and significant challenges in the field of usability and user experience (UX). In addition, the evaluation of applications of the TV ecosystem should not be restricted to the measurement of its instrumental qualities typically belonging to the usability dimension, since it is also important to consider dimensions related to the UX. This paper focuses on the evaluation of a second screen application - the TV content discovery App GUIDER. Two data collection strategies for the evaluation of the App were used: a methodology based on InApp questions (that allows to measure momentary UX) and a more conventional evaluation based on an online questionnaire answered at the end of evaluation sessions (episodic UX). The use of both data collection strategies was intended to get comparative insights to validate if the InApp questions approach could get similar results when compared with those from traditional surveys. The results showed that the response deviation was very low reinforcing the suitability of such alternative data gathering method. The results indicate that triggering InApp questions while users are interacting is a promising method for evaluating applications, namely for UX dimensions, either for functional prototypes or final products. Nevertheless, the validation of such hypothesis requires further and dedicated research.


User experience Usability Evaluation methodologies Interactive television 


  1. 1.
    Zhang, D., Adipat, B.: Challenges, methodologies, and issues in the usability testing of mobile applications. Int. J. Hum. Comput. Interact. 18, 293–308 (2005)CrossRefGoogle Scholar
  2. 2.
    Treder, M., Pachucki, A., Zielonko, A., Lukasiewicz, K.: Mobile Book of Trends (2014)Google Scholar
  3. 3.
    Bank, C., Zuberi, W.: Mobile UI Design Patterns (2014)Google Scholar
  4. 4.
    Abreu, J., Almeida, P., Teles, B.: TV discovery & enjoy: a new approach to help users finding the right TV program to watch. In: Proceedings of the 2014 ACM International Conference on Interactive Experiences for TV and Online Video, pp. 63–70 (2014)Google Scholar
  5. 5.
    Roto, V., Law, E., Vermeeren, A., Hoonhout, J.: User experience white paper. In: Bringing Clarity to Concept User Exp., pp. 1–12 (2011)Google Scholar
  6. 6.
    T.N. Company: Screen wars: the battle for eye space in a TV-everywhere world (2015)., Accessed: 13 June 2015
  7. 7.
    R. B. Media, Second Screen Series - Paper 1: Setting The Scene, vol. 1 (2012)Google Scholar
  8. 8.
    Hsu, S.H., Wen, M.-H., Lin, H.-C., Lee, C.-C., Lee, C.-H.: AIMED-A personalized TV recommendation system. In: European Conference on Interactive Television, pp. 166–174 (2007)Google Scholar
  9. 9.
    BBC Research & Development, “Sibyl Recommender System” (2012). Accessed: 26 Dec 2013
  10. 10.
    Krauss, C., George, L., Arbanowski, S.: New Systems Programming. TV predictor: personalized program recommendations to be displayed on SmartTVs. In: Proceedings of the 2nd Int. Work. Big Data Streams Heterog. Source Min. Algorithms Appl. BigMine 2013. Vol. 23, pp. 63–70. ACM, New York (2013)Google Scholar
  11. 11.
    Obrist, M., Bernhaupt, R., Tscheligi, M.: Interactive television for the home: An ethnographic study on users’ requirements and experiences. Int. J. HCI 24(2) (2008)Google Scholar
  12. 12.
    Gauntlett, D., Hill, A.: TV living: Television, culture and everyday life. Routledge (2002)Google Scholar
  13. 13.
    Law, E.: The measurability and predictability of user experience. In: Proceedings of EICS, pp. 1–10. ACM, New York (2011)Google Scholar
  14. 14.
    I.S.O. (ISO), Ergonomics of Human–System Interaction – Part 210: Human-centred Design for Interactive Systems (formerly known as 13407) (2010)Google Scholar
  15. 15.
    User Experience. Accessed 13 Jan 2017
  16. 16.
    Vermeeren, A.P.O.S., Law, E.L.-C., Roto, V., Obrist, M., Hoonhout, J., Vananen-Vainio-Mattila, K.: User experience evaluation methods: current state and development needs. In: Proceedings of the 6th Nord. Conf. Human Computer Interact. Extending Boundaries, pp. 521–530. ACM (2010)Google Scholar
  17. 17.
    All about UX (2017). Accessed 13 Jan 2017
  18. 18.
    Bernhaupt, R., Pirker, M.: Evaluating user experience for interactive television: towards the development of a domain-specific user experience questionnaire. In: Kotzé, P., Marsden, G., Lindgaard, G., Wesson, J., Winckler, M. (eds.) INTERACT 2013. LNCS, vol. 8118, pp. 642–659. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40480-1_45 CrossRefGoogle Scholar
  19. 19.
    Drouet, D., Bernhaupt, R.: User experience evaluation methods: lessons learned from an interactive TV case-study. In: Bogdan, C., Gulliksen, J., Sauer, S., Forbrig, P., Winckler, M., Johnson, C., Palanque, P., Bernhaupt, R., Kis, F. (eds.) HCSE/HESSD -2016. LNCS, vol. 9856, pp. 351–358. Springer, Cham (2016). doi: 10.1007/978-3-319-44902-9_22 CrossRefGoogle Scholar
  20. 20.
    Abreu, J., Almeida, P., Silva, T.: A UX evaluation approach for second-screen applications. In: Abásolo, M.J., Perales, Francisco J., Bibiloni, A. (eds.) jAUTI/CTVDI -2015. CCIS, vol. 605, pp. 105–120. Springer, Cham (2016). doi: 10.1007/978-3-319-38907-3_9 CrossRefGoogle Scholar
  21. 21.
    Brooke, J., et al.: SUS-A quick and dirty usability scale. Usability Eval. Ind. 189(194), 4–7 (1996)Google Scholar
  22. 22.
    Attrakdiff (2017)., Accessed 13 Jan 2017
  23. 23.
    Jenny Preece, H.S., Rogers, Y.: Interaction Design: Beyond Human-Computer Interaction (2015)Google Scholar
  24. 24.
    Benedek, J., Miner, T.: Measuring Desirability: New methods for evaluating desirability in a usability lab setting. Proc. Usability Prof. Assoc. 2003(8–12), 57 (2002)Google Scholar
  25. 25.
    Burzacca, P., Paternò, F.: Remote usability evaluation of mobile web applications. In: Kurosu, M. (ed.) HCI 2013. LNCS, vol. 8004, pp. 241–248. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-39232-0_27 CrossRefGoogle Scholar
  26. 26.
    Harrison, R., Flood, D., Duce, D.: Usability of mobile applications: literature review and rationale for a new usability model. J. Interact. Sci. 1(1), 1 (2013)CrossRefGoogle Scholar
  27. 27.
    Pagano, D., Maalej, W.: User feedback in the appstore: an empirical study. In: 2013 21st IEEE International Requirements Engineering Conference (RE), pp. 125–134 (2013)Google Scholar
  28. 28.
    Nayebi, F., Desharnais, J.-M., Abran, A.: The state of the art of mobile application usability evaluation. 2012 25th IEEE Canadian Conference on Electrical and Computer Engineering, pp. 1–4, May 2012Google Scholar
  29. 29.
    Abreu, J., Almeida, P., Silva, T., Oliveira, R.: Discovering in a second screen app: perspectives from portuguese and brazilian markets. Procedia Comput. Sci. 64, 1240–1247 (2015)Google Scholar
  30. 30.
    Hassenzahl, M.: The interplay of beauty, goodness, and usability in interactive products. Hum.-Comput. Interact. 19(4), 319–349 (2004)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.University of Aveiro DigiMediaAveiroPortugal

Personalised recommendations