Learning User Preferences in Ubiquitous Systems: A User Study and a Reinforcement Learning Approach

  • Sofia Zaidenberg
  • Patrick Reignier
  • Nadine Mandran
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 339)


Our study concerns a virtual assistant, proposing services to the user based on its current perceived activity and situation (ambient intelligence). Instead of asking the user to define his preferences, we acquire them automatically using a reinforcement learning approach. Experiments showed that our system succeeded the learning of user preferences. In order to validate the relevance and usability of such a system, we have first conducted a user study. 26 non-expert subjects were interviewed using a model of the final system. This paper presents the methodology of applying reinforcement learning to a real-world problem with experimental results and the conclusions of the user study.


Ambient Intelligence Context-aware Computing Personnal Assistant Reinforcement Learning User Study 


  1. 1.
    Assad, M., Carmichael, D., Kay, J., Kummerfeld, B.: PersonisAD: Distributed, active, scrutable model framework for context-aware services. In: LaMarca, A., Langheinrich, M., Truong, K.N. (eds.) Pervasive 2007. LNCS, vol. 4480, pp. 55–72. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  2. 2.
    Barton, J., Pierce, J.: Quantifying magic in ubicomp systems scenarios. In: Position Paper for UbiSys 2006, Orange County, California (2006)Google Scholar
  3. 3.
    Bellotti, V., Edwards, K.: Intelligibility and accountability: human considerations in context-aware systems. HCI 16(2), 193–212 (2001)CrossRefGoogle Scholar
  4. 4.
    Brdiczka, O., Crowley, J.L., Reignier, P.: Learning situation models for providing context-aware services. In: Stephanidis, C. (ed.) UAHCI 2007, Part II. LNCS, vol. 4555, pp. 23–32. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  5. 5.
    Byun, H.E., Cheverst, K.: Exploiting user models and context-awareness to support personal daily activities. In: Workshop in UM 2001 on User Modeling for Context-Aware Applications (2001)Google Scholar
  6. 6.
    Chen, H., Finin, T., Joshi, A.: A context broker for building smart meeting rooms. In: Proceedings of AAAI Spring Symposium, Stanford, California, pp. 53–60. AAAI Press, Menlo Park (2004)Google Scholar
  7. 7.
    Chen, H., Finin, T., Joshi, A.: An ontology for context-aware pervasive computing environments. Special Issue on Ontologies for Distributed Systems, Knowledge Engineering Review 18(3), 197–207 (2004)Google Scholar
  8. 8.
    Crowley, J.L., Coutaz, J., Rey, G., Reignier, P.: Perceptual components for context aware computing. In: Borriello, G., Holmquist, L.E. (eds.) UbiComp 2002. LNCS, vol. 2498, pp. 117–134. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  9. 9.
    Dey, A.K., Abowd, G.D.: The context toolkit: Aiding the development of context-aware applications. In: The Workshop on Software Engineering for Wearable and Pervasive Computing, Limerick, Ireland (June 2000)Google Scholar
  10. 10.
    Doctor, F., Hagras, H., Callaghan, V.: An intelligent fuzzy agent approach for realising ambient intelligence in intelligent inhabited environments. IEEE Tr. on Systems, Man and Cybernetics, Part A 35, 55–65 (2005)CrossRefGoogle Scholar
  11. 11.
    Dourish, P.: What we talk about when we talk about context. Personal Ubiquitous Comput. 8(1), 19–30 (2004)CrossRefGoogle Scholar
  12. 12.
    Ducatel, K., Bogdanowicz, M., Scapolo, F., Leijten, J., Burgelman, J.-C.: Scenarios for ambient intelligence in 2010. Technical report, ISTAG (2001)Google Scholar
  13. 13.
    Godoy, D., Amandi, A.: User profiling for web page filtering. IEEE Internet Computing 9(4), 56–64 (2005)CrossRefGoogle Scholar
  14. 14.
    José, R.: Ubicomp 2.0: From envisioning a future to being part of a new reality. In: Plenary Talk at the UCAmI 2008 conference (2008)Google Scholar
  15. 15.
    Maes, P.: Agents that reduce work and information overload. Commun. ACM 37(7), 30–40 (1994)CrossRefGoogle Scholar
  16. 16.
    Pascoe, J., Thomson, K., Rodrigues, H.: Context-Awareness in the Wild: An Investigation into the Existing Uses of Context in Everyday Life. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM-WS 2007, Part I. LNCS, vol. 4805, pp. 193–202. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  17. 17.
    Rogers, Y.: Moving on from weiser’s vision of calm computing: Engaging ubicomp experiences. In: Dourish, P., Friday, A. (eds.) UbiComp 2006. LNCS, vol. 4206, pp. 404–421. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  18. 18.
    Sutton, R.S.: Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: ICML1990, pp. 216–224 (1990)Google Scholar
  19. 19.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning). The MIT Press, Cambridge (1998)Google Scholar
  20. 20.
    Taylor, A.S., Harper, R., Swan, L., Izadi, S., Sellen, A., Perry, M.: Homes that make us smart. Personal and Ubiquitous Computing 11(5), 383–393 (2007)CrossRefGoogle Scholar
  21. 21.
    Weiser, M.: The computer for the 21st century. Scientific American 265(3), 66–75 (1991)CrossRefGoogle Scholar
  22. 22.
    Zaidenberg, S., Reignier, P., Crowley, J.L.: Reinforcement Learning of Context Models for a Ubiquitous Personal Assistant. In: Advances in Soft Computing, vol. 51, pp. 254–264. Springer, Heidelberg (September 2008)Google Scholar

Copyright information

© IFIP 2010

Authors and Affiliations

  • Sofia Zaidenberg
    • 1
  • Patrick Reignier
    • 1
  • Nadine Mandran
    • 2
  1. 1.INRIA 
  2. 2.LIG 

Personalised recommendations