Reinforcement Learning of Context Models for a Ubiquitous Personal Assistant

  • Sofia Zaidenberg
  • Patrick Reignier
  • James L. Crowley
Part of the Advances in Soft Computing book series (AINSC, volume 51)


Ubiquitous environments may become a reality in a foreseeable future and research is aimed on making them more and more adapted and comfortable for users. Our work consists on applying reinforcement learning techniques in order to adapt services provided by a ubiquitous assistant to the user. The learning produces a context model, associating actions to perceived situations of the user. Associations are based on feedback given by the user as a reaction to the behavior of the assistant. Our method brings a solution to some of the problems encountered when applying reinforcement learning to systems where the user is in the loop. For instance, the behavior of the system is completely incoherent at the be-ginning and needs time to converge. The user does not accept to wait that long to train the system. The user’s habits may change over time and the assistant needs to integrate these changes quickly. We study methods to accelerate the reinforced learning process.


Reinforcement Learn Transition Model Markov Decision Process Context Model Learning Agent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bardram, J.E.: The Java Context Awareness Framework (JCAF) - A Service Infrastructure and Programming Framework for Context-Aware Applications. In: Pervasive Computing (2005)Google Scholar
  2. 2.
    Brdiczka, O., Reignier, P., Crowley, J.L.: Automatic development of an abstract context model for an intelligent environment. In: PerCom (2005)Google Scholar
  3. 3.
    Christensen, J., Sussman, J., Levy, S., Bennett, W.E., Wolf, T.V., Kellogg, W.A.: Too much information. ACM Queue (2006)Google Scholar
  4. 4.
    Crowley, J.L., Coutaz, J., Rey, G., Reigner, P.: Perceptual components for context awareness. In: International conference on ubiquitous computing (2002)Google Scholar
  5. 5.
    Degris, T., Sigaud, O., Wuillemin, P.-H.: Learning the structure of factored markov decision processes in reinforcement learning problems. In: ICML (2006)Google Scholar
  6. 6.
    Dent, L., Boticario, J., Mitchell, T., Sabowski, D., McDermott, J.: A personal learning apprentice. In: AAAI (1992)Google Scholar
  7. 7.
    Dey, A.K., Abowd, G.D.: The context toolkit: Aiding the development of context-aware applications. In: SEWPC (2000)Google Scholar
  8. 8.
    Emonet, R., Vaufreydaz, D., Reignier, P., Letessier, J.: O3MiSCID: an Object Oriented Opensource Middleware for Service Connection, Introspection and Discovery. In: SIPE 2006 (2006)Google Scholar
  9. 9.
    Isbell, C., Shelton, C.R., Kearns, M., Singh, S., Stone, P.: A social reinforcement learning agent. In: AGENTS (2001)Google Scholar
  10. 10.
    Kozierok, R., Maes, P.: A learning interface agent for scheduling meetings. In: IUI (1993)Google Scholar
  11. 11.
    Maes, P.: Agents that reduce work and information overload. ACM, New York (1994)Google Scholar
  12. 12.
    Ricquebourg, V., Menga, D., Durand, D., Marhic, B., Delahoche, L., Log, C.: The smart home concept: our immediate future. In: ICELIE (2006)Google Scholar
  13. 13.
    Roman, M., Hess, C.K., Cerqueira, R., Ranganathan, A., Campbell, R.H., Nahrstedt, K.: Gaia: A middleware infrastructure to enable active spaces. In: IEEE Pervasive Computing (2002)Google Scholar
  14. 14.
    Schiaffino, S., Amandi, A.: Polite personal agent. Intelligent Systems (2006)Google Scholar
  15. 15.
    Sutton, R.S.: Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: ICML (1990)Google Scholar
  16. 16.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction (1998)Google Scholar
  17. 17.
    Vallée, M., Ramparany, F., Vercouter, L.: Dynamic service composition in ambient intelligence environments: a multi-agent approach. In: YRSOC (2005)Google Scholar
  18. 18.
    Weiser, M.: The computer for the 21st century. Scientific American (1991)Google Scholar
  19. 19.
    Zaidenberg, S., Reignier, P., Crowley, J.L.: An architecture for ubiquitous applications. In: IWUC (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Sofia Zaidenberg
    • 1
  • Patrick Reignier
    • 1
  • James L. Crowley
    • 1
  1. 1.Laboratoire LIGSt-Martin d’HèresFrance

Personalised recommendations