Soft Computing

, Volume 21, Issue 23, pp 7067–7075 | Cite as

Fitted Q-iteration and functional networks for ubiquitous recommender systems

  • Matteo Gaeta
  • Francesco Orciuoli
  • Luigi Rarità
  • Stefania Tomasiello
Methodologies and Application


Ubiquitous recommender systems facilitate users on-location by personalized recommendations of items in the proximity via mobile devices. Due to a high variability of situations and preferences, an efficient resource processing is needed in order to assist the user in a proper way. In this paper, we consider a recommender system, able to learn preferences/habits of users through contextual information, such as location and time, using a new offline model-free approximate Q-iteration. Following the basic idea of Fitted Q-Iteration, the paper focuses on a computational scheme, based on functional networks, and that, unlike the well-known neural ones, does not require a large number of training samples. A preliminary case study, which deals with a shopping mall, is useful to show the approximation capabilities of the proposed approach.


Q-learning Functional networks Ubiquitous context Recommender systems 


Compliance with ethical standards

Conflict of interest

All the authors declare that they have no conflict of interest.

Human and animals rights

This article does not contain any studies with human participants or animals performed by any of the authors.


  1. Acampora G, Gaeta M, Tomasiello S (2013) An extended functional network model and its application for a gas sensing system. Soft Comput 17(5):897–908CrossRefGoogle Scholar
  2. Adomavicius G, Tuzhilin A (2008) Context-aware recommender systems. In: Proceedings of the 2008 ACM conference on recommender systems, pp 335–336Google Scholar
  3. Alonso-Betanzos A, Sanchez-Marono N, Carballal-Fortes FM, Suarez-Romero J, Perez-Sanchez B (2007) Classification of computer intrusions using functional networks: a comparative study. In: Proceedings of European symposium on artificial neural networksGoogle Scholar
  4. Antos A, Munos R, Szepesvari C (2008) Fitted Q-iteration in continuous action-space MDPs. Adv Neural Inf Process Syst 20:9–16Google Scholar
  5. Baltrunas L (2008) Exploiting contextual information in recommender systems. In: Proceedings of the 2008 ACM conference on recommender systems, pp 295–298Google Scholar
  6. Bruen M, Yang J (2005) Functional networks in real-time flood forecasting: a novel application. Adv Water Resour 28:899–909CrossRefGoogle Scholar
  7. Castillo E (1998) Functional networks. Neural Process Lett 7:151–159CrossRefMathSciNetGoogle Scholar
  8. Castillo E, Cobo A, Gutiérrez JM, Pruneda E (1998) Working with differential, functional and difference equations using functional networks. Appl Math Model 23(2):89–107CrossRefzbMATHGoogle Scholar
  9. Castillo E, Gutiérrez JM, Cobo A, Castillo C (2000) A minimax method for learning functional networks. Neural Process Lett 11(1):39–49CrossRefGoogle Scholar
  10. Castillo E, Iglesias A, Ruiz-Cobo R (2005) Funct Equ Appl Sci. Elsevier, AmsterdamGoogle Scholar
  11. David VK, Rajasekaran S (2009) Pattern recognition using neural and functional networks. In: Studies in computational intelligence, 160. Springer, BerlinGoogle Scholar
  12. Dey AK, Abowd GD, Salber D (2001) A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum Comput Interact 16:97–166CrossRefGoogle Scholar
  13. El-Sebakhy EA (2008) New computational intelligence paradigm for estimating the software project effort. In: Proceedings of international conference on advanced information networking and applications (AINA)Google Scholar
  14. El-Sebakhy EA, Hadi AS, Faisal KA (2007) Iterative least squares functional networks classifier. IEEE Trans Neural Netw 18(3):844–850CrossRefGoogle Scholar
  15. Ernst D, Geurts P, Wehenkel L (2005) Tree-based batch mode reinforcement learning. J Mach Learn Res 6:503–556zbMATHMathSciNetGoogle Scholar
  16. Fang B, Liao S, Xu K, Cheng H, Zhu C, Chen H (2012) A novel mobile recommender system for indoor shopping. Expert Syst Appl 39(15):11992–12000CrossRefGoogle Scholar
  17. Fano AE (1998) Shopper’s eye: using location-based filtering for a shopping agent in the physical world. In: AGENTS ’98 proceedings of the second international conference on autonomous agents, pp 416–421Google Scholar
  18. Gaeta M, Loia V, Tomasiello S (2013) A generalized functional network for a classifier-quantifiers scheme in a gas-sensing system. Int J Intell Syst 28(10):988–1009CrossRefGoogle Scholar
  19. Gaeta M, Loia V, Miranda S, Tomasiello S (2016) Fitted Q—iteration by functional networks for control problems. Appl Math Model (to appear)Google Scholar
  20. Gediminas A, Tuzhilin A (2005) Toward the next generation of recommender system: a survey of state-of the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):149–156Google Scholar
  21. Gordon GJ (1995) Online fitted reinforcement learning. In: VFA workshop at ML-95Google Scholar
  22. Helmy T, Fatai A (2010) Hybrid computational intelligence models for porosity and permeability prediction of petroleum reservoirs. Int J Comput Intel Appl 9(4):313–337CrossRefGoogle Scholar
  23. Iglesias A, Arcay B, Cotos JM, Taboada JA, Dafonte C (2004) A comparison between functional networks and artificial neural networks for the prediction of fishing catches. Neural Comput Appl 13:24–31CrossRefGoogle Scholar
  24. Lacruz B, Perez-Palomares A, Pruneda RE (2006) Functional networks for classification and regression problems. In: Proceedings of international conference on mathematical and statistical modeling in Honor of Enrique CastilloGoogle Scholar
  25. Mahmood T, Mujtaba G, Venturini A (2014) Dynamic personalization in conversational recommender systems. Inf Syst E-Bus Manag 12(2):213–238CrossRefGoogle Scholar
  26. Mahmood T, Ahmed SH, Mahmood S (2010) Comparing reward-based optimal behaviors in user-adapted recommender systems. In: Proceedings of 3rd IEEE international conference on computer science and information technology, vol 5, pp 332–336Google Scholar
  27. Mettouris C, Papadopoulos GA (2014) Ubiquitous recommender systems. Computing 96(3):223–257CrossRefGoogle Scholar
  28. Neumann G, Peters J (2008) Fitted Q-iteration by advantage weighted regression. Adv Neural Inf Process Syst 21:1177–1184Google Scholar
  29. Ormoneit D, Sen S (2002) Kernel-based reinforcement learning. Mach Learn 49:161–178CrossRefzbMATHGoogle Scholar
  30. Press WH, Teukolsky SA, Vetterling WT, Flannery BP (1992) Numerical recipes in C: the art of scientific computing, 2nd edn. Cambridge University Press, New YorkzbMATHGoogle Scholar
  31. Pruneda RE, Lacruz B, Solares C (2005) A first approach to solve classification problems based on functional networks. In: Artificial neural networks: formal models and their applications. Lecture notes in computer science—ICANN, vol 3697, pp 313–318Google Scholar
  32. Rajasekaran S, Thiruvenkatasamy K, Lee T-L (2006) Tidal level forecasting using functional and sequential learning neural networks. Appl Math Model 30:85–103CrossRefzbMATHGoogle Scholar
  33. Resnick P, Varian HR (1997) Recommender systems. Commun ACM 40(3):56–58CrossRefGoogle Scholar
  34. Riedmiller M (2005) Neural fitted Q-iteration—first experiences with a data efficient neural reinforcement learning method. In: Machine learning: ECML 2005, Volume 3720 of the series Lecture Notes in Computer Science, pp 317–328Google Scholar
  35. Riedmiller M, Braun H (1993) A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In: Proceedings of the IEEE international conference on neural networks (ICNN)Google Scholar
  36. Sanchez-Marono N, Alonso-Betanzos A (2007) Feature selection based on sensitivity analysis. In: Lecture notes in computer science—12th conference Spanish association for artificial intelligence and its associated conference on technology transfer on artificial intelligence (CAEPIA/TTIA), vol 4788, pp 239–248Google Scholar
  37. Singh S, Litman D, Kearns M, Walker M (2002) Optimizing dialogue management with reinforcement learning: experiments with the NJFun system. J Artif Intell Res 16:105–133zbMATHGoogle Scholar
  38. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, CambridgeGoogle Scholar
  39. Taghipour N, Kardan A (2008) A hybrid web recommender system based on Q-learning, In: SAC ’08 proceedings of the 2008 ACM symposium on applied computing, pp 1164–1168Google Scholar
  40. Timmer S, Riedmiller M (2007) Fitted Q–iteration with CMACs. In: Proceedings of IEEE international symposium on approximate dynamic programming and reinforcement learning (ADPRL)Google Scholar
  41. Tomasiello S (2011) A functional network to predict fresh and hardened properties of selfcompacting concretes. Int J Numer Methods Biomed Eng 27(6):840–847CrossRefzbMATHGoogle Scholar
  42. Watkins CJCH, Dayan P (1992) Q-learning. Mach Learn 8:279–292zbMATHGoogle Scholar
  43. Zhou Y, He D-X, Nong Z (2005) Application of functional networks tosolving classification problems. World Acad Sci EngTechnol 12:71–74Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Dipartimento di Ingegneria dell’Informazione, Ingegneria Elettrica e Matematica ApplicataUniversity of SalernoFiscianoItaly
  2. 2.Dipartimento di Scienze Aziendali - Management and Innovation SystemsUniversity of SalernoFiscianoItaly
  3. 3.CO.RI.SA., COnsorzio RIcerca Sistemi ad AgentiUniversity of SalernoFiscianoItaly

Personalised recommendations