Soft Computing

, Volume 21, Issue 23, pp 7067–7075 | Cite as

Fitted Q-iteration and functional networks for ubiquitous recommender systems

  • Matteo Gaeta
  • Francesco Orciuoli
  • Luigi Rarità
  • Stefania Tomasiello
Methodologies and Application


Ubiquitous recommender systems facilitate users on-location by personalized recommendations of items in the proximity via mobile devices. Due to a high variability of situations and preferences, an efficient resource processing is needed in order to assist the user in a proper way. In this paper, we consider a recommender system, able to learn preferences/habits of users through contextual information, such as location and time, using a new offline model-free approximate Q-iteration. Following the basic idea of Fitted Q-Iteration, the paper focuses on a computational scheme, based on functional networks, and that, unlike the well-known neural ones, does not require a large number of training samples. A preliminary case study, which deals with a shopping mall, is useful to show the approximation capabilities of the proposed approach.


Q-learning Functional networks Ubiquitous context Recommender systems 


  1. Acampora G, Gaeta M, Tomasiello S (2013) An extended functional network model and its application for a gas sensing system. Soft Comput 17(5):897–908CrossRefGoogle Scholar
  2. Adomavicius G, Tuzhilin A (2008) Context-aware recommender systems. In: Proceedings of the 2008 ACM conference on recommender systems, pp 335–336Google Scholar
  3. Alonso-Betanzos A, Sanchez-Marono N, Carballal-Fortes FM, Suarez-Romero J, Perez-Sanchez B (2007) Classification of computer intrusions using functional networks: a comparative study. In: Proceedings of European symposium on artificial neural networksGoogle Scholar
  4. Antos A, Munos R, Szepesvari C (2008) Fitted Q-iteration in continuous action-space MDPs. Adv Neural Inf Process Syst 20:9–16Google Scholar
  5. Baltrunas L (2008) Exploiting contextual information in recommender systems. In: Proceedings of the 2008 ACM conference on recommender systems, pp 295–298Google Scholar
  6. Bruen M, Yang J (2005) Functional networks in real-time flood forecasting: a novel application. Adv Water Resour 28:899–909CrossRefGoogle Scholar
  7. Castillo E (1998) Functional networks. Neural Process Lett 7:151–159CrossRefMathSciNetGoogle Scholar
  8. Castillo E, Cobo A, Gutiérrez JM, Pruneda E (1998) Working with differential, functional and difference equations using functional networks. Appl Math Model 23(2):89–107CrossRefMATHGoogle Scholar
  9. Castillo E, Gutiérrez JM, Cobo A, Castillo C (2000) A minimax method for learning functional networks. Neural Process Lett 11(1):39–49CrossRefGoogle Scholar
  10. Castillo E, Iglesias A, Ruiz-Cobo R (2005) Funct Equ Appl Sci. Elsevier, AmsterdamGoogle Scholar
  11. David VK, Rajasekaran S (2009) Pattern recognition using neural and functional networks. In: Studies in computational intelligence, 160. Springer, BerlinGoogle Scholar
  12. Dey AK, Abowd GD, Salber D (2001) A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum Comput Interact 16:97–166CrossRefGoogle Scholar
  13. El-Sebakhy EA (2008) New computational intelligence paradigm for estimating the software project effort. In: Proceedings of international conference on advanced information networking and applications (AINA)Google Scholar
  14. El-Sebakhy EA, Hadi AS, Faisal KA (2007) Iterative least squares functional networks classifier. IEEE Trans Neural Netw 18(3):844–850CrossRefGoogle Scholar
  15. Ernst D, Geurts P, Wehenkel L (2005) Tree-based batch mode reinforcement learning. J Mach Learn Res 6:503–556MATHMathSciNetGoogle Scholar
  16. Fang B, Liao S, Xu K, Cheng H, Zhu C, Chen H (2012) A novel mobile recommender system for indoor shopping. Expert Syst Appl 39(15):11992–12000CrossRefGoogle Scholar
  17. Fano AE (1998) Shopper’s eye: using location-based filtering for a shopping agent in the physical world. In: AGENTS ’98 proceedings of the second international conference on autonomous agents, pp 416–421Google Scholar
  18. Gaeta M, Loia V, Tomasiello S (2013) A generalized functional network for a classifier-quantifiers scheme in a gas-sensing system. Int J Intell Syst 28(10):988–1009CrossRefGoogle Scholar
  19. Gaeta M, Loia V, Miranda S, Tomasiello S (2016) Fitted Q—iteration by functional networks for control problems. Appl Math Model (to appear)Google Scholar
  20. Gediminas A, Tuzhilin A (2005) Toward the next generation of recommender system: a survey of state-of the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):149–156Google Scholar
  21. Gordon GJ (1995) Online fitted reinforcement learning. In: VFA workshop at ML-95Google Scholar
  22. Helmy T, Fatai A (2010) Hybrid computational intelligence models for porosity and permeability prediction of petroleum reservoirs. Int J Comput Intel Appl 9(4):313–337CrossRefGoogle Scholar
  23. Iglesias A, Arcay B, Cotos JM, Taboada JA, Dafonte C (2004) A comparison between functional networks and artificial neural networks for the prediction of fishing catches. Neural Comput Appl 13:24–31CrossRefGoogle Scholar
  24. Lacruz B, Perez-Palomares A, Pruneda RE (2006) Functional networks for classification and regression problems. In: Proceedings of international conference on mathematical and statistical modeling in Honor of Enrique CastilloGoogle Scholar
  25. Mahmood T, Mujtaba G, Venturini A (2014) Dynamic personalization in conversational recommender systems. Inf Syst E-Bus Manag 12(2):213–238CrossRefGoogle Scholar
  26. Mahmood T, Ahmed SH, Mahmood S (2010) Comparing reward-based optimal behaviors in user-adapted recommender systems. In: Proceedings of 3rd IEEE international conference on computer science and information technology, vol 5, pp 332–336Google Scholar
  27. Mettouris C, Papadopoulos GA (2014) Ubiquitous recommender systems. Computing 96(3):223–257CrossRefGoogle Scholar
  28. Neumann G, Peters J (2008) Fitted Q-iteration by advantage weighted regression. Adv Neural Inf Process Syst 21:1177–1184Google Scholar
  29. Ormoneit D, Sen S (2002) Kernel-based reinforcement learning. Mach Learn 49:161–178CrossRefMATHGoogle Scholar
  30. Press WH, Teukolsky SA, Vetterling WT, Flannery BP (1992) Numerical recipes in C: the art of scientific computing, 2nd edn. Cambridge University Press, New YorkMATHGoogle Scholar
  31. Pruneda RE, Lacruz B, Solares C (2005) A first approach to solve classification problems based on functional networks. In: Artificial neural networks: formal models and their applications. Lecture notes in computer science—ICANN, vol 3697, pp 313–318Google Scholar
  32. Rajasekaran S, Thiruvenkatasamy K, Lee T-L (2006) Tidal level forecasting using functional and sequential learning neural networks. Appl Math Model 30:85–103CrossRefMATHGoogle Scholar
  33. Resnick P, Varian HR (1997) Recommender systems. Commun ACM 40(3):56–58CrossRefGoogle Scholar
  34. Riedmiller M (2005) Neural fitted Q-iteration—first experiences with a data efficient neural reinforcement learning method. In: Machine learning: ECML 2005, Volume 3720 of the series Lecture Notes in Computer Science, pp 317–328Google Scholar
  35. Riedmiller M, Braun H (1993) A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In: Proceedings of the IEEE international conference on neural networks (ICNN)Google Scholar
  36. Sanchez-Marono N, Alonso-Betanzos A (2007) Feature selection based on sensitivity analysis. In: Lecture notes in computer science—12th conference Spanish association for artificial intelligence and its associated conference on technology transfer on artificial intelligence (CAEPIA/TTIA), vol 4788, pp 239–248Google Scholar
  37. Singh S, Litman D, Kearns M, Walker M (2002) Optimizing dialogue management with reinforcement learning: experiments with the NJFun system. J Artif Intell Res 16:105–133MATHGoogle Scholar
  38. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, CambridgeGoogle Scholar
  39. Taghipour N, Kardan A (2008) A hybrid web recommender system based on Q-learning, In: SAC ’08 proceedings of the 2008 ACM symposium on applied computing, pp 1164–1168Google Scholar
  40. Timmer S, Riedmiller M (2007) Fitted Q–iteration with CMACs. In: Proceedings of IEEE international symposium on approximate dynamic programming and reinforcement learning (ADPRL)Google Scholar
  41. Tomasiello S (2011) A functional network to predict fresh and hardened properties of selfcompacting concretes. Int J Numer Methods Biomed Eng 27(6):840–847CrossRefMATHGoogle Scholar
  42. Watkins CJCH, Dayan P (1992) Q-learning. Mach Learn 8:279–292MATHGoogle Scholar
  43. Zhou Y, He D-X, Nong Z (2005) Application of functional networks tosolving classification problems. World Acad Sci EngTechnol 12:71–74Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Dipartimento di Ingegneria dell’Informazione, Ingegneria Elettrica e Matematica ApplicataUniversity of SalernoFiscianoItaly
  2. 2.Dipartimento di Scienze Aziendali - Management and Innovation SystemsUniversity of SalernoFiscianoItaly
  3. 3.CO.RI.SA., COnsorzio RIcerca Sistemi ad AgentiUniversity of SalernoFiscianoItaly

Personalised recommendations