Wireless Networks

, Volume 19, Issue 8, pp 1807–1828 | Cite as

A personalized QoE-aware handover decision based on distributed reinforcement learning

  • Behrouz Shahgholi Ghahfarokhi
  • Naser Movahhedinia


Recent developments in heterogeneous mobile networks and growing demands for variety of real-time and multimedia applications have emphasized the necessity of more intelligent handover decisions. Addressing the context knowledge of mobile devices, users, applications, and networks is the subject of context-aware handoff decision as a recent effort to this aim. However, user perception has not been attended adequately in the area of context-aware handover decision making. Mobile users may have different judgments about the Quality of Service (QoS) depending on their environmental conditions, and personal and psychological characteristics. This reality has been exploited in this paper to introduce a personalized user-centric handoff decision method to decide about the time and target of handover based on User Perceived Quality (UPQ) feedbacks. The UPQ degradations are mainly for the sake of (1) exiting the coverage of the serving Point of Attachment (PoA) or (2) QoS degradation of serving access network. Using UPQ metric, the proposed method obviates the necessity of being aware about rapidly varying network QoS parameters and overcomes the complexity and overhead of gathering and managing some other context information. Moreover, considering the underlying network and geographical map, the proposed method is able to inherently exploit the trajectory information of mobile users for handover decision. UPQ degradation is not only due to the user behaviour, but also due to the behaviours of others users. As such, multi-agent reinforcement learning paradigm has been considered for target PoA selection. The employed decision algorithm is based on WoLF-PHC learning method where UPQ is used as a delayed reward for training. The proposed handoff decision has been implemented under IEEE 802.21 framework using NS2 network simulator. The results have shown better performance of the proposed method comparing to conventional methods assuming regular movement of mobile users.


User perceived quality Context-aware handover QoE-aware handover Distributed reinforcement learning 



Adaptive handover decision


Analytic hierarchy process


Information server


Multi attribute decision making


Multi agent reinforcement learning


Mobile controlled handover


MIH independent command service


Media independent event service


Media independent handover


MIH independent information service


Mobile IP


Mobile node


Mean opinion score


Policy hill climbing


Point of attachment


Perceived quality evaluator


Peak signal to noise ratio


Quality of customer experience


Quality of experience


Quality of service


Quality of user experience


Received signal strength


Simple additive weighting


Spatial conceptual map


Stochastic game


Segmental signal to noise ratio




User perceived quality


Way elementary areas


Win or learn fast


  1. 1.
    Prehofer, C., Nafisi, N., & Wei, Q. (2003). A framework for context-aware handover decisions. In 14th IEEE proceedings on personal, indoor and mobile radio communications, PIMRC, 2003, Vol. 3, pp. 2794–2798.Google Scholar
  2. 2.
    Nguyen-Vuong, Q.-T., Agoulmine, N., & Ghamri-Doudane, Y. (2008). A user-centric and context-aware solution to interface management and access network selection in heterogeneous wireless environments. Computer Networks, 52(18), 3358–3372.CrossRefzbMATHGoogle Scholar
  3. 3.
    Jingjing, Z., & Ansari, N. (2011). On assuring end-to-end QoE in next generation networks: Challenges and a possible solution. Communications Magazine, 49(7), 185–191.CrossRefGoogle Scholar
  4. 4.
    Kilkki, K. (2008). Quality of experience in communications ecosystem. Journal of Universal Computer Science, 14(5), 615–624.Google Scholar
  5. 5.
    Stankiewicz, R., & Jajszczyk, A. (2011). A survey of QoE assurance in converged networks. Computer Networks, 55(7), 1459–1473.CrossRefGoogle Scholar
  6. 6.
    Herman, H., Rahman, A. A., Syahbana, Y. A., & Bakar, K. A. (2011). Nonlinearity modelling of QoE for video streaming over wireless and mobile network. In Second international conference on intelligent systems, modelling and simulation (ISMS), 2011, pp. 313–317.Google Scholar
  7. 7.
    Mitra, K., Zaslavsky, A., & Aahlund, C. (2011). A probabilistic context-aware approach for quality of experience measurement in pervasive systems. In Proceedings of the ACM symposium on applied computing, 2011, pp. 419–424.Google Scholar
  8. 8.
    Winkler, S., & Mohandas, P. (2008). The evolution of video quality measurement: from PSNR to hybrid metrics. IEEE Transactions on Broadcasting, 54, 660–668.CrossRefGoogle Scholar
  9. 9.
    Saliba, J., Beresford, A., Ivanovich, M., & Fitzpatrick, P. (2005). User-perceived quality of service in wireless data networks. Personal Ubiquitous Computing, 9(6), 413–422.CrossRefGoogle Scholar
  10. 10.
    Magoulas, G. D., & Ghinea, G. (2001). Neural network-based interactive multicriteria decision making in a quality of perception-oriented management scheme. In International joint conference on neural networks, 2001, Vol. 4, pp. 2536–2541, vol. 2534.Google Scholar
  11. 11.
    Ghahfarokhi, B. S., & Movahhedinia, N. (2011). A contextaware handover decision based on user perceived quality of service trigger. Wireless Communication and Mobile Computing, 11, 723–741.CrossRefGoogle Scholar
  12. 12.
    Piamrat, K. et al. (2011). QoE-aware vertical handover in wireless heterogeneous networks, In Wireless communications and mobile computing conference (IWCMC), 2011, pp. 95–100.Google Scholar
  13. 13.
    Hasswa, A., Nasser, N., & Hassanein, H. (2007). A seamless context-aware architecture for fourth generation wireless networks. Wireless Personal Communications, 43(3), 1035–1049.CrossRefGoogle Scholar
  14. 14.
    Ahmed, T., Kyamakya, K., & Ludwig, M. (2007). Architecture of a context-aware vertical handover decision model and its performance analysis for GPRS–WiFi handover. In 11th IEEE symposium on computers and communications, 2007, pp. 795–801.Google Scholar
  15. 15.
    Kang, J. -M., Ju, H. -T., & Hong, J. (2006). Towards autonomic handover decision management in 4G networks. In Autonomic management of mobile multimedia services, pp. 145–157.Google Scholar
  16. 16.
    Prehofer, C., Nafisi, N., & Wei, Q. (2003). A framework for context-aware handover decisions. In IEEE international symposium on personal, indoor and mobile radio communications, 2003, pp. 2794-2798.Google Scholar
  17. 17.
    Wei, Q., Farkas, K., Prehofer, C., Mendes, P., & Plattner, B. (2006). Context-aware handover using active network technology. Computer Networks, 50(15), 2855–2872.CrossRefGoogle Scholar
  18. 18.
    Ghahfarokhi, B. S., & Movahhedinia, N. (2012). Context-aware handover decision in an enhanced media independent handover framework. Wireless Personal Communications. Published online, doi: 10.1007/s11277-012-0543-4.
  19. 19.
    Neves, P., Soares, J., Sargento, S., Pires, H., & Fontes, F. (2011). Context-aware media independent information server for optimized seamless handover procedures. Computer Networks, 55(7), 1498–1519.CrossRefGoogle Scholar
  20. 20.
    Bowling, M., & Veloso, M. (2002). Multiagent learning using a variable learning rate. Artificial Intelligence, 136(2), 215–250.MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    IEEE 802.21. (2008). IEEE standard for local and metropolitan area networks: Media independent handover services.Google Scholar
  22. 22.
    Cacace, F., & Vollero, L. (2006). Managing mobility and adaptation in upcoming 802.21 enabled devices. In 4th international workshop on wireless mobile applications and services on WLAN hotspots, 2006, pp. 1–10.Google Scholar
  23. 23.
    Oliva, A. d. l., Melia, T., Vidal, A., Bernardos, C. J., Soto, I., & Banchs, A. (2007). IEEE 802.21 enabled mobile terminals for optimized WLAN/3G handovers: a case study. SIGMOBILE Mobile Computing and Communications Review, 11(2), 29–40.Google Scholar
  24. 24.
    ITU-T Rec. P.800. (1996). Methods for subjective determination of transmission quality.Google Scholar
  25. 25.
    ITU-T Rec. J.246. (2008). Perceptual audiovisual quality measurement techniques for multimedia services over digital cable television networks in presence of reduced bandwidth reference.Google Scholar
  26. 26.
    ITU-T Rec. G.1070. (2007). Opinion model for videophone applications. Google Scholar
  27. 27.
    ITU-T Rec. J.247. (2008). Objective perceptual multimedia video quality measurement in the presence of a full reference.Google Scholar
  28. 28.
    ITU-T Rec. p.910. (1999). Subjective video quality assessment method for multimedia applications.Google Scholar
  29. 29.
    Reves, X. (2006). User perceived quality evaluation in a B3G network testbed. 15th IST mobile and wireless summit, 2006. Google Scholar
  30. 30.
    ITU-T Rec. P.861. (1998). Objective quality measurement of telephone-band speech codecs.Google Scholar
  31. 31.
    ITU-T Rec. p.862. (2001). Perceptual evaluation of speech quality (PESQ): An objective method for end to end speech quality assessment of narrowband telephone networks and speech codecs.Google Scholar
  32. 32.
    ITU-T Rec. J.144. (2004). Objective perceptual video quality measurement techniques for digital cable television in presence of full reference.Google Scholar
  33. 33.
    Menkovski, V., Exarchakos, G., & Liotta, A. (2010). Machine learning approach for quality of experience aware networks. In 2nd international conference on intelligent networking and collaborative systems, 2010, pp. 461–466.Google Scholar
  34. 34.
    Fiedler, M., Hossfeld, T., & Tran-Gia, P. (2010). A generic quantitative relationship between quality of experience and quality of service. IEEE Network, 24(2), 36–41.CrossRefGoogle Scholar
  35. 35.
    Brooks, P., & Hestnes, B. r. (2010). User measures of quality of experience: why being objective and quantitative is important. IEEE Network, 24(2), 8–13.CrossRefGoogle Scholar
  36. 36.
    Mahdi, A., & Picovici, D. (2010). New single-ended objective measure for non-intrusive speech quality evaluation. Signal, Image and Video Processing, 4(1), 23–38.CrossRefGoogle Scholar
  37. 37.
    Rein, S., Fitzek, F. H. P., & Reisslein, M. (2005). Voice quality evaluation in wireless packet communication systems: A tutorial and performance results for ROHC. Wireless Communications, 12(1), 60–67.CrossRefGoogle Scholar
  38. 38.
    Jiang, X., Wang, Y., & Wang, C. (2009). No-reference video quality assessment for MPEG-2 video streams using BP neural networks. In 2nd international conference on interaction sciences: Information technology, culture and human, 2009, pp. 307–311.Google Scholar
  39. 39.
    Klaue, J., Rathke, B., & Wolisz, A. (2003). EvalVid—A framework for video transmission and quality evaluation. In Computer performance, pp. 255–272.Google Scholar
  40. 40.
    Takahashi, A., Hands, D., & Barriac, V. (2008). Standardization activities in the ITU for a QoE assessment of IPTV. Communications Magazine, 46, 78–84.CrossRefGoogle Scholar
  41. 41.
    Mitchell, T. M. (1997). Machine learning. New Delhi: McGraw-Hill.zbMATHGoogle Scholar
  42. 42.
    Busoniu, L., Babuska, R., & De Schutter, B. (2008). A Comprehensive Survey of Multiagent Reinforcement Learning. IEEE Transactions on Systems, Man, and Cybernetics Part C: Applications and Reviews, 38(2), 156–172.CrossRefGoogle Scholar
  43. 43.
    Weiss, G. (1999). Multiagent systems: A modern approach to distributed artificial intelligence. Cambridge, MA: MIT Press.Google Scholar
  44. 44.
    Sleem, A., & Kumar, A. (2005). Handoff management in wireless data networks using topography-aware mobility prediction. Journal of Parallel and Distributed Computing, 65(8), 963–982.CrossRefGoogle Scholar
  45. 45.
    Ghahfarokhi, B. S., & Movahhedinia, N. (2007). QoS provisioning by EFuNNs-based handoff planning in cellular MPLS networks. Computer Communications, 30(13), 2676–2685.CrossRefGoogle Scholar
  46. 46.
    Onel, T., Ersoy, C., Cayirci, E., & Parr, G. (2004). A multicriteria handoff decision scheme for the next generation tactical communications systems. Computer Networks, 46(5), 695–708.CrossRefGoogle Scholar
  47. 47.
    Edwards, G., & Sankar, R. (1998). Microcellular handoff using fuzzy techniques. Wireless Networks, 4(5), 401–409.CrossRefGoogle Scholar
  48. 48.
    Sharma, S., Baek, I., & Chiueh, T.-C. (2007). OmniCon: A mobile IP-based vertical handoff system for wireless LAN and GPRS links. Softw. Pract. Exper., 37(7), 779–798.CrossRefGoogle Scholar
  49. 49.
    Zhu, F., & McNair, J. (2006). Multiservice vertical handoff decision algorithms. EURASIP Journal on Wireless Communications and Networks, 2006(2), 52.Google Scholar
  50. 50.
    Stevens-Navarro, E., & Wong, V. W. S. (2006). Comparison between vertical handoff decision algorithms for heterogeneous wireless networks. In IEEE 63rd vehicular technology conference, VTC, 2006, Vol. 2, pp. 947–951.Google Scholar
  51. 51.
    Zhang, W. (2004). Handover decision using fuzzy MADM in heterogeneous networks. In IEEE wireless communications and networking conference, 2004, Vol. 2, pp. 653–658.Google Scholar
  52. 52.
    Hongyan, B., Chen, H., & Lingge, J. (2003). Intelligent signal processing of mobility management for heterogeneous networks. In Proceedings of the international conference on neural networks and signal processing, 2003, Vol. 2, pp. 1578–1581.Google Scholar
  53. 53.
    Chan, P. M. L., Sheriff, R. E., Hu, Y. F., Conforto, P., & Tocci, C. (2002). Mobility management incorporating fuzzy logic for heterogeneous IP environment. Communications Magazine, 39(12), 42–51.CrossRefGoogle Scholar
  54. 54.
    Inoue, M., Mahmud, K., Murakami, H., Hasegawa, M., & Morikawa, H. (2005). Context-based network and application management on seamless networking platform. Wireless Personal Communications, 35(1–2), 53–70.CrossRefGoogle Scholar
  55. 55.
    Wang, C. -Y., Huang, H. -Y., & Hwang, R. -H. (2011). Mobility management in ubiquitous environments. Personal Ubiquitous Computings, 15(3), 235–251.CrossRefGoogle Scholar
  56. 56.
    Indulska, J., & Balasubramaniam, S. (2004). Context-aware vertical handovers between WLAN and 3G networks. In IEEE 59th vehicular technology conference, 2004, Vol. 5, pp. 3019–3023.Google Scholar
  57. 57.
    Kassar, M., Kervella, B., & Pujolle, G. (2008). An overview of vertical handover decision strategies in heterogeneous wireless networks. Computer Communications, 31(10), 2607–2620.CrossRefGoogle Scholar
  58. 58.
    Hong, C.-P., Weems, C. C., & Kim, S.-D. (2008). An effective vertical handoff scheme based on service management for ubiquitous computing. Computer Communications, 31(9), 1739–1750.CrossRefGoogle Scholar
  59. 59.
    Wang, Y., Zhang, P., Zhou, Y., Yuan, J., Liu, F., & Li, G. (2010). Handover management in enhanced MIH framework for heterogeneous wireless networks environment. Wireless Personal Communications, 52(3), 615–636.CrossRefGoogle Scholar
  60. 60.
    Mussabbir, Q. B., Wenbing, Y., Zeyun, N., & Xiaoming, F. (2007). Optimized FMIPv6 using IEEE 802.21 MIH services in vehicular networks. IEEE Transactions on Vehicular Technology, 56(6), 3397–3407.CrossRefGoogle Scholar
  61. 61.
    Samaan, N., & Karmouch, A. (2005). A mobility prediction architecture based on contextual knowledge and spatial conceptual maps. IEEE Transactions on Mobile Computing, 4(6), 537–551.CrossRefGoogle Scholar
  62. 62.
    Sen, S., Sekaran, M., & Hale, J. (1994). Learning to coordinate without sharing information. In Twelfth national conference on artificial intelligence, 1994, Vol. 1, pp. 426–431.Google Scholar
  63. 63.
    Kettani, D., & Moulin, B. (1999). A spatial model based on the notions of spatial conceptual map and of object’s influence areas. Paper presented at the Spatial Information Theory. Cognitive and Computational Foundations of Geographic Information Science, 1999, pp. 401–416.Google Scholar
  64. 64.
    National Institute of Standards and Technology., visited on July 2011.
  65. 65.
    Balasubramaniam, S., & Indulska, J. (2004). Vertical handover supporting pervasive computing in future wireless networks. Computer Communications, 27(8), 708–719.CrossRefGoogle Scholar
  66. 66.
    Rovatsos, M., & Belesiotis, A. (2007). Advice taking in multiagent reinforcement learning, In Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems (AAMAS), 2007, pp. 1342–1344.Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Behrouz Shahgholi Ghahfarokhi
    • 1
  • Naser Movahhedinia
    • 2
  1. 1.Department of Information Technology EngineeringUniversity of IsfahanIsfahanIran
  2. 2.Department of Computer EngineeringUniversity of IsfahanIsfahanIran

Personalised recommendations