Wireless Networks

, Volume 16, Issue 2, pp 481–495 | Cite as

A Markov Decision Process based flow assignment framework for heterogeneous network access

  • Jatinder Pal Singh
  • Tansu Alpcan
  • Piyush Agrawal
  • Varun Sharma


We consider a scenario where devices with multiple networking capabilities access networks with heterogeneous characteristics. In such a setting, we address the problem of efficient utilization of multiple access networks by devices via optimal assignment of traffic flows with given utilities to different networks. We develop and analyze a device middleware functionality that monitors network characteristics and employs a Markov Decision Process (MDP) based control scheme that in conjunction with stochastic characterization of the available bit rate and delay of the networks generates an optimal policy for allocation of flows to different networks. The optimal policy maximizes, under available bit rate and delay constraints on the access networks, a discounted reward which is a function of the flow utilities. The flow assignment policy is periodically updated and is consulted by the flows to dynamically perform network selection during their lifetimes. We perform measurement tests to collect traces of available bit rate and delay characteristics on Ethernet and WLAN networks on a work day in a corporate work environment. We implement our flow assignment framework in ns-2 and simulate the system performance for a set of elastic video-like flows using the collected traces. We demonstrate that the MDP based flow assignment policy leads to significant enhancement in the QoS provisioning (higher rate allocation, lower packet delays and packet loss rates) for the flows and better access network utilization, as compared to policies that allocate flows to different networks using greedy approaches or heuristics like average available bit rate on the networks.


Heterogeneous access networks Flow assignment Markov Decision Process Simulative performance evaluation 



Research Supported by Deutsche Telekom AG. An earlier version of the work appeared in the IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks, June 18–21, 2007, Helsinki, Finland.


  1. 1.
    Vidales, P., Baliosion, J., Serrat, J., Mapp, G., Stejano, F., & Hopper, A. (2005). Autonomic system for mobility support in 4G networks. IEEE Journal on Selected Areas in Communications, 23(12), 2288–2304.CrossRefGoogle Scholar
  2. 2.
  3. 3.
    Cuevas, A., Moreno, J. I., Vidales, P., & Einsiedler, H. (2006). The IMS platform: A solution for next generation network operators to be more than bit pipes. In IEEE Communications Magazine, Issue on Advances of Service Platform Technologies.Google Scholar
  4. 4.
    Bertsekas, D. (2001). Dynamic programming and optimal control (Vol. 2, 2nd ed.). Belmont, MA: Athena Scientific.zbMATHGoogle Scholar
  5. 5.
    The Network Simulator (ns-2).
  6. 6.
    Yaiche, H., Mazumdar, R., & Rosenburg, C. (2000). A game theoretic framework for bandwidth allocation and pricing in broadband networks. IEEE/ACM Transactions on Networking (TON), 8(5), 667–678.CrossRefGoogle Scholar
  7. 7.
    Alpcan, T., & Başar, T. (2005). A utility-based congestion control scheme for internet-style networks with delay. IEEE Transactions on Networking, 13(6), 1261–1274.CrossRefGoogle Scholar
  8. 8.
    Alpcan, T., & Başar, T. (2003). Global stability analysis of an end-to-end congestion control scheme for general topology networks with delay. In Proc. of the 42nd IEEE Conference on Decision and Control (pp. 1092–1097). Maui, HI.Google Scholar
  9. 9.
    Chebrolu, K., & Rao, R. (2002). Communication using multiple wireless interfaces. In Proc. IEEE Wireless Communications and Networking Conference (WCNC 2002) (Vol. 1, pp. 327–331).Google Scholar
  10. 10.
    Fodor, G., Furuksar, A., & Lundsjo, J. (2004). On access selection techniques in always best connected networks. In Proc. ITC Specialist Seminar on Performance Evaluation of Wireless and Mobile Systems.Google Scholar
  11. 11.
    Zemlianov, A., & de Veciana, G. (2005). Cooperation and decision-making in a Wireless multi-provider setting. In Proc. IEEE INFOCOM (pp. 1–14).Google Scholar
  12. 12.
    Shakkottai, S., Altman, E., & Kumar, A. (2005). The case for non-cooperative multihoming of users to access points in IEEE 802.11 WLANs. In Proc. IEEE INFOCOM.Google Scholar
  13. 13.
    Furuksar, A., & Zander, J. (2005). Multiservice allocation for multiaccess wireless systems. IEEE Transactions on Wireless Communications, 4, 174–184.CrossRefGoogle Scholar
  14. 14.
    Mariz, D., Cananea, I., Sadok, D., & Fodor, G. (2006). Simulative analysis of access selection algorithms for multi-access networks. In Proc. IEEE Internation Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM 2006).Google Scholar
  15. 15.
    Thompson, N., He, G., & Luo, H. (2006). Flow scheduling for end-host multihoming. In Proc. IEEE INFOCOM.Google Scholar
  16. 16.
    Kumar, D., Altman, E., & Kelif, J.-M. (2007). Globally optimal user-network association in an 802.11 WLAN and 3G UMTS hybrid cell. In Proc. 20th International Telegraph Congress. Ottawa, Canada.Google Scholar
  17. 17.
    Alpcan, T., Singh, J. P., & Başar, T. (2007). A robust flow control framework for heterogeneous network access. In Proc. 5th International Symposium on Modeling and Optimization in Mobile, Ad-hoc, and Wireless Networks (WiOpt).Google Scholar
  18. 18.
    Zhu, X., Agrawal, P., Singh, J. P., Alpcan, T., & Girod, B. (2007). Rate allocation for multi-user video streaming over heterogeneous access networks. In Proc. ACM Multimedia.Google Scholar
  19. 19.
    Singh, J. P., Alpcan, T., Zhu, X., & Agrawal, P. (2007). Towards heterogeneous network convergence: Policies and middleware architecture for efficient flow assignment, rate allocation and rate control for multimedia applications. In Workshop on Middleware for Next-generation Converged Networks and Application (MNCNA), ACM/IFIP/USENIX 8th International Middleware Conference.Google Scholar
  20. 20.
    Guo, F., Chen, J., Li, W., & Chiueh, T. (2004). Experiences in building a multihoming load balancing system. In: Proc. IEEE INFOCOM.Google Scholar
  21. 21.
    Konrad, A., Zhao, B., Joseph, A., & Ludwig, R. (2003). A markov-based channel model algorithm for wireless networks. Wireless Networks (pp. 189–199).Google Scholar
  22. 22.
    Salmatian, K., & Vaton, S. (2001). Hidden Markov Modeling for network communication channels. In Proc. ACM SIGMETRICS.Google Scholar
  23. 23.
    Froyland, G. (2001). Extracting dynamical behaviour via Markov models. In Nonlinear Dynamics and Statistics: Proceedings, Newton Institute, Cambridge, 1998 (pp. 283–324). Birkhauser.Google Scholar
  24. 24.
    StuhlMuller, K., Farber, N., Link, M., & Girod, B. (2000). Analysis of video transmission over lossy channels. IEEE Journal on Selected Areas in Communication, 18(6), 1012–1032.CrossRefGoogle Scholar
  25. 25.
    Navratil, J., & Cottrell, R. L., Abing.
  26. 26.
    Laboratory, U. N. R., CapProbe.

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Jatinder Pal Singh
    • 1
  • Tansu Alpcan
    • 2
  • Piyush Agrawal
    • 3
  • Varun Sharma
    • 4
  1. 1.Deutsche Telekom LaboratoriesLos AltosUSA
  2. 2.Deutsche Telekom LaboratoriesBerlinGermany
  3. 3.Department of Electrical EngineeringStanford UniversityStanfordUSA
  4. 4.Lehman Brothers Inc.MumbaiIndia

Personalised recommendations