Skip to main content
SpringerLink
Log in

Caching Contents with Varying Popularity Using Restless Bandits

  • Conference paper
  • First Online:
Performance Evaluation Methodologies and Tools (VALUETOOLS 2023)

  • 94 Accesses

Abstract

We study content caching in a wireless network in which the users are connected through a base station that is equipped with a finite capacity cache. We assume a fixed set of contents whose popularity vary with time. Users’ requests for the contents depend on their instantaneous popularity levels. Proactively caching contents at the base station incurs a cost but not having requested contents at the base station also incurs a cost. We propose to proactively cache contents at the base station so as to minimize content missing and caching costs. We formulate the problem as a discounted cost Markov decision problem that is a restless multi-armed bandit problem. We provide conditions under which the problem is indexable and also propose a novel approach to manoeuvre a few parameters to render the problem indexable. We demonstrate efficacy of the Whittle index policy via numerical evaluation.

This work was supported by Centre for Network Intelligence, Indian Institute of Science (IISc), a CISCO CSR initiative.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    There are several instances of content popularity being modelled as Markov chains, e.g., see [9, 19, 20].

  2. 2.

    One can as well consider minimizing expected value of \(\sum _{t=1}^{\infty }\sum _{i \in \mathcal{C}}\beta _i^t c_i(a_i(t-1),r_i(t-1),a_i(t))\). This would model the scenario where the contents have geometrically distributed lifetimes with parameters \(\beta _i\)s. Our RMAB-based solution continues to apply in this case.

  3. 3.

    As in Sect. 3.1, we omit the content index.

References

  1. https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.pdf

  2. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley (1994)

    Google Scholar 

  3. Whittle, P.: Restless bandits: activity allocation in a changing world. J. Appl. Prob. 25(A), 287–298 (1988)

    Google Scholar 

  4. Bertsekas, D.: Dynamic Programming and Optimal Control, I and II, Athena Scientific, Belmont, Massachusetts. New York-San Francisco-London (1995)

    Google Scholar 

  5. Glazebrook, K., Mitchell, H.: An index policy for a stochastic scheduling model with improving/deteriorating jobs. Naval Res. Logistics (NRL) 49(7), 706–721 (2002)

    Article  MathSciNet  Google Scholar 

  6. Glazebrook, K.D., Ruiz-Hernandez, D., Kirkbride, C.: Some indexable families of restless bandit problems. Adv. Appl. Probab. 38(3), 643–672 (2006)

    Article  MathSciNet  Google Scholar 

  7. Ansell, P., Glazebrook, K.D., Nino-Mora, J., O’Keeffe, M.: Whittle’s index policy for a multi-class queueing system with convex holding costs. Math. Methods Oper. Res. 57(1), 21–39 (2003)

    Article  MathSciNet  Google Scholar 

  8. Li, S., Xu, J., Van Der Schaar, M., Li, W.: Popularity-driven content caching. In: IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications. IEEE, pp. 1–9 (2016)

    Google Scholar 

  9. Sadeghi, A., Sheikholeslami, F., Giannakis, G.B.: Optimal and scalable caching for 5G using reinforcement learning of space-time popularities. IEEE J. Selected Topics Signal Process. 12(1), 180–190 (2017)

    Article  Google Scholar 

  10. Gao, J., Zhang, S., Zhao, L., Shen, X.: The design of dynamic probabilistic caching with time-varying content popularity. IEEE Trans. Mob. Comput. 20(4), 1672–1684 (2020)

    Article  Google Scholar 

  11. Abani, N., Braun, T., Gerla, M.: Proactive caching with mobility prediction under uncertainty in information-centric networks. In: Proceedings of the 4th ACM Conference on Information-Centric Networking, pp. 88–97 (2017)

    Google Scholar 

  12. Traverso, S., Ahmed, M., Garetto, M., Giaccone, P., Leonardi, E., Niccolini, S.: Temporal locality in today’s content caching: why it matters and how to model it. ACM SIGCOMM Comput. Commun. Rev. 43(5), 5–12 (2013)

    Article  Google Scholar 

  13. ElAzzouni, S., Wu, F., Shroff, N.,Ekici, E.: Predictive caching at the wireless edge using near-zero caches. In: Proceedings of the Twenty-First International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing, pp. 121–130 (2020)

    Google Scholar 

  14. Xiong, G., Wang, S., Li, J., Singh, R.: Model-free reinforcement learning for content caching at the wireless edge via restless bandits (2022). arXiv preprint arXiv:2202.13187

  15. Le Ny, J., Feron, E.: Restless bandits with switching costs: linear programming relaxations, performance bounds and limited lookahead policies. In: 2006 American Control Conference, p. 6 (2006)

    Google Scholar 

  16. Aalto, S., Lassila, P., Osti, P.: Whittle index approach to size-aware scheduling with time-varying channels. In: Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, pp. 57–69 (2015)

    Google Scholar 

  17. Anand, A., de Veciana, G.: A whittle’s index based approach for QoE optimization in wireless networks. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems 2(1), 1–39 (2018)

    Google Scholar 

  18. Duran, S., Ayesta, U., Verloop, I.M.: On the whittle index of markov modulated restless bandits. Queueing Syst. (2022)

    Google Scholar 

  19. Sadeghi, A., Wang, G., Giannakis, G.B.: Deep reinforcement learning for adaptive caching in hierarchical content delivery networks. IEEE Trans. Cognit. Commun. Netw. 5(4), 1024–1033 (2019)

    Article  Google Scholar 

  20. Wu, P., Li, J., Shi, L., Ding, M., Cai, K., Yang, F.: Dynamic content update for wireless edge caching via deep reinforcement learning. IEEE Commun. Lett. 23(10), 1773–1777 (2019)

    Article  Google Scholar 

  21. Avrachenkov, K.E., Borkar, V.S.: Whittle index based Q-learning for restless bandits with average reward. Automatica 139, 110186 (2022)

    Article  MathSciNet  Google Scholar 

  22. Fu, J., Nazarathy, Y., Moka, S., Taylor, P.G.: Towards Q-learning the whittle index for restless bandits. In: Australian & New Zealand Control Conference (ANZCC). IEEE, vol. 2019, pp. 249–254 (2019)

    Google Scholar 

  23. Robledo, F., Borkar, V., Ayesta, U., Avrachenkov, K.: QWI: Q-learning with whittle index. ACM SIGMETRICS Performance Eval. Rev. 49(2), 47–50 (2022)

    Article  Google Scholar 

  24. Pavamana, K.J., Singh, C.: Caching contents with varying popularity using restless bandits (2023). https://arxiv.org/pdf/2304.12227.pdf

  25. Larranaga, M., Ayesta, U., Verloop, I.M.: Index policies for a multi-class queue with convex holding cost and abandonments. In: The ACM International Conference on Measurement and Modeling of Computer Systems 2014, 125–137 (2014)

    Google Scholar 

  26. Liu, K., Zhao, Q.: Indexability of restless bandit problems and optimality of whittle index for dynamic multichannel access. IEEE Trans. Inf. Theory 56(11), 5547–5567 (2010)

    Article  MathSciNet  Google Scholar 

  27. Meshram, R., Manjunath, D., Gopalan, A.: On the whittle index for restless multiarmed hidden Markov bandits. IEEE Trans. Autom. Control 63(9), 3046–3053 (2018)

    Article  MathSciNet  Google Scholar 

  28. Akbarzadeh, N., Mahajan, A.: Conditions for indexability of restless bandits and an \(\cal{O} \!\left(k^3\right)\) algorithm to compute whittle index. Adv. Appl. Probab. 54(4), 1164–1192 (2022)

    Article  MathSciNet  Google Scholar 

  29. Gast, N., Gaujal, B., Khun, K.: Testing indexability and computing whittle and gittins index in subcubic time. Math. Methods Oper. Res. 97(3), 391–436 (2023). https://doi.org/10.1007/s00186-023-00821-4

Download references

Author information

Authors and Affiliations

  1. Department of Electronic Systems Engineering, Indian Institute of Science, Bengaluru, India

    K. J. Pavamana & Chandramani Singh

Authors
  1. K. J. Pavamana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chandramani Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to K. J. Pavamana .

Editor information

Editors and Affiliations

  1. University of Cambridge, Cambridge, UK

    Evangelia Kalyvianaki

  2. University of Southern California, Los Angeles, CA, USA

    Marco Paolieri

Rights and permissions

Reprints and permissions

Copyright information

© 2024 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pavamana, K.J., Singh, C. (2024). Caching Contents with Varying Popularity Using Restless Bandits. In: Kalyvianaki, E., Paolieri, M. (eds) Performance Evaluation Methodologies and Tools. VALUETOOLS 2023. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 539. Springer, Cham. https://doi.org/10.1007/978-3-031-48885-6_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-48885-6_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-48884-9

  • Online ISBN: 978-3-031-48885-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation