Abstract
Offline logged data is quite common in many web applications such as recommendation, Internet advertising, etc., which offers great potentials to improve online decision making. It is a non-trivial task to utilize offline logged data for online decision making, because the offline logged data is observational and it may mislead online decision making. The VirUCB is one of the latest notable algorithmic frameworks in this research line. This paper studies how to extend VirUCB from upper confidence bound (UCB) based online decision making to Thompson sampling based online decision making, for the purpose of improving the online decision accuracy. We first show that naively applying Thompson sampling to the VirUCB framework is not effective and we reveal fundamental insights on why it is not effective. Based on these insights, we design a filtering algorithm to filter out the logged data corresponding to the optimal arm. To address the challenge that the optimal arm is unknown, we estimate it through the posterior of the reward mean. Putting them together, we obtain our VirTS-DF algorithm. Extensive experiments on two real-world datasets validate the superior performance of VirTS-DF.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Li, Y., Xie, H., Lin, Y., Lui, J.C.: Unifying offline causal inference and online bandit learning for data driven decision. In: WWW (2021)
Imbens, G.W., Rubin, D.B.: Causal inference in statistics, social, and biomedical sciences. Cambridge University Press (2015)
Shivaswamy, P., Joachims, Y.: Multi-armed bandit problems with history. In: AISTAT, pp. 1046–1054 (2012)
Zuo, J., Zhang, X., Joe-Wong, C.: Observe before play: multi-armed bandit with pre-observations. ACM SIGMETRICS Perform. Eval. Rev. 46(2), 89–90 (2019)
Tennenholtz, G., Shalit, U., Mannor, S., Efroni, Y.: Bandits with partially observable offline data. arXiv preprint arXiv:2006.06731 (2020)
Tang, Q., Xie, H.: A robust algorithm to unifying offline causal inference and online multi-armed bandit learning. In: IEEE ICDM (2021)
Lattimore, T., Szepesvári, C.: Bandit algorithms. Cambridge University Press (2020)
Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Mach. Learn. 47(2–3), 235–256 (2002)
Russo, D., Van Roy, B., Kazerouni, A., Osband, I., Wen, Z.: A tutorial on Thompson sampling. arXiv preprint arXiv:1707.02038 (2017)
Chen, W., Wang, Y., Yuan, Y.: Combinatorial multi-armed bandit: general framework and applications. In: ICML, pp. 151–159 (2013)
Bistritz, I., Leshem, A.: Distributed multi-player bandits-a game of thrones approach. In: NIPS (2018)
Wang, C.-C., Kulkarni, S.R., Poor, H.V.: Bandit problems with side observations. IEEE TAC 50(3), 338–355 (2005)
Sharma, N., Basu, S., Shanmugam, K., Shakkottai, S.: Warm starting bandits with side information from confounded data. arXiv preprint arXiv:2002.08405 (2020)
Yun, D., Proutiere, A., Ahn, S., Shin, J., Yi, Y.: Multi-armed bandit with additional observations. Proceed. ACM Measure. Anal. Comput. Syst. 2(1), 1–22 (2018)
Acknowledgment
This work was supported in part by Chongqing Talents: Exceptional Young Talents Project (cstc2021ycjhbgzxm0195), the Chinese Academy of Sciences “Light of West China” Program, the Key Cooperation Project of Chongqing Municipal Education Commission (HZ2021008, HZ2021017), and the “Fertilizer Robot” project of Chongqing Committee on Agriculture and Rural Affairs. Hong Xie is the corresponding author.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Xu, H., Xie, H. (2023). A Thompson Sampling Approach to Unifying Causal Inference and Bandit Learning. In: Kashima, H., Ide, T., Peng, WC. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2023. Lecture Notes in Computer Science(), vol 13936. Springer, Cham. https://doi.org/10.1007/978-3-031-33377-4_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-33377-4_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-33376-7
Online ISBN: 978-3-031-33377-4
eBook Packages: Computer ScienceComputer Science (R0)