Abstract
Online advertising is increasingly switching to real-time bidding on advertisement inventory, in which the ad slots are sold through real-time auctions upon users visiting websites or using mobile apps. To compete with unknown bidders in such a highly stochastic environment, each bidder is required to estimate the value of each impression and to set a competitive bid price. Previous bidding algorithms have done so without considering the constraint of budget limits, which we address in this paper. We model the bidding process as a Constrained Markov Decision Process based reinforcement learning framework. Our model uses the predicted click-through-rate as the state, bid price as the action, and ad clicks as the reward. We propose a bidding function, which outperforms the state-of-the-art bidding functions in terms of the number of clicks when the budget limit is low. We further simulate different bidding functions competing in the same environment and report the performances of the bidding strategies when required to adapt to a dynamic environment.
M. Brorsson—Work done while Mats Brorsson was at OLAmobile, Luxembourg.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The CTR can be seen as the probability of a user clicking on the ad being shown. The predicted CTR is a prediction of this probability based on features of the publisher site/app and the user visiting it.
- 2.
- 3.
- 4.
References
Aggarwal, C.C.: Data Mining: The Textbook. Springer, Cham (2015)
Altman, E.: Constrained Markov Decision Processes. CRC Press, Boca Raton (1999)
Amin, K., Kearns, M., Key, P., Schwaighofer, A.: Budget optimization for sponsored search: censored learning in MDPs. In: Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence. AUAI Press (2012)
Amin, K., Kearns, M., Key, P., Schwaighofer, A.: Budget optimization for sponsored search: censored learning in MDPs. CoRR (2012)
Applebaum, D.: Probability and Information: An Integrated Approach, 2nd edn. Cambridge University Press, Cambridge (2008)
Barber, D.: Bayesian Reasoning and Machine Learning. Cambridge University Press, New York (2012)
Cai, H., Ren, K., Zhag, W., Malialis, K., Wang, J.: Real-time bidding by reinforcement learning in display advertising. In: Proceedings of the 10th ACM International Conference on Web Search and Data Mining (WSDM) (2017)
Chakrabarti, D., Agarwal, D., Josifovski, V.: Contextual advertising by combining relevance with click feedback. In: Proceedings of the 17th International Conference on World Wide Web (WWW) (2008)
Chen, Y., Berkhin, P., Anderson, B., Devanur, N.R.: Real-time bidding algorithms for performance-based display ad allocation. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2011)
Cui, Y., Zhang, R., Li, W., Mao, J.: Bid landscape forecasting in online ad exchange marketplace. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2011)
Geibel, P.: Reinforcement learning for MDPs with constraints. In: European Conference on Machine Learning (2006)
Ghosh, A., Rubinstein, B.I., Vassilvitskii, S., Zinkevich, M.: Adaptive bidding for display advertising. In: Proceedings of the 18th International Conference on World Wide Web, pp. 251–260. ACM (2009)
Hoelzel, M., Ballvé, M.: The programmatic-advertising report: mobile, video, and real-time bidding drive growth in programmatic. BI Intelligence (2015)
Krishna, V.: Auction Theory. Academic Press, San Diego (2009)
Lange, S., Gabel, T., Riedmiller, M.: Batch Reinforcement Learning. Springer, Heidelberg (2012)
Liu, C.: US Ad Spending: eMarketer’s Updated Estimates and Forecast for 2015–2020. Industry report (2016)
Perlich, C., Dalessandro, B., Hook, R., Stitelman, O., Raeder, T., Provost, F.: Bid optimizing and inventory scoring in targeted online advertising. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2012)
Schwartz, E.M., Bradlow, E., Fader, P.: Customer acquisition via display advertising using multi-armed bandit experiments. Ross School of Business Paper (2015)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, vol. 1. MIT Press Cambridge, London (1998)
Xu, J., Lee, K.c., Li, W., Qi, H., Lu, Q.: Smart pacing for effective online ad campaign optimization. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015)
Zhang, W., Yuan, S., Wang, J.: Optimal real-time bidding for display advertising. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2014)
Zhang, W., Yuan, S., Wang, J.: Real-time bidding benchmarking with iPinYou dataset. CoRR (2014)
Acknowledgement
We sincerely thank Prof. Weinan Zhang and his research group from Shanghai Jiaotong University for the short visit. Manxing thanks the National Research Fund (FNR) of Luxembourg for the research support under the AFR PPP scheme and thanks Dr.Tigran Avanesov from OLAmobile for the feedback.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Du, M., Sassioui, R., Varisteas, G., State, R., Brorsson, M., Cherkaoui, O. (2017). Improving Real-Time Bidding Using a Constrained Markov Decision Process. In: Cong, G., Peng, WC., Zhang, W., Li, C., Sun, A. (eds) Advanced Data Mining and Applications. ADMA 2017. Lecture Notes in Computer Science(), vol 10604. Springer, Cham. https://doi.org/10.1007/978-3-319-69179-4_50
Download citation
DOI: https://doi.org/10.1007/978-3-319-69179-4_50
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-69178-7
Online ISBN: 978-3-319-69179-4
eBook Packages: Computer ScienceComputer Science (R0)