Advertisement

Improving Real-Time Bidding Using a Constrained Markov Decision Process

  • Manxing DuEmail author
  • Redouane Sassioui
  • Georgios Varisteas
  • Radu State
  • Mats Brorsson
  • Omar Cherkaoui
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10604)

Abstract

Online advertising is increasingly switching to real-time bidding on advertisement inventory, in which the ad slots are sold through real-time auctions upon users visiting websites or using mobile apps. To compete with unknown bidders in such a highly stochastic environment, each bidder is required to estimate the value of each impression and to set a competitive bid price. Previous bidding algorithms have done so without considering the constraint of budget limits, which we address in this paper. We model the bidding process as a Constrained Markov Decision Process based reinforcement learning framework. Our model uses the predicted click-through-rate as the state, bid price as the action, and ad clicks as the reward. We propose a bidding function, which outperforms the state-of-the-art bidding functions in terms of the number of clicks when the budget limit is low. We further simulate different bidding functions competing in the same environment and report the performances of the bidding strategies when required to adapt to a dynamic environment.

Keywords

Display Advertising Real-time bidding Markov Decision Process Reinforcement Learning 

Notes

Acknowledgement

We sincerely thank Prof. Weinan Zhang and his research group from Shanghai Jiaotong University for the short visit. Manxing thanks the National Research Fund (FNR) of Luxembourg for the research support under the AFR PPP scheme and thanks Dr.Tigran Avanesov from OLAmobile for the feedback.

References

  1. 1.
    Aggarwal, C.C.: Data Mining: The Textbook. Springer, Cham (2015)Google Scholar
  2. 2.
    Altman, E.: Constrained Markov Decision Processes. CRC Press, Boca Raton (1999)Google Scholar
  3. 3.
    Amin, K., Kearns, M., Key, P., Schwaighofer, A.: Budget optimization for sponsored search: censored learning in MDPs. In: Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence. AUAI Press (2012)Google Scholar
  4. 4.
    Amin, K., Kearns, M., Key, P., Schwaighofer, A.: Budget optimization for sponsored search: censored learning in MDPs. CoRR (2012)Google Scholar
  5. 5.
    Applebaum, D.: Probability and Information: An Integrated Approach, 2nd edn. Cambridge University Press, Cambridge (2008)Google Scholar
  6. 6.
    Barber, D.: Bayesian Reasoning and Machine Learning. Cambridge University Press, New York (2012)Google Scholar
  7. 7.
    Cai, H., Ren, K., Zhag, W., Malialis, K., Wang, J.: Real-time bidding by reinforcement learning in display advertising. In: Proceedings of the 10th ACM International Conference on Web Search and Data Mining (WSDM) (2017)Google Scholar
  8. 8.
    Chakrabarti, D., Agarwal, D., Josifovski, V.: Contextual advertising by combining relevance with click feedback. In: Proceedings of the 17th International Conference on World Wide Web (WWW) (2008)Google Scholar
  9. 9.
    Chen, Y., Berkhin, P., Anderson, B., Devanur, N.R.: Real-time bidding algorithms for performance-based display ad allocation. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2011)Google Scholar
  10. 10.
    Cui, Y., Zhang, R., Li, W., Mao, J.: Bid landscape forecasting in online ad exchange marketplace. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2011)Google Scholar
  11. 11.
    Geibel, P.: Reinforcement learning for MDPs with constraints. In: European Conference on Machine Learning (2006)Google Scholar
  12. 12.
    Ghosh, A., Rubinstein, B.I., Vassilvitskii, S., Zinkevich, M.: Adaptive bidding for display advertising. In: Proceedings of the 18th International Conference on World Wide Web, pp. 251–260. ACM (2009)Google Scholar
  13. 13.
    Hoelzel, M., Ballvé, M.: The programmatic-advertising report: mobile, video, and real-time bidding drive growth in programmatic. BI Intelligence (2015)Google Scholar
  14. 14.
    Krishna, V.: Auction Theory. Academic Press, San Diego (2009)Google Scholar
  15. 15.
    Lange, S., Gabel, T., Riedmiller, M.: Batch Reinforcement Learning. Springer, Heidelberg (2012)Google Scholar
  16. 16.
    Liu, C.: US Ad Spending: eMarketer’s Updated Estimates and Forecast for 2015–2020. Industry report (2016)Google Scholar
  17. 17.
    Perlich, C., Dalessandro, B., Hook, R., Stitelman, O., Raeder, T., Provost, F.: Bid optimizing and inventory scoring in targeted online advertising. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2012)Google Scholar
  18. 18.
    Schwartz, E.M., Bradlow, E., Fader, P.: Customer acquisition via display advertising using multi-armed bandit experiments. Ross School of Business Paper (2015)Google Scholar
  19. 19.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, vol. 1. MIT Press Cambridge, London (1998)Google Scholar
  20. 20.
    Xu, J., Lee, K.c., Li, W., Qi, H., Lu, Q.: Smart pacing for effective online ad campaign optimization. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015)Google Scholar
  21. 21.
    Zhang, W., Yuan, S., Wang, J.: Optimal real-time bidding for display advertising. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2014)Google Scholar
  22. 22.
    Zhang, W., Yuan, S., Wang, J.: Real-time bidding benchmarking with iPinYou dataset. CoRR (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Manxing Du
    • 1
    Email author
  • Redouane Sassioui
    • 1
  • Georgios Varisteas
    • 1
  • Radu State
    • 1
  • Mats Brorsson
    • 2
  • Omar Cherkaoui
    • 3
  1. 1.University of LuxembourgLuxembourg CityLuxembourg
  2. 2.Royal Institute of Technology (KTH)StockholmSweden
  3. 3.University of Quebec in MontrealMontrealCanada

Personalised recommendations