Abstract
High turnover of online advertising and especially real time bidding makes this ad market very attractive to beneficiary stakeholders. For publishers, it is as easy as placing some slots in their webpages and sell these slots in the available online auctions. It is important to determine which online auction market to send their slots to. Based on the traditional Waterfall Strategy, publishers have a fixed ordering of preferred online auction markets, and sell the ad slots by trying these markets sequentially. This fixed-order strategy replies heavily on the experience of publishers, and often it does not provide highest revenue. In this paper, we propose a method for dynamically deciding on the ordering of auction markets for each available ad slot. This method is based on reinforcement learning (RL) and learns the state-action through a tabular method. Since the state-action space is sparse, a prediction model is used to solve this sparsity. We analyze a real-time bidding dataset, and then show that the proposed RL method on this dataset leads to higher revenues. In addition, a sensitivity analysis is performed on the parameters of the method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This is the case for online publishers who rely on SSPs to sell their impressions.
References
Afshar, R.R., Zhang, Y., Firat, M., Kaymak, U.: A decision support method to increase the revenue of ad publishers in waterfall strategy. In: IEEE Conference on Computational Intelligence for Financial Engineering and Economics (CIFEr) (2019)
Afshar, R.R., Zhang, Y., Firat, M., Kaymak, U.: A reinforcement learning method to select ad networks in waterfall strategy. In: Proceedings of the 11th International Conference on Agents and Artificial Intelligence, vol. 2, pp. 256–265. SCITEPRESS-Science and Technology Publications (2019)
Austin, D., Seljan, S., Monello, J., Tzeng, S.: Reserve price optimization at scale. In: 2016 IEEE 3rd International Conference on Data Science and Advanced Analytics (DSAA), pp. 528–536. IEEE (2016)
Balseiro, S.R., Feldman, J., Mirrokni, V., Muthukrishnan, S.: Yield optimization of display advertising with ad exchange. Manag. Sci. 60(12), 2886–2907 (2014)
Busch, O.: The programmatic advertising principle. In: Busch, O. (ed.) Programmatic Advertising. MP, pp. 3–15. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-25023-6_1
Cai, H., et al.: Real-time bidding by reinforcement learning in display advertising. In: Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pp. 661–670. ACM (2017)
Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)
Dinodia, P.: Header bidding vs waterfall: how the two revenue optimisation hacks differ (2017). https://www.adpushup.com/blog/header-bidding-vs-waterfall/
Ghosh, A., McAfee, P., Papineni, K., Vassilvitskii, S.: Bidding for representative allocations for display advertising. In: Leonardi, S. (ed.) WINE 2009. LNCS, vol. 5929, pp. 208–219. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-10841-9_20
Japkowicz, N., et al.: Learning from imbalanced data sets: a comparison of various strategies. In: AAAI Workshop on Learning from Imbalanced Data Sets, Menlo Park, CA, vol. 68, pp. 10–15 (2000)
Jones, M.J., Viola, P.: Robust real-time object detection. In: Workshop on Statistical and Computational Theories of Vision, vol. 266, p. 56 (2001)
Li, J., Ni, X., Yuan, Y.: The reserve price of ad impressions in multi-channel real-time bidding markets. IEEE Trans. Comput. Soc. Syst. 5(2), 583–592 (2018)
Muthukrishnan, S.: Ad exchanges: research issues. In: Leonardi, S. (ed.) WINE 2009. LNCS, vol. 5929, pp. 1–12. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-10841-9_1
Quinlan, J.R.: Induction of decision trees. Mach. Learn. 1(1), 81–106 (1986)
Rhuggenaath, J., Akcay, A., Zhang, Y., Kaymak, U.: Fuzzy logic based pricing combined with adaptive search for reserve price optimization in online ad auctions. In: IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) (2019)
Rhuggenaath, J., Akcay, A., Zhang, Y., Kaymak, U.: Optimizing reserve prices for publishers in online ad auctions. In: IEEE Conference on Computational Intelligence for Financial Engineering and Economics (CIFEr) (2019)
Rhuggenaath, J., Akcay, A., Zhang, Y., Kaymak, U.: A PSO-based algorithm for reserve price optimization in online ad auctions. In: 2019 IEEE Congress on Evolutionary Computation (CEC) (2019)
Rhuggenaath, J., Zhang, Y., Akcay, A., Kaymak, U.: Optimal display-ad allocation with guaranteed contracts and supply side platforms, working paper (2018)
Sutton, R.S., Barto, A.G., et al.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Vengerov, D.: A gradient-based reinforcement learning approach to dynamic pricing in partially-observable environments. Future Gene. Comput. Syst. 24(7), 687–693 (2008)
Wang, J., Zhang, W., Yuan, S., et al.: Display advertising with real-time bidding (RTB) and behavioural targeting. Found. Trends® Inf. Retrieval 11(4–5), 297–435 (2017)
Wu, D., et al.: Budget constrained bidding by model-free reinforcement learning in display advertising. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 1443–1451. ACM (2018)
Wu, W.C.H., Yeh, M.Y., Chen, M.S.: Predicting winning price in real time bidding with censored data. In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1305–1314. ACM (2015)
Xie, Z., Lee, K.C., Wang, L.: Optimal reserve price for online ads trading based on inventory identification. In: Proceedings of the ADKDD 2017, p. 6. ACM (2017)
Yuan, S., Wang, J., Chen, B., Mason, P., Seljan, S.: An empirical study of reserve price optimisation in real-time bidding. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1897–1906. ACM (2014)
Zawadziński, M.: What’s the difference between waterfall auctions & header bidding? (2016). http://bit.ly/2KpT5Po
Zhang, W., Yuan, S., Wang, J.: Optimal real-time bidding for display advertising. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1077–1086. ACM (2014)
Acknowledgment
This work was supported by EU EUROSTARS (Project E! 11582).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Refaei Afshar, R., Zhang, Y., Firat, M., Kaymak, U. (2019). Reinforcement Learning Method for Ad Networks Ordering in Real-Time Bidding. In: van den Herik, J., Rocha, A., Steels, L. (eds) Agents and Artificial Intelligence. ICAART 2019. Lecture Notes in Computer Science(), vol 11978. Springer, Cham. https://doi.org/10.1007/978-3-030-37494-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-37494-5_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37493-8
Online ISBN: 978-3-030-37494-5
eBook Packages: Computer ScienceComputer Science (R0)