Skip to main content
SpringerLink
Log in

How to Maximize Clicks for Display Advertisement in Digital Marketing? A Reinforcement Learning Approach

  • Published:
Information Systems Frontiers Aims and scope Submit manuscript

Abstract

One of the core challenges in digital marketing is that the business conditions continuously change, which impacts the reception of campaigns. A winning campaign strategy can become unfavored over time, while an old strategy can gain new traction. In data driven digital marketing and web analytics, A/B testing is the prevalent method of comparing digital campaigns, choosing the winning ad, and deciding targeting strategy. A/B testing is suitable when testing variations on similar solutions and having one or more metrics that are clear indicators of success or failure. However, when faced with a complex problem or working on future topics, A/B testing fails to deliver and achieving long-term impact from experimentation is demanding and resource intensive. This study proposes a reinforcement learning based model and demonstrates its application to digital marketing campaigns. We argue and validate with actual-world data that reinforcement learning can help overcome some of the critical challenges that A/B testing, and popular Machine Learning methods currently used in digital marketing campaigns face. We demonstrate the effectiveness of the proposed technique on real actual data for a digital marketing campaign collected from a firm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. https://bair.berkeley.edu/blog/2019/12/12/mbpo/#fn:naming-conventions

  2. UCB-1 belongs to the family of “follow the perturbed leader” algorithms and has proven to retain the optimal logarithmic rate (but with suboptimal constant). A finite-time analysis of this algorithm has been given in Auer et al. (2002); Auer et al. (2002); Auer et al. (2002/03). Other types of padding functions are considered in Audibert et al. (2007).

  3. The focus of this paper is on an abruptly changing environment, but it is believed that the theoretical tools developed to handle the non-stationarity can be applied in different context.

References

  • Agrawal, R. (1995). Sample mean based index policies by O (log n ) regret for the multi-armed bandit problem. Advances in Applied Probability, 27(4), 1054–1078 {Sutton, 2018 #654}.

    Article  Google Scholar 

  • American Marketing Association (2021). Digital marketing virtual conference. https://www.ama.org/events/virtual-conference/2021-digital-marketing-virtual-conference/

  • Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017). Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6), 26–38.

    Article  Google Scholar 

  • Ascarza, E. (2021). When a/B testing Doesn’t tell you the whole story. Harvard Business Review.

    Google Scholar 

  • Aswani, R., Kar, A. K., & Ilavarasan, P. V. (2018). Detection of spammers in twitter marketing: A hybrid approach using social media analytics and bio inspired computing. Information Systems Frontiers, 20(3), 515–530.

    Article  Google Scholar 

  • Audibert, J.-Y., Munos, R., & Szepesvári, C. (2007). Tuning bandit algorithms in stochastic environments. In M. Hutter, R. A. Servedio, & E. Takimoto (Eds.), Algorithmic Learning Theory (Vol. 4754, pp. 150–165). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-75225-7_15

  • Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2), 235–256.

    Article  Google Scholar 

  • Barone, M. J., et al. (2007). Consumer response to retailer use of cause-related marketing: Is more fit better? Journal of Retailing, 83(4), 437–445.

    Article  Google Scholar 

  • Basse, G. W., & Airoldi, E. M. (2018). Limitations of design-based causal inference and a/B testing under arbitrary and network interference. Sociological Methodology, 48(1), 136–151.

    Article  Google Scholar 

  • Bennett, K. P., & Parrado-Hernández, E. (2006). The interplay of optimization and machine learning research. Journal of Machine Learning Research, 7(46), 1265–1281.

    Google Scholar 

  • Bojinov, I., Rambachan, A., & Shephard, N. (2021). Panel experiments and dynamic causal effects: A finite population perspective. Quantitative Economics, 12(4), 1171–1196.

    Article  Google Scholar 

  • Boone, D. S., & Roehm, M. (2002). Retail Segmentation Using Artificial Neural Networks. International Journal of Research in Marketing, 19(3), 287–301.

    Article  Google Scholar 

  • Botvinick, M., et al. (2019). Reinforcement Learning, Fast and Slow. Trends in Cognitive Sciences, 23(5), 408–422.

    Article  Google Scholar 

  • Chen, S.-S., Chobey, B., & Singh, V. (2021a). A neural network based Price sensitive recommender model to predict customer choices based on Price effect. Journal of Retailing and Consumer Services, 61, 102573.

    Article  Google Scholar 

  • Chen, S.-Y., He, Q.-F., & Lai, C.-F. (2021b). Deep reinforcement learning-based robot exploration for constructing map of unknown environment. Information Systems Frontiers.

  • Chintagunta, P., Hanssens, D. M., & Hauser, J. R. (2016). Marketing science and big data. Marketing Science, 35(3), 341–342.

    Article  Google Scholar 

  • Chiusano, S., Cerquitelli, T., Wrembel, R., & Quercia, D. (2021). Breakthroughs on cross-cutting data management, data analytics, and applied data science. Information Systems Frontiers, 23(1), 1–7.

    Article  Google Scholar 

  • Choi, H., Mela, C. F., Balseiro, S. R., & Leary, A. (2020). Online display advertising markets: A literature review and future directions. Information Systems Research, 31(2), 556–575.

    Article  Google Scholar 

  • Davenport, Thomas H., et al. (2011). Know what your customers want before they do. Harvard Business Review, Dec. 2011. https://hbr.org/2011/12/know-what-your-customers-want-before-they-do. Accessed 30 Apr 2021.

  • Davenport, T., Guha, A., Grewal, D., & Bresgott, T. (2020). How artificial intelligence will change the future of marketing. Journal of the Academy of Marketing Science, 48, 24–42.

    Article  Google Scholar 

  • Du, R. Y., et al. (2021). Capturing Marketing Information to Fuel Growth. Journal of Marketing, 85(1), 163–183.

    Article  Google Scholar 

  • Erevelles, S., Fukawa, N., & Swayne, L. (2016). Big data consumer analytics and the transformation of marketing. Journal of Business Research, 69(2), 897–904.

    Article  Google Scholar 

  • Fabijan, A., Dmitriev, P., McFarland, C., Vermeer, L., Holmström Olsson, H., & Bosch, J. (2018). Experimentation growth: Evolving trustworthy a/B testing capabilities in online software companies. Journal of Software: Evolution and Process, 30(12), e2113.

    Google Scholar 

  • Feit, E. M., & Berman, R. (2019). Test & roll: Profit-maximizing a/b tests. Marketing Science, 38(6), 1038–1058.

    Article  Google Scholar 

  • Fujimoto, S., Hoof, H., & Meger, D. (2018, July). Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (pp. 1587-1596). PMLR.

  • Gallo, A. (2017). A refresher on a/B testing. Harvard Business Review. https://hbr.org/2017/06/a-refresher-on-ab-testing. Accessed 30 Apr 2021.

  • Garivier, A., & Moulines, E. (2008). On upper-confidence bound policies for non-stationary bandit problems (arXiv:0805.3415). arXiv. http://arxiv.org/abs/0805.3415

  • Garivier, A., & Moulines, E. (2011). On upper confidence bound policies for switching bandit problems. In International Conference on Algorithmic Learning Theory (pp. 174–188). Springer.

    Chapter  Google Scholar 

  • Gershman, S. J., & Daw, N. D. (2017). Reinforcement learning and episodic memory in humans and animals: An integrative framework. Annual Review of Psychology, 68(1), 101–128.

    Article  Google Scholar 

  • Goldfarb, A., & Tucker, C. (2011). Online display advertising: Targeting and obtrusiveness. Marketing Science, 30(3), 389–404.

    Article  Google Scholar 

  • Gordini, N., & Veglio, V. (2017). Customers churn prediction and marketing retention strategies. An application of support vector machines based on the AUC parameter-selection technique in B2B e-commerce industry. Industrial Marketing Management, 62, 100–107.

    Article  Google Scholar 

  • Gordon, B. R., Jerath, K., Katona, Z., Narayanan, S., Shin, J., & Wilbur, K. C. (2021). Inefficiencies in digital advertising markets. Journal of Marketing, 85(1), 7–25.

    Article  Google Scholar 

  • Gupta, S., et al. (2020). Optimizing creative allocations in digital marketing. In Advances in Computing and Data Sciences (Vol. 1244, pp. 419–429). Springer Singapore.

    Chapter  Google Scholar 

  • Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018, July). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning (pp. 1861-e1870). PMLR.

  • Hanssens, D. M. (2018). The value of empirical generalizations in marketing. Journal of the Academy of Marketing Science, 46(1), 6–8.

    Article  Google Scholar 

  • Huang, M. H., & Rust, R. T. (2018). Artificial intelligence in service. Journal of Service Research, 21(2), 155–172.

    Article  Google Scholar 

  • Huang, M. H., & Rust, R. T. (2021). A strategic framework for artificial intelligence in marketing. Journal of the Academy of Marketing Science, 49(1), 30–50.

    Article  Google Scholar 

  • Huber, R., Oberländer, A. M., Faisst, U., & Röglinger, M. (2022). Disentangling capabilities for industry 4.0-an information systems capability perspective. Information Systems Frontiers, 1–29. https://doi.org/10.1007/s10796-022-10260-x

  • Iacobucci, D., et al. (2019). The state of marketing analytics in research and practice. Journal of Marketing Analytics, 7(3), 152–181.

    Article  Google Scholar 

  • Jain, D., Dash, M. K., Kumar, A., & Luthra, S. (2021). How is blockchain used in marketing: A review and research agenda. International Journal of Information Management Data Insights, 1(2), 100044.

    Article  Google Scholar 

  • Jang, B., et al. (2019). Q-Learning Algorithms: A Comprehensive Classification and Applications. IEEE Access, 7, 133653–133667.

    Article  Google Scholar 

  • Javanmard, A., & Montanari, A. (2018). Online rules for control of false discovery rate and false discovery exceedance. The Annals of Statistics, 46(2), 526–554.

    Article  Google Scholar 

  • Johnson, G. A., Lewis, R. A., & Nubbemeyer, E. I. (2017). Ghost ads: Improving the economics of measuring online ad effectiveness. Journal of Marketing Research, 54(6), 867–884.

    Article  Google Scholar 

  • Kar, A. K., & Kushwaha, A. K. (2021). Facilitators and Barriers of Artificial Intelligence Adoption in Business–Insights from Opinions Using Big Data Analytics. Information Systems Frontiers, 1-24. https://doi.org/10.1007/s10796-021-10219-4

  • Karuga, G. G., Khraban, A. M., Nair, S. K., & Rice, D. O. (2001). AdPalette: An algorithm for customizing online advertisements on the fly. Decision Support Systems, 32(2), 85–106.

    Article  Google Scholar 

  • Kim, J., Kang, S., & Lee, K. H. (2021). Evolution of digital marketing communication: Bibliometric analysis and network visualization from key articles. Journal of Business Research, 130, 552–563.

    Article  Google Scholar 

  • Kleinberg, R., Slivkins, A., & Upfal, E. (2008). Multi-armed bandits in metric spaces. In Proceedings of the fortieth annual ACM symposium on Theory of computing (pp. 681-690).

  • Kocsis, L., & Szepesvári, C. (2006). Bandit based Monte-Carlo planning. In European conference on machine learning (pp. 282–293). Springer.

    Google Scholar 

  • Kohavi, R., & Longbotham, R. (2017). Online controlled experiments and a/B testing. Encyclopedia of machine learning and data mining, 7(8), 922–929.

    Article  Google Scholar 

  • Kohavi, R., & Thomke, S. (2017). The surprising power of online experiments. Harvard Business Review, 95(5), 74–82.

    Google Scholar 

  • Kushwaha, A. K., & Kar, A. K. (2021). MarkBot–a language model-driven chatbot for interactive marketing in post-modern world. Information Systems Frontiers,1–18. https://doi.org/10.1007/s10796-021-10184-y

  • Kushwaha, A. K., Kar, A. K., & Dwivedi, Y. K. (2021). Applications of big data in emerging management disciplines: A literature review using text mining. International Journal of Information Management Data Insights, 1(2), 100017.

    Article  Google Scholar 

  • Lai, L., El Gamal, H., Jiang, H., & Poor, H. V. (2010). Cognitive medium access: Exploration, exploitation, and competition. IEEE Transactions on Mobile Computing, 10(2), 239–253.

  • Lamberton, C., & Stephen, A. T. (2016). A thematic exploration of digital, social media, and Mobile marketing: Research evolution from 2000 to 2015 and an agenda for future inquiry. Journal of Marketing, 80(6), 146–172.

    Article  Google Scholar 

  • Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., ... & Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971.

  • Liu, X., Derakhshani, M., Lambotharan, S., & Van der Schaar, M. (2020). Risk-aware multi-armed bandits with refined upper confidence bounds. IEEE Signal Processing Letters, 28, 269–273.

    Article  Google Scholar 

  • Liu, R., Gupta, S., & Patel, P. (2021). The application of the principles of responsible AI on social media marketing for digital health. Information Systems Frontiers, 1–25. https://doi.org/10.1007/s10796-021-10191-z

  • Luo, L., et al. (2013). Marketing via Social Media: A Case Study. Library Hi Tech, 31(3), 455–466.

    Article  Google Scholar 

  • Ma, L., & Sun, B. (2020). Machine learning and AI in marketing – Connecting computing power to human insights. International Journal of Research in Marketing, 37(3), 481–504.

    Article  Google Scholar 

  • Merckling, A., et al. (2022). Exploratory state representation learning. Frontiers in Robotics and AI, 9, 1–16.

    Article  Google Scholar 

  • Miklosik, A., Kuchta, M., Evans, N., & Zak, S. (2019). Towards the adoption of machine learning-based analytical tools in digital marketing. IEEE Access, 7, 85705–85718.

    Article  Google Scholar 

  • Misra, K., Schwartz, E. M., & Abernethy, J. (2019). Dynamic online pricing with incomplete information using multiarmed bandit experiments. Marketing Science, 38(2), 226–252.

    Article  Google Scholar 

  • Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., ... & Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning (pp. 1928-1937). PMLR.

  • Nadkarni, S., & Prügl, R. (2021). Digital transformation: A review, synthesis and opportunities for future research. Management Review Quarterly, 71(2), 233–341.

    Article  Google Scholar 

  • Netzer, O., Lattin, J. M., & Srinivasan, V. (2008). A hidden Markov model of customer relationship dynamics. Marketing Science, 27(2), 185–204.

    Article  Google Scholar 

  • Obal, M. W., & Lv, W. (2017). Improving banner ad strategies through predictive modeling. Journal of Research in Interactive Marketing, 11(2), 198–212.

    Article  Google Scholar 

  • Oh, Junhyuk, et al. (2017). ‘Value prediction network’. Advances in Neural Information Processing Systems, vol. 30, Curran Associates, Inc., Neural Information Processing Systems.

  • Pandey, N., Nayal, P., & Rathore, A. S. (2020). Digital marketing for B2B organizations: Structured literature review and future research directions. Journal of Business & Industrial Marketing, 35(7), 1191–1204.

    Article  Google Scholar 

  • Quach, S., Thaichon, P., Martin, K. D., Weaven, S., & Palmatier, R. W. (2022). Digital technologies: Tensions in privacy and data. Journal of the Academy of Marketing Science, 1–25.

  • Rai, A. (2020). Explainable AI: From black box to glass box. Journal of the Academy of Marketing Science, 48(1), 137–141.

    Article  Google Scholar 

  • Rathore, H., Sahay, S. K., Nikam, P., & Sewak, M. (2021). Robust android malware detection system against adversarial attacks using q-learning. Information Systems Frontiers, 23(4), 867–882.

    Article  Google Scholar 

  • Ruiz-Real, J. L., Uribe-Toril, J., Torres, J. A., & De Pablo, J. (2021). Artificial intelligence in business and economics research: Trends and future. Journal of Business Economics and Management, 22(1), 98–117.

    Article  Google Scholar 

  • Rust, R. T. (2020). The future of marketing. International Journal of Research in Marketing, 37(1), 15–26.

    Article  Google Scholar 

  • Rutz, O. J., & Watson, G. F. (2019). Endogeneity and marketing strategy research: An overview. Journal of the Academy of Marketing Science, 47, 479–498.

    Article  Google Scholar 

  • Saura, J. R. (2021). Using data sciences in digital marketing: Framework, methods, and performance metrics. Journal of Innovation & Knowledge, 6(2), 92–102.

    Article  Google Scholar 

  • Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61, 85–117.

    Article  Google Scholar 

  • Schwartz, E. M., Bradlow, E. T., & Fader, P. S. (2017). Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science, 36(4), 500–522.

    Article  Google Scholar 

  • Senz, K. (2021). Is A/B testing effective? Evidence from 35,000 startups. Retrieved November 25, 2021, from https://hbswk.hbs.edu/item/is-ab-testing-effective-evidence-from-35000-startups. Accessed 30 Apr 2021.

  • Singh, V., Chen, S. S., Singhania, M., Nanavati, B., & Gupta, A. (2022). How are reinforcement learning and deep learning algorithms used for big data based decision making in financial industries–A review and research agenda. International Journal of Information Management Data Insights, 2(2), 100094

  • Sridhar, S., & Fang, E. (2019). New vistas for marketing strategy: Digital, data-rich, and developing market (D3) environments. Journal of the Academy of Marketing Science, 47, 977–985.

    Article  Google Scholar 

  • Stourm, V., & Bax, E. (2017). Incorporating hidden costs of annoying ads in display auctions. International Journal of Research in Marketing, 34(3), 622–640.

    Article  Google Scholar 

  • Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (Second ed.). The MIT Press.

    Google Scholar 

  • Verma, S., Sharma, R., Deb, S., & Maitra, D. (2021). Artificial intelligence in marketing: Systematic review and future research direction. International Journal of Information Management Data Insights, 1(1), 100002.

    Article  Google Scholar 

  • Votto, A. M., Valecha, R., Najafirad, P., & Rao, H. R. (2021). Artificial intelligence in tactical human resource management: A systematic literature review. International Journal of Information Management Data Insights, 1(2), 100047.

    Article  Google Scholar 

  • Wang, H., & Hong, M. (2019). Online ad effectiveness evaluation with a two-stage method using a Gaussian filter and decision tree approach. Electronic Commerce Research and Applications, 35, 100852.

    Article  Google Scholar 

  • Youngmann, B., Yom-Tov, E., Gilad-Bachrach, R., & Karmon, D. (2021). Algorithmic copywriting: Automated generation of health-related advertisements to improve their performance. Information Retrieval Journal, 24(3), 205–239.

    Article  Google Scholar 

  • Zhang, Y., et al. (2022). A deep reinforcement learning based hyper-heuristic for combinatorial optimisation with uncertainties. European Journal of Operational Research, 300(2), 418–427.

    Article  Google Scholar 

Download references

Acknowledgements

The authors gratefully acknowledge the team at YourFirstad for their insights and sharing of data for our study. This research paper is submitted as part of the efforts undertaken under a collaboration between the research teams of Indian Institute of Technology Delhi and BASF Germany, funded by BASF Germany.

Author information

Authors and Affiliations

  1. BASF SE, Pfalzgrafenstraße 1, 67061, Ludwigshafen am Rhein, Germany

    Vinay Singh

  2. University of Siegen, Adolf-Reichwein-Straße 2, 57076, Siegen, Germany

    Vinay Singh

  3. Customer Credit Risk Management, BASF Services Europe GmbH, GBE/SO-AET-OBC3, Rotherstrasse 11, 10245, Berlin, Germany

    Brijesh Nanavati

  4. Department of Management Studies and School of Artificial Intelligence, Indian Institute of Technology Delhi, New Delhi, 110016, India

    Arpan Kumar Kar & Agam Gupta

Authors
  1. Vinay Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brijesh Nanavati
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arpan Kumar Kar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Agam Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arpan Kumar Kar.

Ethics declarations

Conflict of Interest

The authors report no conflict of interest in the study undertaken. All contributors have been listed as authors and all authors have contributed.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Singh, V., Nanavati, B., Kar, A.K. et al. How to Maximize Clicks for Display Advertisement in Digital Marketing? A Reinforcement Learning Approach. Inf Syst Front 25, 1621–1638 (2023). https://doi.org/10.1007/s10796-022-10314-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10796-022-10314-0

Keywords

Search

Navigation