Skip to main content
SpringerLink
Log in

An Advert Creation System for 3D Product Placements

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases: Applied Data Science Track (ECML PKDD 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12460))

Abstract

Over the past decade, the evolution of video-sharing platforms has attracted a significant amount of investments on contextual advertising. The common contextual advertising platforms utilize the information provided by users to integrate 2D visual ads into videos. The existing platforms face many technical challenges such as ad integration with respect to occluding objects and 3D ad placement. This paper presents a Video Advertisement Placement & Integration (Adverts) framework, which is capable of perceiving the 3D geometry of the scene and camera motion to blend 3D virtual objects in videos and create the illusion of reality. The proposed framework contains several modules such as monocular depth estimation, object segmentation, background-foreground separation, alpha matting and camera tracking. Our experiments conducted using Adverts framework indicates the significant potential of this system in contextual ad integration, and pushing the limits of advertising industry using mixed reality technologies.

I. Bacher, H. Javidnia and S. Dev—Authors contributed equally.

The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Aurelia, Three.js, Async, Bootstrap, Fontawesome, Whammy, Broadcast ChannelAPI, Canvas API, Fetch API, Storage API.

References

  1. China: short video market revenue 2016–2021. Accessed 15 Mar 2020. https://www.statista.com/statistics/874562/china-short-video-market-size/

  2. Agarwal, S., Snavely, N., Seitz, S.M., Szeliski, R.: Bundle adjustment in the large. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6312, pp. 29–42. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15552-9_3

    Chapter  Google Scholar 

  3. Aksoy, Y., Oh, T.H., Paris, S., Pollefeys, M., Matusik, W.: Semantic soft segmentation. ACM Trans. Graph. (TOG) 37(4), 1–13 (2018)

    Article  Google Scholar 

  4. Basha, T., Avidan, S., Hornung, A., Matusik, W.: Structure and motion from scene registration. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1426–1433. IEEE (2012)

    Google Scholar 

  5. Bazrafkan, S., Javidnia, H., Lemley, J., Corcoran, P.: Semiparallel deep neural network hybrid architecture: first application on depth from monocular camera. J. Electron. Imaging 27(4), 043041 (2018)

    Article  Google Scholar 

  6. Caelles, S., et al.: The 2018 Davis challenge on video object segmentation. arXiv preprint arXiv:1803.00557 (2018)

  7. Chen, Y., Pont-Tuset, J., Montes, A., Van Gool, L.: Blazingly fast video object segmentation with pixel-wise metric learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1189–1198 (2018)

    Google Scholar 

  8. Covell, M., Baluja, S., Fink, M.: Advertisement detection and replacement using acoustic and visual repetition. In: IEEE Workshop on Multimedia Signal Processing, pp. 461–466. IEEE (2006)

    Google Scholar 

  9. Dai, Y., Li, H., He, M.: Projective multiview structure and motion from element-wise factorization. IEEE Trans. Pattern Anal. Mach. Intell. 35(9), 2238–2251 (2013)

    Article  Google Scholar 

  10. Dev, S., et al.: The ALOS dataset for advert localization in outdoor scenes. In: 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–3. IEEE (2019)

    Google Scholar 

  11. Dev, S., et al.: The CASE dataset of candidate spaces for advert implantation. In: 2019 16th International Conference on Machine Vision Applications (MVA), pp. 1–4. IEEE (2019)

    Google Scholar 

  12. Dev, S., et al.: Localizing adverts in outdoor scenes. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 591–594. IEEE (2019)

    Google Scholar 

  13. Fu, H., Gong, M., Wang, C., Batmanghelich, K., Tao, D.: Deep ordinal regression network for monocular depth estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2002–2011 (2018)

    Google Scholar 

  14. Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 270–279 (2017)

    Google Scholar 

  15. Hossari, M., et al.: ADNet: a deep network for detecting adverts. arXiv preprint arXiv:1811.04115 (2018)

  16. Hu, J., Ozay, M., Zhang, Y., Okatani, T.: Revisiting single image depth estimation: toward higher resolution maps with accurate object boundaries. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1043–1051. IEEE (2019)

    Google Scholar 

  17. Hussain, Z., et al.: Automatic understanding of image and video advertisements. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1705–1715 (2017)

    Google Scholar 

  18. Jain, S.D., Xiong, B., Grauman, K.: FusionSeg: learning to combine motion and appearance for fully automatic segmentation of generic objects in videos. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2117–2126. IEEE (2017)

    Google Scholar 

  19. Jang, W.D., Kim, C.S.: Interactive image segmentation via backpropagating refinement scheme. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5297–5306 (2019)

    Google Scholar 

  20. Javidnia, H., Corcoran, P.: Accurate depth map estimation from small motions. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 2453–2461 (2017)

    Google Scholar 

  21. Javidnia, H., Pitié, F.: Background matting. arXiv preprint arXiv:2002.04433 (2020)

  22. Kuznietsov, Y., Stuckler, J., Leibe, B.: Semi-supervised deep learning for monocular depth map prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6647–6655 (2017)

    Google Scholar 

  23. Lasinger, K., Ranftl, R., Schindler, K., Koltun, V.: Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. arXiv preprint arXiv:1907.01341 (2019)

  24. Li, Z., Snavely, N.: MegaDepth: learning single-view depth prediction from internet photos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2041–2050 (2018)

    Google Scholar 

  25. Lim, L.A., Keles, H.Y.: Learning multi-scale features for foreground segmentation. Pattern Anal. Appl. 23, 1369–1380 (2018)

    Google Scholar 

  26. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1150–1157. IEEE (1999)

    Google Scholar 

  27. Maninis, K.K.: Video object segmentation without temporal information. IEEE Trans. Pattern Anal. Mach. Intell. 41(6), 1515–1530 (2018)

    Article  Google Scholar 

  28. Nautiyal, A., et al.: An advert creation system for next-gen publicity. In: Brefeld, U., et al. (eds.) ECML PKDD 2018. LNCS (LNAI), vol. 11053, pp. 663–667. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10997-4_47

    Chapter  Google Scholar 

  29. Oh, S.W., Lee, J.Y., Xu, N., Kim, S.J.: Fast user-guided video object segmentation by interaction-and-propagation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5247–5256 (2019)

    Google Scholar 

  30. Papazoglou, A., Ferrari, V.: Fast object segmentation in unconstrained video. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1777–1784 (2013)

    Google Scholar 

  31. Scharstein, D., Pal, C.: Learning conditional random fields for stereo. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2007)

    Google Scholar 

  32. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47(1–3), 7–42 (2002). https://doi.org/10.1023/A:1014573219977

    Article  MATH  Google Scholar 

  33. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104–4113 (2016)

    Google Scholar 

  34. Tosi, F., Aleotti, F., Poggi, M., Mattoccia, S.: Learning monocular depth estimation infusing traditional stereo knowledge. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9799–9809 (2019)

    Google Scholar 

  35. Wang, Q., Zhang, L., Bertinetto, L., Hu, W., Torr, P.H.: Fast online object tracking and segmentation: a unifying approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1328–1338 (2019)

    Google Scholar 

  36. Wug Oh, S., Lee, J.Y., Sunkavalli, K., Joo Kim, S.: Fast video object segmentation by reference-guided mask propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7376–7385 (2018)

    Google Scholar 

  37. Xu, D., Wang, W., Tang, H., Liu, H., Sebe, N., Ricci, E.: Structured attention guided convolutional neural fields for monocular depth estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3917–3925 (2018)

    Google Scholar 

  38. Yu, F., Gallup, D.: 3D reconstruction from accidental motion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3986–3993 (2014)

    Google Scholar 

  39. Chuang, Y.-Y., Curless, B., Salesin, D.H., Szeliski, R.: A Bayesian approach to digital matting. In: 2001 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 264–271 (2001). https://doi.org/10.1109/CVPR.2001.990970

Download references

Author information

Authors and Affiliations

  1. ADAPT SFI Research Centre, Trinity College Dublin, Dublin, Ireland

    Ivan Bacher, Hossein Javidnia, Soumyabrata Dev, Rahul Agrahari, Murhaf Hossari, Matthew Nicholson, Clare Conran & François Pitié

  2. School of Computer Science, University College Dublin, Dublin, Ireland

    Soumyabrata Dev

  3. Huawei Ireland Research Center, Dublin, Ireland

    Jian Tang, Peng Song & David Corrigan

  4. Department of Electronic and Electrical Engineering, Trinity College Dublin, Dublin, Ireland

    François Pitié

Authors
  1. Ivan Bacher
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hossein Javidnia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Soumyabrata Dev
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rahul Agrahari
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Murhaf Hossari
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Matthew Nicholson
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Clare Conran
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jian Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Peng Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. David Corrigan
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. François Pitié
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ivan Bacher .

Editor information

Editors and Affiliations

  1. Microsoft Research, Redmond, WA, USA

    Yuxiao Dong

  2. Jožef Stefan Institute, Ljubljana, Slovenia

    Dunja Mladenić

  3. Amazon Alexa Knowledge, Cambridge, UK

    Craig Saunders

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bacher, I. et al. (2021). An Advert Creation System for 3D Product Placements. In: Dong, Y., Mladenić, D., Saunders, C. (eds) Machine Learning and Knowledge Discovery in Databases: Applied Data Science Track. ECML PKDD 2020. Lecture Notes in Computer Science(), vol 12460. Springer, Cham. https://doi.org/10.1007/978-3-030-67667-4_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-67667-4_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-67666-7

  • Online ISBN: 978-3-030-67667-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation