Abstract
Immersive 3D media is an emerging type of media that captures, encodes and reconstructs the 3D appearance of people and objects, with applications in tele-presence, teleconference, entertainment, gaming and other fields. In this paper, we discuss a novel concept of live 3D immersive media streaming in a serverless setting. In particular, we present a novel network-centric adaptive streaming framework which deviates from the traditional client-based adaptive streaming used in 2D video. In our framework the decisions for the production of the transcoding profiles are taken in a centralized manner, by considering consumer metrics vs provisioning costs and inferring the expected consumer quality of experience and behavior based on them. In addition, we demonstrate that a naive application of the serverless paradigm might be sub-optimal under some common immersive 3D media scenarios.
Similar content being viewed by others
Notes
A reference implementation of our extended FaaS framework and its integration with MANO, can be found in https://github.com/5g-media/faas-vim-plugin.
In Apache OpenWhisk parlance, functions are termed actions. We will use the terms interchangeably, wherever this does not cause ambiguity
References
Alexiadis D, Chatzitofis A, Zioulis N, Zoidi O, Louizis G, Zarpalas D, Daras P (2017) An integrated platform for live 3D human reconstruction and motion capturing. IEEE Trans Circ Syst Video Technol 27(4):798–813. https://doi.org/10.1109/TCSVT.2016.2576922
Alvarez F, Breitgand D, Griffin D, Andriani P, Rizou S, Zioulis N, Moscatelli F, Serrano J, Keltsch M, Trakadas P et al (2019) An edge-to-cloud virtualized multimedia service platform for 5g networks. IEEE Trans Broadcast 65(2):369–380
Ao L, Izhikevich L, Voelker GM, Porter G (2018) Sprocket: a serverless video processing framework. In: Proceedings of the ACM symposium on cloud computing (SoCC ’18). https://doi.org/10.1145/3267809.3267815, http://par.nsf.gov/biblio/10098946
Apache OpenWhisk: Open Source Serverless Cloud Platform. https://openwhisk.apache.org/
Argo Events Team: Argo Events - The Event-driven Workflow Automation Framework. https://github.com/argoproj/argo-events
Argo Team: Argo Workflows. https://github.com/argoproj/argo
Athanasoulis P, Christakis E, Konstantoudakis K, Drakoulis P, Rizou S, Weit A, Doumanoglou A, Zioulis N, Zarpalas D (2020) Optimizing qoe and cost in a 3d immersive media platform: a reinforcement learning approach. In: MMEDIA 2020: the twelfth international conference on advances in multimedia. IARIA
Ballard T, Griwodz C, Steinmetz R, Rizk A Rats: adaptive 360-degree live streaming. In: Proceedings of the 10th ACM multimedia systems conference, MMSys ’19, p 308–311. Association for Computing Machinery (2019). https://doi.org/10.1145/3304109.3323837. Event-place: Amherst, Massachusetts
Barman N, Martini MG (2019) QoE modeling for HTTP adaptive video streaming–a survey and open challenges. IEEE Access 7:30831–30859. https://doi.org/10.1109/ACCESS.2019.2901778. Conference Name: IEEE Access
Bentaleb A, Taani B, Begen AC, Timmerer C, Zimmermann R (2018) A survey on bitrate adaptation schemes for streaming media over http. IEEE Commun Sur Tutor 21(1):562–585
Bentaleb A, Taani B, Begen AC, Timmerer C, Zimmermann R (2019) A survey on bitrate adaptation schemes for streaming media over HTTP. IEEE Commun Surv Tutor 21(1):562–585. https://doi.org/10.1109/COMST.2018.2862938. Conference Name: IEEE Communications Surveys Tutorials
Bhargava A, Martin J, Babu SV Comparative evaluation of user perceived quality assessment of design strategies for HTTP-based adaptive streaming (2019). https://doi.org/10.1145/3345313
Breitgand D, Weit A (2019) Using gpus with apache openwhisk. https://medium.com/openwhisk/using-gpus-with-apache-openwhisk-c6773efcccfb
Chakareski J, Aksu R, Corbillon X, Simon G, Swaminathan V (2018) Viewport-driven rate-distortion optimized 360∘ video streaming. In: 2018 IEEE International conference on communications (ICC), pp 1–7. https://doi.org/10.1109/ICC.2018.8422859
Chen Y, Zhang F, Wu K, Zhang Q (2015) Qoe-aware dynamic video rate adaptation. In: 2015 IEEE Global communications conference (GLOBECOM). IEEE, pp 1–6
Chou PA, Koroteev M, Krivokuća M (2020) A volumetric approach to point cloud compression—part i: attribute compression. IEEE Trans Image Process 29:2203–2216. https://doi.org/10.1109/TIP.2019.2908095
Christaki K, Apostolakis KC, Doumanoglou A, Zioulis N, Zarpalas D, Daras P (2019) Space wars: an augmentedvr game. In: International conference on multimedia modeling. Springer, pp 566–570
Collet A, Chuang M, Sweeney P, Gillett D, Evseev D, Calabrese D, Hoppe H, Kirk A, Sullivan S High-quality streamable free-viewpoint video (2015). https://doi.org/10.1145/2766945
Crowle S, Doumanoglou A, Poussard B, Boniface M, Zarpalas D, Daras P (2015) Dynamic adaptive mesh streaming for real-time 3d teleimmersion. In: Proceedings of the 20th international conference on 3D web technology, Web3D ’15. https://doi.org/10.1145/2775292.2775296. Event-place: Heraklion, Crete, Greece. Association for Computing Machinery, pp 269–277
Cui L, Mekuria R, Preda M, Jang E (2019) Point-cloud compression: moving picture experts group’s new standard in 2020. IEEE Consum Electron Mag 8(4):17–21. https://doi.org/10.1109/MCE.2019.2905483
Ding AY, Janssen M (2018) 5g applications: requirements, challenges, and outlook. arXiv:1810.06057
Domański M, Stankiewicz O, Wegner K, Grajek T (2017) Immersive visual media — mpeg-i: 360 video, virtual navigation and beyond. In: 2017 International conference on systems, signals and image processing (IWSSIP), pp 1–9
Dou M, Khamis S, Degtyarev Y, Davidson P, Fanello SR, Kowdle A, Escolano SO, Rhemann C, Kim D, Taylor J, Kohli P, Tankovich V, Izadi S (2016) Fusion4D: real-time performance capture of challenging scenes. https://doi.org/10.1145/2897824.2925969
Doumanoglou A, Alexiadis DS, Zarpalas D, Daras P (2014) Toward real-time and efficient compression of human time-varying meshes. IEEE Trans Circ Syst Video Technol 24(12):2099–2116. https://doi.org/10.1109/TCSVT.2014.2319631
Doumanoglou A, Drakoulis P, Zioulis N, Zarpalas D, Daras P (2019) Benchmarking open-source static 3d mesh codecs for immersive media interactive live streaming. IEEE J Emerg Selected Topics Circ Syst 9(1):190–203. https://doi.org/10.1109/JETCAS.2019.2898768
Doumanoglou A, Griffin D, Serrano J, Zioulis N, Phan TK, Jiménez D, Zarpalas D, Alvarez F, Rio M, Daras P (2018) Quality of experience for 3-D immersive media streaming. IEEE Trans Broadcast 64(2):379–391. https://doi.org/10.1109/TBC.2018.2823909. Conference Name: IEEE Transactions on Broadcasting
Doumanoglou A, Zioulis N, Griffin D, Serrano J, Phan TK, Jiménez D, Zarpalas D, Alvarez F, Rio M, Daras P (2018) A system architecture for live immersive 3d-media transcoding over 5g networks. In: 2018 IEEE International symposium on broadband multimedia systems and broadcasting (BMSB). IEEE, pp 11–15
El Marai O, Taleb T, Menacer M, Koudil M (2018) On improving video streaming efficiency, fairness, stability, and convergence time through client–server cooperation. IEEE Trans Broadcast 64(1):11–25. https://doi.org/10.1109/TBC.2017.2781146
ETSI: Multi-access Edge Computing (MEC); Framework and Reference Architecture; GS MEC 003 V2.1.1 (2019-01) (2019). https://www.etsi.org/deliver/etsi_gs/MEC/001_099/003/02.01.01_60/gs_MEC003v020101p.pdf
Fan CL, Lo WC, Pai YT, Hsu CH (2019) A survey on 360∘ video streaming: acquisition, transmission, and display. ACM Comput Surv, 52(4). https://doi.org/10.1145/3329119
Flannel, Kubernetes Networking. https://github.com/coreos/flannel#flannel
Girinathan J, Breckinridge R (2018) Simple serverless video on demand (vod) workflow. https://aws.amazon.com/blogs/networking-and-content-delivery/serverless-video-on-demand-vod-workflow/
Graf M, Timmerer C, Mueller C (2017) Towards bandwidth efficient adaptive streaming of omnidirectional video over HTTP: design, implementation, and evaluation. In: Proceedings of the 8th ACM on multimedia systems conference - MMSys’17. https://doi.org/10.1145/3083187.3084016. ACM Press, Taipei, pp 261–271
Hannuksela MM, Wang YK, Hourunranta A An overview of the OMAF standard for 360∘ video. In: 2019 Data compression conference (DCC), pp 418–427 (2019). https://doi.org/10.1109/DCC.2019.00050. ISSN: 2375-0359
He J, Qureshi M, Qiu L, Li J, Li F, Han L (2018) Rubiks: practical 360-degree streaming for smartphones, p 482–494. https://doi.org/10.1145/3210240.3210323
Hoßfeld T, Seufert M, Sieber C, Zinner T, Tran-Gia P (2015) Identifying QoE optimal adaptation of HTTP adaptive streaming based on subjective studies. Comput Netw 81:320–332. https://doi.org/10.1016/j.comnet.2015.02.015, http://www.sciencedirect.com/science/article/pii/S1389128615000626
Hosseini M (2019) Adaptive rate allocation for view-aware point-cloud streaming. arXiv:1911.008121911.00812
Hosseini M, Swaminathan V (2016) Adaptive 360 vr video streaming: divide and conquer. In: 2016 IEEE International symposium on multimedia (ISM). https://doi.org/10.1109/ISM.2016.0028, pp 107–110
IBM Cloud Functions. https://cloud.ibm.com/functions/
ISO/IEC: Information technology — Dynamic adaptive streaming over HTTP (DASH) — Part 5: Server and network assisted DASH (SAND). https://www.iso.org/standard/69079.html
Jain R (1990) The art of computer systems performance analysis: techniques for experimental design, measurement, simulation, and modeling. Wiley. https://books.google.co.il/books?id=eOR0kJjgMqkC
Karakottas A, Papachristou A, Doumanoqlou A, Zioulis N, Zarpalas D, Daras P (2018) Augmented vr. In: 2018 IEEE Conference on virtual reality and 3d user interfaces (VR), pp 1–1
Kazhdan M (2005) Reconstruction of solid models from oriented point sets. In: Proceedings of the third eurographics symposium on geometry processing. Eurographics Association, p 73
Kazhdan M, Bolitho M, Hoppe H (2006) Poisson surface reconstruction. In: Proceedings of the fourth eurographics symposium on geometry processing, vol 7
Kleinrock L (1976) Queueing systems. Wiley
Kritikos K, Skrzypek P (2018) A review of serverless frameworks. In: 2018 IEEE/ACM international conference on utility and cloud computing companion (UCC companion). IEEE, pp 161–168
Krivokuća M, Chou PA, Koroteev M (2020) A volumetric approach to point cloud compression–part ii: geometry compression. IEEE Trans Image Process 29:2217–2229. https://doi.org/10.1109/TIP.2019.2957853
Kubernetes: Production-grade container orchestrator. https://kubernetes.io/
Li Z, Zhu X, Gahm J, Pan R, Hu H, Begen AC, Oran D (2014) Probe and adapt: rate adaptation for HTTP video streaming at scale. IEEE J Selected Areas Commun 32(4):719–733. https://doi.org/10.1109/JSAC.2014.140405. ArXiv:1305.0510
Maglo A, Lavoué G., Dupont F, Hudelot C (2015) 3d mesh compression: survey, comparisons, and emerging trends. ACM Comput Surv 47(3). https://doi.org/10.1145/2693443
Mao H, Netravali R, Alizadeh M (2017) Neural adaptive video streaming with pensieve. In: Proceedings of the conference of the ACM special interest group on data communication, SIGCOMM ’17. https://doi.org/10.1145/3098822.3098843. Association for Computing Machinery, Los Angeles, pp 197–210
Mehrabi A, Siekkinen M, Ylä-Jääski A (2017) Joint optimization of qoe and fairness through network assisted adaptive mobile video streaming. In: 2017 IEEE 13th International Conference On Wireless And Mobile Computing, Networking And Communications (WiMob). IEEE, pp 1–8
Mehrabi A, Siekkinen M, Yla-Jaaski A (2019) Edge computing assisted adaptive mobile video streaming. IEEE Trans Mob Comput 18(4):787–800. https://doi.org/10.1109/TMC.2018.2850026
Mekuria R, Blom K, Cesar P (2017) Design, implementation, and evaluation of a point cloud codec for tele-immersive video. IEEE Trans Circ Syst Video Technol 27(4):828–842. https://doi.org/10.1109/TCSVT.2016.2543039
Misra K, Segall A, Horowitz M, Xu S, Fuldseth A, Zhou M (2013) An overview of tiles in HEVC. IEEE J Selected Topics Signal Process 7(6):969–977. https://doi.org/10.1109/JSTSP.2013.2271451. Conference Name: IEEE Journal of Selected Topics in Signal Processing
New European Media (NEM): 5G-MEDIA Slice Definition. https://bscw.5g-ppp.eu/pub/bscw.cgi/d322688/NEM%20Networld2020%205G%20media%20slice%20V1-2_24092019.pdf (2019)
NVIDIA: Multi-process service. https://docs.nvidia.com/deploy/mps/
Orts-Escolano S, Rhemann C, Fanello S, Chang W, Kowdle A, Degtyarev Y, Kim D, Davidson PL, Khamis S, Dou M, Tankovich V, Loop C, Cai Q, Chou PA, Mennicken S, Valentin J, Pradeep V, Wang S, Kang SB, Kohli P, Lutchyn Y, Keskin C, Izadi S (2016) Holoportation: virtual 3D teleportation in real-time. In: Proceedings of the 29th annual symposium on user interface software and technology, UIST ’16. https://doi.org/10.1145/2984511.2984517. Association for Computing Machinery, Tokyo, pp 741–754
Pantos R, May W (2017) HTTP live streaming. Tech. Rep. RFC8216, RFC Editor. https://doi.org/10.17487/RFC8216. https://www.rfc-editor.org/info/rfc8216
Park J, Chou PA, Hwang JN (2018) Volumetric media streaming for augmented reality. In: 2018 IEEE Global communications conference (GLOBECOM), pp 1–6. https://doi.org/10.1109/GLOCOM.2018.8647537
Paudyal P, Battisti F, Carli M (2016) Impact of video content and transmission impairments on quality of experience. Multimed Tools Applic 75(23):16461–16485. https://doi.org/10.1007/s11042-015-3214-0
Robitza W, Garcia MN, Raake A (2017) A modular http adaptive streaming qoe model—candidate for itu-t p 1203 (“p nats”). In: 2017 Ninth international conference on quality of multimedia experience (QoMEX). IEEE, pp 1–6
Sami Kekki EA (2018) MEC in 5G networks, ETSI White Paper No. 28. https://www.etsi.org/images/files/ETSIWhitePapers/etsi_wp28_mec_in_5G_FINAL.pdf
Schatz R, Sackl A, Timmerer C, Gardlo B (2017) Towards subjective quality of experience assessment for omnidirectional video streaming. In: 2017 Ninth international conference on quality of multimedia experience (QoMEX), pp 1–6. https://doi.org/10.1109/QoMEX.2017.7965657. ISSN: 2472-7814
Schatz R, Zabrovskiy A, Timmerer C (2019) Tile-based streaming of 8K omnidirectional video: subjective and objective QoE evaluation. In: 2019 Eleventh international conference on quality of multimedia experience (QoMEX), pp 1–6. https://doi.org/10.1109/QoMEX.2019.8743230. ISSN: 2472-7814
Schoeffelen T (2020) Designing a serverless video streaming pipeline. https://medium.com/@tschoffelen/designing-a-serverless-video-streaming-pipeline-2d3828f3ccf8
Schreer O, Feldmann I, Renault S, Zepp M, Worchel M, Eisert P, Kauff P (2019) Capture and 3D video processing of volumetric video. In: 2019 IEEE International conference on image processing (ICIP), pp 4310–4314. https://doi.org/10.1109/ICIP.2019.8803576. ISSN: 2381-8549
Service Development Kit for Media-Type Virtualized Network Services in 5G Networks (to appear). IEEE Communication Magazine (2020)
Seufert M, Egger S, Slanina M, Zinner T, Hoßfeld T, Tran-Gia P (2015) A survey on quality of experience of HTTP adaptive streaming. IEEE Commun Surveys Tutor 17(1):469–492. https://doi.org/10.1109/COMST.2014.2360940. Conference Name: IEEE Communications Surveys Tutorials
Singla A, Göring S, Raake A, Meixner B, Koenen R, Buchholz T (2019) Subjective quality evaluation of tile-based streaming for omnidirectional videos. In: Proceedings of the 10th ACM multimedia systems conference, MMSys ’19. https://doi.org/10.1145/3304109.3306218. Association for Computing Machinery, Amherst, pp 232–242
Skupin R, Sanchez Y, Podborski D, Hellge C, Schierl T (2017) Viewport-dependent 360 degree video streaming based on the emerging Omnidirectional Media Format (OMAF) standard. In: 2017 IEEE international conference on image processing (ICIP), pp 4592–4592. https://doi.org/10.1109/ICIP.2017.8297155. ISSN: 2381-8549
Sodagar I (2011) The MPEG-DASH standard for multimedia streaming over the internet. IEEE MultiMedia 18(4):62–67. https://doi.org/10.1109/MMUL.2011.71. Conference Name: IEEE MultiMedia
Soltanian A, Naboulsi D, Salahuddin MA, Glitho R, Elbiaze H, Wette C (2018) Ads: adaptive and dynamic scaling mechanism for multimedia conferencing services in the cloud. In: 2018 15th IEEE Annual consumer communications & networking conference (CCNC). IEEE, pp 1–6
Spiteri K, Urgaonkar R, Sitaraman RK (2016) BOLA: near-optimal bitrate adaptation for online videos. arXiv:1601.06748 [cs]
Sterzentsenko V, Karakottas A, Papachristou A, Zioulis N, Doumanoglou A, Zarpalas D, Daras P (2018) A low-cost, flexible and portable volumetric capturing system, 200–207. https://doi.org/10.1109/SITIS.2018.00038
Sterzentsenko V, Karakottas A, Papachristou A, Zioulis N, Doumanoglou A, Zarpalas D, Daras P (2018) A low-cost, flexible and portable volumetric capturing system. In: 2018 14th International conference on signal-image technology & internet-based systems (SITIS). IEEE, pp 200–207
Sullivan G, Ohm JR, Han WJ, Wiegand T (2012) Overview of the high efficiency video coding (HEVC) standard. IEEE Trans Circ Syst Video Technol 22(12):1649–1668. https://doi.org/10.1109/TCSVT.2012.2221191
Sun L, Duanmu F, Liu Y, Wang Y, Ye Y, Shi H, Dai D (2018) Multi-path multi-tier 360-degree video streaming in 5g networks. In: Proceedings of the 9th ACM multimedia systems conference, MMSys ’18. Association for Computing Machinery, pp 162–173. https://doi.org/10.1145/3204949.3204978. Event-place: Amsterdam, Netherlands
Tian Y, Babcock R, Taylor C, Ji Y (2018) A new live video streaming approach based on amazon s3 pricing model. In: 2018 IEEE 8th Annual computing and communication workshop and conference (CCWC), pp 321–328. https://doi.org/10.1109/CCWC.2018.8301615
van der Hooft J, Wauters T, De Turck F, Timmerer C, Hellwagner H (2019) Towards 6DoF HTTP adaptive streaming through point cloud compression. In: Proceedings of the 27th ACM international conference on multimedia, MM ’19. https://doi.org/10.1145/3343031.3350917. Association for Computing Machinery, Nice, pp 2405–2413
Wien M, Boyce JM, Stockhammer T, Peng WH (2019) Standardization status of immersive video coding. IEEE J Emerg Selected Topics Circ Syst 9(1):5–17
Xie L, Xu Z, Ban Y, Zhang X, Guo Z (2017) 360probdash: improving qoe of 360 video streaming using tile-based http adaptive streaming. In: Proceedings of the 2017 ACM on multimedia conference, MM 2017, Mountain View, CA, USA, October 23-27, 2017. ACM, pp 315–323. https://doi.org/10.1145/3123266.3123291
Xin Z, Fu S (2019) User-centric qoe model of visual perception for mobile videos. Vis Comput 35(9):1245–1254
Yamasaki T, Aizawa K (2010) Patch-based compression for time-varying meshes. In: 2010 IEEE International conference on image processing, pp 3433–3436. https://doi.org/10.1109/ICIP.2010.5652911
Yates RD, Goodman DJ (2014) Probability and stochastic processes: a friendly introduction for electrical and computer engineers. Wiley
Zadtootaghaj S, Schmidt S, Möller S (2018) Modeling gaming qoe: towards the impact of frame rate and bit rate on cloud gaming. In: 2018 Tenth international conference on quality of multimedia experience (QoMEX). IEEE, pp 1–6
Zhang G, Lee JYB (2019) Ensemble adaptive streaming - a new paradigm to generate streaming algorithms via specializations. IEEE Transactions on Mobile Computing, pp 1–1. https://doi.org/10.1109/TMC.2019.2909202. Conference Name: IEEE Transactions on Mobile Computing
Zhang M, Zhu Y, Zhang C, Liu J (2019) Video processing with serverless computing: a measurement study. In: Proceedings of the 29th ACM workshop on network and operating systems support for digital audio and video, pp 61–66
Zhang W, Chen Q, Fu K, Zheng N, Huang Z, Leng J, Li C, Zheng W, Guo M (2020) Towards QoS-aware and resource-efficient GPU microservices based on spatial multitasking GPUs in datacenters. arXiv:2005.02088
Zheng Y, Wu D, Ke Y, Yang C, Chen M, Zhang G (2016) Online cloud transcoding and distribution for crowdsourced live game video streaming. IEEE Trans Circ Syst Video Technol 27(8):1777–1789
Zioulis N, Alexiadis D, Doumanoglou A, Louizis G, Apostolakis K, Zarpalas D, Daras P (2016) 3D tele-immersion platform for interactive immersive experiences between remote users. In: 2016 IEEE International conference on image processing (ICIP), pp 365–369. https://doi.org/10.1109/ICIP.2016.7532380. ISSN: 2381-8549
Acknowledgements
This work has been realized in the context of the 5G-MEDIA project (www.5gmedia.eu), which has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 761699.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Konstantoudakis, K., Breitgand, D., Doumanoglou, A. et al. Serverless streaming for emerging media: towards 5G network-driven cost optimization. Multimed Tools Appl 81, 12211–12250 (2022). https://doi.org/10.1007/s11042-020-10219-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-10219-7