Skip to main content
SpringerLink
Log in

QoE Evaluation in Adaptive Streaming: Enhanced MDT with Deep Learning

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

We propose an architecture for performing virtual drive tests for mobile network performance evaluation by facilitating radio signal strength data from user equipment. Our architecture comprises three main components: (i) pattern recognizer that learns a typical (nominal) behavior for application KPIs (key performance indicators); (ii) predictor that maps from network KPIs to application KPIs; (iii) anomaly detector that compares predicted application performance with said typical pattern. To simulate user-traces, we utilize a commercial state-of-the-art network optimization tool, which collects application and network KPIs at different geographical locations at various times of the day, to train an initial learning model. Although the collected data is related to an adaptive video streaming application, the proposed architecture is flexible, autonomous and can be used for other applications. We perform extensive numerical analysis to demonstrate key parameters impacting video quality prediction and anomaly detection. Playback time is shown to be the most important parameter affecting video quality, most likely due to video packet buffering during playback. We additionally observe that network KPIs, which characterize the cellular connection strength, improve QoE (quality of experience) estimation in anomalous cases diverging from the nominal. The efficacy of our approach is demonstrated with a mean-maximum F1-score of 77%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. according to "similarweb.com"

  2. Recommendation ITU-T P.1203.1 (2016), Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport - Video quality estimation module.

  3. 3GPP TS 37.320: “Radio measurement collection for Minimization of Drive Tests (MDT); Overall description; Stage 2 (Release 16)"

  4. https://youtu.be/A1OpjNq6zGE

  5. refer to "statista.com"

  6. Each test session is at most 60 s long of measurement of radio and application KPIs collected for the same Youtube video.

  7. dBm indicates power level expressed in decibels with reference to one milliwatt.

  8. Adam is a stochastic gradient method that is based on adaptive estimation of first-order and second-order moments and it is widely used in the literature for its fast convergence properties.

References

  1. Tsolkas, D., Liotou, E., Passas, N., Merakos, L.: A survey on parametric qoe estimation for popular services. J. Netw. Comput. Appl. (2017). https://doi.org/10.1016/j.jnca.2016.10.016

  2. Casas, P., Seufert, M., Schatz, R.: Youqmon: a system for on-line monitoring of youtube qoe in operational 3g networks. ACM SIGMETRICS Perform. Eval. Rev. 41(2), 44–46 (2013)

    Article  Google Scholar 

  3. Mushtaq, M.S., Augustin, B., Mellouk, A.: Empirical study based on machine learning approach to assess the qos/qoe correlation. In: 2012 17th European Conference on Networks and Optical Communications, pp. 1–7 (2012). https://doi.org/10.1109/NOC.2012.6249939

  4. Bouraqia, K., Sabir, E., Sadik, M., Ladid, L.: Quality of experience for streaming services: measurements, challenges and insights. IEEE Access 8, 13341–13361 (2020). https://doi.org/10.1109/ACCESS.2020.2965099

    Article  Google Scholar 

  5. Frnda, J., Nedoma, J., Vanus, J., Martinek, R.: A hybrid qos-qoe estimation system for iptv service. Electronics 8(5), 585 (2019)

    Article  Google Scholar 

  6. Kim, H.J., Choi, S.G.: A study on a qos/qoe correlation model for qoe evaluation on iptv service. In: 2010 The 12th International Conference on Advanced Communication Technology (ICACT), vol. 2, pp. 1377–1382 (2010)

  7. Orsolic, I., Pevec, D., Suznjevic, M., Skorin-Kapov, L.: Youtube qoe estimation based on the analysis of encrypted network traffic using machine learning. In: 2016 IEEE Globecom Workshops (GC Wkshps), pp. 1–6. IEEE (2016)

  8. Orsolic, I., Suznjevic, M., Skorin–Kapov, L.: Youtube qoe estimation from encrypted traffic: comparison of test methodologies and machine learning based models. In: 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–6. IEEE (2018)

  9. Nightingale, J., Salva-Garcia, P., Calero, J.M.A., Wang, Q.: 5g-qoe: Qoe modelling for ultra-hd video streaming in 5g networks. IEEE Trans. Broadcast. 64(2), 621–634 (2018). https://doi.org/10.1109/TBC.2018.2816786

    Article  Google Scholar 

  10. Brunnström, K., Dima, E., Qureshi, T., Johanson, M., Andersson, M., Sjöström, M.: Latency impact on quality of experience in a virtual reality simulator for remote control of machines. Signal Process. 89, 116005 (2020). https://doi.org/10.1016/j.image.2020.116005

    Article  Google Scholar 

  11. Ibarrola, E., Davis, M., Voisin, C., Close, C., Cristobo, L.: Qoe enhancement in next generation wireless ecosystems: a machine learning approach. IEEE Commun. Stand. Magaz. 3(3), 63–70 (2019). https://doi.org/10.1109/MCOMSTD.001.1900001

    Article  Google Scholar 

  12. Pérez, P., García, N., Villegas, Á.: Subjective assessment of adaptive media playout for video streaming. In: 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–6 (2019). https://doi.org/10.1109/QoMEX.2019.8743320

  13. Bampis, C.G., Li, Z., Katsavounidis, I., Huang, T.-Y., Ekanadham, C., Bovik, A.C.: Towards Perceptually Optimized End-to-end Adaptive Video Streaming. pp. 1808–03898 (2018). arXiv:1808.03898 [eess.IV]

  14. Gahbiche Msakni, H., Youssef, H.: Is qoe estimation based on qos parameters sufficient for video quality assessment? In: 2013 9th International Wireless Communications and Mobile Computing Conference (IWCMC), pp. 538–544 (2013). https://doi.org/10.1109/IWCMC.2013.6583615

  15. Yamagishi, K.: Qoe-estimation models for video streaming services. In: 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 357–363 (2017). https://doi.org/10.1109/APSIPA.2017.8282058

  16. Seufert, M., Wehner, N., Casas, P.: Studying the impact of has qoe factors on the standardized qoe model p. 1203. In: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), pp. 1636–1641 (2018). https://doi.org/10.1109/ICDCS.2018.00185

  17. Robitza, W., Göring, S., Raake, A., Lindegren, D., Heikkilä, G., Gustafsson, J., List, P., Feiten, B., Wüstenhagen, U., Garcia, M.-N., Yamagishi, K., Broom, S.: Http adaptive streaming qoe estimation with itu-t rec. p. 1203: open databases and software. In: Proceedings of the 9th ACM Multimedia Systems Conference. MMSys ’18, pp. 466–471. Association for Computing Machinery, New York, NY (2018). https://doi.org/10.1145/3204949.3208124

  18. Amour, L., Boulabiar, M.I., Souihi, S., Mellouk, A.: An improved qoe estimation method based on qos and affective computing. In: 2018 International Symposium on Programming and Systems (ISPS), pp. 1–6 (2018). https://doi.org/10.1109/ISPS.2018.8379009

  19. Chen, Y., Wu, K., Zhang, Q.: From qos to qoe: a tutorial on video quality assessment. IEEE Commun. Surv. Tutor. 17(2), 1126–1165 (2015). https://doi.org/10.1109/COMST.2014.2363139

    Article  Google Scholar 

  20. García-Pineda, M., Segura-García, J., Felici-Castell, S.: A holistic modeling for qoe estimation in live video streaming applications over lte advanced technologies with full and non reference approaches. Comput. Commun. 117, 13–23 (2018). https://doi.org/10.1016/j.comcom.2017.12.010

    Article  Google Scholar 

  21. Pokhrel, J., Wehbi, B., Morais, A., Cavalli, A., Allilaire, E.: Estimation of qoe of video traffic using a fuzzy expert system. In: 2013 IEEE 10th Consumer Communications and Networking Conference (CCNC), pp. 224–229. IEEE (2013)

  22. Kang, Y., Chen, H., Xie, L.: An artificial-neural-network-based qoe estimation model for video streaming over wireless networks. In: 2013 IEEE/CIC International Conference on Communications in China (ICCC), pp. 264–269. IEEE (2013)

  23. Pierucci, L., Micheli, D.: A neural network for quality of experience estimation in mobile communications. IEEE Multimedia 23(4), 42–49 (2016). https://doi.org/10.1109/MMUL.2016.21

    Article  Google Scholar 

  24. Banović-Ćurguz, N., Ilišević, D.: Mapping of qos/qoe in 5g networks. In: 2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 404–408 (2019). https://doi.org/10.23919/MIPRO.2019.8757034

  25. Pierucci, L.: The quality of experience perspective toward 5g technology. IEEE Wirel. Commun. 22(4), 10–16 (2015). https://doi.org/10.1109/MWC.2015.7224722

    Article  Google Scholar 

  26. Uthansakul, P., Anchuen, P., Uthansakul, M., Ahmad Khan, A.: Estimating and synthesizing qoe based on qos measurement for improving multimedia services on cellular networks using ann method. IEEE Trans. Netw. Serv. Manage. 17(1), 389–402 (2020). https://doi.org/10.1109/TNSM.2019.2946091

    Article  Google Scholar 

  27. Imran, A., Zoha, A., Abu-Dayya, A.: Challenges in 5g: how to empower son with big data for enabling 5g. IEEE Netw. 28(6), 27–33 (2014). https://doi.org/10.1109/MNET.2014.6963801

    Article  Google Scholar 

  28. Garcia-Reinoso, J., Roselló, M.M., Kosmatos, E., Landi, G., Bernini, G., Legouable, R., Contreras, L.M., Lorenzo, M., Trichias, K., Gupta, M.: The 5g eve multi-site experimental architecture and experimentation workflow. In: 2019 IEEE 2nd 5G World Forum (5GWF), pp. 335–340 (2019). https://doi.org/10.1109/5GWF.2019.8911624

  29. Maia, O.B., Yehia, H.C., de Errico, L.: A concise review of the quality of experience assessment for video streaming. Comput. Commun. 57, 1–12 (2015). https://doi.org/10.1016/j.comcom.2014.11.005

    Article  Google Scholar 

  30. Liotou, E., Tsolkas, D., Passas, N.: A roadmap on qoe metrics and models. In: 2016 23rd International Conference on Telecommunications (ICT), pp. 1–5. IEEE (2016)

  31. Chang, H.-S., Hsu, C.-F., Hoßfeld, T., Chen, K.-T.: Active learning for crowdsourced qoe modeling. IEEE Trans. Multimed. 20(12), 3337–3352 (2018). https://doi.org/10.1109/TMM.2018.2831639

    Article  Google Scholar 

  32. Alimpertis, E., Markopoulou, A., Butts, C.T., Bakopoulou, E., Psounis, K.: A unified prediction framework for signal maps. CoRR (2022). arXiv:2202.03679

  33. Phillips, C., Ton, M., Sicker, D., Grunwald, D.: Practical radio environment mapping with geostatistics. In: 2012 IEEE International Symposium on Dynamic Spectrum Access Networks, pp. 422–433 (2012). https://doi.org/10.1109/DYSPAN.2012.6478166

  34. Yun, Z., Iskander, M.F.: Ray tracing for radio propagation modeling: principles and applications. IEEE Access 3, 1089–1100 (2015). https://doi.org/10.1109/ACCESS.2015.2453991

    Article  Google Scholar 

  35. Ray, A., Deb, S., Monogioudis, P.: Localization of lte measurement records with missing information. In: IEEE INFOCOM 2016—The 35th Annual IEEE International Conference on Computer Communications, pp. 1–9 (2016). https://doi.org/10.1109/INFOCOM.2016.7524370

  36. Fida, M.-R., Lutu, A., Marina, M.K., Alay, O.: Zipweave: Towards efficient and reliable measurement based mobile coverage maps. In: IEEE INFOCOM 2017 - IEEE Conference on Computer Communications, pp. 1–9 (2017). https://doi.org/10.1109/INFOCOM.2017.8057098

  37. Chakraborty, A., Rahman, M.S., Gupta, H., Das, S.R.: Specsense: Crowdsensing for efficient querying of spectrum occupancy. In: IEEE INFOCOM 2017 - IEEE Conference on Computer Communications, pp. 1–9 (2017). https://doi.org/10.1109/INFOCOM.2017.8057113

  38. He, S., Shin, K.G.: Steering crowdsourced signal map construction via bayesian compressive sensing. In: IEEE INFOCOM 2018 - IEEE Conference on Computer Communications, pp. 1016–1024 (2018). https://doi.org/10.1109/INFOCOM.2018.8485972

  39. Enami, R., Rajan, D., Camp, J.: Raik: Regional analysis with geodata and crowdsourcing to infer key performance indicators. In: 2018 IEEE Wireless Communications and Networking Conference (WCNC), pp. 1–6 (2018). https://doi.org/10.1109/WCNC.2018.8377405

  40. Alimpertis, E., Markopoulou, A., Butts, C., Psounis, K.: City-wide signal strength maps: Prediction with random forests. In: The World Wide Web Conference. WWW ’19, pp. 2536–2542. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3308558.3313726

  41. Ghorbani, A., Zou, J.: Data shapley: Equitable valuation of data for machine learning. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 2242–2251. PMLR, (2019). https://proceedings.mlr.press/v97/ghorbani19c.html

  42. Bampis, C.G., Li, Z., Katsavounidis, I., Bovik, A.C.: Recurrent and dynamic models for predicting streaming video quality of experience. IEEE Trans. Image Process. 27(7), 3316–3331 (2018). https://doi.org/10.1109/TIP.2018.2815842

    Article  MathSciNet  MATH  Google Scholar 

  43. Dimopoulos, G., Leontiadis, I., Barlet-Ros, P., Papagiannaki, K.: Measuring video qoe from encrypted traffic. In: Proceedings of the 2016 Internet Measurement Conference. IMC ’16, pp. 513–526. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2987443.2987459

  44. Akhtar, Z., Nam, Y.S., Govindan, R., Rao, S., Chen, J., Katz-Bassett, E., Ribeiro, B., Zhan, J., Zhang, H.: Oboe: Auto-tuning video abr algorithms to network conditions. In: Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. SIGCOMM ’18, pp. 44–58. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3230543.3230558

  45. De Cicco, L., Cilli, G., Mascolo, S.: Erudite: A deep neural network for optimal tuning of adaptive video streaming controllers. In: Proceedings of the 10th ACM Multimedia Systems Conference. MMSys ’19, pp. 13–24. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3304109.3306216

  46. Liu, L., Hu, H., Luo, Y., Wen, Y.: When wireless video streaming meets ai: A deep learning approach. IEEE Wirel. Commun. 27(2), 127–133 (2020). https://doi.org/10.1109/MWC.001.1900220

    Article  Google Scholar 

  47. Menkovski, V., Exarchakos, G., Liotta, A.: Online qoe prediction. In: 2010 Second International Workshop on Quality of Multimedia Experience (QoMEX), pp. 118–123 IEEE (2010).

  48. Mongay Batalla, J., Cuadra Sanchez, A., Ruiz Alonso, J., Kopertowski, Z., Guardalben, L., Garcia, N., Ophir, S.: On assuring survivability of network operator’s services in evolving network environment. IEEE Access 6, 35646–35656 (2018). https://doi.org/10.1109/ACCESS.2018.2851066

    Article  Google Scholar 

  49. Keyhl, M., Obermann, M., Satti, S., Rousell, G.: Perceptual quality evaluation of ott streaming video tv services. In: Proceedings of the 68th NAB Broadcast Engineering Conference, vol. 2014, pp. 252–260 (2014)

  50. Gui, J., Zhou, K.: Flexible adjustments between energy and capacity for topology control in heterogeneous wireless multi-hop networks. J. Netw. Syst. Manage. 24(4), 789–812 (2016)

    Article  Google Scholar 

  51. 3GPP: Telecommunication management; Subscriber and equipment trace; Trace concepts and requirements. Technical Specification (TS) 32.421, 3rd Generation Partnership Project (3GPP) (April 2021). Version 17.1.1. https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=2008

  52. Jennings, B., Stadler, R.: Resource management in clouds: Survey and research challenges. J. Netw. Syst. Manage. 23(3), 567–619 (2015)

    Article  Google Scholar 

  53. Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 4765–4774. Curran Associates, Inc., (2017). http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf

Download references

Acknowledgements

This work is supported in part by The Scientific and Technological Research Council of Turkey (TUBITAK), funded by under 1001 Programme Project No: 119E198, and “5G-PERFECTA: 5G and Next Generation Mobile Performance Compliance Testing Assurance” under EUREKA Programme funded by TEYDEB 1509, and also supported within the framework of 5G and Beyond Joint Graduate Support Programme coordinated by Information and Communication Technologies Authority. Further, Turkcell 6GEN-LAB contributed to this study within the 1515 Frontier R&D Laboratory Support Program funded by TUBITAK.

Funding

This study is partially supported by Turkcell within the framework of 5G and Beyond Joint Graduate Support Programme coordinated by Information and Communication Technologies Authority. This work is supported in part by The Scientific and Technological Research Council of Turkey (TÜBİTAK), funded by under 1001 Programme Project No: 119E198, and “5G-PERFECTA: 5G and Next Generation Mobile Performance Compliance Testing Assurance” under EUREKA Programme, funded by under TEYDEB 1509 - International Industry R&D Support Program, Project No: 9190006. Turkcell 6GEN-LAB contributed to this study within the 1515 Frontier R&D Laboratory Support Program funded by TUBITAK.

Author information

Authors and Affiliations

  1. Department of Electrical and Electronics Engineering, Bilkent University, Ankara, Turkey

    Hakan Gokcesu

  2. Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Turkey

    Ozgur Ercetin

  3. Network Technologies, Turkcell Technology, Istanbul, Turkey

    Hakan Gokcesu, Gokhan Kalem & Salih Ergut

  4. Data Science and R&D, Oredata, Istanbul, Turkey

    Salih Ergut

Authors
  1. Hakan Gokcesu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ozgur Ercetin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gokhan Kalem
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Salih Ergut
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors wrote the main manuscript text, prepared the figures, and reviewed the manuscript.

Corresponding author

Correspondence to Hakan Gokcesu.

Ethics declarations

Conflict of interest

The authors were affiliated with Turkcell via employment or consultancy. There exists a patent application in regard to this research.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gokcesu, H., Ercetin, O., Kalem, G. et al. QoE Evaluation in Adaptive Streaming: Enhanced MDT with Deep Learning. J Netw Syst Manage 31, 41 (2023). https://doi.org/10.1007/s10922-023-09730-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10922-023-09730-7

Keywords

Search

Navigation