Crowdsourcing Quality of Experience Experiments

  • Sebastian Egger-LamplEmail author
  • Judith Redi
  • Tobias Hoßfeld
  • Matthias Hirth
  • Sebastian Möller
  • Babak Naderi
  • Christian Keimel
  • Dietmar Saupe
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10264)


Crowdsourcing enables new possibilities for QoE evaluation by moving the evaluation task from the traditional laboratory environment into the Internet, allowing researchers to easily access a global pool of workers for the evaluation task. This makes it not only possible to include a more diverse population and real-life environments into the evaluation, but also reduces the turn-around time and increases the number of subjects participating in an evaluation campaign significantly, thereby circumventing bottle-necks in traditional laboratory setups. In order to utilise these advantages, the differences between laboratory-based and crowd-based QoE evaluation are discussed in this chapter.



The authors want to thank Schloss Dagstuhl Leibniz-Zentrum für Informatik, the participants of Dagstuhl Seminar 15481 Evaluation in the Crowd: Crowdsourcing and Human-Centred Experiments as well as the Qualinet members that participated in the creation of Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force [38]. Furthermore, this work was supported by the Deutsche Forschungsgemeinschaft (DFG) under Grants HO4770/2-1 and TR257/38-1.


  1. 1.
    Alonso, O., Rose, D.E., Stewart, B.: Crowdsourcing for relevance evaluation. In: ACM SigIR Forum, vol. 42, pp. 9–15. ACM (2008)Google Scholar
  2. 2.
    Becker, M., Borchert, K., Hirth, M., Mewes, H., Hotho, A., Tran-Gia, P.: Microtrails: comparing hypotheses about task selection on a crowd sourcing platform. In: International Conference on Knowledge Technologies and Data-driven Business (I-KNOW), Graz, Austria, October 2015Google Scholar
  3. 3.
    Bhatti, N., Bouch, A., Kuchinsky, A.: Integrating User-Perceived quality into web server design. In: 9th International World Wide Web Conference, pp. 1–16 (2000)Google Scholar
  4. 4.
    Bossuyt, P.: A Comparison of Probabilistic Unfolding Theories for Paired Comparisons Data. Springer, Heidelberg (1990). doi: 10.1007/978-3-642-84172-9 CrossRefGoogle Scholar
  5. 5.
    Bouch, A., Kuchinsky, A., Bhatti, N.: Quality is in the eye of the beholder: meeting users’ requirements for internet quality of service. In: CHI 2000: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 297–304. ACM, New York (2000)Google Scholar
  6. 6.
    Bradley, R.A., Terry, M.E.: Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39(3/4), 324–345 (1952)CrossRefzbMATHMathSciNetGoogle Scholar
  7. 7.
    Callet, P.L., Möller, S., Perkis, A. (eds.): Qualinet white paper on definitions of Quality of Experience (2012)Google Scholar
  8. 8.
    Chen, K.T., Chang, C.J., Wu, C.C., Chang, Y.C., Lei, C.L.: Quadrant of euphoria: a crowdsourcing platform for QoE assessment. Network 24(2) (2010)Google Scholar
  9. 9.
    Chen, K.T., Wu, C.C., Chang, Y.C., Lei, C.L.: A crowdsourceable QoE evaluation framework for multimedia content. In: Proceedings of the 17th ACM international conference on Multimedia, MM 2009, pp. 491–500. ACM (2009)Google Scholar
  10. 10.
    Cooke, M., Barker, J., Lecumberri, G., Wasilewski, K.: Crowdsourcing in Speech Perception. Crowdsourcing for Speech Processing: Applications to Data Collection, Transcription and Assessment, pp. 137–172 (2013)Google Scholar
  11. 11.
    Corriveau, P., Gojmerac, C., Hughes, B., Stelmach, L.: All subjective scales are not created equal: the effects of context on different scales. Sig. Process. 77(1), 1–9 (1999)CrossRefzbMATHGoogle Scholar
  12. 12.
    Strohmeier, D., Jumisko-Pyykkö, S., Raake, A.: Toward task-dependent evaluation of Web-QoE: free exploration vs. who ate what? In: Globecom Workshops, pp. 1309–1313. IEEE (2012)Google Scholar
  13. 13.
    David, H.A.: The Method of Paired Comparisons. Griffin’s statistical monographs, vol. 41, 2nd edn. Charles Griffin & Company Limited, London (1988)zbMATHGoogle Scholar
  14. 14.
    Downs, J.S., Holbrook, M.B., Sheng, S., Cranor, L.F.: Are your participants gaming the system?: screening mechanical turk workers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2399–2402. ACM (2010)Google Scholar
  15. 15.
    Egger, S., Gardlo, B., Seufert, M., Schatz, R.: The impact of adaptation strategies on perceived quality of HTTP adaptive streaming. In: Proceedings of the 2014 Workshop on Design, Quality and Deployment of Adaptive Video Streaming, pp. 31–36. ACM (2014)Google Scholar
  16. 16.
    Egger, S., Reichl, P., Hosfeld, T., Schatz, R.: time is bandwidth? narrowing the gap between subjective time perception and quality of experience. In: 2012 IEEE International Conference on Communications (ICC), pp. 1325–1330. IEEE (2012)Google Scholar
  17. 17.
    Egger, S., Schatz, R.: Interactive content for subjective studies on web browsing QoE: A Kepler derivative. In: ETSI STQ Workshop on Selected Items on Telecommunication Quality Matters, pp. 27–28 (2012)Google Scholar
  18. 18.
    Eichhorn, A., Ni, P., Eg, R.: Randomised pair comparison: an economic and robust method for audiovisual quality assessment. In: Proceedings of the 20th International Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 63–68. ACM (2010)Google Scholar
  19. 19.
    Engeldrum, P.G.: Psychometric Scaling: A Toolkit for Imaging Systems Development. Imcotek Press, Winchester (2000)Google Scholar
  20. 20.
    ETSI: Speech Processing, Transmission and Quality Aspects (STQ); Reference webpage for subjective testing. ETSI Standard TR 103 256, October 2014Google Scholar
  21. 21.
    Fliegel, K.: Qualinet multimedia databases v5. 5 (2014)Google Scholar
  22. 22.
    Freitas, P.G., Redi, J.A., Farias, M.C., Silva, A.F.: Video quality ruler: a new experimental methodology for assessing video quality. In: 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX), pp. 1–6. IEEE (2015)Google Scholar
  23. 23.
    Gardlo, B.: Quality of experience evaluation methodology via crowdsourcing. Ph.D. thesis, University of Zilina (2012)Google Scholar
  24. 24.
    Gardlo, B., Egger, S., Hossfeld, T.: Do scale-design and training matter for video QoE assessments through crowdsourcing? In: Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia, pp. 15–20. ACM (2015)Google Scholar
  25. 25.
    Gardlo, B., Egger, S., Seufert, M., Schatz, R.: Crowdsourcing 2.0: enhancing execution speed and reliability of web-based QoE testing. In: International Conference on Communications, Sydney, AU, June 2014Google Scholar
  26. 26.
    Guse, D., Egger, S., Raake, A., Möller, S.: Web-QoE under real-world distractions: two test cases. In: 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX), pp. 220–225. IEEE (2014)Google Scholar
  27. 27.
    Hands, D., Wilkins, M.: A study of the impact of network loss and burst size on video streaming quality and acceptability. In: Diaz, M., Owezarski, P., Sénac, P. (eds.) IDMS 1999. LNCS, vol. 1718, pp. 45–57. Springer, Heidelberg (1999). doi: 10.1007/3-540-48109-5_5 CrossRefGoogle Scholar
  28. 28.
    Hanhart, P., Korshunov, P., Ebrahimi, T.: Crowd-based quality assessment of multiview video plus depth coding. In: IEEE International Conference on Image Processing, ICIP 2014. Paris France, April 2014Google Scholar
  29. 29.
    Hanhart, P., Korshunov, P., Ebrahimi, T.: Crowdsourcing evaluation of high dynamic range image compression. In: SPIE Optical Engineering + Applications. International Society for Optics and Photonics, San Diego, CA, USA, August 2014Google Scholar
  30. 30.
    Hanhart, P., Krasula, L., Le Callet, P., Ebrahimi, T.: How to benchmark objective quality metrics from paired comparison data? In: Quality of Multimedia Experience (QoMEX), pp. 1–6. IEEE (2016)Google Scholar
  31. 31.
    Harris, C.: You’re hired! an examination of crowdsourcing incentive models in human resource tasks. In: WSDM Workshop on Crowdsourcing for Search and Data Mining (CSDM), pp. 15–18 (2011)Google Scholar
  32. 32.
    Hirth, M., Hoßfeld, T., Tran-Gia, P.: Anatomy of a crowdsourcing platform - using the example of In: Workshop on Future Internet and Next Generation Networks (FINGNet), Seoul, Korea, June 2011
  33. 33.
    Hossfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., Tran-Gia, P.: Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. Trans. Multimedia 16(2), 541–558 (2014)CrossRefGoogle Scholar
  34. 34.
    Hoßfeld, T., Seufert, M., Hirth, M., Zinner, T., Tran-Gia, P., Schatz, R.: Quantification of YouTube QoE via crowdsourcing. In: Symposium on Multimedia, Dana Point, USA, December 2011Google Scholar
  35. 35.
    Hossfeld, T.: On training the crowd for subjective quality studies. VQEG eLetter 1(1), 8 (2014)Google Scholar
  36. 36.
    Hoßfeld, T., Egger, S., Schatz, R., Fiedler, M., Masuch, K., Lorentzen, C.: Initial delay vs. interruptions: between the devil and the deep blue sea. In: QoMEX 2012, Yarra Valley, Australia, July 2012Google Scholar
  37. 37.
    Hoßfeld, T., Hirth, M., Korshunov, P., Hanhart, P., Gardlo, B., Keimel, C., Timmerer, C.: Survey of web-based crowdsourcing frameworks for subjective quality assessment. In: 2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP), pp. 1–6. IEEE (2014)Google Scholar
  38. 38.
    Hoßfeld, T., Hirth, M., Redi, J., Mazza, F., Korshunov, P., Naderi, B., Seufert, M., Gardlo, B., Egger, S., Keimel, C.: Best practices and recommendations for crowdsourced QoE - lessons learned from the qualinet task force “Crowdsourcing” October 2014., lessons learned from the Qualinet Task Force “Crowdsourcing” COST Action IC1003 European Network on Quality of Experience in Multimedia Systems and Services (QUALINET)
  39. 39.
    Hoßfeld, T., Schatz, R., Biedermann, S., Platzer, A., Egger, S., Fiedler, M.: The memory effect and its implications on web QoE modeling. In: 23rd International Teletraffic Congress (ITC 2011), San Francisco, CA, USA (2011)Google Scholar
  40. 40.
    Hoßfeld, T., Schatz, R., Seufert, M., Hirth, M., Zinner, T., Tran-Gia, P.: Quantification of YouTube QoE via Crowdsourcing. In: IEEE International Workshop on Multimedia Quality of Experience - Modeling, Evaluation, and Directions (MQoE 2011), Dana Point, CA, USA, December 2011Google Scholar
  41. 41.
    Hsueh, P.Y., Melville, P., Sindhwani, V.: Data quality from crowdsourcing: a study of annotation selection criteria. In: Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, pp. 27–35. Association for Computational Linguistics (2009)Google Scholar
  42. 42.
    Huynh-Thu, Q., Garcia, M.N., Speranza, F., Corriveau, P., Raake, A.: Study of rating scales for subjective quality assessment of high-definition video. IEEE Trans. Broadcast. 57(1), 1–14 (2011)CrossRefGoogle Scholar
  43. 43.
    International Telecommunication Union: Methods for Subjective Determination of Transmission Quality. ITU-T Recommendation P.800, August 1996Google Scholar
  44. 44.
    International Telecommunication Union: Interactive test methods for audiovisual communications. ITU-T Recommendation P.920, May 2000Google Scholar
  45. 45.
    International Telecommunication Union: Vocabulary and effects of transmission parameters on customer opinion of transmission quality, amendment 2. ITU-T Recommendation P.10/G.100 (2006)Google Scholar
  46. 46.
    International Telecommunication Union: ITU-T recommendation e.800. Quality of Telecommunication Services: Concepts, models, objectives and dependability planning. terms and definitions related to the quality of telecommunication services. ITU-T Recommendation E.800, September 2008Google Scholar
  47. 47.
    International Telecommunication Union: Subjective video quality assessment methods for multimedia applications. ITU-T Recommendation P.910, April 2008Google Scholar
  48. 48.
    International Telecommunication Union: Methodology for the Subjective Assessment of the Quality of Television Pictures. ITU-R Recommendation BT.500-12, March 2009Google Scholar
  49. 49.
    International Telecommunication Union: Subjective Testing Methodology for web browsing. ITU-T Recommendation P.1501 (2013)Google Scholar
  50. 50.
    International Telecommunication Union: Subjective Methods for the Assessment of stereoscopic 3DTV Systems. ITU-R Recommendation BT.2021, July 2015Google Scholar
  51. 51.
    Janowski, L., Pinson, M.: Subject bias: introducing a theoretical user model. In: 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX), pp. 251–256. IEEE (2014)Google Scholar
  52. 52.
    Jones, B.L., McManus, P.R.: Graphic scaling of qualitative terms. SMPTE J. 95(11), 1166–1171 (1986)CrossRefGoogle Scholar
  53. 53.
    Jumisko-Pyykkö, S., Hannuksela, M.M.: Does context matter in quality evaluation of mobile television?. In: Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services, pp. 63–72. ACM (2008)Google Scholar
  54. 54.
    Keelan, B.: Handbook of Image Quality: Characterization and Prediction. CRC Press, Boca Raton (2002)CrossRefGoogle Scholar
  55. 55.
    Keelan, B.W., Urabe, H.: ISO 20462: a psychophysical image quality measurement standard. In: Electronic Imaging 2004, pp. 181–189. International Society for Optics and Photonics (2003)Google Scholar
  56. 56.
    Keimel, C., Habigt, J., Diepold, K.: Challenges in crowd-based video quality assessment. In: Forth International Workshop on Quality of Multimedia Experience (QoMEX 2012), Yarra Valey, Australia, July 2012Google Scholar
  57. 57.
    Keimel, C., Habigt, J., Horch, C., Diepold, K.: QualityCrowd - a framework for crowd-based quality evaluation. In: Picture Coding Symposium, Krakow, PL, May 2012Google Scholar
  58. 58.
    Keimel, C., Habigt, J., Horch, C., Diepold, K.: Video quality evaluation in the cloud. In: Packet Video Workshop, Munich, DE, May 2012Google Scholar
  59. 59.
    Korshunov, P., Cai, S., Ebrahimi, T.: Crowdsourcing approach for evaluation of privacy filters in video surveillance. In: 1st International ACM workshop on Crowdsourcing for Multimedia (CrowdMM 2012). ACM, Nara, October 2012Google Scholar
  60. 60.
    Korshunov, P., Nemoto, H., Skodras, A., Ebrahimi, T.: The effect of HDR images on privacy: crowdsourcing evaluation. In: SPIE Photonics Europe 2014, Optics, Photonics and Digital Technologies for Multimedia Applications, Brussels, Belgium, April 2014Google Scholar
  61. 61.
    Kraft, S., Zölzer, U.: BeaqleJS: HTML5 and JavaScript based framework for the subjective evaluation of audio quality. In: Linux Audio Conference, Karlsruhe, DE, May 2014Google Scholar
  62. 62.
    Kubey, R., Csikszentmihalyi, M.: Television and the Quality of Life: How Viewing Shapes Everyday Experience. A Volume in the Communication Series. L. Erlbaum Associates (1990).
  63. 63.
    Laghari, K., Crespi, N., Connelly, K.: Toward total quality of experience: a QoE model in a communication ecosystem. IEEE Commun. Mag. 50(4), 58–65 (2012)CrossRefGoogle Scholar
  64. 64.
    Lebreton, P.R., Mäki, T., Skodras, E., Hupont, I., Hirth, M.: Bridging the gap between eye tracking and crowdsourcing. In: Human Vision and Electronic Imaging XX, San Francisco, California, USA, 9–12 February 2015, p. 93940W (2015)Google Scholar
  65. 65.
    Lee, J.S., De Simone, F., Ebrahimi, T.: Subjective quality evaluation via paired comparison: application to scalable video coding. IEEE Trans. Multimedia 13(5), 882–893 (2011)CrossRefGoogle Scholar
  66. 66.
    Li, J., Barkowsky, M., Le Callet, P.: Boosting paired comparison methodology in measuring visual discomfort of 3DTV: performances of three different designs. In: Proceeding SPIE Electronic Imaging-Stereoscopic Displays and Applications XXIV (2013)Google Scholar
  67. 67.
    Little, G., Chilton, L., Goldman, M., Miller, R.: TurKit: tools for iterative tasks on mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 29–30. ACM (2009)Google Scholar
  68. 68.
    Mayo, C., Aubanel, V., Cooke, M.: Effect of prosodic changes on speech intelligibility. In: Interspeech. Citeseer (2012)Google Scholar
  69. 69.
    Mazza, F., Da Silva, M.P., Le Callet, P.: Would you hire me? Selfie portrait images perception in a recruitment context. In: IS&T/SPIE Electronic Imaging, p. 90140X. International Society for Optics and Photonics (2014)Google Scholar
  70. 70.
    Möller, S.: Quality Engineering - Qualität kommunikationstechnischer Systeme. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-11548-6 Google Scholar
  71. 71.
    Möller, S., Raake, A.: Telephone speech quality prediction: towards network planning and monitoring models for modern network scenarios. Speech Commun. 38, 47–75 (2002)., ACM ID: 638082CrossRefzbMATHGoogle Scholar
  72. 72.
    Mosteller, F.: Remarks on the method of paired comparisons: I. The least squares solution assuming equal standard deviations and equal correlations. Psychometrika 16(1), 3–9 (1951)CrossRefGoogle Scholar
  73. 73.
    Naderi, B., Polzehl, T., Beyer, A., Pilz, T., Möller, S.: Crowdee: mobile crowdsourcing micro-task platform - for celebrating the diversity of languages. In: Proceeding of 15th Annual Conference of the International Speech Communication Assocation (Interspeech 2014) (2014)Google Scholar
  74. 74.
    Naderi, B., Polzehl, T., Wechsung, I., Köster, F., Möller, S.: Effect of trapping questions on the reliability of speech quality judgments in a crowdsourcing paradigm. In: 16th Annual Conference of the International Speech Communication Assocation (Interspeech 2015), ISCA, pp. 2799–2803 (2015)Google Scholar
  75. 75.
    Ouyang, Y., Yan, T., Wang, G.: CrowdMi: scalable and diagnosable mobile voice quality assessment through wireless analytics. IEEE Internet Things J. 2(4), 287–294 (2015)CrossRefGoogle Scholar
  76. 76.
    Parasuraman, A., Zeithaml, V.A., Berry, L.L.: A conceptual model of service quality and its implications for future research. J. Market. 49, 41–50 (1985)CrossRefGoogle Scholar
  77. 77.
    Pitrey, Y., Engelke, U., Barkowsky, M., Pépion, R., Le Callet, P.: Aligning subjective tests using a low cost common set. In: Euro ITV, IRCCyN-Contribution (2011)Google Scholar
  78. 78.
    Polzehl, T., Naderi, B., Köster, F., Möller, S.: Robustness in speech quality assessment and temporal training expiry in mobile crowdsourcing environments. In: 16th Annual Conference of the International Speech Communication Assocation (Interspeech 2015), ISCA, pp. 2794–2798 (2015)Google Scholar
  79. 79.
    Raake, A.: Speech Quality of VoIP: Assessment and Prediction. Wiley, New York (2006)CrossRefGoogle Scholar
  80. 80.
    Rainer, B., Waltl, M., Timmerer, C.: A web based subjective evaluation platform. In: Workshop on Quality of Multimedia Experience, Klagenfurth, AT, July 2013Google Scholar
  81. 81.
    Redi, J., Hoßfeld, T., Korshunov, P., Mazza, F., Povoa, I., Keimel, C.: Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal. In: Workshop on Crowdsourcing for Multimedia, Barcelona, ES, October 2013Google Scholar
  82. 82.
    Redi, J., Liu, H., Alers, H., Zunino, R., Heynderickx, I.: Comparing subjective image quality measurement methods for the creation of public databases. In: IS&T/SPIE Electronic Imaging, p. 752903. International Society for Optics and Photonics (2010)Google Scholar
  83. 83.
    Redi, J., Povoa, I.: Crowdsourcing for rating image aesthetic appeal: Better a paid or a volunteer crowd? In: 3rd International ACM workshop on Crowdsourcing for Multimedia (CrowdMM 2014), Orlando, FL, USA, November 2014Google Scholar
  84. 84.
    Redi, J., Siahaan, E., Korshunov, P., Habigt, J., Hossfeld, T.: When the crowd challenges the lab: lessons learnt from subjective studies on image aesthetic appeal. In: Proceedings of the Fourth International Workshop on Crowdsourcing for Multimedia, pp. 33–38. ACM (2015)Google Scholar
  85. 85.
    Reichl, P.: From charging for Quality of Aervice to charging for Quality of Experience. Annales des Télécommunications 65(3–4), 189–199 (2010)CrossRefGoogle Scholar
  86. 86.
    Rerabek, M., Yuan, L., Krasula, L., Korshunov, P., Fliegel, K., Ebrahimi, T.: Evaluation of privacy in high dynamic range video sequences. In: SPIE Optical Engineering + Applications. International Society for Optics and Photonics, San Diego, CA, USA, August 2014Google Scholar
  87. 87.
    Ribeiro, F., Florencio, D., Nascimento, V.: Crowdsourcing subjective image quality evaluation. In: Image Processing. Brussels, BE, September 2011Google Scholar
  88. 88.
    Ribeiro, F., Florencio, D., Zhang, C., Seltzer, M.: CrowdMOS: an approach for crowdsourcing mean opinion score studies. In: International Conference on Acoustics, Speech and Signal Processing. Prague, CZ, May 2011Google Scholar
  89. 89.
    de Ridder, H.: Cognitive issues in image quality measurement. J. Electron. Imaging 10(1), 47–55 (2001)CrossRefMathSciNetGoogle Scholar
  90. 90.
    Rossi, P.E., Gilula, Z., Allenby, G.M.: Overcoming scale usage heterogeneity. J. Am. Stat. Assoc. 96(453), 20–31 (2001). CrossRefGoogle Scholar
  91. 91.
    Rubino, G.: Quantifying the quality of audio and video transmissions over the internet: the PSQA approach. In: Design and Operations of Communication Networks: A Review of Wired and Wireless Modelling and Management Challenges. Imperial College Press (2005)Google Scholar
  92. 92.
    Sackl, A., Schatz, R.: Evaluating the impact of expectations on end-user quality perception. In: Proceedings of International Workshop Perceptual Quality of Systems (PQS), pp. 122–128 (2013)Google Scholar
  93. 93.
    Sanchez-Iborra, R., JPC Rodrigues, J., Cano, M.D., Moreno-Urrea, S.: QoE measurements and analysis for VoIP services. Emerging Research on Networked Multimedia Communication Systems, p. 285 (2015)Google Scholar
  94. 94.
    Schatz, R., Egger, S.: Vienna surfing - assessing mobile broadband quality in the field. In: Taft, N., Wetherall, D. (eds.) Proceedings of the 1st ACM SIGCOMM Workshop on Measurements Up the STack (W-MUST). ACM (2011)Google Scholar
  95. 95.
    Schatz, R., Hoßfeld, T., Janowski, L., Egger, S.: From packets to people: quality of experience as a new measurement challenge. In: Biersack, E., Callegari, C., Matijasevic, M. (eds.) Data Traffic Monitoring and Analysis. LNCS, vol. 7754, pp. 219–263. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-36784-7_10 CrossRefGoogle Scholar
  96. 96.
    Schnitzer, S., Rensing, C., Schmidt, S., Borchert, K., Hirth, M., Tran-Gia, P.: Demands on task recommendation in crowdsourcing platforms - the workers perspective. In: CrowdRec Workshop, Vienna, Austria (9 2015)Google Scholar
  97. 97.
    Shaikh, J., Fiedler, M., Paul, P., Egger, S., Guyard, F.: Back to normal? Impact of temporally increasing network disturbances on QoE. In: 2013 IEEE Globecom Workshops (GC Workshops), pp. 1186–1191. IEEE (2013)Google Scholar
  98. 98.
    Silverstein, D.A., Farrell, J.E.: Quantifying perceptual image quality. In: PICS, vol. 98, pp. 242–246 (1998)Google Scholar
  99. 99.
    Soldani, D., Li, M., Cuny, R.: QoS and QoE management in UMTS cellular systems. Wiley, West Sussex (2006)CrossRefGoogle Scholar
  100. 100.
    Thurstone, L.L.: A law of comparative judgment. Psychol. Rev. 34(4), 273 (1927)CrossRefGoogle Scholar
  101. 101.
    Tsukida, K., Gupta, M.R.: How to analyze paired comparison data. Technical report, DTIC Document (2011)Google Scholar
  102. 102.
    Varela, M., Mäki, T., Skorin-Kapov, L., Hoßfeld, T.: Increasing payments in crowdsourcing: don’t look a gift horse in the mouth. In: 4th International Workshop on Perceptual Quality of Systems (PQS 2013), Vienna, Austria (2013)Google Scholar
  103. 103.
    Varela, M., Mäki, T., Skorin-Kapov, L., Hoßfeld, T.: Towards an understanding of visual appeal in website design. In: 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), pp. 70–75. IEEE (2013)Google Scholar
  104. 104.
    Varela, M., Skorin-Kapov, L., Mäki, T., Hoßfeld, T.: QoE in the web: a dance of design and performance. In: 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX), pp. 1–7. IEEE (2015)Google Scholar
  105. 105.
    Virtanen, M., Gleiss, N., Goldstein, M.: On the use of evaluative category scales in telecommunications. In: Human Factors in Telecommunications (1995)Google Scholar
  106. 106.
    Wickelmaier, F., Schmid, C.: A matlab function to estimate choice model parameters from paired-comparison data. Behav. Res.h Methods Instrum. Comput. 36(1), 29–40 (2004)CrossRefGoogle Scholar
  107. 107.
    Winkler, S., Mohandas, P.: The evolution of video quality measurement: from PSNR to hybrid metrics. IEEE Trans. Broadcast. 54(3), 660–668 (2008). CrossRefGoogle Scholar
  108. 108.
    Wolters, M.K., Isaac, K.B., Renals, S.: Evaluating speech synthesis intelligibility using Amazon mechanical turk. In: 7th Speech Synthesis Workshop (2010)Google Scholar
  109. 109.
    Wu, C.C., Chen, K.T., Chang, Y.C., Lei, C.L.: Crowdsourcing multimedia QoE evaluation: a trusted framework. IEEE Trans. Multimedia 15(5), 1121–1137 (2013)CrossRefGoogle Scholar
  110. 110.
    Xu, Q., Huang, Q., Jiang, T., Yan, B., Lin, W., Yao, Y.: HodgeRank on random graphs for subjective video quality assessment. Trans. Multimedia 14(3), 844–857 (2012)CrossRefGoogle Scholar
  111. 111.
    Yu-Chuan, Y., Chu, C.Y., Yeh, S.L., Chu, H.H., Huang, P.: Lab experiment vs. crowdsourcing: a comparative user study on Skype call quality. In Proceedings of the 9th Asian Internet Engineering Conference, pp. 65–72 (2013)Google Scholar
  112. 112.
    Zinner, T., Hirth, M., Fischer, V., Hohlfeld, O.: Erwin - enabling the reproducible investigation of waiting times for arbitrary workflows. In: 8th International Conference on Quality of Multimedia Experiene (QoMEX), Lisbon, Portugal, June 2016Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Sebastian Egger-Lampl
    • 1
    Email author
  • Judith Redi
    • 2
  • Tobias Hoßfeld
    • 3
  • Matthias Hirth
    • 4
  • Sebastian Möller
    • 5
  • Babak Naderi
    • 5
  • Christian Keimel
    • 6
  • Dietmar Saupe
    • 7
  1. 1.Austrian Institute of TechnologyViennaAustria
  2. 2.Delft University of TechnologyDelftNetherlands
  3. 3.University of Duisburg-EssenDuisburgGermany
  4. 4.University of WürzburgWürzburgGermany
  5. 5.TU BerlinBerlinGermany
  6. 6.Technische Universität MünchenMunichGermany
  7. 7.University of KonstanzKonstanzGermany

Personalised recommendations