YouTube QoE evaluation tool for Android wireless terminals

Abstract

In this paper, we present an Android application which is able to evaluate and analyze the perceived quality of experience (QoE) for YouTube service in wireless terminals. To achieve this goal, the application carries out measurements of objective quality of service (QoS) parameters, which are then mapped onto subjective QoE (in terms of mean opinion score, MOS) by means of a utility function. Our application also informs the user about potential causes that lead to a low MOS as well as provides some hints to improve it. After each YouTube session, the users may optionally qualify the session through an online opinion survey. This information has been used in a pilot experience to correlate the theoretical QoE model with real user feedback. Results from such an experience have shown that the theoretical model (taken from the literature) provides slightly more pessimistic results compared to user feedback. Users seem to be more indulgent with wireless connections, increasing the MOS from the opinion survey in about 20% compared to the theoretical model, which was obtained from wired scenarios.

1. Introduction

Real-time entertainment services (comprised mostly of streaming video and audio) are becoming one of the dominant web-based services in telecommunications networks. In particular, YouTube service is currently the largest single source of real-time entertainment traffic and the third most visited Internet site (preceded by Google and Facebook). It has emerged to account for more Internet traffic than any other service. Mobile networks have the highest proportion of real-time entertainment traffic. Nowadays, YouTube leads the way, accounting for 20% to 25% of total traffic in mobile networks. Additionally, 27.8% of all YouTube traffic (first half 2012) has been consumed on a smartphone or tablet [1].

The combination of increasing device capabilities, high-resolution content, and longer video duration (largely due to live content) means that YouTube’s growth will continue for the foreseeable future. Driven by higher bitrates and enhanced capabilities of mobile devices, the trend is also going towards high-definition (HD) video, which considerably enhances quality demand. That is the reason mobile network operators are following this trend, as it will be hugely influential on network requirements and subscriber quality of experience. In the context of video streaming services, quality of service (QoS) measurements become non-sufficient for evaluating the overall quality because they do not take into account user satisfaction. As a consequence, many network operators are starting to study the evaluation of quality of experience (QoE), which can be considered as the overall performance of a system from a user’s perspective.

The QoE has been usually evaluated through subjective tests carried out on the users in order to assess their degree of satisfaction with a mean opinion score (MOS) indicator [2]. This type of approach is obviously quite expensive, as well as annoying to the user. That is why in recent years, new methods have been used to estimate the QoE based on certain performance indicators associated with services. The evaluation methodology used by most network operators to obtain statistical QoE is based on field testing. These tests often use mobile handsets as a modem, with laptop computers that perform the tests and collect statistics. However, this process is expensive in terms of resources and staff, and it does not use the entire protocol stack implemented in the terminal. These drawbacks are solved by integrating QoE analyzers in the mobile terminal itself so that quality measurements are specific to each terminal. Using this approach, additional measurements can be collected (along the protocol stack) to allow for enhanced analysis of the performance of each service. Furthermore, if mobile terminals are able to report the measurements to a server, the QoE assessment process is simplified significantly.

Recently, a number of works have focused on developing subjective QoE evaluation frameworks for mobile users. For instance, an implementation of a QoE measurement framework on Android platform is presented in [3, 4], although results are limited to a laboratory environment. The works in [5–7] present a framework for measuring the QoE for distorted videos in terms of peak signal-to-noise ratio (PSNR) or a modified metric called cPSNR, respectively. A non-intrusive video quality estimator based on parametric models that considers the group of pictures (GoP) length, type of frames, packet loss rate, etc. is presented in [8]. A QoE framework for multimedia services (named as QoM) for run time quality evaluation of video streaming services is presented in [9]; this approach is based on the influence of QoE factors and various network and application-level QoS parameters, although no evaluation of the proposed framework in a context of a real wireless network has been performed. In [10], the problem of YouTube QoE monitoring from an access provider’s perspective is investigated, showing that it is possible to detect application-level stalling events by using network-level passive probing only. The work in [11] describes a tool that monitors YouTube application comfort, making it possible to estimate the time when the YouTube player is stalling.

Other works are focused on specific YouTube models to compute the QoE. In [12, 13], different QoE YouTube models that take into account the key influence factors in the quality perception (such as stalling events caused by network bottlenecks) are presented. They quantify the impact of initial delays on the user-perceived QoE by means of subjective laboratory and crowdsourcing studies. Other works are devoted to estimate the MOS for video services [3, 14, 15]; among them, the analysis presented in [15] provides a utility function for HTTP video streaming as a function of three application performance metrics: initial buffering time, mean rebuffering time, and rebuffering frequency.

However, none of the previous works has performed a deep validation of existing models through real tests over different radio technologies. In this work, we describe an Android application that carries out measurements of objective QoS indicators associated to YouTube service; these performance indicators are then mapped onto subjective QoE (in terms of MOS). Our application also informs the user about possible causes that lead to a low MOS, and provides some hints to improve it. After each YouTube session, the users may optionally qualify the session through an opinion survey. This information has been used in a pilot experience to correlate the theoretical QoE model with real user feedback.

The remainder of this paper is structured as follows. A description of the YouTube QoE evaluation method is given in Section 2, specifying its main performance indicators. In Section 3, we describe our Android application for YouTube QoE evaluation. The results from a YouTube evaluation pilot experience are analyzed in Section 4. Finally, some concluding remarks are given in Section 5.

2. YouTube QoE evaluation method

YouTube service employs progressive download technique, which enables the playback of the video before the content downloaded is completely finished [16]. Old YouTube delivery service for mobile terminals (through the mobile YouTube link http://m.youtube.com) was based on conventional video streaming architecture, i.e., Real Time Streaming Protocol (RTSP) and Real Time Protocol (RTP), the latter being transported over User Datagram Protocol (UDP). However, the current delivery service (both for smartphones and PCs) uses progressive video download via HyperText Transfer Protocol (HTTP) over Transmission Control Protocol (TCP).

Nowadays, TCP is the preferred transport protocol for YouTube and other video servers since the majority of video content delivery over the Internet is not live and most users’ bandwidth is usually greater than the video coding rate [17, 18]. The HTTP/TCP architecture also solves the problem of access blocking carried out by many firewalls for unknown UDP ports. Additionally, the continuous improvements in latency reduction and throughput maximization achieved in new cellular technologies have allowed the use of TCP for minimizing the impact of errors without reducing severely the effective throughput.

The video clip download process is started by the end user when a request (with a link to the desired video clip) is sent to the YouTube web server (see Figure 1). When the client web browser receives the YouTube web page, the embedded player initiates the required signaling with the media server indicating the video to be played out along with some setup parameters [17]. Then, the server starts progressively sending the video data over an HTTP response. The video data is then stored in a play-out buffer at the client side before being displayed. Once the download has been started, there is no further client-to-server signaling (unless the user interacts with the player).

Figure 1

Signaling flow chart of a YouTube session (via web browser).

The video data transfer from the media server to the client consists of two phases: initial burst of data and throttling algorithm [17]. In the initial phase, the media server sends an initial burst of data (whose size is determined by one of the setup parameters) at the maximum available bandwidth. Then, the server starts the throttling algorithm, where the data are sent at a constant rate (normally at the video clip encoding rate multiplied by the throttle factor, also denoted in the setup parameters). In a network congestion episode, the data that are not able to be delivered at this constant rate are buffered in the server and released as soon as the congestion is alleviated. When this occurs, data are sent at the maximum available bandwidth. Whenever the player’s buffer runs out of data, the playback will be paused, leading to a rebuffering event.

Like quality of Internet services in general, Internet video streaming quality is mainly depending on throughput. However, quality requirements in terms of throughput are more demanding than those for other popular Internet applications as file download, web browsing, and messaging. The main differences are that throughput has to meet rather precise requirements and that these requirements are stream-specific, i.e., if data are not transmitted according to the playing rate (corrected by the influence of initial buffering), a rebuffering will likely occur and user QoE will drop down rapidly. It is therefore essential not only to measure the download throughput, but also to check against the bitrate the individual stream is encoded with.

There exist many quality metrics to characterize the video quality. Some of them are based on comparing the received (and degraded) video with the original video (usually called ‘reference’). Examples of this type of quality metrics are as follows: mean square error (MSE) [19], peak signal-to-noise ratio (PSNR) [19], video structural similarity (VSSIM) [20], perceptual evaluation of video quality (PEVQ) [21], and video quality metric (VQM) [22]. This type of metrics is useful for obtaining objective metrics in controlled experiments, but these metrics are not applicable for online (real-time) procedures as the full reference is not available. Furthermore, they are suited to measure the image quality degradation, e.g., due to packet losses or compression algorithms. Since using TCP, packet losses are recovered, this type of metrics is less useful for YouTube.

That is why other works are oriented to provide a model for estimating the video quality without a reference. For instance, the work described in [23] presents a regression model to estimate the visual perceptual quality in terms of MOS for MPEG-4 videos over wireless networks. However, this algorithm requires an image reconstruction process to evaluate the differences between the original and the resulting images (after network transmission), which makes it not adequate for online quality estimations. In [24], the impact of delay and delay variation on user’s perceived quality for video streaming is analyzed. However, it does not consider other objective metrics such as resolution, frame rate, or packet losses, which are also important for obtaining an accurate QoE estimation. In [25], a no-reference subjective metric to evaluate the video quality is presented, which considers the frame rate or the picture resolution, although their computation is complex to be used real-time.

Our implementation is based on the work presented in [15], which studied how the network QoS affects the QoE of HTTP video streaming. In this work, they propose a generic procedure to estimate the end user’s perceived quality following three steps:

• 1) Estimate (through modeling) or measure network QoS (e.g., throughput, round trip time, loss rate, etc.).

• 2) Convert network QoS metrics onto application QoS (application performance metrics) by means of protocols’ modeling.

• 3) Map application QoS onto end user’s QoE (in terms of MOS).

It should be noticed that the first step might not be needed if the mobile terminal is equipped with a customized YouTube client that directly monitors and reports the application performance metrics. Otherwise, the mobile terminal shall be able to convert the network QoS onto application QoS by specific protocol modeling. For instance, there are different TCP performance models to estimate TCP throughput from network QoS [26, 27]. Afterwards, application performance metrics can be estimated at the receiver from performance indicators at lower layers (e.g., TCP throughput) as well as other parameters like the video coding rate, video length, buffer size at the receiver, or the minimum buffer threshold that triggers a rebuffering event (see [15] for further details).

The third step is performed by applying a utility function for HTTP video streaming as a function of three application performance metrics:

• Initial buffering time (T_init): time elapsed until a certain buffer occupancy threshold has been reached so the playback can start

• Rebuffering frequency (f_rebuf): frequency of interruption events during the playback

• Mean rebuffering time (T_rebuf): average duration of a rebuffering event

In [15], each application performance metric (T_init, f_rebuf, and T_rebuf) is divided into three levels (low, medium, high) which are based on the 25th, 50th, and 75th percentiles of the actual metric. The quantified performance metrics L_ti, L_fr, and L_tr take the numerical values 1, 2, and 3 to represent the ‘low’, ‘medium’, and ‘high’ levels, respectively. The mapping from T_init, f_rebuf, and T_rebuf to L_ti, L_jr, and L_tr is performed according to Table 1[15].

Table 1 Three valued levels of application performance metrics

The quantified performance metrics L_ti, L_fr, and L_tr were used in [15] to obtain the MOS estimation from linear regression:

$${\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{QoSmodel}}=4.23-0.0672\cdot L_{\mathrm{t}\mathrm{i}}-0.742\cdot L_{\mathrm{f}\mathrm{r}}-0.106\cdot L_{\mathrm{t}\mathrm{r}}\text{.}$$

(1)

From the previous equation, it can be seen that the maximum predicted MOS is 4.23. Coefficients for all service levels (L_ti, L_fr, and L_tr) are negative as any increase in them would lead to worse service quality. Moreover, the rebuffering frequency metric (f_rebuf, quantized as L_fr) has the highest impact on the end user’s QoE (regression coefficient −0.742), compared to the initial buffering time (regression coefficient −0.06) and the rebuffering duration (regression coefficient −0.106). In this respect, it is reasonable to think that the perceived quality does not only depend on the pause intensity (percentage of time in the pause state), since a higher number of pauses (with lower pause durations) seems more annoying to the user.

3. Android application for YouTube QoE evaluation

The model for estimating YouTube QoE has been implemented as an Android application. Our QoE tool is able to run in two different modes:

• 1) Intrusive mode: Our application includes an embedded video player, thus providing access to the content being consumed through the YouTube Application Programming Interface (API).

• 2) Transparent mode: Our application runs in the background, so monitoring functionalities are associated to YouTube sessions established either through the native YouTube application or through the web browser.

Our Android application includes the following modules:

• Monitoring: This module is responsible for monitoring network QoS parameters as well as other configuration parameters as required to estimate the application performance monitoring (listed in the previous section). It makes use of the Android Networking and YouTube Data API to get a number of parameters associated to the session, as detailed next.

• QoE estimation: This is in charge of (automatically) computing the QoE of a YouTube session (in terms of MOS) from QoS parameters, according to Equation 1.

• QoE advices: This informs the user about possible causes that lead to a low MOS and provides some hints to improve it.

• QoE user feedback: This allows users to qualify the session through an opinion survey. This information is used to correlate the QoE model with real user feedback.

• QoE reporting: This module is responsible for reporting all the performance indicators to a QoE server for post-processing purposes.

A general overview of our YouTube QoE framework is depicted in Figure 2. In addition to the MOS value automatically estimated by the application, users are requested to qualify the session (video, audio, and general feedback) manually in the same MOS scale (from 1 to 5). We have used both types of QoE evaluations to validate the theoretical model proposed in [15], as well as to propose a modified function according to the results of our pilot experience.

Figure 2

YouTube QoE framework.

Figure 3 shows some snapshots of our Android QoE tool in its intrusive mode version, which includes the media player. Once the YouTube session is over, estimated quality results are shown to the user and reported to the QoE server.

Figure 3

Snapshots of our Android QoE tool (intrusive mode).

In order to estimate the QoE, the monitoring module must collect a set of performance indicators to be subsequently mapped onto QoE. In addition to the three application performance metrics (T_init, f_rebuf, T_rebuf) required to compute the MOS, the monitoring module gathers other relevant information related to the mobile terminal, session information (date, type of player, etc.), location of the measurements, or network information. All this information is reported, together with the estimated QoE and subjective quality specified by the users, to the QoE server. The complete list of parameters that are reported (from the terminal) to the QoE server is given in Table 2.

Table 2 List of parameters that are reported (from the terminal) to the QoE server

Our application uses principally standard libraries from Android SDK. Regarding the collection of network statistics, Android terminals allow to get radio-related parameters and performance indicators like the received signal strength as well as traffic-related statistics via standard libraries, like android.telephony or android.net. Time tagging (in milliseconds) associated to performance indicators is performed via standard time class, whereas location information is obtained from Android Location Manager (selecting network providers and GPS, if available).

However, other relevant information is not accessible through the standard Android API, which is more oriented towards applications of general interest. For instance, YouTube application performance metrics (T_init, f_rebuf, T_rebuf) and some parameters related to the YouTube session (like video duration) cannot be directly obtained from Android API. We have discarded the use of Android packet sniffing capabilities (in the form of .pcap files) as they require the terminal to run with root privileges.

When our QoE tool runs in intrusive mode (i.e., player embedded in the application), the measurement of the three application performance metrics is straightforward. For this purpose, we use YouTube API, which provides the capability not only to embed a YouTube player in your own application or to control a standalone YouTube player, but also to get certain information about the ongoing YouTube session. Concretely, application performance metrics can be obtained with this API as described next.

The YouTube player has internally several states: unstarted, buffering, playing, paused, and ended. The media player is on unstarted state when it is first loaded, but as soon as it starts downloading the video file, the player changes its state to buffering. When the buffer has enough data, the playback starts, thus changing to playing state. The paused state is given if the video play-out is paused by the user. The player may go back to the buffering state if the buffer runs out of data during the playback due to congestion problems (called rebuffering event); otherwise, the playback will continue until achieving the ended state. Therefore, since the YouTube API provides information about the player state at each moment, it is easy to compute: 1) the initial buffering time (T_init) as the time elapsed in the buffering state for the first time, 2) rebuffering frequency (f_rebuf) from the number of times that the player enters the buffering state during playback, and 3) mean rebuffering time (T_rebuf) from the time elapsed in the buffering state.

However, when our QoE tool runs in transparent mode, the computation of these metrics is not so easy as we do not have access to the transitions among states, so they have to be estimated from network-level metrics, as detailed in [15]. In particular, the following basic information is required: average TCP throughput, average playing rate, and player buffer size. The model to estimate application QoS metrics from network QoS is valid under certain assumptions: 1) The network bandwidth, round trip time (RTT), and packet loss rate are assumed to be constant during the video download, and 2) the client does not interact with the video during the playback, such as pausing and forward/backward. However, these assumptions may not be very realistic due to the fact that, as throughput and playing rate may vary along the time, player’s buffer utilization depends on the instantaneous throughput and play-out rate rather than their average values. Therefore, this approach is expected to provide slightly optimistic results.

Regarding the QoE advices module, its role is to analyze possible causes that provide a low QoE and subsequently to provide particular advices to the user when certain conditions are given. As an example, Table 3 shows potential causes of low QoE, their associated evidences, and advices.

Table 3 Examples of causes of low QoE and advices to users (QoE advices module)

The application needs permission from the user to read the phone state, to access location information, to access external storage, and to connect to the Internet. Information is stored in the device until it is reported to the server. In the device, the SQLite3 database, included in the operating system, is used for storing all retrieved information.

The proposed Android application is useful for both network operators and YouTube users. There are many use cases that would benefit the rollout of a YouTube QoE monitoring solution as proposed in this paper, such as:

• 1. The knowledge of the instantaneous YouTube QoE per user may help the operator to perform actions like, e.g.: a) to modify the subscriber priority when a poor performance in a specific location or particular subscriber is detected; b) to set dynamically different bandwidth limits depending on a number of factors like usage patterns, subscriber, location, time of day, and so forth; c) to send notifications to subscribers like advices to improve their QoE.

• 2. Network capacity planning: The proposed QoE solution could help on the identification of network bottlenecks and re-dimension the network to ensure the targeted QoE.

• 3. Traffic forecasting process based on historical data traffic stored in its database.

• 4. Handset and service performance benchmarking: With the growing number of mobile handsets and multimedia content launched onto the market, it is becoming increasingly important for operators to benchmark each individual terminal and measure its performance. This process enables the identification of problematic handsets and the analysis of the cause for the faults. By identifying problematic handsets, operators can quickly make the required adjustments to their network, thus improving the customer experience

• 5. Network monitoring and reporting: The proposed QoE solution use passive methods to infer automatically from passive measurements the user perception on the network. The goal is to automatically derive user perception from specific indicators being accessed purely from monitoring (eliminating the need for customer surveys) both from the network and terminal sides.

• 6. Customer care: The ability of linking perceived (subjective) experience with measured (objective) QoE indicators may lead to significant benefits in terms of achieving a better insight onto customer perceived quality in a much more wide approach than the current one based on sampling of specific customers. QoE monitoring solutions are linked onto customer care centers by means of simplified interfaces and overall status for real-time access to customer-specific information, enhancing the response to customer quality and thus satisfaction. Customer care teams can rapidly diagnose problems and identify whether the root cause is linked to a badly performing network, mobile terminal, or application.

4. YouTube QoE pilot experiment

A set of 17 users (engineers from Telefónica company) were selected to participate in a pilot experiment, which consisted in periodically testing our YouTube QoE tool (installed on different Android smartphones) for 1 month. Every YouTube native session was transparently monitored and evaluated in two ways: 1) automatically by the application (from the QoE model previously described) and 2) by the users through an online opinion survey. A total number of 1,435 YouTube sessions were evaluated during the pilot. The data collected from each user device was sent to a server for post-processing purposes.The pilot experience was carried out in Madrid (Spain), covering both rural and urban environments (as shown in Figure 4 on the left). Different colors represent the associated subjective quality (from the opinion survey) for a set of YouTube sessions. Such a survey (related to the video quality, audio quality, and overall quality) was requested to be filled after each YouTube session. Figure 4 on the right shows the probability density distribution of the feedback associated to video, audio, and general quality.

Figure 4

Geographical distribution of the users (left) and quality feedback distribution (right).

According to the statistics collected at the QoE server, the majority of videos consumed by the users are short: nearly 90% of the videos are shorter than 5 min, and average duration is 160 s (see Table 4). Regarding the video characteristics, users had free access to the YouTube repository, so a wide variety of videos with different average bitrates (from 75 kbps to 3 Mbps depending on the resolution and codec) have been downloaded.

Table 4 Summary for reproduction time (in minutes)

Next, statistics related to the application performance metrics (mainly referred to as T_init, f_rebuf, and T_rebuf, which are required to evaluate MOS) are analyzed in detail. Later, their effect on the experienced quality will be described.

First, a box-and-whiskers plot of the initial buffering time (T_init) per technology is shown in Figure 5. This non-parametric representation depicts quartiles as a box with median drawn as a vertical line inside the box; that is, 50% of values for T_init are included in the interval inside the box. Moreover, lines extending from the boxes (whiskers) indicate variability outside the upper and lower quartiles. Outliers lying further 1.5 times the inter-quartile range are plotted as individual points, and those further three times that range (extreme points) are besides filled up. Average value is shown as a red cross. A box-and-whiskers plot can be seen as a kind of summary of the cumulative distribution function (CDF) (whose estimation for T_init is shown in Figure 6) and a graphical representation of numerical measures, some of which are presented in Table 5.

Figure 5

Box-and-whiskers plots for the initial buffering time (in seconds) per technology.

Figure 6

Estimated CDF of the initial buffering time ( T _init ).

Table 5 Summary for initial buffering time ( T _init ) in seconds

Results from Figures 5 and 6 show that T_init values for WiFi connections are lower than those for UMTS. For 3G sessions, the estimated coefficient of variation (CV) is higher than 2, i.e., the standard deviation of T_init for UMTS connections is more than twice its average value. For WiFi, this dispersion measure is reduced to about 1.2. This comes from the fact that T_init samples are concentrated around the median for WiFi sessions, whereas UMTS presents a higher range. Such a heavy tail results in a higher average located in the last quartile. In any case, 50% of the videos have experienced an initial buffering time shorter than 7 s. In most connections, no rebuffering is necessary; thus, the median for the rebuffering frequency (f_rebuf) is 0 (see Table 6). However, in this case, f_rebuf is higher for WiFi than for UMTS; the reason is that, although the number of pauses is smaller for WiFi (Table 7), videos were shorter (see Table 4), thus boosting the frequency of interruption events even if the mean rebuffering time (T_rebuf) is lower (Table 8).

Table 6 Summary for rebuffering frequency ( f _rebuf )

Table 7 Summary for number of pauses ( N _pauses )

Table 8 Summary for mean rebuffering time ( T _rebuf ) in seconds

Now, we describe the effect of performance indicators in the reported MOS. Figure 7 shows the initial buffering time (T_init) box-and-whiskers plot per MOS. As shown in the results, lower T_init values are associated with higher MOS. Although a higher feedback quality could be expected for WiFi than for UMTS, it can be observed that users do not assign a significantly lower MOS for UMTS than for WiFi (see Figure 8 and Table 9) even if, from an objective point of view, their performance is better (a summary of the three studied performance indicators can be found in Tables 10, 11 and 12). That is, although WiFi connections achieve better QoS figures, MOS values are very similar to those of 3G. The reason for that might be that subjective users’ expectations could be influenced by the type of connection being used. Hence, users might penalize the QoE of WiFi connections due to a higher expected quality. We expect similar or even more demanding user attitude from 4G users (not extensively available at the time of the survey).

Figure 7

Box-and-whiskers plot for T _init per reported MOS.

Figure 8

Box-and-whiskers plot for MOS per used technology.

Table 9 Percentage of reported MOS per technology

Table 10 Average, max, and standard deviation values (in seconds) for T _init as per technology and MOS

Table 11 Average, max, and standard deviation values for f _rebuf as per technology and MOS

Table 12 Average, max, and standard deviation values (in seconds) for mean rebuffering time as per technology and MOS

Next, the appropriateness of the theoretical model in Equation 1 is analyzed by evaluating the correlation between the theoretical MOS and the MOS reported by users, resulting in a correlation factor of 0.97, i.e., the coefficient of determination R² for linear regression through origin is 93.93%. Figure 9 on the left shows the difference between MOS results provided by the theoretical model (see Equation 1) and MOS reported from the users’ opinion survey. It can be observed that the QoE model provides an estimation which falls within ±0.5 of the reported score in 23% of sessions. In general, the model in [23] provides more pessimistic results than the opinion of users as the estimated MOS is lower than that reported for about 68% of sessions. A simple modification results from taking a linear regression between the modeled MOS and MOS as reported by users, yielding to:

$${\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{mod}}=1.1935\cdot {\mathrm{M}\mathrm{O}\mathrm{S}}_{\mathrm{QoSmodel}}$$

(2)

Figure 9

Histogram for the difference between original QoS model (left) or modified QoS model (right) and as reported by users.

Note that this measurement indicates that MOS is about 20% higher than that given in Equation 1. The reason could be that users could be more indulgent with wireless connections than for wired scenarios under which the original model was obtained.

Due to regression properties, the average value for the difference between MOS as obtained by Equation 2 and that reported by users (that is, the residuals) is 0, although no symmetry around 0 exists (see Figure 9 on the right). Differences between subgroups per technology are not significant (estimated slope of 1.1995 for WiFi connections and 1.2089 for UMTS). It was explored whether a multivariant regression could improve those results. Only linear regression was analyzed as a modification of numerical quantities as those proposed in [28] cannot be easily included in the multivariant procedure. The adjusted R² including all available parameters results in 90.5%, only a bit lower (90.46%) if the total rebuffering time is taken out from regression. As this value is lower than that obtained with Equation 1, the heuristical measurement quantization proposed in [23] increased by 20% seems to be able to predict well the users’ expectations.

5. Conclusions

This work has presented a QoE evaluation tool for Android terminals which is able to estimate the QoE (in terms of MOS) for YouTube service based on theoretical models. In particular, this tool makes it possible to map network QoS onto the QoE of YouTube sessions. Additionally, a QoE advices module analyzes possible causes that lead to low QoE and subsequently provides particular advices to the user under certain conditions.

Our application has been tested on a pilot experience over 17 Android terminals for 1 month. According to the statistics, most of the responses from the users’ survey match up with theoretical estimations; however, the QoE model provides slightly more pessimistic results than the opinion of the wireless users, probably as the model was initially generated under wired scenarios. In that sense, we propose a modified utility function from taking a linear regression between the theoretical MOS and the MOS reported by users.

In our opinion, it is critical that application developers provide access to the main key performance indicators (KPIs) associated to their services in order to ease the evaluation and analysis of the QoE.

Authors’ information

GG received his B.Sc. and Ph.D. degrees in Telecommunications Engineering from the University of Málaga (Spain) in 1999 and 2009, respectively. From 2000 to 2005, he worked at Nokia Networks and Optimi Corporation (recently acquired by Ericsson), leading the area of QoS for 2G and 3G cellular networks. Since 2005, he is an associate professor at the University of Málaga. His research interests include the field of mobile communications, especially QoS/QoE evaluation for multimedia services and radio resource management strategies for LTE and LTE-Advanced.

LH is a telecommunications engineer from E.T.S.I.T - UPM, specialized in computer science. He has worked mainly in defense aerospace industry as a military avionics engineer and in wireless telecommunication industry as systems engineer. He also worked for public administration as a consultant in computer science and also as a contract reviewer. Since 1996, he has worked for Telefónica I  + D, always involved in mobile network R&D. Recently, he worked at the area of PDI - Enabling Platforms (SLA) as a computer technology specialist. Currently, he is working at the area of PTI - Capacity & Traffic Analysis & Solutions, developing tools based on knowledge extraction, machine learning, and analytic prediction techniques.

QP is a physicist for the Complutense University of Madrid from 1986. In 1987, he started working in Telefónica I  + D, in aspects related with reliability predictions, failure analysis, and components electronic testing, used in telecommunication systems. In 2011, he was working in QoE issues for mobile broadband networks, analyzing and developing tools and client applications to control and manage the QoS and QoE of mobile services. Currently, he is working in GCTO (Global Chief Technology Office) of Telefónica, analyzing and defining the requirements needed for residential cellular gateways.

JL received his B.Sc. in Telecommunications in 1998 in Universidad Politécnica de Madrid. In 1999, he worked for Teldat on ciphering techniques, and since 2000, he is working in Telefónica I  + D on several areas related to mobile communications and physical layer performance, involving terminal specifications and testing, link-level and system-level simulations of 3G/3.5G systems, LTE digital signal processing, and LTE-Advanced. He is currently working on quality of experience in wireless networks.

RG received her B.Sc. degree in Telecommunications Engineering in 1998 from Madrid Polytechnics University. Between 1997 and 1998, she staged at the Digital Communication Systems Department at the Technical University Hamburg-Harburg (Germany), where she carried out her master thesis about multipath fading channel modeling. In July 1998, she signed for Telefónica I  + D, becoming part of the Radio Communication Systems Department. Her career has oriented towards mobile communications, especially on radio planning and optimization and QoE in LTE.

MCAT received M.S. and Ph.D. degrees in Telecommunications Engineering from the University of Malaga, Spain, in 1994 and 2001, respectively. Currently, she is working at the Department of Communications Engineering, at the same university. Her main research interests include adaptive modulation and coding for fading channels, multi-user OFDM, SC-FDMA, cross-layer design, and probabilistic QoS guarantees for wireless communications.