Modelling user quality of experience from objective and subjective data sets using fuzzy logic

This contribution is a follow-up of our past research, which was aimed at investigating how packet loss-related issues affect user perception of video quality. In that study, we tested user perception using 72 different test sequences, which were prepared in advance in an emulated network environment. The same type of content was used in all test sequences (1-h documentary film about the solar system); however, in each test sequence, packet loss rate (PLR), number of packet loss occurrences (PLOs), and duration of PLOs varied.

Details of the subjective test are discussed in the remainder of this chapter, but here, it is first necessary to recognize that we adopted the test methodology from [5]. The authors designed the experiment, which allows a researcher to prepare the test sequences in advance, thus retaining control over audiovisual quality. The sequences are then stored on, for instance, an optical drive or removable storage, and distributed to the subjects for rating in their home environments. Hence, subjective evaluation of the sequences is conducted in uncontrolled conditions, which is important for QoE evaluation, as emphasized in the Introduction.

Note that we have analyzed other approaches that include uncontrolled environments. For instance, in [7, 8, 9], the authors employed QoE crowdtesting, while in [10, 11], user network performances were remotely monitored during streaming sessions. These two approaches were not suitable for this study, because it would have been difficult to pursue the test subjects to download or stream 1-h video (i.e., several gigabytes of data) to their devices at home, which was necessary in [7, 8, 9] and [10, 11], respectively.

In [12], Staelens et al. prepared the test sequences in advance and stored them on tablet computers. The tablets were then distributed to the subjects, who watched the sequences in everyday conditions. The subjects rated the quality of the sequences directly on the tablets. The authors collected the rating data after the subjects returned the tablets. With this approach, video downloading or streaming is avoided, yet we decided not to follow it due for two reasons: (a) we did not have a sufficient number of tablet devices to conduct a large-scale study such as ours (602 test subjects) and (b) the player used for watching the test sequences contained a video quality rating scale; thus, the purpose of the test was revealed to the subjects. When the purpose of the test is known to the subjects, they are more focused on quality degradation during the test and less focused on content. This is unlike everyday service usage scenarios and also affects the user QoE rating. Finally, the group of authors in [13] implemented a QoE rating scale in the user interface of the VLC Media Player. The player was installed on the subjects’ devices, who used it for streaming multimedia content. This approach was also rejected since the visibility of the rating scale during playback reveals the purpose of the test to the subjects.

2.1 The experiment setup and creation of test sequences

In this experiment, test subjects evaluated the quality of 1-h video about the solar system in a home environment. We have used entertainment-oriented content selection [14], since we assumed that, while at home, the subjects usually watch video content that interests them. An additional reason for choosing the content that can entertain the subjects was the duration of the test, i.e., the video, and the need to capture the subjects’ concentration for a full hour. We believe that if different content would have been used, that would have been perceived by the subjects as, for instance, boring, then their willingness to participate in our study would decline. Note that our test conditions were unlike controlled environment experimenting in laboratory, where the tests usually last 20–30 min, during which observers rate the quality of several short video clips. In such shorter tests, it is easier to retain the subjects’ focus on the task at hand.

The experiment was performed using longer test sequences; thereby, we have acknowledged the findings presented in [15, 16, 17]; the authors in that study demonstrate how QoE evaluation requires an increase in the duration of the test sequences, because user perception cannot be entirely shaped when using shorter video clips. The original video used in the experiment was encoded with Advanced Video Coding (H.264/AVC) and Advanced Audio Coding (AAC). The video bitrate, audio bitrate and frame rate were 9.8 Mbps, 256 kbps and 50 fps, respectively. The video resolution was 1920 × 1080 pixels. Note that the video contained video subtitles.

To create test sequences, first, the original video was streamed six times between two computers, which were connected in a peer-to-peer connection (Fig. 1). The Network Emulator Client was installed on Computer 1. The client dropped the packets on the outgoing stream toward Computer 2. PLR varied between 0.05, 0.1, 0.5, 1, 1.5 and 2%, while the burst packet loss length was set to 1. VLC Media Player was used to stream the video between the computers. Each incoming (degraded) video signal was stored on Computer 2 in the same format as the original video. During streaming of the video between the computers, UDP was used on the transport layer. This differs from Hyper Text Transfer Protocol (HTTP)-based streaming, which uses TCP (Transport Control Protocol). Nowadays, HTTP-based adaptive streaming is the relevant scenario; however, UDP-based streaming is still used for delivering Internet Protocol Television (IPTV), in particular for those services that use set top boxes.

Open image in new window

Second, we have imported the stored video signals into CyberLink Power Director and extracted 1, 4, 7 or 10 short video clips from a degraded video signal and inserted them into the original video signal. The duration of a single inserted clip, i.e., a single packet loss occurrence (PLO), varied between 1, 4 and 7 s. By varying the number of inserted PLOs and the duration of a single PLO, we were able to generate different total durations for all PLOs in a test sequence that equaled 1, 4, 7, 10, 16, 28, 40, 49 or 70 s. The total duration is derived by multiplying the number of inserted PLOs in a sequence (1, 4, 7 or 10) with the duration of a single PLO (1, 4 or 7 s). When selecting these particular values of the parameters, the objective was to generate a wide range of total duration of the distortions in the sequences. Additionally, we wanted to create sequences with equal total duration of distortion while containing a different number of inserted PLOs and different durations of a single PLO. For instance, two sequences can contain 28 s of total quality distortions, but one can have 7 PLOs each lasting 4 s, while the other has 4 PLOs each lasting 7 s. This enabled us to evaluate the impact of each parameter individually.

In [18], the authors revealed that when the distortions are grouped into the first few minutes of the video, the quality scores of the subjects are observed to increase. Conversely, if the distortions are grouped into the last few minutes, the scores are observed to decrease. This kind of subject reasoning is influenced by human short-term memory [19] and the psychological effect of recency [18, 20]. Thus, the PLOs were evenly distributed over the entire duration of all test sequences. However, in each test sequence, the first and last 7 min and 17 s were unaffected by the degradations, which allowed the test subjects to get involved with the content in the beginning of the session as well as critically think about the audiovisual quality toward the end.

In this study, the methodology used for the subjective evaluation of the sequences was adopted from [5]. Thus, the sequences were distributed to the subjects on a DVD. This format was chosen due to the following reasons: (a) the test sequences were easily distributed to the subjects; (b) DVD players are more available compared to Blu-ray players (this was important since the survey was conducted among a student population); (c) DVD disks are cheaper compared to, e.g., memory sticks; and (d) according to [21], while evaluating other services, test subjects often use the quality of DVDs as a reference.

During the encoding of the test sequences to the DVD format, the PAL system, MPEG-2 video encoding format and variable bitrate encoding method were used. All video encoding settings were set to maintain the best possible video quality. Furthermore, all video enhancement features of the CyberLink Power Director software were turned off and the software did not use any error concealment methods. The resulting video bitrate, audio bitrate and frame rate were equal to 9.51 Mbps, 256 kbps and 25 fps, respectively.

Although we have used the settings which allowed the best possible video quality for the conversion to the DVD format, this process downgraded the quality of the sequences, since the original video (encoded with H.264/AVC) was re-encoded into the MPEG-2 format. In general, this decline in the video quality is hard to notice in low-motion scenes but can be noticed in high-motion scenes when moving objects, appearing in a scene, can become blocky. The difference in the quality becomes even more visible to the subjects if they are provided with the original sequence for the comparison (that was not the case in our study). Since we conducted the experiment using the documentary film, most of the scenes in the film were low-motion scenes (for instance, the presenter’s monologue or dialogue with other persons appearing in the film). Thus, the quality degradation caused by the re-encoding process was not apparent. Furthermore, we need to highlight that in our test conditions, which were mimicking a lifelike viewing experience, the subjects were not focused on keeping track of the video quality, they were focused on the content instead. Thus, we believe that the decline in the video quality due to the re-encoding of the video did not impact the QoE of our test subjects.

Revealing user perceptions and QoE for specific services can be a resource-consuming task. Thus, various objective quality assessment models are developed that can estimate subjective perception of quality based on the values of the network-related parameters. ITU-T in [28] categorized the objective video quality assessment models as full reference (FR), reduced reference (RR) and no reference (NR) models, depending on the availability of a reference (unprocessed) video signal for the assessment. To measure, assess or predict the quality of a video signal, the objective models employ the following statistical models [29]: (a) media-layer models; (b) packet-layer models; (c) bitstream-layer models; (d) hybrid models; and (e) planning models. This ITU-T classification provides a generally accepted and commonly referenced framework for classification of different metrics and quality assessment models.

In the following overview of the related work, the focus will be on the objective assessment models that combine objective and subjective data sets to deliver the output. Note that some authors call this type of model a hybrid model (for instance, [30]), but they must be distinguished from the aforementioned hybrid statistical models. The focus of this literature overview means that different metrics and models, such as MSE (mean square error), PSNR (peak signal-to-noise ratio), MPQM (moving picture quality metrics), MSAD (modified sum of absolute difference), SSIM (structural similarity) Index, VQM (video quality model) and others, will not be reviewed since their output is calculated without taking into account user perception, which is critical for QoE evaluation. In addition, this review includes the work of other researchers who use different techniques to make a correlation between objective and subjective data sets, regardless of the transport layer protocol that has been used in the test.

Different authors have attempted to improve standard video quality metrics. For instance, Chan et al. in [31] introduced three types of modifications to the PSNR metrics using subjective test results to derive more accurate MOS assessments. The subjective evaluation of different videos involved 21 test subjects, and the obtained results were used for the modifications. Another example of this type of metrics modification (for MSE metrics and SSIM Index) is presented in [32, 33], respectively.

In [34], the authors showed how the random neural network can be trained with a subjective data set to assess the quality of a video stream on the receiver side. The subjective quality tests were conducted with a panel of 20 test subjects who evaluated the impact of stream bitrate, frame rate, packet loss rate, the burst packet loss length and the ratio of the encoded intra to inter macro-blocks on their perceived video quality. Similarly to [34], in [35], the authors also use the neural network for assessing the QoE of HTTP video streaming, taking into account the effects of pause position. Subjective evaluation was conducted among 54 test subjects (60 s video clips were used for testing). Another neural network application for QoE assessment for HTTP video streaming can be found in [36], while in [37], the authors use the network for QoE assessment of 3D video.

Mok et al. in [38] correlate user QoE with network QoS (quality of service) also for the HTTP video streaming service. Application layer QoS is expressed with three parameters (initial buffering time, mean re-buffering duration and re-buffering frequency), while the MOS are obtained by surveying a panel of test subjects who evaluated an 87-s video clip under different network conditions. Further information about the QoE of HTTP adaptive streaming can be found in the extensive literature overview presented in [39].

Apart from the abovementioned neural networks, other techniques can be used to correlate between user perception and objective parameters. A machine learning approach was employed by Menkovski et al. to develop the objective model, which can determine the extent of video quality degradations that may lead to unacceptable video quality as perceived by test subjects (see [40, 41]). The test subjects watched the sequences several times (the extent of quality distortions was increased each time) and indicated what they perceived as the acceptable limit of quality degradation in the sequence. Another example of testing the user acceptance of video quality can be found in [42]. The authors tested the user acceptance of mobile video against different objective parameters (re-buffering frequency, bitrate, frame rate, etc.) and developed a model for the acceptability of the quality of a mobile video session.

In [43, 44], the authors construct a \(k\)-dimensional Euclidian space, where \(k\) represents the number of network dependent and independent parameters that may affect user QoE. The space is then divided into \(N\) zones, and each zone is assigned with a QoE index; i.e., the authors define zones, where different values of various parameters lead to the same QoE rating. The QoE index is derived from subjective quality tests that were conducted among 77 test subjects who evaluated 18 test sequences. Later, Robalo and Velez in [45] use the results presented in [44] for mapping between the QoS and QoE.

Nguyen et al. in [46] apply a mixed effects model to predict user QoE for World Wide Web-based multimedia services. It is noteworthy that the authors include the state of mind parameter in their model (values of the parameter were: normal, bored, and stressed), proving how the QoE concept assumes a holistic approach to service evaluation. A further example of how network independent parameters may impact user QoE is presented in [47, 48], where different authors examine user perception of quality degradations while watching different types of video content. The obtained knowledge is used for development of the content-aware QoE assessment model.

In [49], the authors carried out subjective tests for the purpose of examining the perceptual experience of time-varying video quality. Based on the obtained results, the authors proposed an asymmetric adaptation model capable of mimicking human opinions when watching video with time-varying quality. Zhang et al. in [50] use fuzzy decision trees to predict user QoE from the log data collected from different Internet video service providers in China. After processing the raw data sets (the sets contained information about video ID, content and video type, client IP address and location, access device, join time, frame rate, bandwidth, buffering times and buffering ratio), the authors model user engagement as a key aspect of viewer behavior that, they claim, in some sense reflects users’ QoE. Hameed et al. in [51] also use decision trees for the construction of a low-complexity video quality model that predicts user QoE. For this purpose, the authors prepared 288 test sequences, which were evaluated by 100 test subjects under controlled conditions.

The impact of viewing distance on user QoE was analyzed in [52], and no reference QoE assessment model is proposed. However, the authors explored the regularities between image QoE and viewing distance for different types of images but not for video. The QoE crowdtesting platform is used in [9] to collect subjects’ opinions about the quality of adaptive media playout. The results were used to develop a nonlinear model that is able to describe user QoE relative to the audio/video distortions. Although the test is conducted in uncontrolled environments, the test sequence used in the study lasted only 51 s. Note that we have previously underlined that QoE evaluation requires using longer test sequences (based on the findings presented in [15, 16, 17]).

The models and metrics presented here are all based on subjective test results obtained in controlled laboratory environments, with two exceptions: [9, 50]. However, in [9], only short video clips were used for testing user perception, while in [50], the QoE is predicted from user engagement behavior and not from actual subjective quality tests. This lack of assessment models that originate from uncontrolled experiments was our primary motivation in this study.

The model presented in this contribution assesses the level of user QoE using the values of the three objective parameters as inputs, namely, the PLR, the number and total duration of PLOs in a sequence. For the purpose of fuzzification of scalar values of these three parameters, we asked the subjects to evaluate the level of their annoyance in relation to: (a) the observed quality distortions, i.e., perceived video artifacts (in Fig. 2, we correlated these responses with the scalar values of PLRs); (b) the number of PLOs that they have noticed (Fig. 3); and (c) the total duration of all quality distortions in the video (Fig. 4). The subjects rated the level of their annoyance on an 11-point scale, designed by ITU-T in [22], which allows the linguistic meanings of different grades to be added as an aid during rating. The meanings that are used in this study are depicted on the secondary \(y\)-axes of Figs. 2, 3 and 4.

Open image in new window

Open image in new window

Open image in new window

It can be observed that, for a given value on the \(x\)-axis of the figures, the test subjects’ level of annoyance is sometimes spread over all the annoyance level categories. This is most noticeable for the data presented in Fig. 2 and can be explained by acknowledging that, for instance, a PLR of 2% was perceived as imperceptible quality distortion when the sequence contained only one PLO. However, the same PLR was perceived as very annoying quality distortion if the sequence contained 10 PLOs, with 70 s of quality degradations in total. Similar results are presented in [53], where the authors also discuss the ambiguity between the PLR and the MOS values.

In Sect. 2.2.1, we announced that this chapter will elaborate further the importance of using 11-point continuous scales for collecting the subjects’ opinions. Thus, let us assume that, instead of 11-point scales, five-level discrete rating scales have been used in the survey. In that case, the results shown in Figs. 2, 3, and 4 would differ. For instance, all data points depicted in Fig. 2, for a given value of PLR, would be grouped in no more than five discrete values on the \(y\)-axis. The results would appear less ambiguous and, due to the loss of the fuzziness of the data, we could conclude that the impact of PLR on the annoyance level of the subjects is clearer than it is. This also applies to the results shown in Figs. 3 and 4.

After correlating the values of the three objective parameters with the test subjects’ subjective rating, the analysis of the user QoE for each of the 72 test sequences was conducted. The results are presented in Table 1. The subjects also rated the quality of the sequences on an 11-point quality scale that contained the following linguistic meanings: 0–2 bad quality; 2–4 poor quality; 4–6 fair quality; 6–8 good quality; and 8–10 excellent quality.

The obtained MOS varied on the interval [4.16, 8.96], meaning that even the most degraded test sequence with 70 s of quality distortions caused by PLR of 2% was still evaluated as a test sequence of fair quality (the average rating for the sequence number 72). In general, we can observe that the average QoE ratings, i.e., MOS, remain high despite the quality degradations of the specific sequences. For instance, from Fig. 4, it can be seen that almost all test subjects, who rated the sequences with 70 s of quality distortions, perceived that duration as annoying (ranging from slightly to very annoying); yet, this was not entirely reflected on their QoE level. As discussed in Sects. 2.3 and 2.4, the methodology for the subjective evaluation of the sequences highly impacted these results. Specifically, the subjects’ unawareness of the purpose of the test, familiar home environment, longer test sequences, content that was mainly evaluated as mostly and very entertaining, affected the subjects, making them more forgiving to the occasional advent of video degradations.

We also emphasize that the video contained subtitles. In our previous research, we found reasons to believe that the sequences with subtitles can be rated higher by the subjects compared with the sequences without subtitles since the text can draw the viewer’s attention away from the picture to the bottom of the screen. Hence, some quality distortions may remain unnoticed by the subjects.

The objective of this chapter is to discuss the first stage of the model development. Namely, the chapter will show how the scalar values of the input and the output parameters are converted into the fuzzy variables which are then used by the inference system of the model. However, first, we want to justify the decision to use fuzzy logic for the modelling of user QoE.

As has been pointed out in Sect. 4, when the results presented in Figs. 2, 3, and 4 were discussed, the impact of the three objective parameters on the subjects’ perception is not decisive. Yet, the ambiguity of the obtained results was expected mainly due to the following reasons.

To cope with the uncertainty in the results, we have used fuzzy logic which is useful when trying to characterize concepts and phenomena with natural ambiguity [54]. Additionally, the logic allowed us to model the relationships between the objective and the subjective data sets which will be demonstrated in the remainder of this chapter.

5.1 Defining the clusters and fuzzy membership functions for the input parameters

In this step, the fuzzification of the crisp values shown in Figs. 2, 3, and 4 was conducted using the Fuzzy C-Means (FCM) clustering approach [55]. The objective of using this procedure was to group the data points presented on the figures into fuzzy clusters and to find centers of those clusters. Note that the FCM method allows the clusters to overlap. Therefore, the \(i\)th data point (\({x_i}\)) can be a member of several clusters (\(j\)) with different degrees of membership (\({u_{ij}}\)). According to [55], the method requires minimizing the objective function \({J_m}\):

$${J_m}=\mathop \sum \limits_{{i=1}}^{L} \mathop \sum \limits_{{j=1}}^{C} {u_{ij}}^{m} \cdot {\left\| {{x_i} - {c_j}} \right\|^2},$$

(1)

where \({c_j}\) denotes the \(d\)-dimension center of the cluster, \(||{\text{*}}||\) is any norm expressing the similarity between any measured data and the center, and \(m\) is any real number > 1. Note that in this study, \(m=2\). Using this method for grouping the data points from the figures into fuzzy clusters and finding the centers of those clusters assumed the iterative process, where the \({u_{ij}}\) and the \({c_j}\) are updated with Eqs. 2 and 3, respectively. The iterative process finishes when the stopping criteria (\(\varepsilon\)) from Eq. 4 is met (\(k\) are the iteration steps). In our case, we used the value of \(\varepsilon\) that is set by default in MATLAB to \({10^{ - 5}}\).

$${u_{ij}}=\frac{1}{{\mathop \sum \nolimits_{{k=1}}^{C} {{\left( {\frac{{\left\|{x_i} - {c_j}\right\|}}{{\left\|{x_i} - {c_k}\right\|}}} \right)}^{\frac{2}{{m - 1}}}}}}$$

(2)

$${c_j}=\frac{{\mathop \sum \nolimits_{{i=1}}^{N} {u_{ij}}^{m} \cdot {x_i}}}{{\mathop \sum \nolimits_{{i=1}}^{N} {u_{ij}}^{m}}}$$

(3)

$${\text{ma}}{{\text{x}}_{ij}}\left\{ {\left| {{u_{ij}}^{{(k)}} - {u_{ij}}^{{(k - 1)}}} \right|} \right\}<\varepsilon$$

(4)

For each of the three input parameters, the subjects’ responses are grouped into three fuzzy clusters. The procedure required 24, 69 and 33 iterations before the stopping criteria \(\varepsilon\) was met and the centers of the clusters were defined for the PLR, Number of PLOs and Total duration of PLOs, respectively. The results are presented in Figs. 5a, 6a, and 7a, respectively. Note that thickly printed plus, × and circle signs on the figures represent the cluster centers, and their coordinates can be found in Table 2.

Open image in new window

Open image in new window

Open image in new window

Table 2

Centers of the clusters shown in Figs. 5a, 6a, and 7a

Figure	Cluster 1 (plus sign)		Cluster 2 (× sign)		Cluster 3 (circle sign)
	\(x\)	\(y\)	\(x\)	\(y\)	\(x\)	\(y\)
Figure 5a	0.4545	0.3492	0.8758	2.9503	1.3937	6.4334
Figure 6a	1.6513	1.1024	6.5083	2.1201	9.3728	6.0449
Figure 7a	6.4254	1.7545	33.0713	3.6854	67.1134	6.2761

The final step toward defining the boundaries of the clusters was taken when the values of the data points on the \(x\) axis were correlated with their degrees of membership \({u_{ij}}\) to the specific cluster \(~j\) (see the Appendix 2, Figs. 11, 12, 13). In his book about fuzzy logic and its engineering applications [54], Ross reports how the normal distribution corresponds better with the changes in human perception, because the transitions between different opinions and attitudes are usually happening gradually (“smoothly”). Thus, bell-shaped functions can mimic that behavior better compared with, for instance, triangular functions. For this reason, we used the normal distribution for the approximation of the membership functions. Note also that the “smooth” transitions in human perception were captured with the use of continuous 11-point numerical scales for subjective data collection. The “smoothness” would be lost if the five-level discrete scales would have been used or questions with two-alternative options (as discussed in Sects. 2.2.1 and  4).

The obtained functions are presented in Figs. 5b, 6b, and 7b. Since the fuzzy systems use linguistic variables to infer conclusions, each cluster presented on the figures is named. These linguistic variables will be used in Sect. 6.1 for defining a set of fuzzy rules of the model.

Properties of the fuzzy membership functions shown in Figs. 5b, 6b, and 7b can be found in Table 3. Note that the dashed and the dashed–dotted Gaussian functions of each input parameter had to be modified so that \({u_{ij}}\) would be equal to 1 in cases when \({x_i}<\bar {x}\) (for the dashed functions) and \({x_i}>\bar {x}\) (for the dashed-dotted functions).

Table 3

Properties of the Gaussian functions shown in Figs. 5b, 6b, and 7b

Figure	First membership function (dashed line)		Second membership function (solid line)		Third membership function (dashed–dotted line)
	\(\bar {x}\)	\(\sigma\)	\(\bar {x}\)	\(\sigma\)	\(\bar {x}\)	\(\sigma\)
Figure 5b	0.4545	0.6574	0.8758	0.5398	1.3937	0.4887
Figure 6b	1.6513	2.4	6.5083	1.748	9.3728	2.061
Figure 7b	6.4254	13.73	33.0713	10.92	67.1134	16.33

The mean values (\(\bar {x}\)) of the Gaussian functions depicted in Figs. 5b, 6b, and 7b correspond with the \(x\)-coordinates of the centers of the clusters presented in Table 2 and Figs. 5a, 6a, and 7a, respectively. This means that the data points have a higher degree of membership (\({u_{ij}}\)) to a specific cluster \(~j\) if they are located closer to the center of that cluster. Therefore, the shift in the sample population mean in this case is actually impacted by the number of clusters and the location of their respective centers.

We can observe how the fuzzy membership functions are overlapping due to the dispersion of the data points around the centers of the clusters. The overlapping of the membership functions is a key feature of the FCM procedure since it enables grouping a specific data point into more than one cluster. This is important, because it is not possible to unambiguously define, for instance, whether PLR of 1% is causing imperceptible, slightly or very annoying quality distortion from a user point of view. In fact, our results show that this particular PLR is a member of all three clusters (Fig. 5b) with different degrees of membership.

The most distinctive overlapping of the membership functions is present for the first input parameter (Fig. 5b). The stretch of these functions (by a factor of \(\sigma\)) shapes their slopes, which are not so steep compared with the functions depicted on the other two figures (Figs. 6b, 7b). This is due to the previously discussed fact that certain PLRs were evaluated differently by the subjects, depending on the number of PLOs in a test sequence and their total duration. These results give us solid arguments for inclusion of the three parameters in the inference system of the model since they clearly create an affiliated effect on user perception.

5.2 Development of fuzzy membership functions for the output parameter

We noted in Sect. 4, how the subjects evaluated the quality of the video and their viewing experience on an 11-point quality scale that also contained linguistic meanings of the specific ratings. These linguistic meanings were used to name the fuzzy clusters of the output parameter. However, to increase the accuracy of the model, instead of five clusters, the output parameter is modelled with eight clusters. While experimenting with different settings of the model, we have observed that it is better to increase the number of clusters of the output parameter, because the model became more responsive to the changes of the input values and assessed the QoE with more accuracy.

The membership functions of the eight fuzzy clusters of the output parameter are depicted in Fig. 8. Again, the Gaussian functions are used (their properties can be found in Table 4). For the clusters bad and excellent quality, the functions are modified so that \({u_{ij}}\) would be equal to 1 in cases when \({x_i}<\bar {x}\) and \({x_i}>\bar {x}\), respectively. Note that the membership function of the excellent quality cluster has \(\bar {x}\) = 8.44, so that the ratings ≥ 8 would have a higher degree of membership to this cluster compared with the cluster good quality 2.

Open image in new window

Table 4

Properties of the Gaussian functions shown in Fig. 8

Membership function of the cluster	\(\bar {x}\)	\(\sigma\)
Bad quality	1.42	0.648
Poor quality 1	2.5	0.5308
Poor quality 2	3.5	0.5308
Fair quality 1	4.5	0.5308
Fair quality 2	5.5	0.5308
Good quality 1	6.5	0.5308
Good quality 2	7.5	0.5308
Excellent quality	8.44	0.648

The previous chapter showed how the crisp values of the input and the output parameters are converted into the fuzzy variables needed for development of the inference system of the model. However, the model has to produce a quantifiable output, i.e., the assessed QoE rating in a numeric form. Therefore, the fuzzy values have to be defuzzified. This requires defining a set of fuzzy rules, choosing between the conjunctive or disjunctive system of rules and employing a defuzzification method that calculates the model output (e.g., max membership principle, centroid method, weighted average method, center of sums or other).

The subjects rated 72 different test sequences in a home environment. The properties of each test sequence were previously presented in Table 1, and they include the following objective parameters: (a) PLR; (b) the number of PLOs in a sequence; and (c) total duration of PLOs. The values of these three parameters are plotted in Fig. 9, where each marker represents one test sequence, while the shape and the color of the marker represents the value of the assessed QoE by the model (according to the attached legend). While previously discussing the results presented in Table 1, we have mentioned how the MOS of the test subjects varied on the interval [4.16, 8.96], staying within the fair quality set even for the most degraded test sequence. The assessed QoE rating of the model varies between [4.48, 8.74]. This motivates us to investigate in the future to what extent a 1-h video has to be degraded to evoke higher dissatisfaction from the subjects.

Open image in new window

The output of the model was tested for each test sequence by comparing it with the corresponding MOS of the subjects. Results of this analysis are presented in Fig. 10. The Pearson’s correlation coefficient for this set of data equals 0.8841, indicating a strong positive linear relationship between the two variables.

Open image in new window

To allow comparison between our model and similar models, Table 6 tabulates the properties of the models developed by other authors and the achieved correlation coefficients. The comparison includes only related work in which the effect of packet loss-related issues on user perception is investigated. In addition, since we used UDP on the transport layer, the comparison excludes models developed for assessing the QoE of HTTP-based video streaming. Note that the models developed by the authors referenced in Table 6 are all based on the controlled environment experiments (using only short test sequences), where the objective was to eliminate different QoE influential factors, which were the essential part of our study.

It could be argued that the models, which originate from the uncontrolled tests, produce outputs that are better suited for remote evaluation of service quality if the service is used in everyday, lifelike scenarios. This is due to the already mentioned fact that the subjective evaluation of service quality, when conducted in the uncontrolled environment, produces different results if compared with the results of the controlled environment experimenting. Considering these differences, it is not unexpected to find that certain authors question the usability of laboratory testing in controlled conditions (e.g., see the work presented in [27, 56]). Thus, the objective assessment models that are based solely on the subjective test results obtained from a controlled environment experiment have this mismatch integrated in them.

In this study, we have used the UDP-based multimedia streaming service to demonstrate how the objective and the subjective data sets can be correlated and the obtained fuzzy dependencies used for development of a no reference objective video quality assessment model for assessing user QoE. The primary use of the developed model is for the assessment of QoE for UDP-based IPTV services.

We are aware of certain usability limitations of the model. Namely, a subjective evaluation was conducted among the student population; hence, we cannot claim that the model output would have strong correlation with MOS of a more versatile panel of test subjects. Furthermore, only one type of video content was used for the subjective evaluation. Testing with other types of content of various duration may evoke different levels of user dissatisfaction for the same values of the objective parameters which we have tested. Hence, the inference system of our model would have to be modified with a purpose of making it applicable for the assessment of user QoE for other types of content of various duration. We also cannot ignore the impact of certain properties of the video, used in the study, on the test results. As emphasized, the video lasted 1 h and it contained subtitles. Thus, the recency effect affected the subjects’ perception and some quality degradations became harder to notice by the subjects, respectively. This was reflected on the obtained results and the inference system of the model. Finally, the model cannot assess user QoE if PLR, number of PLOs and their total duration exceeds 2%, 10 and 70 s, respectively. These limitations of the model opened multiple paths for our future research and the opportunity to improve the model.

The results of the fuzzification process showed how a crisp value of one input parameter cannot be unambiguously related to the specific user rating; overlapping of the fuzzy clusters of the same input parameter exists as well as the fact that all three objective input parameters create an affiliated effect on user QoE. The latter proved that it was meaningful to include the three chosen objective parameters as inputs of the model.

It must be emphasized that the developed inference system of the model is predominantly influenced by the methodology used for the subjective evaluation of the sequences. In our study format, the subjects were unaware of the purpose of the test; they watched and evaluated 1-h video at their homes, in a familiar environment, with or without company at any time of the day (depending on their liking). These test conditions shifted the subjects’ focus away from noticing and memorizing the video quality distortions to the actual content of the video. We believe that these test conditions provide more lifelike evaluation results of user QoE.

The subjective evaluation of user QoE that was conducted in uncontrolled test conditions represents the origin of one of the models most distinctive features; we showed that only a few such assessment models exist today. Usually, other authors develop similar models from the results of the controlled experiment. Due to the disparity between the results obtained in the controlled and uncontrolled test environments, it could be argued that, in the context of everyday service usage, the models based on the uncontrolled experiments assess user QoE with more accuracy compared to metrics and assessment models that are based solely on laboratory test results.

Our future research will strive toward developing a similar QoE assessment model that will be able to make assessments based on the video frame rate, bitrate and re-buffering frequency, possibly for 4K video streaming. Again, we plan to build the model from the results of uncontrolled experimenting. Thus, the new model will enable more lifelike assessments of user QoE compared to currently available models, which are mainly based on laboratory test results. However, considering the abovementioned limitations of the current model, this new subjective evaluation of video quality will test a more diverse group of test subjects (e.g., by employing QoE crowdtesting), who will evaluate different types of content (e.g., music videos or TV shows). Additionally, learning from the research of other authors presented in this paper (for instance, from those listed in Table 6), the new model could also combine different input parameters with the results obtained from known metrics for objective evaluation of video quality such as PSNR and SSIM. The model would then become an FR model.

We would also like to continue our investigation of how video subtitles affect user QoE. To the best of our knowledge, this particular field of research is still insufficiently explored. For this purpose, we will use eye-tracking glasses and inspect the eye movements of the subjects’ when they watch the video with time-varying quality and subtitles. Then, we will be able to correlate the subjects’ perception of video quality, their eye movements, the amount of text on the screen and specific quality distortions.